Skip to content
This repository was archived by the owner on Jan 12, 2025. It is now read-only.

Commit 3b08524

Browse files
hl2guidehl2guide
authored
v0.2
1 parent 4691c22 commit 3b08524

2 files changed

Lines changed: 177 additions & 16 deletions

File tree

‎README.md‎

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,22 +2,28 @@
22

33
An All-In-One AdGuard Home blocklist and allowlist generated locally on your PC using Python.
44

5-
Version: 0.1.25
5+
Version: 0.2
66

7-
Updated (AEST): 2024-05-16 02:41:49PM
7+
Updated (AEST): 2024-06-14 01:32:25AM
88

9-
## Repo Status
9+
## ✨ What's New in 0.2
10+
- Added new must-have requirement of a Python package named [__requests__](https://pypi.org/project/requests/)
11+
- Added new lists
12+
- More cleanup of output rules
13+
14+
## 🔬 Repo Status
1015

1116
- Written in **Python** it now has been updated for 2024 and have new lists added
12-
- Currently around **1.2 million unique rules** are included
13-
- The blocklist will make AdGuard Home's RAM usage increase to around **330MB**
17+
- Currently around **5.2 million unique rules** are included
18+
- The blocklist will make AdGuard Home's RAM usage increase to around **775MB**
1419
- Do not use these lists on older hardware with fewer than **4 CPU cores** or **IOT** devices
1520
- Be prepared to make your own whitelist additions within AdGuard Home, depending on your needs
1621
- I attempted and tested using **GitHub Codespaces** and **github LFS** but moved away from both due to filesize restrictions
1722

18-
## Included Blocklists
23+
## ⛔ Included Blocklists
1924

2025
- [1Hosts (Lite)](https://adguardteam.github.io/HostlistsRegistry/assets/filter_24.txt)
26+
- [AdAway Default Blocklist](https://adguardteam.github.io/HostlistsRegistry/assets/filter_2.txt)
2127
- [AdGuard DNS filter](https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt)
2228
- [AdGuard Simplified Domain Names filter](https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt) by AdGuard
2329
- [Anudeep's Blacklist](https://github.com/anudeepND/blacklist) - [RAW](https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt) by Anudeep
@@ -31,7 +37,7 @@ Updated (AEST): 2024-05-16 02:41:49PM
3137
- [Ransom](https://blocklistproject.github.io/Lists/adguard/ransomware-ags.txt)
3238
- [Scam](https://blocklistproject.github.io/Lists/adguard/scam-ags.txt)
3339
- [Tracking](https://blocklistproject.github.io/Lists/adguard/tracking-ags.txt)
34-
- [AdAway Default Blocklist](https://adguardteam.github.io/HostlistsRegistry/assets/filter_2.txt)
40+
- [Blocklists by sefinek](https://sefinek.net/blocklist-generator/adguard) - ✨ 0.2
3541
- [Dan Pollock's List](https://adguardteam.github.io/HostlistsRegistry/assets/filter_4.txt)
3642
- [Dandelion Sprout's Anti Push Notifications](https://adguardteam.github.io/HostlistsRegistry/assets/filter_39.txt)
3743
- [Dandelion Sprout's Anti-Malware List](https://adguardteam.github.io/HostlistsRegistry/assets/filter_12.txt)
@@ -55,7 +61,14 @@ Updated (AEST): 2024-05-16 02:41:49PM
5561
- [stamparm's ipsum](https://raw.githubusercontent.com/stamparm/ipsum/master/levels/3.txt)
5662
- [uBlock₀ filters - Badware risks](https://adguardteam.github.io/HostlistsRegistry/assets/filter_50.txt)
5763

58-
## Setup
64+
## ⚒️ Setup
65+
66+
1. Install [Python](https://www.python.org) on your device
67+
2. Install a required Python package "requests" by running:
68+
69+
```
70+
python -m pip install requests
71+
```
5972

6073
- The python script `create_list.py` can be used to generate `aio_blocklist_final.txt` on a local machine that has Python
6174
- I have tested it on Windows 11, however it should work fine on other OSs too (as long as you have Python installed)
@@ -70,7 +83,7 @@ Updated (AEST): 2024-05-16 02:41:49PM
7083

7184
To regenerate the list simply run `create_list.py` when desired (as a scheduled task etc)
7285

73-
# Archived Old Info
86+
# 📦 Archived Old Info
7487

7588
The below is old info that shouldn't be needed anymore.
7689

‎create_list.py‎

Lines changed: 155 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,19 @@
77

88
# FOR ADGUARD HOME ONLY
99

10-
# Version: 0.1.25
11-
# Date: 2024-05-16 02:41:49PM
10+
# Required Packages
11+
# A python package called "requests" is required to handle 403 errors (install that package first by running: python -m pip install requests)
12+
13+
# Version: 0.2
14+
# Date: 2024-06-14 01:20:58AM
1215

1316
# Imports
1417
from time import gmtime, strftime
1518
from urllib.request import urlretrieve
19+
import requests
20+
#from urllib.request import Request, urlopen
21+
#import requests
22+
#from random import seed
1623
import glob
1724
import os
1825

@@ -92,6 +99,126 @@
9299
"https://raw.githubusercontent.com/borestad/blocklist-abuseipdb/main/abuseipdb-s100-60d.ipv4",
93100
# ipsum : https://github.com/stamparm/ipsum
94101
"https://raw.githubusercontent.com/stamparm/ipsum/master/levels/3.txt",
102+
# Sefinek-Blocklist-Collection (many lists) - https://sefinek.net/blocklist-generator/adguard
103+
"https://blocklist.sefinek.net/generated/v1/adguard/abuse/blocklistproject/hosts.fork.txt",
104+
"https://blocklist.sefinek.net/generated/v1/adguard/abuse/urlhaus.abuse.ch/hostfile.fork.txt",
105+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/0Zinc/easylist.fork.txt",
106+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/DandelionSprout.GameConsoleAdblockList.txt",
107+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/FadeMind/UncheckyAds.fork.txt",
108+
#"https://blocklist.sefinek.net/generated/v1/adguard/ads/MajkiIT/SmartTV-ads.fork.txt", # Testing Samsung TV login issues 05.06.2024
109+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/ShadowWhisperer/Ads.fork.txt",
110+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/adaway/hosts.fork.txt",
111+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/anudeepND/adservers.fork.txt",
112+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/blocklistproject/hosts.fork.txt",
113+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/blocklistproject/youtube.fork.txt",
114+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/craiu/mobiletrackers.fork.txt",
115+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/crazy-max/spy.fork.txt",
116+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/disconnectme/simple-ad.fork.txt",
117+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/firebog/AdguardDNS.fork.txt",
118+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/firebog/Admiral.fork.txt",
119+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/firebog/Easylist.fork.txt",
120+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/firebog/Prigent-Ads.fork.txt",
121+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/jerryn70/GoodbyeAds.fork.txt",
122+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/r-a-y/AdguardMobileAds.fork.txt",
123+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/sefinek.hosts.txt",
124+
"https://blocklist.sefinek.net/generated/v1/adguard/ads/yoyo/ads-trackers-etc.fork.txt",
125+
"https://blocklist.sefinek.net/generated/v1/adguard/amp/developerdan/amp-hosts-extended.fork.txt",
126+
"https://blocklist.sefinek.net/generated/v1/adguard/amp/ente-dev/google-amp-hosts.fork.txt",
127+
"https://blocklist.sefinek.net/generated/v1/adguard/anime/myanimelist.net.txt",
128+
"https://blocklist.sefinek.net/generated/v1/adguard/anime/shinden.pl.txt",
129+
"https://blocklist.sefinek.net/generated/v1/adguard/apps/whatsapp.txt",
130+
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/cryptojacking/Snota418/Crypto-streams.fork.txt",
131+
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/cryptojacking/firebog/Prigent/Crypto.fork.txt",
132+
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/cryptojacking/hoshsadiq/adblock-nocoin-list.fork.txt",
133+
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/cryptojacking/zerodot1/CoinBlockerLists-hosts.fork.txt",
134+
"https://blocklist.sefinek.net/generated/v1/adguard/crypto/sites/sefinek.hosts.txt",
135+
"https://blocklist.sefinek.net/generated/v1/adguard/dating-services/developerdan/extended.fork.txt",
136+
"https://blocklist.sefinek.net/generated/v1/adguard/dead-domains/jarelllama/dead-domains.fork.txt",
137+
"https://blocklist.sefinek.net/generated/v1/adguard/drugs/blocklistproject/drugs.fork.txt",
138+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/FadeMind/add-2o7Net.fork.txt",
139+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/MajkiIT/adguard-host.fork.txt",
140+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/MajkiIT/easy-privacy-host.fork.txt",
141+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/cbuijs/adult-domains-24733.fork.txt",
142+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/deathbybandaid/CountryCodesLists-France.fork.txt",
143+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/deathbybandaid/ParsedBlacklists-EasyList-Liste-FR.fork.txt",
144+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/deathbybandaid/ParsedBlacklists-EasyList.fork.txt",
145+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/hagezi/pro.fork.txt",
146+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/justdomains/adguarddns-justdomains.fork.txt",
147+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/oisd/big.fork.txt",
148+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/r-a-y/AdguardApps.fork.txt",
149+
"https://blocklist.sefinek.net/generated/v1/adguard/extensions/r-a-y/AdguardMobileSpyware.fork.txt",
150+
"https://blocklist.sefinek.net/generated/v1/adguard/fakenews/StevenBlack/hosts.fork.txt",
151+
"https://blocklist.sefinek.net/generated/v1/adguard/fakenews/marktron/hosts.fork.txt",
152+
"https://blocklist.sefinek.net/generated/v1/adguard/fraud/blocklistproject/hosts.fork.txt",
153+
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/MajkiIT/gambling-hosts.fork.txt",
154+
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/StevenBlack/hosts.fork.txt",
155+
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/TrustPositif/gambling-indonesia.fork.txt",
156+
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/blocklistproject/hosts.fork.txt",
157+
"https://blocklist.sefinek.net/generated/v1/adguard/gambling/sefinek.hosts.txt",
158+
"https://blocklist.sefinek.net/generated/v1/adguard/games/league-of-legends.txt",
159+
"https://blocklist.sefinek.net/generated/v1/adguard/games/valorant.txt",
160+
"https://blocklist.sefinek.net/generated/v1/adguard/hate-and-junk/developerdan/extended.fork.txt",
161+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/AssoEchap/stalkerware-indicators.fork.txt",
162+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/DandelionSprout-AntiMalwareHosts.fork.txt",
163+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/RPiList/Malware.fork.txt",
164+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/Spam404/main-blacklist.fork.txt",
165+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/bigdargon/hostsVN.fork.txt",
166+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/blocklistproject/malware.fork.txt",
167+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/digitalside/latestdomains.fork.txt",
168+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/disconnectme/simple-malvertising.fork.txt",
169+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/malware-filter/urlhaus-filter-hosts-online.fork.txt",
170+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/phishing.txt",
171+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/quidsup/notrack-malware.fork.txt",
172+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/reported-by-norton.txt",
173+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/sefinek.hosts1.txt",
174+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/sefinek.hosts2.txt",
175+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/suspicious.txt",
176+
"https://blocklist.sefinek.net/generated/v1/adguard/malicious/web-attacks.txt",
177+
"https://blocklist.sefinek.net/generated/v1/adguard/phishing/Dogino/Discord-Phishing-URLs-phishing.fork.txt",
178+
"https://blocklist.sefinek.net/generated/v1/adguard/phishing/RPiList/Phishing-Angriffe.fork.txt",
179+
"https://blocklist.sefinek.net/generated/v1/adguard/phishing/blocklistproject/phishing.fork.txt",
180+
"https://blocklist.sefinek.net/generated/v1/adguard/phishing/phishing.army/blocklist-extended.fork.txt",
181+
"https://blocklist.sefinek.net/generated/v1/adguard/porn/4skinSkywalker/hosts.fork.txt",
182+
"https://blocklist.sefinek.net/generated/v1/adguard/porn/Sinfonietta/pornography-hosts.fork.txt",
183+
"https://blocklist.sefinek.net/generated/v1/adguard/porn/StevenBlack/porn.fork.txt",
184+
"https://blocklist.sefinek.net/generated/v1/adguard/porn/blocklistproject/porn.fork.txt",
185+
"https://blocklist.sefinek.net/generated/v1/adguard/porn/chadmayfield/pi-blocklist-porn-all.fork.txt",
186+
"https://blocklist.sefinek.net/generated/v1/adguard/porn/oisd/nsfw.fork.txt",
187+
"https://blocklist.sefinek.net/generated/v1/adguard/porn/sefinek.hosts.txt",
188+
"https://blocklist.sefinek.net/generated/v1/adguard/ransomware/blocklistproject/ransomware.fork.txt",
189+
"https://blocklist.sefinek.net/generated/v1/adguard/redirect/blocklistproject/redirect.fork.txt",
190+
"https://blocklist.sefinek.net/generated/v1/adguard/scam/Dogino/Discord-Phishing-URLs-scam.fork.txt",
191+
"https://blocklist.sefinek.net/generated/v1/adguard/scam/blocklistproject/scam.fork.txt",
192+
"https://blocklist.sefinek.net/generated/v1/adguard/scam/durablenapkin/scamblocklist.fork.txt",
193+
"https://blocklist.sefinek.net/generated/v1/adguard/scam/jarelllama/scam.fork.txt",
194+
"https://blocklist.sefinek.net/generated/v1/adguard/sites/booth.pm.txt",
195+
"https://blocklist.sefinek.net/generated/v1/adguard/sites/gamebanana.txt",
196+
"https://blocklist.sefinek.net/generated/v1/adguard/sites/omegle.txt",
197+
"https://blocklist.sefinek.net/generated/v1/adguard/sites/patreon.txt",
198+
"https://blocklist.sefinek.net/generated/v1/adguard/sites/pinterest.txt",
199+
"https://blocklist.sefinek.net/generated/v1/adguard/sites/pixiv.txt",
200+
"https://blocklist.sefinek.net/generated/v1/adguard/sites/riotgames.txt",
201+
"https://blocklist.sefinek.net/generated/v1/adguard/social/facebook.txt",
202+
"https://blocklist.sefinek.net/generated/v1/adguard/social/instagram.txt",
203+
"https://blocklist.sefinek.net/generated/v1/adguard/social/snapchat.txt",
204+
"https://blocklist.sefinek.net/generated/v1/adguard/social/tiktok.txt",
205+
"https://blocklist.sefinek.net/generated/v1/adguard/social/twitter.txt",
206+
"https://blocklist.sefinek.net/generated/v1/adguard/spam/FadeMind/add-Spam.fork.txt",
207+
"https://blocklist.sefinek.net/generated/v1/adguard/spam/RPiList/spam-mails.fork.txt",
208+
"https://blocklist.sefinek.net/generated/v1/adguard/spam/stopforumspam/toxic-domains-whole.fork.txt",
209+
"https://blocklist.sefinek.net/generated/v1/adguard/suspicious/FadeMind/add-Risk.fork.txt",
210+
"https://blocklist.sefinek.net/generated/v1/adguard/suspicious/firebog/w3kbl.fork.txt",
211+
"https://blocklist.sefinek.net/generated/v1/adguard/test.txt",
212+
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/0Zinc/easyprivacy.fork.txt",
213+
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/MajkiIT/adguard-mobile-host.fork.txt",
214+
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/ente-dev/tv.fork.txt",
215+
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/frogeye/firstparty-trackers-hosts.txt",
216+
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/mitchellkrogza/INACTIVE.fork.txt",
217+
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/neodevpro/host.fork.txt",
218+
"https://blocklist.sefinek.net/generated/v1/adguard/tracking-and-telemetry/quidsup/trackers-hosts.fork.txt",
219+
"https://blocklist.sefinek.net/generated/v1/adguard/useless-websites/jarelllama/parked-domains.fork.txt",
220+
"https://blocklist.sefinek.net/generated/v1/adguard/useless-websites/sefinek.hosts.txt",
221+
95222
#"https://hosts.anudeep.me/mirror/adservers.txt"
96223
# LostAd [TOO BIG]
97224
#"https://raw.githubusercontent.com/lennihein/LostAd/main/lostad_dns.txt",
@@ -117,9 +244,26 @@
117244
for URL in URLS:
118245
LIST_INDEX = URLS.index(URL)
119246
FILENAME = CURRENTWORKINGDIRECTORY + "downloaded_lists\\blocklist" + str(LIST_INDEX) + ".txt"
120-
print(NOW+" - "+"Downloading: " + URL + " >> " + FILENAME)
121-
urlretrieve(URL, FILENAME)
122-
NOW = strftime("%Y-%m-%d %H:%M:%S", gmtime())
247+
# print(NOW+" - "+"Downloading: " + URL + " >> " + FILENAME)
248+
# urlretrieve(URL, FILENAME)
249+
# NOW = strftime("%Y-%m-%d %H:%M:%S", gmtime())
250+
251+
# Uses a workaround for 403 errors by applying a better "User-Agent" header to the request
252+
if URL.startswith("https://blocklist.sefinek.net/"):
253+
print(NOW+" - "+"Downloading: " + URL + " >> " + FILENAME)
254+
HEADERS = {
255+
'User-Agent': 'Mozilla 5.0',
256+
}
257+
RESPONSE = requests.get(URL, headers=HEADERS)
258+
CONTENT = RESPONSE.text
259+
if RESPONSE.status_code == 200:
260+
with open(FILENAME, "w", encoding='utf8') as file:
261+
file.write(CONTENT)
262+
NOW = strftime("%Y-%m-%d %H:%M:%S", gmtime())
263+
else:
264+
print(NOW+" - "+"Downloading: " + URL + " >> " + FILENAME)
265+
urlretrieve(URL, FILENAME)
266+
NOW = strftime("%Y-%m-%d %H:%M:%S", gmtime())
123267

124268
# Merges lists into one file
125269
# Source: https://bobbyhadz.com/blog/merge-text-files-in-python#how-to-merge-text-files-in-python
@@ -158,7 +302,8 @@
158302
with open(CURRENTWORKINGDIRECTORY + "aio_blocklist_final.txt", "w", encoding='utf-8') as new_f:
159303
for line in LINES:
160304
if not line.startswith("!") and not line.startswith("#") and not line.startswith("@@") \
161-
and not line.startswith("$removeparam="):
305+
and not line.startswith("$removeparam=") and not line.startswith("*") \
306+
and not line.startswith("-") and not line.startswith(".") and not line.startswith("/"):
162307
# Removes line prefixes
163308
line = line.replace('0.0.0.0 ','')
164309
line = line.replace('127.0.0.1 ','')
@@ -169,8 +314,11 @@
169314
line = '||' + line
170315
line = line.replace('||||','||')
171316
line = line.replace('^','^$important')
172-
if not (line == '') and not (line == ' ') and not (line == '||') and not '##' in line:
317+
if not (line == '') and not (line == ' ') and not (line == '||') and not (line == '##'):
173318
line = line.replace('^','^$important')
319+
line = line.replace('||||','||')
320+
line = line.replace('|||','||')
321+
line = line.replace('$important$important','$important')
174322
new_f.write(line)
175323
os.remove(CURRENTWORKINGDIRECTORY + "aio_blocklist.txt")
176324

0 commit comments

Comments
 (0)