AdGuardHome Max rules in AdGuardHome to prevent router freezing?

[CB7]

Hi,

Was wondering if there's a maximum of rules a list can have to stop AdGuardHome from "crashing" when updating?
I have a couple of lists on AGH, including DBL OISD FULL which contains about 1.1 million lines. I noticed that whenever AGH does its weekly update: the whole thing freezes (and by the looks of it: when processing DBL OISD). The RAM spikes, the CPU goes to 100% and it seems to be running out of memory. This causes DNS to become unavailable for hours on end and thus internet not functioning adequately. (Strangely, the memory issue I thought would be addressed by adding swap; but it doesn't really seem to remediate the issue.). The router has to be rebooted to solve it.

Curious if I have a configuration issue or the router is simply not "powerful" enough to handle such a large blocklist?
Kinda sucks because the whole point of running AGH on the router was to ensure that during a power failure all essential services (internet + dns (with filtering) + VPN) would continue running on the UPS.

I could consider upgrading from AX3000 to AX6000 (or RT-AX88U to solve the scarceness of ethernet ports), but not sure if 1GB of RAM will actually solve the issue; that'd be an expensive gamble.

 

[Gary_Dexter]

OISD Big only has 140k entries - not 1.1 million.

Are you using the correct list formats?

 

[Tech9]

or RT-AX88U to solve the scarceness of ethernet ports

If you need more GbE LAN ports - get a switch for $15-20 instead. This router is from 2018 and two hardware generations behind. No 3006 firmware support and perhaps few years left before End-of-Life. Newer models with better hardware and 3006 firmware are RT-AX88U Pro* and RT-BE88U**.

* - 2.0GHz CPU and 1GB RAM, 3004 based Asuswrt-Merlin at the moment
** - 2.6GHz CPU and 2GB RAM, 3006 based Asuswrt-Merlin available now

 

[CB7]

OISD Big only has 140k entries - not 1.1 million.

Are you using the correct list formats?

Not OISD Big, but OISD Full; has 1.074.864 entries. (https://gitlab.com/morganas/dbl-oisd-nl/-/raw/master/dbl.txt)
I think its in the correct format, yes

If you need more GbE LAN ports - get a switch for $15-20 instead. This router is from 2018 and two hardware generations behind. No 3006 firmware support and perhaps few years left before End-of-Life. Newer models with better hardware and 3006 firmware are RT-AX88U Pro* and RT-BE88U**.

* - 2.0GHz CPU and 1GB RAM, 3004 based Asuswrt-Merlin at the moment
** - 2.6GHz CPU and 2GB RAM, 3006 based Asuswrt-Merlin available now

Yep, got a managed switch - but prefer eliminating devices. Thanks for the advice; that BE88U with 2GB RAM sounds like a dream; price is a nightmare. Wish they'd have an entry model with just more RAM. But I appreciate that's probably a niche market. So I guess options are getting AX88U Pro, BE88U or offload DNS to a container on the NAS and take the uptime hit with power outages. (Currently often due to construction work) Maybe I can make a cron job on the Asus that checks whether or not DNS is responsive on the NAS and if not: start AdGuard Home. (Or temporarily simply redirect traffic to a filtered external DNS). If it comes back up: switch back.

Well either that or use smaller lists in AdGuard Home I suppose to cut down on its resource usage when running list updates.

Thanks!

 

[dave14305]

Not OISD Big, but OISD Full; has 1.074.864 entries. (https://gitlab.com/morganas/dbl-oisd-nl/-/raw/master/dbl.txt)
I think its in the correct format, yes

That list hasn't updated in 3 years.

#  Entries: 1074864
#  TimeUpdated: 2021-05-27T06:51:56+02:00

According to OISD.nl, the proper format for AGH is https://big.oisd.nl

 

[CB7]

That list hasn't updated in 3 years.

#  Entries: 1074864
#  TimeUpdated: 2021-05-27T06:51:56+02:00

According to OISD.nl, the proper format for AGH is https://big.oisd.nl

Huh, that's stupid. I hadn't thought of checking whether or not those are still being maintained. I got rid of a couple of 'em, including the old hostnet backup (fredprod). Maybe it'll have a better time now.

Do disabled lists still get updated by AdGuard or does it simply skip over them if they're not enabled?

 

[Gary_Dexter]

Huh, that's stupid. I hadn't thought of checking whether or not those are still being maintained. I got rid of a couple of 'em, including the old hostnet backup (fredprod). Maybe it'll have a better time now.

Do disabled lists still get updated by AdGuard or does it simply skip over them if they're not enabled?

It will ignore disabled lists. Just delete them though as there’s no point keeping them (it will also keep a cache of the list which isn’t needed)

 

[thelonelycoder]

Not OISD Big, but OISD Full; has 1.074.864 entries. (https://gitlab.com/morganas/dbl-oisd-nl/-/raw/master/dbl.txt)
I think its in the correct format, yes

Why would you use a stale third party link when there‘s the real thing? Obviously this is not oisd.nl

 

[EmeraldDeer]

For DNS blocking on my RT-AX88U, I am using Diversion in Dnsmasq format. I am using two different big, general, but balanced lists and two NSFW and gambling lists. Diversion is reporting a coalesced list of 1.2 million. I am not experiencing any issues.

https://big.oisd.nl/dnsmasq2
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt

https://nsfw.oisd.nl/dnsmasq2
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/gambling.txt

 

[Gary_Dexter]

Why would you use a stale third party link when there‘s the real thing? Obviously this is not oisd.nl

It’s a fork. And an unmaintained one at that.

Hagezi’s lists are miles better than OISD

 

[Gary_Dexter]

For DNS blocking on my RT-AX88U, I am using Diversion in Dnsmasq format. I am using two different big, general, but balanced lists and two NSFW and gambling lists. Diversion is reporting a coalesced list of 1.2 million. I am not experiencing any issues.

https://big.oisd.nl/dnsmasq2
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt

https://nsfw.oisd.nl/dnsmasq2
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/gambling.txt

Good list choices.

If you’re using Hagezi Pro you don’t need OISD as 99% of the domains are in Hagezi’s lists anyway

Diversion is different to AGH

 

[CB7]

For DNS blocking on my RT-AX88U, I am using Diversion in Dnsmasq format. I am using two different big, general, but balanced lists and two NSFW and gambling lists. Diversion is reporting a coalesced list of 1.2 million. I am not experiencing any issues.

https://big.oisd.nl/dnsmasq2
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt

https://nsfw.oisd.nl/dnsmasq2
https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/gambling.txt

Thanks! I'll check 'em out. I don't use Diversion (I prefer something with a UI and an app (+ I can link AGH to HomeBridge)), but will absolutely check that list out. -edit- Turns out I already had Hagezi enabled in AGH

Why would you use a stale third party link when there‘s the real thing? Obviously this is not oisd.nl

Because I once added them years ago and never bothered to look whether or not they're still being maintained. They "just work".

It will ignore disabled lists. Just delete them though as there’s no point keeping them (it will also keep a cache of the list which isn’t needed)

Thanks, I'll get going with that then!

 

[SomeWhereOverTheRainBow]

Good list choices.

If you’re using Hagezi Pro you don’t need OISD as 99% of the domains are in Hagezi’s lists anyway

Diversion is different to AGH

Lately I use:

1Hosts Lite
Hagezi Multi-Normal
Hagezi TIF
xRuffKeyz NRD Phishing 30-days
OISD Big

But you can also get all this curated from my lists.

GitHub - jumpsmm7/GeneratedAdblock

Contribute to jumpsmm7/GeneratedAdblock development by creating an account on GitHub.

github.com

Theres about 10 lists to choose from. I generate RPZ for bind and unbound. There is also several flavors compatible with dnsmasq. Also, there is unbound NXDOMAIN, or unbound NULL. There are also filters compatible for AdGuardhome and pihole. I recommend if you use the Filters, that you choose the conservative lists only if you are on a limited RAM situation because they have been condensed down to just the TLD and domain without the extra subdomains, but be warned this can also cause more need to allowlist.

 

[Gary_Dexter]

Lately I use:

1Hosts Lite
Hagezi Multi-Normal
Hagezi TIF
xRuffKeyz NRD Phishing 30-days
OISD Big

But you can also get all this curated from my lists.

GitHub - jumpsmm7/GeneratedAdblock

Contribute to jumpsmm7/GeneratedAdblock development by creating an account on GitHub.

github.com

Theres about 10 lists to choose from. I generate RPZ for bind and unbound. There is also several flavors compatible with dnsmasq. Also, there is unbound NXDOMAIN, or unbound NULL. There are also filters compatible for AdGuardhome and pihole. I recommend if you use the Filters, that you choose the conservative lists only if you are on a limited RAM situation because they have been condensed down to just the TLD and domain without the extra subdomains, but be warned this can also cause more need to allowlist.

I think the NRD phishing domains are included in Hagez’s TIF list anyway.

Used to use 1hosts but there was a period where the lists weren’t maintained/maintained regularly and the maintainer wasn’t as active on GH to fix any issues etc.

And I think OISD is not needed as Hagezi includes the domains/sources that make up that list in the TIF and Multi lists anyway.

 

[SomeWhereOverTheRainBow]

I think the NRD phishing domains are included in Hagez’s TIF list anyway.

Used to use 1hosts but there was a period where the lists weren’t maintained/maintained regularly and the maintainer wasn’t as active on GH to fix any issues etc.

And I think OISD is not needed as Hagezi includes the domains/sources that make up that list in the TIF and Multi lists anyway.

I use them to catch any missing domains. It would be a different story if each list was updates one after the other so there was no room for overlap, but I use all of them for redundancy. And from what I can tell the NRD lists is not yet been included. Hagezi points people to use it if they like though. The list is relatively new.

GitHub - hagezi/dns-blocklists: DNS-Blocklists: For a better internet - keep the internet clean!

DNS-Blocklists: For a better internet - keep the internet clean! - hagezi/dns-blocklists

github.com

 

[Gary_Dexter]

I use them to catch any missing domains. It would be a different story if each list was updates one after the other so there was no room for overlap, but I use all of them for redundancy. And from what I can tell the NRD lists is not yet been included. Hagezi points people to use it if they like though. The list is relatively new.

GitHub - hagezi/dns-blocklists: DNS-Blocklists: For a better internet - keep the internet clean!

DNS-Blocklists: For a better internet - keep the internet clean! - hagezi/dns-blocklists

github.com

The phishing NRD’s are included:

dns-blocklists/sources.md at main · hagezi/dns-blocklists

DNS-Blocklists: For a better internet - keep the internet clean! - hagezi/dns-blocklists

github.com

The normal NRD ones aren’t (14 and 30 day) as they are MASSIVE lists and have to be split into two separate ones (over 7million domains on 1 part of the 30 day list…!!)

 

[SomeWhereOverTheRainBow]

The phishing NRD’s are included:

dns-blocklists/sources.md at main · hagezi/dns-blocklists

DNS-Blocklists: For a better internet - keep the internet clean! - hagezi/dns-blocklists

github.com

The normal NRD ones aren’t (14 and 30 day) as they are MASSIVE lists and have to be split into two separate ones (over 7million domains on 1 part of the 30 day list…!!)

I think you are partially right. I manually downloaded both the 30 day phishing list, and the tif list. Then I used sort command to count the number of common lines:
sort tif.txt nrd-phishing-30day.txt | uniq -d | wc -l.
It turns out there are only 6984 common entries between the two lists. The 30 day phishing list itself contains 8600 entries. So technically the entire 30 day phishing list isnt entirely included inside the tif list.

 

[Gary_Dexter]

I believe Hagezi removes any NX or dead domains in his compilations to keep things clean and any unnecessary lines out of his lists.

 

[Amwjujo]

Lately I use:

1Hosts Lite
Hagezi Multi-Normal
Hagezi TIF
xRuffKeyz NRD Phishing 30-days
OISD Big

But you can also get all this curated from my lists.

GitHub - jumpsmm7/GeneratedAdblock

Contribute to jumpsmm7/GeneratedAdblock development by creating an account on GitHub.

github.com

Theres about 10 lists to choose from. I generate RPZ for bind and unbound. There is also several flavors compatible with dnsmasq. Also, there is unbound NXDOMAIN, or unbound NULL. There are also filters compatible for AdGuardhome and pihole. I recommend if you use the Filters, that you choose the conservative lists only if you are on a limited RAM situation because they have been condensed down to just the TLD and domain without the extra subdomains, but be warned this can also cause more need to allowlist.

Hi,
Are you using all 5 lists stated above together?
I recently moved my adguard + Unbound server on a Proxmox LXC and I'm trying to understand what is best to use on my recently configuration.
Any advice is highly appreciated.
Thank you.

 

[Toinks]

Sharing lists I use, in case others find it useful.
Note rule counts are always as of certain date.

1Hosts (Lite) - 88,802 rules
https://badmojr.github.io/1Hosts/
List for blocking pesky ads, trackers, and malware. Balanced version: set and forget, doesn't hamper user experience.

HaGeZi's Pro mini - 87,134 rules
https://github.com/hagezi/dns-blocklists?tab=readme-ov-file#promini
Size-optimised version for DNS/Browser adblockers. This list only contains domains from the Pro full that have been found on Top 1M lists (Umbrella, Cloudflare, Tranco, Chrome, ...) in the last 12 months.

HaGeZi's TIF mini - 94,218 rules
https://github.com/hagezi/dns-blocklists?tab=readme-ov-file#tifmini
A size-optimised version of the Threat Intelligence Feeds Medium list. Designed for Adblockers that have problems with the size of the TIF Medium list.

xRuffKeyz NRD Phishing 30-days - 8,634 rules
https://github.com/xRuffKez/NRD/tree/main/lists/30-day_phishing
NRD list useful for blocking recently registered domains, many of which are associated with malicious behavior, advertisements, or unwanted tracking.

OISD Big Safe Top-N - 121,874 rules
https://github.com/cbuijs/oisd/tree/master/big
Optimized version of OISD Big blocklist. Safe list, suggested excludes removed. Domains actually used according TOP-N lists.

OISD NSFW Safe Top-N - 68,734 rules
https://github.com/cbuijs/oisd/tree/master/nsfw
Optimized version of OISD NSFW blocklist. Safe list, suggested excludes removed. Domains actually used according TOP-N lists.