Diversion Diversion large list from oisd.nl crushes my GT-AX6000

[vlord]

I love the large list from oisd. It works (near) perfectly in eliminating trackers and adverts while allowing applications to work without having to fiddle with whitelists. With the standard medium list, links from slickdeals won’t work without whitelisting them and some Roku apps break.

Unfortunately I can’t use the large list. On my router, after about 8-10 hours, DHCP stops working, and then clients drop off wifi. I suspect a memory or IO issue because the CPUs are humming at 25% (1 cpu generally gets pegged at 100%). For now I’ve dropped back to using a custom combination of StevenBlack and oisd.nl/basic which seems to catch a high percentage of the adverts.

Are there things I can tweak to enable the large list to run without breaking my router? Currently am running:
1. Diversion w/ pixelserv
2. Skynet
3. Unbound (with no addons)
4. ntpMerlin using Chrony
5. scMerlin
6. uiDivStats
7. vnStat

This is a snapshot after switching to the custom list above.

Mem: 973064K used, 45720K free, 2964K shrd, 10512K buff, 424588K cached
CPU:  3.8% usr  1.3% sys  0.2% nic 94.5% idle  0.0% io  0.0% irq  0.1% sirq
Load average: 0.14 0.12 0.29 2/199 26745
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4204     1 nobody   S    81436  7.9   0  0.0 chronyd -r -u nobody -f /jffs/addons/ntpmerlin.d/chrony.conf
 4207     1 nobody   S    76364  7.4   3  0.0 unbound -c /opt/var/lib/unbound/unbound.conf
 6468     1 nobody   S    65332  6.3   2  0.0 pixelserv-tls 192.168.1.2
 2979     1 admin    S    23136  2.2   0  0.0 amas_lib
18665     1 nobody   S    20476  2.0   3  3.4 dnsmasq --log-async
 2917     1 admin    S    19520  1.9   2  0.0 conn_diag
 3291     1 admin    S    18256  1.7   1  0.0 amas_portstatus
 2915     1 admin    S    18256  1.7   3  0.0 roamast

This is from before I reverted and was on the large list

Mem: 978052K used, 40732K free, 3028K shrd, 10044K buff, 372048K cached
CPU: 25.1% usr  1.4% sys  0.2% nic 73.1% idle  0.0% io  0.0% irq  0.1% sirq
Load average: 1.03 1.15 1.18 3/198 13062

Clearly a difference in the load averages but nothing too concerning from this output.

Is anyone successfully running the large list from OISD w/ Diversion?

 

[bluzfanmr1]

What about creating a swap file via amtm to see if that helps?

 

[vlord]

What about creating a swap file via amtm to see if that helps?

I currently have a 2GB swap file which is why I thought maybe it was an IO issue

 

[dave14305]

dnsmasq 2.86 and higher are better optimized for large lists, but not in the hosts file format currently used by Diversion.

I had adapted a small script to use oisd.nl on Merlin using address= and local= options. I was using the small list without any problems. I don’t recall if I tried the big list, but I only had an AC86U at the time, so not as much RAM to play with.

Which Ad-Blocker

Out of curiosity.

www.snbforums.com

 

[bluzfanmr1]

I currently have a 2GB swap file which is why I thought maybe it was an IO issue

I've found my GT-AX6000 doesn't use the swap file because swappiness is set at a default of 0. You can check by using the free command. I ended up modifying the default swappiness according to this post:

https://www.snbforums.com/threads/b...-issues-when-swap-is-needed.80355/post-785361

After this change the swap file is being used. I just loaded the large list and everything seems to be fine. I will run it for awhile and see if there are any issues.

BTW, the blocklist has about 845,xxx domains, correct?

 

[SomeWhereOverTheRainBow]

I love the large list from oisd. It works (near) perfectly in eliminating trackers and adverts while allowing applications to work without having to fiddle with whitelists. With the standard medium list, links from slickdeals won’t work without whitelisting them and some Roku apps break.

Unfortunately I can’t use the large list. On my router, after about 8-10 hours, DHCP stops working, and then clients drop off wifi. I suspect a memory or IO issue because the CPUs are humming at 25% (1 cpu generally gets pegged at 100%). For now I’ve dropped back to using a custom combination of StevenBlack and oisd.bl/basic which seems to catch a high percentage of the adverts.

Are there things I can tweak to enable the large list to run without breaking my router? Currently am running:
1. Diversion w/ pixelserv
2. Skynet
3. Unbound (with no addons)
4. ntpMerlin using Chrony
5. scMerlin
6. uiDivStats
7. vnStat

This is a snapshot after switching to the custom list above.

Mem: 973064K used, 45720K free, 2964K shrd, 10512K buff, 424588K cached
CPU:  3.8% usr  1.3% sys  0.2% nic 94.5% idle  0.0% io  0.0% irq  0.1% sirq
Load average: 0.14 0.12 0.29 2/199 26745
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4204     1 nobody   S    81436  7.9   0  0.0 chronyd -r -u nobody -f /jffs/addons/ntpmerlin.d/chrony.conf
 4207     1 nobody   S    76364  7.4   3  0.0 unbound -c /opt/var/lib/unbound/unbound.conf
 6468     1 nobody   S    65332  6.3   2  0.0 pixelserv-tls 192.168.1.2
 2979     1 admin    S    23136  2.2   0  0.0 amas_lib
18665     1 nobody   S    20476  2.0   3  3.4 dnsmasq --log-async
 2917     1 admin    S    19520  1.9   2  0.0 conn_diag
 3291     1 admin    S    18256  1.7   1  0.0 amas_portstatus
 2915     1 admin    S    18256  1.7   3  0.0 roamast

This is from before I reverted and was on the large list

Mem: 978052K used, 40732K free, 3028K shrd, 10044K buff, 372048K cached
CPU: 25.1% usr  1.4% sys  0.2% nic 73.1% idle  0.0% io  0.0% irq  0.1% sirq
Load average: 1.03 1.15 1.18 3/198 13062

Clearly a difference in the load averages but nothing too concerning from this output.

Is anyone successfully running the large list from OISD w/ Diversion?

See my post below for the best advice... https://www.snbforums.com/threads/d...isd-nl-crushes-my-gt-ax6000.84958/post-840945

 

[vlord]

I've found my GT-AX6000 doesn't use the swap file because swappiness is set at a default of 0. You can check by using the free command. I ended up modifying the default swappiness according to this post:

https://www.snbforums.com/threads/b...-issues-when-swap-is-needed.80355/post-785361

After this change the swap file is being used. I just loaded the large list and everything seems to be fine. I will run it for awhile and see if there are any issues.

BTW, the blocklist has about 845,xxx domains, correct?

That’s correct and I was able to run it for about 8hrs before everything came to a screeching halt. Clients would drop off wifi and DHCP stopped responding to client requests.

 

[vlord]

I've found my GT-AX6000 doesn't use the swap file because swappiness is set at a default of 0. You can check by using the free command. I ended up modifying the default swappiness according to this post:

https://www.snbforums.com/threads/b...-issues-when-swap-is-needed.80355/post-785361

After this change the swap file is being used. I just loaded the large list and everything seems to be fine. I will run it for awhile and see if there are any issues.

BTW, the blocklist has about 845,xxx domains, correct?

Sure enough.. not using any swap. I’ll add that and test again later:

admin@RT-AX86U:/jffs/scripts# free
             total       used       free     shared    buffers     cached
Mem:       1018784     978704      40080       3100      12236     415960
-/+ buffers/cache:     550508     468276
Swap:      2097148          0    2097148
admin@RT-AX86U:/jffs/scripts#

 

[SomeWhereOverTheRainBow]

Sure enough.. not using any swap. I’ll add that and test again later:

admin@RT-AX86U:/jffs/scripts# free
             total       used       free     shared    buffers     cached
Mem:       1018784     978704      40080       3100      12236     415960
-/+ buffers/cache:     550508     468276
Swap:      2097148          0    2097148
admin@RT-AX86U:/jffs/scripts#

It does use the swap, (but at the last minute when your router is already about to crash). To capture its use, you would have to capture it at the moments right before your router crashes.
Here is my advice:
Use diversion without pixelserv-tls. (use diversion lite). And sharing the words of the man, the myth, and the legend @Martinski - here is how you enable swappiness:
https://www.snbforums.com/threads/adguard-home-uses-97-of-ram.82212/post-808363
The way the current swappiness is setup, is great for routers that will almost never use a swap. It has been shown in many threads that it still "will" use a swap with the swappiness set to zero; however, it comes at a cost of putting you closer to OOM than most people (and their routers) will be able to tolerate.

 

[SomeWhereOverTheRainBow]

Sure enough.. not using any swap. I’ll add that and test again later:

admin@RT-AX86U:/jffs/scripts# free
             total       used       free     shared    buffers     cached
Mem:       1018784     978704      40080       3100      12236     415960
-/+ buffers/cache:     550508     468276
Swap:      2097148          0    2097148
admin@RT-AX86U:/jffs/scripts#

And here is my divine revelation to you...

Diversion - pixelserv-tls high CPU usage

Hi all, pixelserv-tls is consuming about 36% of my AC88u's CPU. Is this to be expected? Both my cores are close to maxed out at 100%. I'm getting slow internet speeds, and wondering if this is potentially part of the problem. Everything is updated... Thanks in advance...

www.snbforums.com

It is not the list, but is the fact you are using the list in combination with pixelserv-tls.

 

[vlord]

And here is my divine revelation to you...

Diversion - pixelserv-tls high CPU usage

Hi all, pixelserv-tls is consuming about 36% of my AC88u's CPU. Is this to be expected? Both my cores are close to maxed out at 100%. I'm getting slow internet speeds, and wondering if this is potentially part of the problem. Everything is updated... Thanks in advance...

www.snbforums.com

It is not the list, but is the fact you are using the list in combination with pixelserv-tls.

Thanks so much for the suggestions. I disabled pixelserv, enabled swappiness, and am running the large list. Will see how the next 24 hours goes and report back.

 

[bluzfanmr1]

Thanks so much for the suggestions. I disabled pixelserv, enabled swappiness, and am running the large list. Will see how the next 24 hours goes and report back.

24 hours later running the large list and I have had no issues. I do run pixelserv-tls and no issues there either. Swappping is happening, though not much:

             total       used       free     shared    buffers     cached
Mem:       1018784     977664      41120       3848      51572     281440
-/+ buffers/cache:     644652     374132
Swap:      2097148       2048    2095100

 

[vlord]

24 hours later running the large list and I have had no issues. I do run pixelserv-tls and no issues there either. Swappping is happening, though not much:

             total       used       free     shared    buffers     cached
Mem:       1018784     977664      41120       3848      51572     281440
-/+ buffers/cache:     644652     374132
Swap:      2097148       2048    2095100

I’m glad it’s working for you because I had to turn it back off again. DHCP assignments were taking 5+ minutes and one CPU was pegged at 100%. I tried disabling Diversion stats but didn’t seem to help. For reference I have a 18 Amazon devices and 8 Roku devices that are the two highest diversion blocks out there (in the thousands each day). I’m now suspecting that Diversion is single threaded and that a queue is building in processing requests which is causing overall slower networking performance. One test I happened upon was playing Crossy Road. Repeated fast forward action movement would result in glitches or pauses of Crossy Road. Disabling Diversion allowed fluid game play. Highly subjective, I know, but for me it was a good indicator. Have you tried going from wifi to cellular and back or rebooting your router to see if clients are able to quickly obtain a DHCP address? What does Top - C show for your CPU’s. Thanks for willing to be a Guinea pig in helping sort this.

As an aside, I removed diversion, unbound, and NTP Merlin and installed Adguard Home (also keeping Skynet). Granted this is a much smaller blocking list to start and I’ve not yet added OISD large list in yet. Just wanted to compare the two to see if Adguard might be a suitable alternative if I can’t get Diversion sorted, or perhaps even if I can.

 

[bluzfanmr1]

I had one Roku device and it was notorious for flooding the router with requests when it couldn’t phone home. I’ll bet with 8 of them being blocked that it’s too much for the router. Can you look at the stats and prove or disprove this theory?

I ended up sinkholing the Roku requests by mapping them to a nonexistent IP. That seemed to reduce the request significantly. Maybe try that?

 

[SomeWhereOverTheRainBow]

I had one Roku device and it was notorious for flooding the router with requests when it couldn’t phone home. I’ll bet with 8 of them being blocked that it’s too much for the router. Can you look at the stats and prove or disprove this theory?

I ended up sinkholing the Roku requests by mapping them to a nonexistent IP. That seemed to reduce the request significantly. Maybe try that?

I think that is one solution @dave14305 talked about in one of these other threads. When the client domains requests were causing issues with pixelserv-tls, one option was to send them individually to 0.0.0.0 instead of pixelserv-tls address. At that rate it would be easier to send them all to 0.0.0.0 and save on the extra cpu bombardment and memory strain.

 

[vlord]

The domains were blocked with the “medium” block list as well, but Adguard home is showing about the same with 10K block requests within 24 hours using their default block list. By sinkholing them, do you mean setting the hosts file to resolve them to 127.0.0.1 or something different?

Can diversion use the default block lists from AGH? I’ve been pretty impressed with what AGH blocks out of the box, even though it’s a much smaller list than oisd.

 

[SomeWhereOverTheRainBow]

The domains were blocked with the “medium” block list as well, but Adguard home is showing about the same with 10K block requests within 24 hours using their default block list. By sinkholing them, do you mean setting the hosts file to resolve them to 127.0.0.1 or something different?

Can diversion use the default block lists from AGH? I’ve been pretty impressed with what AGH blocks out of the box, even though it’s a much smaller list than oisd.

View attachment 50109

This is what I mean:

Diversion - pixelserv-tls high CPU usage

Hi all, pixelserv-tls is consuming about 36% of my AC88u's CPU. Is this to be expected? Both my cores are close to maxed out at 100%. I'm getting slow internet speeds, and wondering if this is potentially part of the problem. Everything is updated... Thanks in advance...

www.snbforums.com

 

[Mutzli]

What USB drive do you use for the swap file? Not sure, but it could be that a slow USB drive is causing your problems.

I have an USB-SSD drive attached and diversion with pixelserv works great. My blocklist has over 1.235 million entries incl. oisd.nl big list.

 

[bluzfanmr1]

The domains were blocked with the “medium” block list as well, but Adguard home is showing about the same with 10K block requests within 24 hours using their default block list. By sinkholing them, do you mean setting the hosts file to resolve them to 127.0.0.1 or something different?

Can diversion use the default block lists from AGH? I’ve been pretty impressed with what AGH blocks out of the box, even though it’s a much smaller list than oisd.

View attachment 50109

If I remember correctly, this is what I did. Create the hosts.add file and enter the line below. Save and when finished, restart dnsmasq.

nano /jffs/configs/hosts.add

0.0.0.0 scribe.logs.roku.com

service restart_dnsmasq

You should now see it in your hosts file.

cat /etc/hosts

 

[vlord]

Thanks! I will add that to my hosts file to cut down on the chatter and processing.