Diversion Diversion - the Router Ad-Blocker

[Butterfly Bones]

I too have seen, and regularly do see my pixelserv-tls thread count go up over 100, and the memory grows too. Then it stops and works it way down to 2 or 3 threads. No idea what is causing this yet.

That was explained by kvic in the pixelserv-tls thread back in the earlier active days of development. That is by design when you get say 15-20 people all web browsing and all those ads hitting so pixelserv-tls increases the number requests per service thread and then the number of active service threads. When / if activity slows, the number of service threads will decrease.

It all shows in the pixelserv stats.

pixelserv-tls 2.3.1 (compiled: Jan 31 2020 13:27:14 flags: tfo tls1_3) options: 192.168.1.2

uts 0d 00:44 process uptime
log 1 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc 1 number of active service threads
kmx 1 maximum number of service threads
kvg 0.00 average number of requests per service thread
krq 0 max number of requests by one service thread

 

[thelonelycoder]

@thelonelycoder quick question for you. I noticed that in your rotate-logs script you still maintain dnsmasq.log2 (weekly) and dnsmasq.log1 (yesterday) even if weekly stats is turned off. Curious what else you use the dnsmasq.log1/2 for? Seems like a waste to keep around unless other parts are using it, and it is extra disk and computation.

personally my dnsmasq logs grow very quickly, so I am turning off stats generation. It took over 11 minutes last night to generate the stats, and that is on my RT-AX88U which is quite fast.

Thank you for your time.

Logging is enabled.

 

[juched]

Logging is enabled.

Gotcha. Thanks.

 

[GregS]

I'm seeing an odd issue on my RT-AX88U, after awhile no DNS requests get resolved. nslookup from any client on the network will time out, following the dnsmasq log shows no queries. If I disable diversion then DNS resolution starts working again, enable and it stops again. A reboot will resolve the issue so dns will work while diversion is enabled; but the issue will return days later.
Any ideas what could be wrong? Or what I should check on the next time this happens. I did force a diversion update during the last issue but that didn't help.

For background I have been using Diversion for a few years on an AC68U and only recently upgraded to an AX88U as repeated requests to pixelserv from my ShieldTV kept overloading the router. I'm using the Standard+ blocking list. I'm using v384.15_0 of Merlin's firmware.

 

[L&LD]

@GregS, did you reuse your USB drive that was running Diversion on the RT-AC68U to the RT-AX88U?

If you did, I would recommend formatting the USB drive on a PC and 'Format JFFS partition on next reboot' to completely wipe out your previous configuration. Make sure to save anything of importance on the JFFS partition first, if needed.

Then, use amtm to format the USB drive to Ext4 w/journaling and create a swap file. Now, do a clean install of the scripts you use, beginning with Diversion.

 

[dave14305]

Hi,

when installing diversion on a AC86 running latest merlin's beta, i get this message.
Router Config is default. do i need to disable uPnP?
thanks

i Checking port 443 availability
____________________________________________________

✖ Port 443 is in use by unknown application
(AiCloud port is OK, check other applications)

Here's the netstat output:

tcp 0 0 0.0.0.0:44300 0.0.0.0:* LISTEN 9736/miniupnpd

1. Continue, pixelserv-tls will work in your opinion
2. Abort installation

Enter your selection [1-2]

You will be OK to continue. The install function is catching 44300 instead of just 443 (partial match).

 

[ugandy]

You will be OK to continue. The install function is catching 44300 instead of just 443 (partial match).

thanks. upnp media server was on.

 

[bitmonster]

I'm not sure if this is the right place to ask - but is it possible to implement a Murdoch Media blocklist? I am just disgusted by their behaviour and want all Murdoch sites blocked so I can do my bit to starve this parasite of clicks and revenue. Diversion blocklists would be the perfect way to do it, as I also have my phone pretty much permanently VPN'd through home so it would protect my phone from this disease too.

https://iview.abc.net.au/show/media-watch/series/0/video/FA1935H008S00

 

[maghuro]

I'm not sure if this is the right place to ask - but is it possible to implement a Murdoch Media blocklist? I am just disgusted by their behaviour and want all Murdoch sites blocked so I can do my bit to starve this parasite of clicks and revenue. Diversion blocklists would be the perfect way to do it, as I also have my phone pretty much permanently VPN'd through home so it would protect my phone from this disease too.

https://iview.abc.net.au/show/media-watch/series/0/video/FA1935H008S00

AFAIK you can create your own blacklist on top of any list you're using.

Just add the domain(s) you want to block after installing diversion

 

[jrmwvu04]

I stopped using Pixelserv a week or so ago after I noticed 184 pixelserv-tls processes (really threads) stemming from a blocked domain triggered while my Termius iOS ssh client was running. It never let up. So some apps may be persistent and repeatedly try to access a blocked domain rapidly in succession, spawning many pixelserv-tls threads visible in htop. I believe I read before that Instagram was another such app.

Otherwise, Diversion itself is very memory friendly if using anything smaller than the Large blocking list. Plus hosts are also of questionable value for me personally and add a lot of hosts to the memory footprint.

Start small, start Lite and observe.

It likes to ping e.crashlyrics.com and doesn’t accept no for an answer. Not that uncommon in iOS apps in my experience

 

[Butterfly Bones]

I stopped using Pixelserv a week or so ago after I noticed 184 pixelserv-tls processes (really threads) stemming from a blocked domain triggered while my Termius iOS ssh client was running. It never let up. So some apps may be persistent and repeatedly try to access a blocked domain rapidly in succession, spawning many pixelserv-tls threads visible in htop. I believe I read before that Instagram was another such app.

Otherwise, Diversion itself is very memory friendly if using anything smaller than the Large blocking list. Plus hosts are also of questionable value for me personally and add a lot of hosts to the memory footprint.

Start small, start Lite and observe.

It likes to ping e.crashlyrics.com and doesn’t accept no for an answer. Not that uncommon in iOS apps in my experience

Back during pixelserv active development, kvic gave an alternate way to handle those pesky carrier domains. I've run this way since those early days.

# /jffs/configs/hosts.add

0.0.0.0 t.appflyer.com
0.0.0.0 x.flyme.com
0.0.0.0 e.crashlytics.com
0.0.0.0 app-measurement.com
0.0.0.0 gateway-carry.icloud.com                 
0.0.0.0 e12930.ksd.akamaiedge.net                 
0.0.0.0 ssl.google-analytics.com
0.0.0.0 mesu.g.aaplimg.com                       
0.0.0.0 gsp64-ssl.ls-apple.com.akadns.net         
0.0.0.0 p3-buy.itunes-apple.com.akadns.net       
0.0.0.0 e673.dsce9.akamaiedge.net                 
0.0.0.0 settings.crashlytics.com
0.0.0.0 reports.crashlytics.com
0.0.0.0 onesignal.com
0.0.0.0 data.flurry.com
0.0.0.0 api.branch.io
0.0.0.0 www.google-analytics.com
0.0.0.0 adservice.google.com
0.0.0.0 ad.doubleclick.net
0.0.0.0 ssl.google-analytics.com
0.0.0.0 securepubads.g.doubleclick.net
0.0.0.0 ib.adnxs.com
0.0.0.0 sb.scorecardresearch.com
0.0.0.0 js-agent.newrelic.com
0.0.0.0 www.googleadservices.com
0.0.0.0 ads.servebom.com
0.0.0.0 api.amplitude.com

 

[GregS]

@GregS, did you reuse your USB drive that was running Diversion on the RT-AC68U to the RT-AX88U?

If you did, I would recommend formatting the USB drive on a PC and 'Format JFFS partition on next reboot' to completely wipe out your previous configuration. Make sure to save anything of importance on the JFFS partition first, if needed.

Then, use amtm to format the USB drive to Ext4 w/journaling and create a swap file. Now, do a clean install of the scripts you use, beginning with Diversion.

Basically yes, it's from an AC86U that I returned. I'll try your idea and report back. Thanks

 

[Ubimo]

@GregS
Are you also running Skynet?

 

[GregS]

@GregS
Are you also running Skynet?

Yes, though during the last issue toggling it on/off didn't make a noticeable difference.

 

[dave14305]

Yes, though during the last issue toggling it on/off didn't make a noticeable difference.

Is it any better if you disable logging in Diversion? Maybe it’s too slow to write the log to usb.

 

[Wrkdbf_Guy]

Stupid question probably, but I tried to install Diversion Lite on my ASUS RT-AC68U router but I got a message telling me the USB drive is "readable but not writable". So I exited the install.

As background, I formatted the USB drive via Windows to NTSF. Then using AMTM, I formatted the drive as EXT4 with Journalling; also creating a 2GB swap file. Everything formatted fine. (Not sure it's relevant, but I do see a "recovering Journal" message in the AMTM Disk Check log.)

I am a career, now retired, Software Developer - though on a different platform than Unix/Linux - so I understand profiles, permissions, etc. But not sure how to change the USB drive via the router, to make it writable. TIA, Bill

 

[L&LD]

@Wrkdbf_Guy if trying to format it once again proves ineffective, just use/buy a different USB drive.

 

[GregS]

Is it any better if you disable logging in Diversion? Maybe it’s too slow to write the log to usb.

I'll give that a try next time but it seems unlikely as it's a new/fast USB drive and the log doesn't move that fast normally, maybe a few queries per second during busy times and only 1 query per few seconds during slow times. Plus after I disable and re-enable there's nothing coming through in the logs. Also the AC68U instance that it replaced has a much slower USB and was handling the load fine for several years. Worth a try though, thanks.

 

[GregS]

@dave14305 , @Ubimo , @L&LD
It just happened again, after the full format/re-install. Disabling the log didn't seem to make a difference. But I noticed if I turned off Skynet and Diversion, then turned Diversion back on, it would work. I tried many variations but I can't seem to get both Diversion and Skynet to work together unless I reboot; though each will work fine on it's own while in this state. To test after each change I just ran an nslookup from my computer. When I can get responses consistently without timeouts I call that a success, but there were 2 types of failure. The first is what happens initially when this problem starts, nslookup always times out and there is no activity in the log. The second type happens after I turn both Diversion and Skynet back on, responses frequently time out once or twice before returning an answer, though the log shows these requests. So basically any dns lookup takes about 4-8seconds. If while both are on I turn off Skynet then I get that first type of failure again, I must then turn Diversion off then on to get things working again (minus skynet). Not sure if it matters but I was turning these on and off via scMerlin and not through their respective GUIs. I haven't rebooted yet, is there something else I should try? Or a setting to change so it won't break again after the next reboot?

 

[dave14305]

@dave14305 , @Ubimo , @L&LD
It just happened again, after the full format/re-install. Disabling the log didn't seem to make a difference. But I noticed if I turned off Skynet and Diversion, then turned Diversion back on, it would work. I tried many variations but I can't seem to get both Diversion and Skynet to work together unless I reboot; though each will work fine on it's own while in this state. To test after each change I just ran an nslookup from my computer. When I can get responses consistently without timeouts I call that a success, but there were 2 types of failure. The first is what happens initially when this problem starts, nslookup always times out and there is no activity in the log. The second type happens after I turn both Diversion and Skynet back on, responses frequently time out once or twice before returning an answer, though the log shows these requests. So basically any dns lookup takes about 4-8seconds. If while both are on I turn off Skynet then I get that first type of failure again, I must then turn Diversion off then on to get things working again (minus skynet). Not sure if it matters but I was turning these on and off via scMerlin and not through their respective GUIs. I haven't rebooted yet, is there something else I should try? Or a setting to change so it won't break again after the next reboot?

What are your WAN DNS settings? LAN DHCP Server DNS settings? LAN DNSFilter settings?

What is the full output of running nslookup snbforums.com on your computer?