Diversion Diversion 5.1.3 - the Router Ad-Blocker, May 09, 2024

[dave14305]

Guessing I didn't need to do that?

Postconf scripts aren’t really meant to be run manually. They run during the generation of the conf file by the firmware.

 

[JimbobJay]

Postconf scripts aren’t really meant to be run manually. They run during the generation of the conf file by the firmware.

OK, that's good to know. So can you confirm that that does indeed mean I didn't need to change the

CONFIG="$1"

line in your iCloud PR script (I know, off-topic for this thread. Sorry!) to make it

CONFIG="/etc/dnsmasq.conf"

? Which is what I did do to resolve the sed -i error I was getting when manually testing dnsmasq.postconf to see if your PR script was working.

That was what caused me to come across the sed -i error and line 42 errors that were occuring for the Diversion portion of the script.

Can you confirm that I do not need to worry about seeing these when running it manually, because they aren't supposed to be run manually in the first place?

 

[bibikalka]

Diversion 5.1.3 is now available

What's new

• Deregisters additional Dnsmasq instance IPs when rebooting. Thanks to @bibikalka for reporting

On some router models a reboot does not deregister virtual IP interfaces. This fixes it. @bibikalka you can remove the line in /jffs/scripts/unmount I asked to add.

I had to do another manual reboot, so the freq is about every 5-7 days. Seeking further tweaks!

I switched non-blocking IP to 16, and see this in the logs:

May 17 08:25:23 RT-AC86U-9988 avahi-daemon[7424]: Withdrawing address record for 192.168.1.17 on br0.
May 17 08:25:23 RT-AC86U-9988 avahi-daemon[7424]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.17.

When I reboot, I only see this:

May 17 04:05:05 RT-AC86U-9988 avahi-daemon[7140]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.17.
May 17 04:06:52 RT-AC86U-9988 avahi-daemon[3294]: Registering new address record for 192.168.1.17 on br0.IPv4.

I wonder if it'd make sense to "Withdrawing address record" as well upon shutdown.

 

[ComputerSteve]

So i'm noticing that even going down to the standard blocklist my Macbook Pro running Sonoma 14.5 is having an issue that sometimes when a website page loads it will not have any clickable links. I have disabled diversion and that problem doesn't happen anymore !

 

[Celo]

Hey! Just did a factory reset on my router (GT-AXE11000) and installed Diversion for the first time.
Looks to be working fine except that it seems like it's ignoring my allowed list. I've added just one domain to test, battle.net, and it appears in the blocked lists and graphs in the Diversion UI "module" (as telemetry-in.battle.net, to be more precise). I've tried adding both to the allowed list with no success. I'm using the default block list.
Thank you!

 

[HorseCalledHorse]

Hey! Just did a factory reset on my router (GT-AXE11000) and installed Diversion for the first time.
Looks to be working fine except that it seems like it's ignoring my allowed list. I've added just one domain to test, battle.net, and it appears in the blocked lists and graphs in the Diversion UI "module" (as telemetry-in.battle.net, to be more precise). I've tried adding both to the allowed list with no success. I'm using the default block list.
Thank you!

Did you update the blocking list after you added the above domain to your allow list? Diversion will prompt you to do so, but you might have missed it. If you did, go into the blocking list (b) and update it (bu).

You can also do it through the router GUI: the LAN/Diversion tab, then Update blocking list.

 

[thelonelycoder]

I had to do another manual reboot, so the freq is about every 5-7 days. Seeking further tweaks!

I switched non-blocking IP to 16, and see this in the logs:

May 17 08:25:23 RT-AC86U-9988 avahi-daemon[7424]: Withdrawing address record for 192.168.1.17 on br0.
May 17 08:25:23 RT-AC86U-9988 avahi-daemon[7424]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.17.

When I reboot, I only see this:

May 17 04:05:05 RT-AC86U-9988 avahi-daemon[7140]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.17.
May 17 04:06:52 RT-AC86U-9988 avahi-daemon[3294]: Registering new address record for 192.168.1.17 on br0.IPv4.

I wonder if it'd make sense to "Withdrawing address record" as well upon shutdown.

I am as frustrated and confused as you are about this and want it gone sooner than later.

Clearly something is grabbing that ad-blocking exclusion IP before Diversion can start the separate Dnsmasq instance.

Good to know: Some devices do not respond to ping, especially Windows 11 computers. Mine does not, so it makes it harder to diagnose what's going on.

I have added exhaustive logging to the routers System Log to help track this anomaly down for the ad-blocking exclusion feature:
When the router reboots you MUST see entries like this when it shuts down:

May 19 11:01:24 Diversion: disabling services for unmount
May 19 11:01:24 Diversion: unmount event, unregistered ad-blocking exclusion IP 192.168.3.3

During the startup some ifconfig, ping, nslookup and arp tests are done, depending on if the ad-blocking exclusion IP is found.
This is a correct boot-up sequence snippet, the way as it should be:

May 19 11:02:07 Diversion: ad-blocking exclusion IP 192.168.3.3 correctly not found with ifconfig before starting separate Dnsmasq instance
May 19 11:02:08 rc_service: cfg_server 1630:notify_rc start_bsd
May 19 11:02:08 rc_service: waitting "restart_dnsmasq" via  ...
May 19 11:02:10 Diversion: ad-blocking exclusion IP 192.168.3.3 correctly did not respond to ping before starting separate Dnsmasq instance
May 19 11:02:10 Diversion: creating br0:ad_blocking_excl for 192.168.3.3 now
May 19 11:02:10 avahi-daemon[1533]: Registering new address record for 192.168.3.3 on br0.IPv4.
May 19 11:02:10 Diversion: created br0:ad_blocking_excl for 192.168.3.3
May 19 11:02:10 dnsmasq[2702]: started, version 2.90 cachesize 150
May 19 11:02:10 dnsmasq[2702]: read /etc/hosts - 22 names
May 19 11:02:10 dnsmasq[2702]: using nameserver 192.168.2.1#53
May 19 11:02:10 dnsmasq[2702]: using nameserver 192.168.2.1#53
May 19 11:02:10 Diversion: started separate Dnsmasq instance for ad-blocking exclusion on IP 192.168.3.3
May 19 11:02:10 Diversion: restarted Dnsmasq to apply settings

If ifconfig or ping detects the ad-blocking exclusion IP during boot-up, a lot more info will be logged.
Make sure to save the complete System Log when one of these two lines appear. It will log a lot more than just this:

ad-blocking exclusion IP found with ifconfig before starting separate Dnsmasq instance, collecting data
ad-blocking exclusion IP responded to ping before starting separate Dnsmasq instance, doing nslookup and arp checks

To get the new files enter 1233 into the Diversion SSH UI, it will prompt to confirm I instructed you to do so. Confirm (Continue) it with 1.
Then maybe reboot the router and get familiar with the new System Log entries.

Hope we can track it down now with these verbose logging entries.

 

[thelonelycoder]

A new installation of diversion makes the dnsmasq.postconf file look like this

#!/bin/sh

. /opt/share/diversion/file/post-conf.div # Added by Diversion

with no CONFIG line nor reference to /etc/dnsmasq.conf, and brings up the sed -i error as well as the line 42 errors I included above (#598).

/opt/share/diversion/file/post-conf.div includes variable CONFIG definition of CONFIG=$1 in the file which is passed on when running certain scripts. In this case "$1" is /etc/dnsmasq.conf

 

[thelonelycoder]

Hey! Just did a factory reset on my router (GT-AXE11000) and installed Diversion for the first time.
Looks to be working fine except that it seems like it's ignoring my allowed list. I've added just one domain to test, battle.net, and it appears in the blocked lists and graphs in the Diversion UI "module" (as telemetry-in.battle.net, to be more precise). I've tried adding both to the allowed list with no success. I'm using the default block list.
Thank you!

What blocking list type are you using?
How are you adding the allow list entries?

These two are blocked in the Standard and Large blocking list. Obviously they are telemetry gathering sleazebags:

telemetry-in.battle.net
telemetry.battle.net

The domain battle.net however is not blocked by these two blocking list types and the website works fine with the Large list here.

If you wish, you can add the complete two telemetry domains to the allowlist and they will then get through.

 

[thelonelycoder]

So i'm noticing that even going down to the standard blocklist my Macbook Pro running Sonoma 14.5 is having an issue that sometimes when a website page loads it will not have any clickable links. I have disabled diversion and that problem doesn't happen anymore !

Good for you, ad-blocking prevents you from viewing various entertainment elements - who wants to miss that!

But seriously, I have no idea. Are you using Safari to browse the web? Did you ever clear the caches for that browser?
The Standard and Large blocking lists are extremely well curated.

 

[ComputerSteve]

Good for you, ad-blocking prevents you from viewing various entertainment elements - who wants to miss that!

But seriously, I have no idea. Are you using Safari to browse the web? Did you ever clear the caches for that browser?
The Standard and Large blocking lists are extremely well curated.

Yes I have reset the browser. I actually tried creating a brand new user on my Mac to see if something was up with my profile. The problem seems to only occur while using an adblock -- Maybe it's an apple bug idk? I just never had that problem before.

 

[Celo]

What blocking list type are you using?
How are you adding the allow list entries?

These two are blocked in the Standard and Large blocking list. Obviously they are telemetry gathering sleazebags:

telemetry-in.battle.net
telemetry.battle.net

The domain battle.net however is not blocked by these two blocking list types and the website works fine with the Large list here.

If you wish, you can add the complete two telemetry domains to the allowlist and they will then get through.

Hello, thanks for the reply!

I'm using the Standard list. I thought that by adding the main domain would also allow subdomains, seems it's not the case

Just added both subdomains as you said and now it's not blocking. Just fyi, i noticed that everything was fine whit the blocking, i just wanted to allow it because i play WoW A LOT (haha) and the domain was completely messing up the graphs in uiDivstats (and inflating the logs).

Thank you very much for your work, we appreciate it a lot!!

 

[HorseCalledHorse]

Yes I have reset the browser. I actually tried creating a brand new user on my Mac to see if something was up with my profile. The problem seems to only occur while using an adblock -- Maybe it's an apple bug idk? I just never had that problem before.

Can you give some URL examples? I'm also running Sonoma 14.5 and would be happy to check.

 

[ComputerSteve]

Can you give some URL examples? I'm also running Sonoma 14.5 and would be happy to check.

The thing is its random, & sometimes the same exact link that didn't work with reload works.. So for example.. I'm browsing on facebook. I click something on the news feed it takes me to another webpage.. The webpage loads but I can't click on anything.. So then I refresh the page and the links work. I even have the problem just typing something in the safari bar and it loads google search.. Sometimes I can't click on any of the links. I think it has something to do with the website isn't loading fully or a script isn't loading because of the adblock since if I disable the adblock of diversion I don't get the problem.

 

[RMerlin]

Good to know: Some devices do not respond to ping, especially Windows 11 computers. Mine does not, so it makes it harder to diagnose what's going on.

Windows often defaults to Public network profiles when first connected to a network. Switching it to Private will make ping respond.

Dunno what Microsoft where thinking, back in the day they would prompt you to select if you wanted to use a Public or Private profile when connecting for the first time to a new network or router. Now they just quietly use a public profile.

 

[L&LD]

For (default) safety, I would assume. I don't know how many people I've helped get their devices to 'see' their printers by changing this setting for them. But again, most devices on a home network I've seen do not connect to anything else (just to the 'net).

 

[thelonelycoder]

Windows often defaults to Public network profiles when first connected to a network. Switching it to Private will make ping respond.

Dunno what Microsoft where thinking, back in the day they would prompt you to select if you wanted to use a Public or Private profile when connecting for the first time to a new network or router. Now they just quietly use a public profile.

StackOverflow suggested to change firewall rules and some other things. I stopped at reading after the second sentence mentioning ICMP packets being whatever. Now I know the real reason. Will confirm later.

Edit: All four network adapters on my primary Windows 11 PC were set to Public network profile - go figure. Even says this is the recommended setting...
Switched them to Private and all is good now with ping.

 

[bibikalka]

I am as frustrated and confused as you are about this and want it gone sooner than later.

...

To get the new files enter 1233 into the Diversion SSH UI, it will prompt to confirm I instructed you to do so. Confirm (Continue) it with 1.
Then maybe reboot the router and get familiar with the new System Log entries.

Hope we can track it down now with these verbose logging entries.

Alright, I did enter your double secret 1233 code into Diversion, it downloaded the updates. Will monitor things over the week, and will let you know if anything strange comes up.

Many thanks for continuing to look into this!

 

[RMerlin]

StackOverflow suggested to change firewall rules and some other things. I stopped at reading after the second sentence mentioning ICMP packets being whatever. Now I know the real reason. Will confirm later.

Edit: All four network adapters on my primary Windows 11 PC were set to Public network profile - go figure. Even says this is the recommended setting...
Switched them to Private and all is good now with ping.

If you trust your LAN, and you don't want to deal with random issues communicating with other LAN devices (like networked printers), using the Private profile is a better choice.

 

[bibikalka]

I am as frustrated and confused as you are about this and want it gone sooner than later.

...

To get the new files enter 1233 into the Diversion SSH UI, it will prompt to confirm I instructed you to do so. Confirm (Continue) it with 1.
Then maybe reboot the router and get familiar with the new System Log entries.

Hope we can track it down now with these verbose logging entries.

Alright, had a episode during this morning's reboot! Here is the relevant section of the log:

May 22 04:06:55 RT-AC86U-9988 Diversion: Mounting Diversion WebUI page
May 22 04:06:55 RT-AC86U-9988 vnstatd[4146]: vnStat daemon 2.12 (pid:4146 uid:0 gid:0 64-bit, SQLite 3.41.2)
May 22 04:06:55 RT-AC86U-9988 vnstatd[4146]: Data retention: 720 5MinuteHours, 31 HourlyDays, 31 DailyDays, 13 MonthlyMonths, -1 YearlyYears, 15
May 22 04:06:55 RT-AC86U-9988 vnstatd[4146]: Monitoring (1): eth0 (1000 Mbit)
May 22 04:06:55 RT-AC86U-9988 Diversion: Mounted Diversion WebUI as user1.asp in LAN/Diversion
May 22 04:06:56 RT-AC86U-9988 Diversion: ad-blocking exclusion IP 192.168.1.16 correctly not found with ifconfig before starting separate Dnsmasq
May 22 04:06:59 RT-AC86U-9988 Diversion: ad-blocking exclusion IP 192.168.1.16 correctly did not respond to ping before starting separate Dnsmasq
May 22 04:06:59 RT-AC86U-9988 Diversion: creating br0:ad_blocking_excl for 192.168.1.16 now
May 22 04:06:59 RT-AC86U-9988 avahi-daemon[3487]: Registering new address record for 192.168.1.16 on br0.IPv4.
May 22 04:06:59 RT-AC86U-9988 Diversion: created br0:ad_blocking_excl for 192.168.1.16
May 22 04:06:59 RT-AC86U-9988 dnsmasq[4416]: failed to create listening socket for 192.168.1.16: Address already in use
May 22 04:06:59 RT-AC86U-9988 dnsmasq[4416]: FAILED to start up
May 22 04:06:59 RT-AC86U-9988 Diversion: started separate Dnsmasq instance for ad-blocking exclusion on IP 192.168.1.16
May 22 04:06:59 RT-AC86U-9988 Diversion: restarted Dnsmasq to apply settings
May 22 04:06:59 RT-AC86U-9988 (dnsmasq.postconf): Updating /etc/dnsmasq.conf for unbound.....
May 22 04:06:59 RT-AC86U-9988 uiDivStats: dnsmasq has restarted, restarting taildns
May 22 04:06:59 RT-AC86U-9988 dnsmasq[4446]: failed to create listening socket for 192.168.1.1: Address already in use
May 22 04:06:59 RT-AC86U-9988 dnsmasq[4446]: FAILED to start up

Here is a good reboot from yesterday:

May 21 04:06:55 RT-AC86U-9988 Diversion: Mounted Diversion WebUI as user2.asp in LAN/Diversion
May 21 04:06:55 RT-AC86U-9988 uiScribe: Mounting WebUI page for uiScribe
May 21 04:06:55 RT-AC86U-9988 uiScribe: Mounted uiScribe WebUI page as Main_LogStatus_Content.asp
May 21 04:06:55 RT-AC86U-9988 Diversion: ad-blocking exclusion IP 192.168.1.16 correctly not found with ifconfig before starting separate Dnsmasq
May 21 04:06:58 RT-AC86U-9988 Diversion: ad-blocking exclusion IP 192.168.1.16 correctly did not respond to ping before starting separate Dnsmasq
May 21 04:06:58 RT-AC86U-9988 Diversion: creating br0:ad_blocking_excl for 192.168.1.16 now
May 21 04:06:58 RT-AC86U-9988 avahi-daemon[3242]: Registering new address record for 192.168.1.16 on br0.IPv4.
May 21 04:06:58 RT-AC86U-9988 Diversion: created br0:ad_blocking_excl for 192.168.1.16
May 21 04:06:58 RT-AC86U-9988 dnsmasq[5252]: started, version 2.90 cachesize 150
May 21 04:06:58 RT-AC86U-9988 dnsmasq[5252]: compile time options: IPv6 GNU-getopt no-RTC no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP
May 21 04:06:58 RT-AC86U-9988 dnsmasq[5252]: read /etc/hosts - 22 names
May 21 04:06:58 RT-AC86U-9988 dnsmasq[5252]: using nameserver 172.16.1.254#53
May 21 04:06:58 RT-AC86U-9988 dnsmasq[5252]: using nameserver 172.16.1.254#53 for domain attlocal.net
May 21 04:06:58 RT-AC86U-9988 dnsmasq[5252]: using nameserver 172.16.1.254#53
May 21 04:06:58 RT-AC86U-9988 dnsmasq[5252]: using nameserver 172.16.1.254#53 for domain attlocal.net
May 21 04:06:58 RT-AC86U-9988 Diversion: started separate Dnsmasq instance for ad-blocking exclusion on IP 192.168.1.16
May 21 04:06:58 RT-AC86U-9988 Diversion: restarted Dnsmasq to apply settings
May 21 04:06:58 RT-AC86U-9988 (dnsmasq.postconf): Updating /etc/dnsmasq.conf for unbound.....
May 21 04:06:58 RT-AC86U-9988 uiDivStats: dnsmasq has restarted, restarting taildns