Diversion Diversion - the Router Ad-Blocker

[thelonelycoder]

@thelonelycoder

OK,
Run into an issue and need to cancel the install and choose Lite.
View attachment 14535

I can not change 192.168.1.2 as that is another WiFi router on my network and can not change it.

Best way to cancel the install? or at least have the option to go back?

Instead of changing the start of the IP pool address, you can change the end of it.
For example "IP Pool Ending Address" would be 192.168.1.253. This would free up 192.168.1.254 for use with pixelserv-tls.

 

[thelonelycoder]

@thelonelycoder, I think I found another insect to be squashed... I whitelisted reddit.com which was in the wildcard blacklist and when proceeding to whitelist www.reddit.com (which I didn't have to, as the wildcard entry ws removed by my previous action), the following happened:

 1. Add domain
 2. Delete domain
 3. Process whitelist
 4. Sort and verify whitelist
 5. Restore whitelist from backup
 6. Set domain active/inactive (#! )

 Select what to do [1-6 e=Exit] 1
____________________________________________________

 Add domain like so: example.com or www.example.com

 An explanatory comment can be added after this step

 Enter domain  [e=Exit] www.reddit.com
____________________________________________________

 no exact match found in blocking file for
 www.reddit.com
 no need to add it to the whitelist

 no near matches found either
  !  match(es) will be removed from file(s)
 when added to whitelist.

 Add it to whitelist? [1=Yes 2=No] 2 <<==== No, thank you, I don't want you to.
____________________________________________________

 www.reddit.com
 was not found in the blocking file or blacklist
 If you add it anyway, it may have no effect unless
 you select a larger blocking file.

 The domain will be marked #(forced-entry) and is
 not re-added to the blocking file if you remove it.

 Add it to whitelist anyway? [1=Yes 2=No] 2 <<==== No, I'm absolutely sure, I don't want you to. Thank you.
____________________________________________________

Regardless of the fact that there was no need to whitelist www.reddit.com if I had read the text properly, I'd expect Diversion to return to the previous menu after hitting 2 for the first time when asked 'Add it to the whitelist?'. Instead, Diversion again confirms it's really not in the blocking file or blacklist and asks me if I'm sure whether I want to whitelist it. Looks like there's some duplicate code here?

It's a like a frickin' Micro$oft product asking me to confirm twice whether I'm really sure I don't want to do something

That is fixed now, no version change as it was such a simple modification.
In the Diversion UI use 12 or do a force Update in d to update the affected functions.div.

 

[tomsk]

No need to speculate. So far I can rule out that this - or anything else in that file for that matter - causes dnsmasq to repeatedly restart.

This line comes before the 3 seconds wait, so it would show up in the Syslog:

logger -t Diversion "diversion.conf is locked by another process, waiting 3secs, from $0"

Ah yes--- i noticed the logger statement in there but i didn't consider it would be written before the 3 sec delay. I was pawing though the various files in /opt/share/diversion/files and the also opt/bin/diversion file
but i couldn't see how the lock file gets generated in the first place. Could you explain the mechanism and what its for?.... As you mentioned not likely a cause of the dnsmasq restarts people are experiencing... but nice to know if it ever pops up in a log. Thanks

 

[thelonelycoder]

Ah yes--- i noticed the logger statement in there but i didn't consider it would be written before the 3 sec delay. I was pawing though the various files in /opt/share/diversion/files and the also opt/bin/diversion file
but i couldn't see how the lock file gets generated in the first place. Could you explain the mechanism and what its for?.... As you mentioned not likely a cause of the dnsmasq restarts people are experiencing... but nice to know if it ever pops up in a log. Thanks

Look in write-config.div. The lock is to prevent two instances (or more) writing to the diversion.conf at the same time.

 

[Adamm]

Found a small bug in update-bf.div

    if [ -f "${DIVERSION_DIR}/list/blockinglist" ] || [ -f "${DIVERSION_DIR}/list/blockinglist_fs" ]; then
        pgl='https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext&useip=0.0.0.0'
        pglfile="${DIVERSION_DIR}/backup/hostsfile_$(echo $pgl | sed 's#http[s]*://##;s#/#-#g;s#?#-#g;s#=.*##g')"
        if [ ! -f "${pglfile}" ] || [ "$(find ${pglfile} -mtime +6)" ]; then
            echo " getting pgl.yoyo.org file to lower memory usage while updating"
            curl_dl "$pgl" | sed 's/^0.0.0.0/'$blockingIP'/g;s/\r$//' | grep -w ^$blockingIP | awk '{print $1 " " $2}' >"${DIVERSION_DIR}/backup/blockinglist.tmp"
            if [ -s "${DIVERSION_DIR}/backup/blockinglist.tmp" ] && grep -q "^$blockingIP" "${DIVERSION_DIR}/backup/blockinglist.tmp"; then
                mv -f "${DIVERSION_DIR}/backup/blockinglist.tmp" "${pglfile}"
            elif [ -s "${pglfile}" ]; then
                echo " download temporary pgl.yoyo.org file failed, using old file"
                rm -f "${DIVERSION_DIR}/backup/blockinglist.tmp"
            fi
        fi

The yoyo host file is unnecessarily downloaded during the update process, I assume this is legacy code left over from AB where this list was manually included rather then part of the unified list. Would also be nice to see a feature we previously discussed, where if you whitelist a domain it also whitelists the non www variant (and vise versa).

 

[58chev]

Instead of changing the start of the IP pool address, you can change the end of it.
For example "IP Pool Ending Address" would be 192.168.1.253. This would free up 192.168.1.254 for use with pixelserv-tls.

ThanX.

I will give this a try on the weekend (only time I can muck around with the network), My IP pool is small from 50 - 150. So I don't see an issue.

 

[IT Guy]

I would really like to have the function -to be able too "select my own host file source" -insted being force to use the preselected ones of "small, standard, large....". To clearify I only want to use the approved host file that my company has approved. So no other hostfiles from diffrent unapproved sources like the ones that is set as default in Diversion. Hope you understand

 

[M@rco]

I would really like to have the function -to be able too "select my own host file source" -insted being force to use the preselected ones of "small, standard, large....". To clearify I only want to use the approved host file that my company has approved. So no other hostfiles from diffrent unapproved sources like the ones that is set as default in Diversion. Hope you understand

That's possible by using a Custom hostfile. It doesn't have to use one of the predefined blocking files, you can start from scratch and enter a link to the host file your company has approved. As long as you make sure it is in the correct format for Diversion to process it. From Diversion main menu, select b, 1, 2 to Customize.

 

[Skeptical.me]

The Syslog will have entries giving a hint what's going on. Post it.

It appears this issue was related to my VPN profiles. DNS was leaking, and WiFi devices were not appearing on the VPN iptables list for some reason. I deleted ALL profiles, then rebooted first. Then I set up only 1 profile with a .ovpn. then I made sure all devices were connected correctly and no DNS leaks were occurring. Someone on here was helping with this but it appears I frustrated them with my lack of CL experience (it wasn't M@rco ). However, the issue has stopped since addressing the VPN issue. I'm not sure how they were related but it has stopped.

I have one question. Is it possible to block social media sharing widgets on webpages. All ads are blocked even in the Apple News App, which is terrific. However, on webpages it would be great to get rid of those social media things. If not, its all good, I can just use a script blocker.

 

[thelonelycoder]

Found a small bug in update-bf.div

    if [ -f "${DIVERSION_DIR}/list/blockinglist" ] || [ -f "${DIVERSION_DIR}/list/blockinglist_fs" ]; then
        pgl='https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext&useip=0.0.0.0'
        pglfile="${DIVERSION_DIR}/backup/hostsfile_$(echo $pgl | sed 's#http[s]*://##;s#/#-#g;s#?#-#g;s#=.*##g')"
        if [ ! -f "${pglfile}" ] || [ "$(find ${pglfile} -mtime +6)" ]; then
            echo " getting pgl.yoyo.org file to lower memory usage while updating"
            curl_dl "$pgl" | sed 's/^0.0.0.0/'$blockingIP'/g;s/\r$//' | grep -w ^$blockingIP | awk '{print $1 " " $2}' >"${DIVERSION_DIR}/backup/blockinglist.tmp"
            if [ -s "${DIVERSION_DIR}/backup/blockinglist.tmp" ] && grep -q "^$blockingIP" "${DIVERSION_DIR}/backup/blockinglist.tmp"; then
                mv -f "${DIVERSION_DIR}/backup/blockinglist.tmp" "${pglfile}"
            elif [ -s "${pglfile}" ]; then
                echo " download temporary pgl.yoyo.org file failed, using old file"
                rm -f "${DIVERSION_DIR}/backup/blockinglist.tmp"
            fi
        fi

The yoyo host file is unnecessarily downloaded during the update process, I assume this is legacy code left over from AB where this list was manually included rather then part of the unified list. Would also be nice to see a feature we previously discussed, where if you whitelist a domain it also whitelists the non www variant (and vise versa).

That's not a bug. While updating the blocking file, a smaller temp file is used to lower the memory footprint so that the router does not run out of memory. Sorting, removing duplicates and whitelisted domains is memory intensive. The yoyo file is perfect as a temp blocking file.

 

[dugaduga]

2.2.0-rc.4 is available

• NEW indicator of TLS 1.3 support status on servstats page.
• FIXED failed to log server name on unsuccessful handshakes. Garbage may be captured instead. When it happens it may lead to crash or a hung process. (issue reported from @Asad Ali @Protik @jrmwvu04)

For details, pls read the release page as usual.

Looks like my best time in this thread. I have ppl handling the frontline in the thread and have other ppl working hard in silos on tests! thank you. Time for me to hit the sac now.

This update has appeared to have fixed my issue with pixelserv-tls hanging/crashing; Since reverting to Ab-Solution, with the latest entware packages, I have had zero problems... where previously diversion would restart dnsmasq over and over, and load and unload diversion into memory after turning off samba, sig checks, and often while updating blocklists.... often breaking browsing and nullifying adblocking altogether, and sometimes even upon booting.

 

[dugaduga]

2.2.0-rc.4 is available

• NEW indicator of TLS 1.3 support status on servstats page.
• FIXED failed to log server name on unsuccessful handshakes. Garbage may be captured instead. When it happens it may lead to crash or a hung process. (issue reported from @Asad Ali @Protik @jrmwvu04)

For details, pls read the release page as usual.

Looks like my best time in this thread. I have ppl handling the frontline in the thread and have other ppl working hard in silos on tests! thank you. Time for me to hit the sac now.

This update has appeared to have fixed my issue with pixelserv-tls hanging/crashing; Since reverting to Ab-Solution, with the latest entware packages, I have had zero problems... where previously diversion would restart dnsmasq over and over, and simultaneously diversion would load and unload into memory, over and over again, after turning off samba, sig checks, and often, while updating blocklists. This persisted with a factory default and a clean /jffs folder with no previous scripts using the latest meriln beta.

Here is the debug log: https://pastebin.com/uEyKsAcN

Tomsk may be onto something:

i noticed that in the post-conf.div file you have this piece of code

if [ -s "${DIVERSION_DIR}/.conf/diversion.conf" ]; then

    if [ -f /tmp/diversion.lock ]; then
        logger -t Diversion "diversion.conf is locked by another process, waiting 3secs, from $0"
        sleep 3
        rm -f /tmp/diversion.lock
    fi

seeing as post-conf.div is called from dnsmasq.postconf does this mean that dnsmasq won't restart for 3 seconds? .... I'm wondering if there is an edge case where some other process sees dnsmasq not started during that 3 secs, and then tries to restart it.... and rinse and repeat..... *pure speculation mode ON*

I wonder if testing by removing "sleep 3" this could be factored out,

 

[Adamm]

That's not a bug. While updating the blocking file, a smaller temp file is used to lower the memory footprint so that the router does not run out of memory. Sorting, removing duplicates and whitelisted domains is memory intensive. The yoyo file is perfect as a temp blocking file.

My mistake, I assumed its purpose wrong when quickly reading the output

 

[thelonelycoder]

This update has appeared to have fixed my issue with pixelserv-tls hanging/crashing; Since reverting to Ab-Solution, with the latest entware packages, I have had zero problems... where previously diversion would restart dnsmasq over and over, and simultaneously diversion would load and unload into memory, over and over again, after turning off samba, sig checks, and often, while updating blocklists. This persisted with a factory default and a clean /jffs folder with no previous scripts using the latest meriln beta.

Here is the debug log: https://pastebin.com/uEyKsAcN

Tomsk may be onto something:



I wonder if testing by removing "sleep 3" this could be factored out,

If that were the the problem, the Syslog would show the log entry. And since I have not seen it in any of the posted logs, I can rule that out.

 

[Xentrk]

It appears this issue was related to my VPN profiles. DNS was leaking, and WiFi devices were not appearing on the VPN iptables list for some reason. I deleted ALL profiles, then rebooted first. Then I set up only 1 profile with a .ovpn. then I made sure all devices were connected correctly and no DNS leaks were occurring. Someone on here was helping with this but it appears I frustrated them with my lack of CL experience (it wasn't M@rco ). However, the issue has stopped since addressing the VPN issue. I'm not sure how they were related but it has stopped.

I have one question. Is it possible to block social media sharing widgets on webpages. All ads are blocked even in the Apple News App, which is terrific. However, on webpages it would be great to get rid of those social media things. If not, its all good, I can just use a script blocker.

Here is a snip from https://x3mtek.com/torguard-openvpn-2-4-client-setup-for-asuswrt-merlin-firmware/ regarding the DNS Leak issue with Diversion + dnsmasq + OpenVPN Client. Based on recent testing, I need to update this work-around solution to also include the setting Accept DNS Configuration=None. I also need to update AB-Solution references to Diversion!

Spoiler: Diversion+dnsmasq+OpenVPN

AB-Solution is the ad blocking solution for Asus routers using Asuswrt-Merin firmware. AB-Solution requires DNSmasq to work properly. With Asuswrt-Merlin firmware, OpenVPN clients use the VPN tunnel’s DNS. As a result, AB-Solution will not work for LAN clients connected to the VPN tunnel when using Policy Rules since DNSmasq is by-passed. AB-Solution will still work for devices connected to the WAN though.

John9547 LTS fork has implemented DNS differently than Asuswrt-Merlin. The DNS rules are reversed. With Accept DNS Configuration set to Exclusive, the VPN clients will use DNSmasq and AB-Solution will work. There is also a check box on how you want to handle the WAN clients. If you leave it unchecked, the WAN clients will also use the VPN DNS servers (but not the tunnel) and they can use AB-Solution. If you check the box, the WAN client requests are sent directly to the WAN DNS servers and AB-Solution will not be available.

To resolve the DNS and routing issues when using Policy Rules with Asuswrt-Merlin, set Accept DNS Configuration to “Strict” and specify the DNS server for the VPN tunnel to use by adding the dhcp-option DNS command in the Custom Configuration section. Without the dhcp-option command, AB-Solution updates will fail, the AB-Solution email function will no longer work and the wget command will not able to resolve the domain name. The downside with these settings is that DNS will leak. Having my DNS leak has not caused me any issues for my use case.

 

[sentinelvdx]

@thelonelycoder I don't know if it's just me but everytime I restart my router for X reason, Diversion is loaded, but pixelserv does not start. I have to get in Diversion's menu an restart pixelserv manually.
That was not happening on abs

Sent from S.G. S9+ Duos

 

[bluzfanmr1]

I switched from a gmail address to an Apple iCloud email address and now I can't get the email function to work as it did with the gmail address. I am running 384.7_beta2 and the latest Diversion.

Here are the settings I have in Diversion:

  1. Edit From address:   xxx@icloud.com
  2. Edit To name:        xxxx
  3. Edit To address:     xxx@icloud.com
  4. Edit Router name:    RT-AC66U_B1
  5. Edit User name:      xxx@icloud.com
  6. Edit Password:       Apple ID App-Specific Pwd
  7. Edit SMTP Server:    smtp.mail.me.com
  8. Edit Server port:    587
  9. Edit Protocol:       smtp
 10. Edit SSL flag:       --insecure

Each time I get this curl error message:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   299    0     0  100   299      0    328 --:--:-- --:--:-- --:--:--   328
curl: (56) Failure when receiving data from the peer

 [ ✖ ] sending testmail failed

 Note the curl: error above and check your settings

 [ ! ] Press [Enter] to return to menu

As seen above I did generate an Apple App-Specific Password and I also tried various different SSL flags without any luck. I also tried using just the user name without the @icloud.com and that doesn't matter either. I searched around everywhere looking for an answer but couldn't find one so I'm hoping someone can help.

Thanks for this forum and any help you may provide.

 

[Waffull]

Thank you for all your hard work. I sent you a donation.
Suggestion, since Paypal redirects to https://www.ab-solution.info/thank-you.html, setup a cname record to redirect ab-solution.info to diversion.ch.

Great work... Even though my port didn't work, it didn't properly take the storage location... So I just uninstall and started fresh. I assume, although I haven't tried yet, that I can import an old whitelist backup.

 

[Skeptical.me]

Am I able to add any tracking blockers hots files, and if so where would I find them and how would I load them?

 

[M@rco]

I have one question. Is it possible to block social media sharing widgets on webpages. All ads are blocked even in the Apple News App, which is terrific. However, on webpages it would be great to get rid of those social media things. If not, its all good, I can just use a script blocker.

Short answer: no. Long answer: only if they're not embedded in the page and originate from a domain that you can block, without blocking relevant content. You can try taking a look at the source of a page to see where it originates, which might be the same as the domain you're loading the view the page (hence, you can't block it). You probably need a script blocker for this.

Am I able to add any tracking blockers hots files, and if so where would I find them and how would I load them?

(Domains with) several trackers are already included. If you use something like Privacy Badger for instance, you'll still see hits, simply because they recognize the tracking url in the webpage. However, Diversion blocks them from actually contacting their HQ as the request is re-routed. If you do see trackers calling home, you can either blacklist them or search for a more extensive list which includes them.