Diversion Diversion - the Router Ad-Blocker

[Dabombber]

Looks like diversion is reading /proc/swaps and messed up the logic a bit. Just ignore the message for now.

@thelonelycoder
From line 451

    else
        # No reference in post-mount
        swaplocation="$(find /tmp/mnt/*/myswap.swp 2> /dev/null)"
        if [ "$(echo "$swaplocation" | wc -l)" -ge 2 ]; then
            swtext=" Multiple swap files found, only one file is\\n supported. Run sw to delete."
        elif [ -f "$swaplocation" ]; then
            # File found without a reference
            writeswaploc
            swtext=" Added missing swap file entry to\\n /jffs/scripts/post-mount"
#        elif [ "$(wc -l < /proc/swaps)" -ge "2" ]; then # should be -eq
#            #Swap mounted without a reference (should be ignored, likely handled outside diversion)
#            swaplocation="$(sed -n '2p' /proc/swaps | awk '{print $1}')"
#            if [ -f "$swaplocation" ]; then
#                writeswaploc
#                swtext=" Added missing swap file entry to\\n /jffs/scripts/post-mount"
#            else
#                # Swap mounted with no file, ie a partition
#                sed -i '\~swapon ~d' /jffs/scripts/post-mount
#                swtext=" No swap file found, reference removed in\\n /jffs/scripts/post-mount"
#            fi
        #elif [ "$swaplocation" ]; then
        elif [ "$(wc -l < /proc/swaps)" -gt 2 ]
            # Multiple swaps mounted
        fi
    fi

EDIT: Oh wait, I see what the last elif was for now. maybe use

"$(echo "$swaplocation" | wc -l)" -ge 2

before the -f check to detect multiple files.

 

[thelonelycoder]

No, it doesn't remove anything, it just prints that message.

Maybe I would need to run the script differently, so Diversion and AMTM doesn't see it as a swapfile that doesn't exist. Any recommendation?

I got this, let me do some testing. If @Adamm can do basic support of a swap partition in Skynet, so can I in Diversion and amtm

Thanks @Dabombber for your suggestions above. I'm on it right now.

 

[Adamm]

If @Adamm can do basic support of a swap partition in Skynet, so can I in Diversion and amtm

The emphasis being the word basic, I still firmly believe a swap file is a significantly better solution.

 

[thelonelycoder]

The emphasis being the word basic, I still firmly believe a swap file is a significantly better solution.

Same here, that's why I didn't even bother looking into it when I wrote the self-fixing check. But looking at your code I found a simple additional test to at least acknowledge the use of a swap partition and honor and respect what the user did.

 

[EeK]

yes, thanks you. good find

I DO NOT recommend using such a large hosts file in Diversion.

Listen to @thelonelycoder. I was doing some testing using different blacklists (for unrelated issues - my Xbox achievements still haven't unlocked, even without any scripts running), and such a massive hosts file will cause severe issues, making the router eat pretty much all of the RAM (I only had 20 MB free) and reducing everything to a crawl when connecting via SSH. I had to format both my USB drive and jffs partition and start from scratch to "fix" it.

Speaking of which, I found out the answer to one my previous questions. Only two remain.

1. What is the predefined blocking list in b: Small, Standard, Medium or Large? A = Standard (and, as far as I can tell, there's no point n using either Medium or Large)
2. The www/non-www domain should remain in its default setting ("off")?
3. No domains in the hosted whitelist can start with "www"? For some reason, in all the Xbox whitelists I found, two of the several domains in them included the full address ("www.domain.com").

 

[thelonelycoder]

2. The www/non-www domain should remain in its default setting ("off")?

I answered that already. With a hosted whitelist that setting does not matter.

3. No domains in the hosted whitelist can start with "www"? For some reason, in all the Xbox whitelists I found, two of the several domains in them included the full address ("www.domain.com").

Whitelistimg is done with the exact term. www.domain.com and domain.com are treated separately. If the domain you want to whitelist uses www, then you have to add it as www.domain.com to the whitelist.

 

[Jumpstarter]

Speaking of such crazy stuff.. I am interested in turning on the DS feature and using the diversion-dnsmasq.... or is that just quick buttons that change the values of the routers dnsmasq.conf? --- or does it help to benefit the performance of ad-blocking.....

Also can I route my traffic through Diversion if Pixel-servtls address is 192.168.1.2........

or can I serve it up through DNS-Filter Global mode router??????

 

[EeK]

I answered that already. With a hosted whitelist that setting does not matter.

Indeed, you've mentioned that it doesn't affect hosted whitelists. I was just wondering how it works when you're not using a hosted whitelist. If the blocklist is hardcoded into Diversion, does that setting ever matter? It's set to "off" by default for a reason, right?

Whitelistimg is done with the exact term. www.domain.com and domain.com are treated separately. If the domain you want to whitelist uses www, then you have to add it as www.domain.com to the whitelist.

I believe I saw in one of your FAQs that the hosted whitelist required all domains to be in the "domain.com/otherdomain.com" format, without "www". It's okay to have required domains in the full "www.domain.com" format, then?

 

[EeK]

On a separate note, I can confirm that the issue with Xbox achievements has been resolved - the two "queued" achievements finally popped up, after playing a lot more of the same game.

The following whitelist restores all basic functionality to the console, and is the most up-to-date, as of now (Microsoft has been known for requiring additional domains to be whitelisted after some time):

activity.windows.com
attestation.xboxlive.com
cert.mgt.xboxlive.com
client-s.gateway.messenger.live.com
clientconfig.passport.net
continuum.dds.microsoft.com
ctldl.windowsupdate.com
def-vef.xboxlive.com
device.auth.xboxlive.com
eds.xboxlive.com
help.ui.xboxlive.com
licensing.xboxlive.com
mobile.pipe.aria.microsoft.com
notify.xboxlive.com
settings-win.data.microsoft.com
title.auth.xboxlive.com
title.mgt.xboxlive.com
v10.events.data.microsoft.com
v10.vortex-win.data.microsoft.com
v20.events.data.microsoft.com
v20.vortex-win.data.microsoft.com
www.msftncsi.com
www.xboxlive.com
xbox.ipv6.microsoft.com
xboxexperiencesprod.experimentation.xboxlive.com
xflight.xboxlive.com
xkms.xboxlive.com
xsts.auth.xboxlive.com

The list was compiled from Pi-hole's FAQ, Kamil's Lab, several Reddit threads and posts on Xbox forums, and by my own experimentation.

Despite claims by some users, domains like "bing.com" and "strict.bing.com" are not required to be whitelisted at this moment.

Again, that may change in the future, but, at least for now, that's the most comprehensive list you'll need for everything related to your Xbox One family of devices (Xbox One, Xbox One S, Xbox One X) - online multiplayer, messaging, Achievements, etc.

 

[thelonelycoder]

I am interested in turning on the DS feature and using the diversion-dnsmasq.... or is that just quick buttons that change the values of the routers dnsmasq.conf

None of the available options in ds are configurable through the UI AFAIK. Read carefully Simon Kelleys explanations before letting the dog loose on these.
What I recommend for better Diversion stats accuracy is bogus-priv and domain-needed. Look them up and read for yourself.
Anything else keep your hands off unless you actually need it set.

Also can I route my traffic through Diversion if Pixel-servtls address is 192.168.1.2........

No, pixelserv-tls is not a resolver/forwarder. Only Dnsmasq does this on your router and this is where Diversion ties in.

 

[thelonelycoder]

If the blocklist is hardcoded into Diversion, does that setting ever matter? It's set to "off" by default for a reason, right?

Please, give me a break and look up 'hosts based ad bocking' and read how it works. I'm not your personal teacher.

I believe I saw in one of your FAQs that the hosted whitelist required all domains to be in the "domain.com/otherdomain.com" format, without "www". It's okay to have required domains in the full "www.domain.com" format, then?

For the last time: When using a hosted whitelist, you have to add the www or non-www domain yourself.
If using the non-hosted whitelist, you can toggle the setting to auto-add www and non-www domain when adding one through the Diversion UI.

Some websites only run on the www prefix, some don't and redirect to the non-www website. Diversion does that.
Technically speaking, www.domain.com is a sub-domain of domain.com. Some sites use these two for separate websites, hence why some domains need the www to be blocked, others not.

 

[WuTang LAN]

On a separate note, I can confirm that the issue with Xbox achievements has been resolved - the two "queued" achievements finally popped up, after playing a lot more of the same game.

The following whitelist restores all basic functionality to the console, and is the most up-to-date, as of now (Microsoft has been known for requiring additional domains to be whitelisted after some time):

activity.windows.com
attestation.xboxlive.com
cert.mgt.xboxlive.com
client-s.gateway.messenger.live.com
clientconfig.passport.net
continuum.dds.microsoft.com
ctldl.windowsupdate.com
def-vef.xboxlive.com
device.auth.xboxlive.com
eds.xboxlive.com
help.ui.xboxlive.com
licensing.xboxlive.com
mobile.pipe.aria.microsoft.com
notify.xboxlive.com
settings-win.data.microsoft.com
title.auth.xboxlive.com
title.mgt.xboxlive.com
v10.events.data.microsoft.com
v10.vortex-win.data.microsoft.com
v20.events.data.microsoft.com
v20.vortex-win.data.microsoft.com
www.msftncsi.com
www.xboxlive.com
xbox.ipv6.microsoft.com
xboxexperiencesprod.experimentation.xboxlive.com
xflight.xboxlive.com
xkms.xboxlive.com
xsts.auth.xboxlive.com

The list was compiled from Pi-hole's FAQ, Kamil's Lab, several Reddit threads and posts on Xbox forums, and by my own experimentation.

Despite claims by some users, domains like "bing.com" and "strict.bing.com" are not required to be whitelisted at this moment.

Again, that may change in the future, but, at least for now, that's the most comprehensive list you'll need for everything related to your Xbox One family of devices (Xbox One, Xbox One S, Xbox One X) - online multiplayer, messaging, Achievements, etc.

I did offer the solution to your issue 8 days ago.

Diversion - the Router Ad-Blocker

Anyway, I'm glad you finally got it resolved.

P.s - Quite a few of the domains you listed are not required to be whitelisted.

 

[EeK]

I did offer the solution to your issue 8 days ago.

Diversion - the Router Ad-Blocker

Anyway, I'm glad you finally got it resolved.

P.s - Quite a few of the domains you listed are not required to be whitelisted.

I did try whitelisting only the domains you mentioned, as well as following the procedures of clearing persistent storage, saved games and power cycling the device, but, sadly, that wasn't enough to restore full functionality to all Xbox-related services. I even created a Google spreadsheet to compare all the domains I could find (notice the date when the spreadsheet was last edited) and tested after adding each one to the whitelist individually, in order to come up with a list that had the minimum number of required domains.

Some articles on the subject (like the blog I linked) are also somewhat outdated. Microsoft has been continuously tweaking how their Windows 10-based devices interact with endpoints (here's an article on it), and quite a few necessary domains have been added in the last few months. Messaging, for example, requires "client-s.gateway.messenger.live.com", as noted on this Reddit post. There's also "v20.vortex-win.data.microsoft.com" and "v20.events.data.microsoft.com", as noted here and here, respectively.

As I've said, even though mine's a comprehensive list as of today, that may change in the future.

I just wanted to end this write-up by saying that I don't intend to bother other users on the SNBForums with questions anymore, even though I strongly believe that that's exactly what discussion boards are for. In an earlier comment of mine, I mentioned that not everyone on a dedicated forum is an expert, and most are likely looking to educate themselves on subjects with which they may not be entirely familiar.

No, I don't expect to find "personal teachers" anywhere on the internet, or to have others doing all of the work for me. However, as an empathic person, I try to engage in all conversations with respect and courtesy, and provide assistance whenever humanly possible. I'm actually a lawyer and a law professor in "real" life, and constantly deal with clients and students messaging me outside of office/school hours, seeking advice (not to mention complete strangers in online forums dedicated to my areas of expertise - to which I respond for free).

In fact, I was eager to share the Xbox whitelist I compiled because I felt greatly indebted to this community and wanted to give it something back. Others have had similar issues as recent as last year, and if I manage to help someone with my (admittedly) very limited knowledge of networks, then I see this as an absolute win.

 

[thelonelycoder]

No, I don't expect to find "personal teachers" anywhere on the internet, or to have others doing all of the work for me. However, as an empathic person, I try to engage in all conversations with respect and courtesy, and provide assistance whenever humanly possible. I'm actually a lawyer and a law professor in "real" life, and constantly deal with clients and students messaging me outside of office/school hours, seeking advice (not to mention complete strangers in online forums dedicated to my areas of expertise - to which I respond for free).

Look, I do my scripts for free, in my spare time. I'm a coder at home and like to concentrate on my tasks. I am easily distracted with more interesting things or stuff that bothers me.

My full-time job as an employee that pays the bills does not involve coding directly. But here I also face direct face to face contact with our clients. A big part of my job is troubleshooting and finding solutions for our systems when they don't work. While doing that I sometimes get very irritated as my current goal (and in my mind my only goal) is to get the system up and running again ASAP.
Bothering me with repeated questions at a time when I'm highly concentrated and want to solve a problem is never good.
I may over react at such times and hope you understand my side of the story too.

 

[thelonelycoder]

Diversion 4.1.1 is now available

What's new in Diversion 4.1.1
- Bug fixes and various improvements
- Renamed update-bf.div to update-bl.div (to reflect wording 'blocking list')
- Added bu as option. Runs blocking list update in the background as opposed to b, 4.
- Greyed out option in el to auto-add www and non-www domain when hosted whitelist is in use
- Additional checks for hosted lists in el
- Basic Swap partition support added, supporting fstab or post-mount entry
- Improved logic for regular swap file detection and (auto) management
- Swap file/partition size is now shown in the o options screen

How to update to Diversion 4.1.1
Use u or d and select Update.

Screenshot shows expanded options o menu:

 

[thelonelycoder]

Maybe I would need to run the script differently, so Diversion and AMTM doesn't see it as a swapfile that doesn't exist. Any recommendation?

You should be fully covered now with Diversion 4.1.1

 

[Mandarina]

You should be fully covered now with Diversion 4.1.1

Thank you, already updated. The only thing it seems that's not working, is the automatic detection of the swap partition:

 

[L&LD]

Thank you, already updated. The only thing it seems that's not working, is the automatic detection of the swap partition:

Try rebooting the router and waiting for 10 minutes before checking again.

 

[thelonelycoder]

Thank you, already updated. The only thing it seems that's not working, is the automatic detection of the swap partition:

Aargh, I asked you why it's not showing in amtm. How exactly is the swap file activated?

Edit:
1. It looks in /jffs/configs/fstab for the 'swap' entry.
2. It checks /jffs/scripts/post-mount for the 'swapon ' line. It HAS to be at the beginning of the line, no spaces before 'swapon'.
3. It also double checks with /proc/swaps to make sure it is active.
To check use:

cat /proc/swaps

 

[Mandarina]

Aargh, I asked you why it's not showing in amtm. How exactly is the swap file activated?

/jffs/scripts/swap-enable:

#!/bin/sh
swapon /dev/sda1

/jffs/scripts/post-mount:

#!/bin/sh
. /jffs/scripts/post-mount.div # Added by Diversion
. /jffs/scripts/swap-enable

Do I revert back to putting the swapon line directly on post-mount? I thought of leaving it this way because it seemed like a clean way of doing it.