Diversion Diversion - the Router Ad-Blocker

[dave14305]

And my browser shows the certificate error, as expected. What's weird, though, is that if I click on "continue to this website", it actually shows the Play store link... which means that my browser still managed to connect to the obfuscated Safelinks URL and got redirected to the Play store by the Microsoft redirection. I was expecting the usual "blank page", the one served by pixelserv, instead

What am I missing?

You mentioned earlier that on your LAN 192.168.1.2 is usually another router. But if Diversion is expecting 192.168.1.2 to be the Pixelserv IP but instead forwards it to another router, it could possibly be interfering with the normal blocking.

What's also weird is that Diversion is installed on my 192.168.1.1 router, but somehow Diversion seems to redirect to 192.168.1.2 which does not exist on my network currently (usually it's another router)

What happens if you browse to http://192.168.1.2/servstats

You should see the Pixelserv stat page. Otherwise we need to revisit how Diversion is configured on your router.

 

[thelonelycoder]

Just recently added my old Asus RT-AC56U as a wired AP to my main router. Do I need to install Diversion with DNSCrypt & Skynet on the old router or it's not needed?

As @skeal writes, no. Even if you tried, Diversion would not install as this router is in AP mode. Some services are disabled in that mode.

 

[thelonelycoder]

Finally after 1 day of use, i see a lot of adds in Facebook and also in my gmail, i did not have any with Addblock on chrome

Use f to follow the logfile and try to find out what additional domains need to be blacklisted. Usually users complain the other way round, that FB is not working. So this is new . Also know that router based ad-blocking is not capable to block ads that are served from the same domain as the content is.

 

[dave14305]

I re-ran my test and also see the redirect happening when it did not before. Definitely routing to the blocking IP but still redirecting per the remainder of the URL.

Nov 29 20:35:10 pixelserv-tls[741]: 192.168.1.161 nam01.safelinks.protection.outlook.com GET /?url=http%3A%2F%2Fsimplisafe.com%2Fandroid%2Fapp%2F&data=02%7C01%7C%7C05666ffee5934815180608d65035ac31%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636784588779611271&sdata=Tn5EPqsDVZqofj%2FPZ5n%2BZAee5AEYrqq7p%2FX6VRNGpBs%3D&reserved=0 HTTP/1.1 tls_1.2

 

[dave14305]

I re-ran my test and also see the redirect happening when it did not before. Definitely routing to the blocking IP but still redirecting per the remainder of the URL.

Nov 29 20:35:10 pixelserv-tls[741]: 192.168.1.161 nam01.safelinks.protection.outlook.com GET /?url=http%3A%2F%2Fsimplisafe.com%2Fandroid%2Fapp%2F&data=02%7C01%7C%7C05666ffee5934815180608d65035ac31%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636784588779611271&sdata=Tn5EPqsDVZqofj%2FPZ5n%2BZAee5AEYrqq7p%2FX6VRNGpBs%3D&reserved=0 HTTP/1.1 tls_1.2

I think I get it. The very first time you go to the blacklisted site, Pixelserv has not yet generated a certificate for the blocked domain, so you get the not found message. On subsequent visits, the embedded redirect in the URL is allowed to proceed. I tested this by deleting the existing cert for the safelinks name and restarted Pixelserv.

So while Diversion is working correctly, you may want to ask about this specific case in the Pixelserv thread. This may be a host you want to send to 0.0.0.0 instead of Pixelserv IP.

 

[thelonelycoder]

I think I get it. The very first time you go to the blacklisted site, Pixelserv has not yet generated a certificate for the blocked domain, so you get the not found message. On subsequent visits, the embedded redirect in the URL is allowed to proceed. I tested this by deleting the existing cert for the safelinks name and restarted Pixelserv.

So while Diversion is working correctly, you may want to ask about this specific case in the Pixelserv thread. This may be a host you want to send to 0.0.0.0 instead of Pixelserv IP.

I doubt that a pixelserv-tls or null address redirect works in this case. It might just be the browser that goes to the end-domain anyway.

 

[dave14305]

I doubt that a pixelserv-tls or null address redirect works in this case. It might just be the browser that goes to the end-domain anyway.

Pixelserv logs it as a redirect (rdr) in its stats and the code actually checks for a referrer redirect in the passed url, but I don’t quite get what it does after that.
https://github.com/kvic-z/pixelserv-tls/blob/master/socket_handler.c#L888

          } else {
            // pick out encoded urls (usually advert redirects)
            if (do_redirect && strcasestr(path, "=http")) {
              char *decoded = malloc(strlen(path)+1);
              urldecode(decoded, path);

Edit:
I see now that this behavior can be altered by adding the “R” switch to Pixelserv in ep in Diversion. Learning something new every time I post!

 

[robca]

You mentioned earlier that on your LAN 192.168.1.2 is usually another router. But if Diversion is expecting 192.168.1.2 to be the Pixelserv IP but instead forwards it to another router, it could possibly be interfering with the normal blocking.

What happens if you browse to http://192.168.1.2/servstats

You should see the Pixelserv stat page. Otherwise we need to revisit how Diversion is configured on your router.

I changed the IP address of all other routers, and I confirmed that 192.168.1.2 is just pixelserv. Indeed accessing the stats page, as you suggest, gives me the pixelserv status

 

[robca]

I think I get it. The very first time you go to the blacklisted site, Pixelserv has not yet generated a certificate for the blocked domain, so you get the not found message. On subsequent visits, the embedded redirect in the URL is allowed to proceed. I tested this by deleting the existing cert for the safelinks name and restarted Pixelserv.

So while Diversion is working correctly, you may want to ask about this specific case in the Pixelserv thread. This may be a host you want to send to 0.0.0.0 instead of Pixelserv IP.

Ok, thanks for confirming it was not just me seeing it. For now, I think I'm happy enough with being blocked at least once. My main goal is to get a warning if Safelinks gets reactivated without my knowledge, and as long as I don't store the certificate, I'm ok

Thanks everyone who looked into it, really appreciated

 

[j0shimi]

Hi all, absolutely newbie and first time user of Diversion or any router based ad blocking.

Can I just confirm I've set everything up correctly on my Merlin based Asus RT-AC87U using an ext4 USB. I've installed standard with pixelserv-tls setup to 192.168.1.2, with an IP startup pool of 192.168.1.3/

It all seems to be working with 1 Host file, do I need to change any other settings or is it set and forget?

Also, how do I add more hosts/block lists to diversion, I see many users have far more ads blocked than I do?

Many thanks for the help, absolute beginner here.

 

[thelonelycoder]

Hi all, absolutely newbie and first time user of Diversion or any router based ad blocking.

Can I just confirm I've set everything up correctly on my Merlin based Asus RT-AC87U using an ext4 USB. I've installed standard with pixelserv-tls setup to 192.168.1.2, with an IP startup pool of 192.168.1.3/

It all seems to be working with 1 Host file, do I need to change any other settings or is it set and forget?

Also, how do I add more hosts/block lists to diversion, I see many users have far more ads blocked than I do?

Many thanks for the help, absolute beginner here.

If it blocks ads, then it's working .
The blocking file type can be changed in b. There are four preset types to choose from. Or you can customize any of the types with your own selection of hosts files to use. The blocking file is updated once per week, using the set type.

 

[j0shimi]

If it blocks ads, then it's working .
The blocking file type can be changed in b. There are four preset types to choose from. Or you can customize any of the types with your own selection of hosts files to use. The blocking file is updated once per week, using the set type.

That's the sort of answer I like, cheers for all the hard work you've put in, I'll be sure to make a donation!

 

[Kronyx]

Use f to follow the logfile and try to find out what additional domains need to be blacklisted. Usually users complain the other way round, that FB is not working. So this is new . Also know that router based ad-blocking is not capable to block ads that are served from the same domain as the content is.

ok, i'll take a look at that, i also have a problem with the amazon app on my phone, it report an error when browsing, do i just whitelist amazon.ca ?

 

[robca]

ok, i'll take a look at that, i also have a problem with the amazon app on my phone, it report an error when browsing, do i just whitelist amazon.ca ?

The Amazon app uses a lot of trackers and websites. You will need to whitelist quite a few sites, and those sites change over time. I personally prefer to use the Amazon website instead.

The best way to know what to whitelist is to monitor the blocked sites while the Amazon app is running. Launch Diversion, choose O for more options, choose F to follow the log, option 3 to filter only blocked sites. Start the Amazon app, and start noting all the blocked websites. Start adding them to the whitelist, and repeat until everything works. If the app stops working again, repeat all. The main caveat is that you will enable all the ad trackers for Amazon, which will also be used for other things (Amazon now makes a lot of money out of ads, it's really becoming a big business for them)

 

[robca]

Pixelserv logs it as a redirect (rdr) in its stats and the code actually checks for a referrer redirect in the passed url, but I don’t quite get what it does after that.
https://github.com/kvic-z/pixelserv-tls/blob/master/socket_handler.c#L888

          } else {
            // pick out encoded urls (usually advert redirects)
            if (do_redirect && strcasestr(path, "=http")) {
              char *decoded = malloc(strlen(path)+1);
              urldecode(decoded, path);

Edit:
I see now that this behavior can be altered by adding the “R” switch to Pixelserv in ep in Diversion. Learning something new every time I post!

I missed the edit until now. Great find!

For other folks who find this by searching: I wanted to block Safelinks/ATP and get a warning when Microsoft once more re-enabled Safelinks (they seem to do it every few months, lately they added a user configurable setting, hopefully this won't happen again). I added the Safelinks URL to the wildcard blacklist, but I found that I was still redirected to the target destination, so I wrongly assumed that Diversion was not working properly (Heresy! Diversion is truly awesome, I'll pay proper penance for the next few days . In reality Diversion was doing its job, passing the Safelinks URL to Pixelserv, which in turn was stripping away the blocked domain, and decoding the rest of the URL to bring me to the destination.

This is, in general, highly desirable for all the links that have a referrer (Pixelserv blocks the tracker, but still allows to get to the destination). In my case, I wanted to be warned about Safelinks more than anything else, and I got confused by the fact that if I accept the browser warning about the wrong certificate and hig "proceed tot the website", that address gets stored and I won't get a warning next time

Many thanks Dave14305 for your investigations and the information shared

 

[user1085]

Something weird seems to be happening to me. Every few days/hours (?? Not sure), the diversion install seems to be lost. When I try to reinstall by logging into the CLI, I get the following message. Even though the original diversion install was on an ext2 formatted 1 GB USB drive. And I don't remove the USB drive or do anything with it between the failures. I have to reformat the USB as ext2 and reinstall diversion until the cycle repeats. How can I debug this further?

[ Error ] No compatible device(s) found to install
Diversion on. A device formatted with one of
these file systems is required:
ext2, ext3, ext4​

 

[john9527]

How can I debug this further?

Get a new USB drive.....I had a drive like this that would just 'disappear' after a while when hooked to the router. Worked fine on a Windows system.

 

[user1085]

Get a new USB drive.....I had a drive like this that would just 'disappear' after a while when hooked to the router. Worked fine on a Windows system.

Weird! Thanks. Will try out another drive. Although it's a pity it's so hard to find a dirt cheap 1/2GB drive these days :+ 16GB seems to be the cheapest these days

 

[loveleeyoungae]

Something weird seems to be happening to me. Every few days/hours (?? Not sure), the diversion install seems to be lost. When I try to reinstall by logging into the CLI, I get the following message. Even though the original diversion install was on an ext2 formatted 1 GB USB drive. And I don't remove the USB drive or do anything with it between the failures. I have to reformat the USB as ext2 and reinstall diversion until the cycle repeats. How can I debug this further?

[ Error ] No compatible device(s) found to install
Diversion on. A device formatted with one of
these file systems is required:
ext2, ext3, ext4​

Btw, what app and on which platform did you format the USB drive?

 

[kolyan]

Is this normal to go through the setup/install everytime router rebooted? I didn't seem to have to issue on AC66 and that's what it does now on AC86