AB-Solution - The Ad Blocking Solution

[thelonelycoder]

Guys,
Out of curiosity, do following stats look "normal"?
Especially slu, tmo and cls.
slh 20603 # of accepted HTTPS requests
slm 322 # of rejected HTTPS requests (missing certificate)
sle 0 # of rejected HTTPS requests (certificate available but bad)
slu 857 # of dropped HTTPS requests (unknown error)

err 0 # of dropped requests (failed to accept client connection)
tmo 2624 # of dropped requests (client timeout before connection accepted)
cls 5517 # of dropped requests (client disconnect before connection accepted)

Yup numbers look about the same on mine, except for the tmo, which is 0.
Turn on ps logging in ps and watch the syslog. Or it might just be one domain from one device.

 

[apokrif]

Yup numbers look about the same on mine, except for the tmo, which is 0.
Turn on ps logging in ps and watch the syslog. Or it might just be one domain from one device.

ps logging is on already.
Could you tell what to look for in syslog exactly, please?

 

[thelonelycoder]

ps logging is on already.
Could you tell what to look for in syslog exactly, please?

Since I dont have any of those I would not know.

 

[apokrif]

Since I dont have any of those I would not know.

There is nothing outstanding in syslog...
Lot’s of pixelserv messages like:
Jun 23 20:48:31 pixelserv[6632]: 1435575.fls.doubleclick.net _.fls.doubleclick.net missing
Jun 23 20:48:33 pixelserv[2726]: cert _.fls.doubleclick.net generated and saved
And no any errors.

dnsmasq.log is irrelevant and didn’t have anything interesting either.

Is there a requests log against pixelserv-tls showing client IP and status?
I assume, the pixelserv-tls stats is aggregated based the status above, right?

According to https://github.com/kvic-z/pixelserv-tls
$ pixelserv-tls --help
Usage: pixelserv-tls
-l (log access to syslog)
It can be enabled (in S80pixelserv-tls)

 

[thelonelycoder]

There is nothing outstanding in syslog...
Lot’s of pixelserv messages like:
Jun 23 20:48:31 pixelserv[6632]: 1435575.fls.doubleclick.net _.fls.doubleclick.net missing
Jun 23 20:48:33 pixelserv[2726]: cert _.fls.doubleclick.net generated and saved
And no any errors.

dnsmasq.log is irrelevant and didn’t have anything interesting either.

Is there a requests log against pixelserv-tls showing client IP and status?
I assume, the pixelserv-tls stats is aggregated based the status above, right?

According to https://github.com/kvic-z/pixelserv-tls
$ pixelserv-tls --help
Usage: pixelserv-tls
-l (log access to syslog)
It can be enabled (in S80pixelserv-tls)

Dnsmasq redirects the blocked domains to pixelserv-tls which does it's own logging.
The logging you enable in AB is exactly what it says, the -l switch for pixelserv-tls. Don't change the S80... file, it gets rewritten by AB whenever you change someting in the ps menu.

I would not be bothered by these tmo entries. If you have no problems browsing, don't make one over nothing.
If you must know, post in kvic's pixelserv-tls thread.

 

[apokrif]

The logging you enable in AB is exactly what it says, the -l switch for pixelserv-tls. Don't change the S80... file, it gets rewritten by AB whenever you change someting in the ps menu.

They keep growing steadily. I was thinking:
add -l to S80... file
restart pixelserv-tls via S80
Monitor syslog ~1 hour
remove -l to S80... file
Am I missing anything?

If you must know, post in kvic's pixelserv-tls thread.

If I dig something out, I'll ask all further questions there!

 

[thelonelycoder]

They keep growing steadily. I was thinking:
add -l to S80... file
restart pixelserv-tls via S80
Monitor syslog ~1 hour
remove -l to S80... file
Am I missing anything?

If I dig something out, I'll ask all further questions there!

I don't get it, AB has a menu option to enable logging and I explained it.

 

[iManuB]

- close the ORIGIN application
- in AB, select f option 1
- open ORIGIN
- look for the blocked domain that might be ORIGIN related
- in AB, use the el function to whitelist domain

Thanks so much, theonlycoder. I try that!

 

[truglodite]

thelonelycoder,
ok if apokrif's stats look OK, then mine seem off:

uts    1d 04:05    pixelserv uptime
log    1    logging access to syslog (0=disabled 1=enabled)
req    31646    total # of requests (HTTP, HTTPS, success, failure etc)
avg    3544 bytes    average length of request URL
rmx    119717 bytes    maximum length of request URL
tav    583 ms    average processing time (per request)
tmx    10076 ms    maximum processing time (per request)
slh    1    # of accepted HTTPS requests
slm    0    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slu    0    # of dropped HTTPS requests (unknown error)
nfe    539    # of GET requests for server-side scripting
gif    3    ...

I thought 'slh' should be way higher. Also pixelserver appears to be working well in the browsers it is serving, and the dnsmasq logs show lots of lines indicating pixelserv-tls IP is returning pixels instead of ads. Should I be worried?

Kevin

[edit: I think I may have answered my initial question about why the slh hits were not incrementing... after disabling ps 'log to system log', the various 'HTTPS' stats started climbing again.

Not sure if this is expected behavior or a bug. Regardless, I prefer having my ps logs separate from system logs, with the web stats page working... versus having combined ps+sys logs and no web stats (I show the webstats to curious family/friends). Not sure hot the piping is setup, and if it's even possible to pipe the HTTPS stuff to both the webpage and the syslog at the same time? I'm a bit rusty with bash, otherwise I'd totally help fix it... if is even fixable.

Another observation I made this morning... req=5547, slh=4795, but cls=4845. Is that normal? I figured I should see approximately req ~= slh+cls, but that's clearly not the case with my setup. Is there some info available on these web stats that expands on the ps wiki? I'm pretty much a networking newb.]

 

[thelonelycoder]

[edit: I think I may have answered my initial question about why the slh hits were not incrementing... after disabling ps 'log to system log', the various 'HTTPS' stats started climbing again.

Not sure if this is expected behavior or a bug. Regardless, I prefer having my ps logs separate from system logs, with the web stats page working... versus having combined ps+sys logs and no web stats (I show the webstats to curious family/friends). Not sure hot the piping is setup, and if it's even possible to pipe the HTTPS stuff to both the webpage and the syslog at the same time? I'm a bit rusty with bash, otherwise I'd totally help fix it... if is even fixable.

Another observation I made this morning... req=5547, slh=4795, but cls=4845. Is that normal? I figured I should see approximately req ~= slh+cls, but that's clearly not the case with my setup. Is there some info available on these web stats that expands on the ps wiki? I'm pretty much a networking newb.]

For lack of deeper understanding I am unable to answer these questions. Kvic is the one to ask, in this thread.
Pixelserv-tls has only one place built in to log activities, the Syslog. There are no switches to direct it elsewhere.
I would have built this in if there were an option.

 

[tomsk]

Regardless, I prefer having my ps logs separate from system logs, with the web stats page working... versus having combined ps+sys logs and no web stats (I show the webstats to curious family/friends). Not sure hot the piping is setup, and if it's even possible to pipe the HTTPS stuff to both the webpage and the syslog at the same time?

One method of separating out the logs is to use syslog-ng instead of the built in syslog. @kvic has written a nice article here http://kazoo.ga/migrate-to-syslog-ng-3-8/ Entware syslog-ng is now at version 3.9 though so the config file will have to be tweaked a little to suit.

 

[Toad004]

ab-solution is preventing Microsoft office from connecting to onedrive and other servers properly.
What are the appropriate domain(s) I need to whitelist?

 

[elorimer]

ab-solution is preventing Microsoft office from connecting to onedrive and other servers properly.
What are the appropriate domain(s) I need to whitelist?

I had this problem; it wasn't ab-solution but the iptables script I am running (skynet). I whitelisted the {mydomain}-mysharepoint.com domain, it looked up the IP and added it to the whitelist. That domain wasn't in the ab-solution domain list anyway. But after that my onedrive was back.

 

[truglodite]

Toad, not sure the exact domains... try this.

Ssh\ab-solution, [f], [Enter], [2], [Enter], [Enter]... now try and use your onedrive as usual. If dnsmasq is the cause of your problem, you will see some lines containing m$oft domains that you can try to whitelist. That should git 'er choochin' proper once again... if it is in fact dnsmasq at play and not something else. I say dnsmasq, because it is what does the heavy lifting for ab-solution (ab-soltution just makes it easier for us n00bs).

I usually add domains to the whitelist one by one, starting low level then working up (ie try 'rover.ebay.com' before opening up all of 'ebay.com'... this way if the rover.ebay.com works, then ads.ebay.com still get blocked ). There may be more than one domain that needs white listing before your office/onedrive stuff works 100%, but following this procedure it shouldn't take long to fix.

Note... if you have a noisy network, it's best to try this during a quieter time. It's also best to kill processes that may be grabbing real ads in the background (like a browser displaying boston.com). This will help isolate exactly which 'hostfileified' domains office is trying to call.

 

[Toad004]

Toad, not sure the exact domains... try this.

Ssh\ab-solution, [f], [Enter], [2], [Enter], [Enter]... now try and use your onedrive as usual. If dnsmasq is the cause of your problem, you will see some lines containing m$oft domains that you can try to whitelist. That should git 'er choochin' proper once again... if it is in fact dnsmasq at play and not something else. I say dnsmasq, because it is what does the heavy lifting for ab-solution (ab-soltution just makes it easier for us n00bs).

I usually add domains to the whitelist one by one, starting low level then working up (ie try 'rover.ebay.com' before opening up all of 'ebay.com'... this way if the rover.ebay.com works, then ads.ebay.com still get blocked ). There may be more than one domain that needs white listing before your office/onedrive stuff works 100%, but following this procedure it shouldn't take long to fix.

Note... if you have a noisy network, it's best to try this during a quieter time. It's also best to kill processes that may be grabbing real ads in the background (like a browser displaying boston.com). This will help isolate exactly which 'hostfileified' domains office is trying to call.

Thank you!
It appears that nexus.officeapps.live.com is what office connects to see if the servers are up (and that it's activated). As far as I can tell, that's litterally ALL it does. I'm betting one of the hosts files thinks it's an adserver, or at least a telemetry server - I've seen blanket "anti-telemetry" files blocking things like bing.com and hotmail.com.

Nice to know that ab-solution has such a useful feature. I'll be putting it to good use!

 

[thelonelycoder]

Nice to know that ab-solution has such a useful feature. I'll be putting it to good use!

There's also f option 4 for an enhanced view of unblocked and blocked domains. It's use is a tad more complicated but is explained in the AB UI step by step as you go.
Adding a whitelist entry removes it from the blocking_file, while removing a whitelist entry re-enters it into the blocking_file.
This way ad-blocking always blocks only entries in the blocking file and the blacklist.

 

[steelskinz]

I discovered yesterday that blacklist is more important than whitelist.

I had an url in blacklist which i forget i put in and when i tried to whitelist it it doesn't work.

I realized when i see log file that abs was intelligent enough to put in green line that the website was in blacklist.txt..

 

[thelonelycoder]

I discovered yesterday that blacklist is more important than whitelist.

If you use the built in function es in AB to black/whitelist then you will be warned if you add a domain that is in another file.
This is not possible if you edit them by hand, of course.

If you manually block it in the blacklist, adding whitelist entries will not unblock the ones you have in the blacklist. The whitelist only removes entries in the blocking file.
Unless, as said above, you use the UI function, then you will be warned.

Edit and correction: If you add a domain in the whitelist that is in the blacklist, no warning is given.
The whitelist check only looks at the blocking file.
And since the blacklist is separately loaded to dnsmasq from the blocking file, whatever you have in these files is blocked.

 

[thelonelycoder]

I discovered yesterday that blacklist is more important than whitelist.

I had an url in blacklist which i forget i put in and when i tried to whitelist it it doesn't work.

I changed some code in the functions.add file to include a search in the blacklist when adding a domain to the whitelist.
Seems to work fine. Will release as soon as I'm sure it will not break other things.

 

[thelonelycoder]

Before I upload the updated functions.add (see posts above), is there anything else quick fix wise that is worth looking into?
Bugs, improvements, typos or some such things that make it worth changing the version number file on the server?
Thanks