[Release] AB-Solution 3 - The Ad Blocking Solution

[thelonelycoder]

Enough horror stories for one day ...Now you are scaring the children!

Well, then don't tell them, some secrets are best kept to yourself...

 

[Beherit]

AB-Solution stops working each time it does its automatic weekly update. I have to manually update it or reboot my RT-AC87U router to get it working again. The router is running merlinwrt v380.63_2.

Any idea what's causing this or how I can disable auto-updates as a workaround?

 

[tomsk]

Do you have any messages in your syslog to help t/shoot?... and what is your update day (Sundays seem to be an issue) ....you can try selecting a different update day and see if that helps.

 

[Beherit]

As it happens, yep I've set it to Sunday. I'll try changing it and let you all know if it works.

The syslog is empty since I just rebooted the router to get ABS working again. I'll keep it in mind for next time.

 

[thelonelycoder]

AB-Solution stops working each time it does its automatic weekly update. I have to manually update it or reboot my RT-AC87U router to get it working again. The router is running merlinwrt v380.63_2.

Any idea what's causing this or how I can disable auto-updates as a workaround?

As @tomsk writes: More info is needed to find the cause of this.
But before you do anything at all, answer these, it provides info I'm eager to know:
- Enter cru l (lower case L) in a terminal and post output
- What update day is set and what language is your router WebUI
- How much RAM is left
- Your router model

 

[thelonelycoder]

As it happens, yep I've set it to Sunday. I'll try changing it and let you all know if it works.

The syslog is empty since I just rebooted the router to get ABS working again. I'll keep it in mind for next time.

Do as I said before you do anything, I need to know.

 

[thelonelycoder]

There is something fishy going on with cron jobs, at least on my 380.64 beta routers, have to fire up another router that is not on the latest fw test it.
Cron entries get overwritten by older entries for some reason when they are run:

tlc@RT-AC1900P-CA08:/tmp/home/root# cru l
00 5 * * 0 /tmp/mnt/absolution/adblocking/addon/update-hosts.add cronjob #UpdateHosts#
20 5 * * * /tmp/mnt/absolution/adblocking/addon/rotate-logs.add #RotateLogs#

tlc@RT-AC1900P-CA08:/tmp/home/root# cru d RotateLogs
tlc@RT-AC1900P-CA08:/tmp/home/root# cru l
00 5 * * 0 /tmp/mnt/absolution/adblocking/addon/update-hosts.add cronjob #UpdateHosts#

tlc@RT-AC1900P-CA08:/tmp/home/root# cru a RotateLogs "*/2 * * * 0 /tmp/mnt/absolution/adblocking/addon/rotate-logs.add"

tlc@RT-AC1900P-CA08:/tmp/home/root# cru l
00 5 * * 0 /tmp/mnt/absolution/adblocking/addon/update-hosts.add cronjob #UpdateHosts#
*/2 * * * 0 /tmp/mnt/absolution/adblocking/addon/rotate-logs.add #RotateLogs#

tlc@RT-AC1900P-CA08:/tmp/home/root# cru l
00 5 * * 0 /tmp/mnt/absolution/adblocking/addon/update-hosts.add cronjob #UpdateHosts#
20 5 * * * /tmp/mnt/absolution/adblock

- First cru l shows two jobs, with RotateLogs running at 20 past 5 in the morning every day.
- I delete the job (cru d RotateLogs)
- Job is deleted, only one job is listed
- I manually add the job to run every two minutes on Sundays
- Job is listed correctly (*/2 * * * 0)
- After job is run by cron, the old entry is established (20 5 * * *)
Something for @RMerlin to ponder? The crontabs file reflects that change.

Edit: Forget the above post. This is expected and I get fooled by my own programming.
Rotating the logs restarts dnsmasq and therefore the job gets added again through ab_dnsmasq_postconf.sh.

All good, still need to know what language the Sunday update job users have set from those that see the error.

 

[tomsk]

There is something fishy going on with cron jobs, at least on my 380.64 beta routers, have to fire up another router that is not on the latest fw test it.
Cron entries get overwritten by older entries for some reason when they are run:

tlc@RT-AC1900P-CA08:/tmp/home/root# cru l
00 5 * * 0 /tmp/mnt/absolution/adblocking/addon/update-hosts.add cronjob #UpdateHosts#
20 5 * * * /tmp/mnt/absolution/adblocking/addon/rotate-logs.add #RotateLogs#

tlc@RT-AC1900P-CA08:/tmp/home/root# cru d RotateLogs
tlc@RT-AC1900P-CA08:/tmp/home/root# cru l
00 5 * * 0 /tmp/mnt/absolution/adblocking/addon/update-hosts.add cronjob #UpdateHosts#

tlc@RT-AC1900P-CA08:/tmp/home/root# cru a RotateLogs "*/2 * * * 0 /tmp/mnt/absolution/adblocking/addon/rotate-logs.add"

tlc@RT-AC1900P-CA08:/tmp/home/root# cru l
00 5 * * 0 /tmp/mnt/absolution/adblocking/addon/update-hosts.add cronjob #UpdateHosts#
*/2 * * * 0 /tmp/mnt/absolution/adblocking/addon/rotate-logs.add #RotateLogs#

tlc@RT-AC1900P-CA08:/tmp/home/root# cru l
00 5 * * 0 /tmp/mnt/absolution/adblocking/addon/update-hosts.add cronjob #UpdateHosts#
20 5 * * * /tmp/mnt/absolution/adblock

- First cru l shows two jobs, with RotateLogs running at 20 past 5 in the morning every day.
- I delete the job (cru d RotateLogs)
- Job is deleted, only one job is listed
- I manually add the job to run every two minutes on Sundays
- Job is listed correctly (*/2 * * * 0)
- After job is run by cron, the old entry is established (20 5 * * *)
Something for @RMerlin to ponder? The crontabs file reflects that change.

That really is odd.... is not even the full original job but a truncated version of it.

the OP just now was on an RT-AC87U so he would have 256Mb RAM. But i think you are on to something here with the cron entry rather than running out of RAM. He is on 380.62_2 firmware though, so probably not related to changes in the beta if it is the same issue.

Does it matter that one entry has "cronjob" and the other doesn't?

 

[thelonelycoder]

No, the 'cronjob' is a parameter I send along with the job so that update-hosts.add knows to delete the weekly stats. It is not run when you run it manually with (u).

Anyways, this is obsolete, cron seems fine, see my added info to the post.
But something with the date function within asuswrt-merlin seems to have changed. I add the update day with the date function $(date +"%w") which returns the day of week (0..6); 0 is Sunday.
I'm thinking now that it may have something to do with the 'export TZ=$(cat /etc/TZ)' I add in the S80pixelserv-tls file.
Don't know what influence the TimeZone has for the date function, but it may reverse the change I made long ago from %u (day of week (1..7); 1 is Monday).
Have to investigate that.

 

[tomsk]

Its a strange bug... i have my update at 2am on a Tuesday and stats get emailed to me like clockwork..nothing unusual in the syslog. I don't understand why setting the pixelserv timezone would mess with the stats though... baffling

Dec  6 01:30:02 RT-AC68U-4690 tOmsK: AB-Solution blocked 153'942 total 22'142 week 1'414 new ads
Dec  6 01:32:18 RT-AC68U-4690 tOmsK: AB-Solution compiled stats
Dec  6 02:01:26 RT-AC68U-4690 tOmsK: AB-Solution updated hosts file, 214569 Domains are now blocked
Dec  6 02:01:29 RT-AC68U-4690 rc_service: service 11894:notify_rc restart_dnsmasq
Dec  6 02:01:29 RT-AC68U-4690 script: Running /jffs/scripts/dnsmasq.postconf (args: /etc/dnsmasq.conf)
Dec  6 02:01:30 RT-AC68U-4690 tOmsK: AB-Solution added entries via ab_dnsmasq_postconf.sh
Dec  6 02:01:30 RT-AC68U-4690 tOmsK: AB-Solution linked dnsmasq.postconf to ab_dnsmasq_postconf.sh
Dec  6 02:01:37 RT-AC68U-4690 tOmsK: AB-Solution rotated dnsmasq log files
Dec  6 02:01:39 RT-AC68U-4690 rc_service: service 11953:notify_rc restart_dnsmasq
Dec  6 02:01:40 RT-AC68U-4690 script: Running /jffs/scripts/dnsmasq.postconf (args: /etc/dnsmasq.conf)
Dec  6 02:01:41 RT-AC68U-4690 tOmsK: AB-Solution added entries via ab_dnsmasq_postconf.sh
Dec  6 02:01:41 RT-AC68U-4690 tOmsK: AB-Solution linked dnsmasq.postconf to ab_dnsmasq_postconf.sh

 

[thelonelycoder]

I just changed WebUI languages and 'Fus orar' (time zone in, lets see: Romanian, looks like a lovely language!) forth and back, always returns 0 for Sunday. This would be correct.
One would think this would change, but no, not when I want to get to the bottom of it.

 

[thelonelycoder]

@tomsk: You would have set yours to GMT+3?

 

[john9527]

I'm thinking now that it may have something to do with the 'export TZ=$(cat /etc/TZ)' I add in the S80pixelserv-tls file.

Just an FYI.....I now add the export TZ automatically when starting a shell in my fork.....

 

[tomsk]

@tomsk: You would have set yours to GMT+3?

Im at GMT +4

 

[thelonelycoder]

Just an FYI.....I now add the export TZ automatically when starting a shell in my fork.....

Thats for the logging only or for crontab as well?
How can I check that, because this might be the reason why 0 is no longer valid for Sun and it should be 1 (for the american system).

 

[thelonelycoder]

The US needs to change their ways, too many (in)significant differences to the rest of the world...
And it needs to be back-ported immediately in all software releases.
(just joking)

 

[sfx2000]

But this, like other solutions whitelists the domain, but doesn't necessarily whitelist the ad-providers - and here it does become a challenge, as one might whitelist providers that support SNB sites, but that would open up things for those ad providers on other sites.

People have different reasons for implementing an ad-blocking kind of solution - for some it's privacy, for others, it's a security aspect, and some have bandwidth caps that unwanted content blocked is bandwidth saved...

There is a way to fix adblockers - where it's friendly to both the site operator and the ad provider, and still keep a user experience that would address both privacy and security concerns.

 

[tomsk]

Just an FYI.....I now add the export TZ automatically when starting a shell in my fork.....

Yes i saw that in your changelog ... but afaik the Merlin main fork hasn't done that

 

[sfx2000]

But the discussion we are having here is for white-listing. Serving Ads from the same domain may still work with the major browser based ad-blockers, provided the relative path to the ads on that server does not change.
White-listing does work that way. But on a hosts based blocker there are only two ways: Accept the ads or block the domain and go look for content elsewhere (for the hardcore user).

One of the more clever solutions intercept the ad mid-stream by rendering the entire session in memory, replacing the ad uri's with another, returning the right result codes to the ad provider, using an anonymized UUID as needed - basically the solution deconstructs the page to the various info elements and rebuilds on the fly to the end user - so clickthrus still work, impressions still work, and just about every ad blocker is defeated - both javascript browser based as well as host driven via dns..

the advantage here is that whitelist/blacklist can occur in the event of malware, etc pretty much on the fly.

solutions like this are already out there...

 

[thelonelycoder]

One of the more clever solutions intercept the ad mid-stream by rendering the entire session in memory, replacing the ad uri's with another, returning the right result codes to the ad provider, using an anonymized UUID as needed - basically the solution deconstructs the page to the various info elements and rebuilds on the fly to the end user - so clickthrus still work, impressions still work, and just about every ad blocker is defeated - both javascript browser based as well as host driven via dns..

the advantage here is that whitelist/blacklist can occur in the event of malware, etc pretty much on the fly.

solutions like this are already out there...

That may very well work on your BHR, but I doubt privoxy or any other software can do that on a consumer router.