[Release] AB-Solution 3 - The Ad Blocking Solution

[thelonelycoder]

00 5 * * 5 /tmp/mnt/usbstick/adblocking/addon/update-hosts.add cronjob #UpdateHosts#
20 5 * * * /tmp/mnt/usbstick/adblocking/addon/rotate-logs.add #RotateLogs#
30 4 * * 5 /tmp/mnt/usbstick/adblocking/scripts/weekly-log-stats.sh #WeeklyStats#

5 is Friday for cron, so your update really runs at that day.
MET (or MEZ in german speaking countries such as mine) stands for Middle European Time, just another confusing acronym for saying GMT +1, or UTC +1, or CET.
Official is UTC+1 (which is, thanks to the French, the wrong way around: Coordinated Universal Time (UTC).

Anyway, I wonder, maybe you should set that new and get new files. Although the manual run works (u) it could be an error in the update-hosts.add.
Enter (cu) and then enter 1 for Redownload all add-on scripts.
Then enter (h) and select the same hosts file you use at the moment, then select to change the day (and time) when the update should run.
Better not select Sunday, I need to fix things first in an update.
This should fix whatever the problem was.
If not, post output from the Syslog in the WebUI.

 

[sfx2000]

How do I know cron will honor the TZ as well?
And if it is unset, it will be set when pixelserve is added as addon, but by that time the cron job and numerical day is set in AB. Hence my above words about having to double check and change dates afterwards.

Safer to just put the crontab at a specific point in localtime (time on the device) - don't worry so much about tzdata, as this changes all the time, and not everyone has the same tzfile... plus then you a little bit of load spreading across different timezones, so the hosting server doesn't get a big traffic spike...

 

[thelonelycoder]

Safer to just put the crontab at a specific point in localtime (time on the device) - don't worry so much about tzdata, as this changes all the time, and not everyone has the same tzfile... plus then you a little bit of load spreading across different timezones, so the hosting server doesn't get a big traffic spike...

I don't get it, the cron job still needs to be written with a day. And that is my problem. If I write 0 for sunday it works for some TZ's and not for others.
But there seems to be no reliable way to find out what day format cron actually runs on, and ot throws an error if it's set wrong (0-6 or 1-7).
As for the load spreading, that was always my reason to use Sunday at 2 a.m. Local time.
I added the choice to select the day and time as a convenience for the users. Because the stats function runs half an hour before that and you may want the stats at a specific time and day.

 

[john9527]

But there seems to be no reliable way to find out what day format cron actually runs on, and ot throws an error if it's set wrong (0-6 or 1-7).

Did you try specifying 'Sun' instead of a number? from crontab man

The time and date fields are:

           field         allowed values
           -----         --------------
           minute        0-59
           hour          0-23
           day of month  1-31
           month         1-12 (or names, see below)
           day of week   0-7 (0 or 7 is Sun, or use names)

Names can also be used for the ``month'' and ``day of week'' fields. 
Use the first three letters of the particular day or month (case does not matter).  
Ranges or lists of names are not allowed.

 

[dcb]

I stumbled upon this thread and have a big thank you! What a great app.

 

[sfx2000]

I don't get it, the cron job still needs to be written with a day. And that is my problem. If I write 0 for sunday it works for some TZ's and not for others.

basic crontab - this will run every day at midnight... and that's safe enough, let the different time zones help out with server load for updates..

00 00 * * * (some task)

From the man page:

man -S 5 crontab

   cron(8) examines cron entries once every minute.

   The time and date fields are:

         field          allowed values
         -----          --------------
         minute         0-59
         hour           0-23
         day of month   1-31
         month          1-12 (or names, see below)
         day of week    0-7 (0 or 7 is Sun, or use names)

 

[thelonelycoder]

Did you try specifying 'Sun' instead of a number? from crontab man

basic crontab - this will run every day at midnight... and that's safe enough, let the different time zones help out with server load for updates..

00 00 * * * (some task)

I feel stupid, so don't blame me for being ignorant: I missed @john9527's comment hint earlier on. Just tested it, three letter day and month abbreviations works in cron.
This solves my problem.

I cant just run it daily as that would truncate the stats function, it is tied to the update day and time and deletes old dnsmasq log files when run as a cron job (hence the 'cronjob' I send along with 00 5 * * 5 /tmp/mnt/usbstick/adblocking/addon/update-hosts.add cronjob #UpdateHosts#).
But I can sleep well now, sun mon or whatever day came to the rescue.

 

[sfx2000]

I feel stupid, so don't blame me for being ignorant: I missed @john9527's comment hint earlier on. Just tested it, three letter day and month abbreviations works in cron.
This solves my problem.

For the win!

Happy to assist...

 

[john9527]

This solves my problem.

Gooooooooooooaaaaaaaaaal !!!!

I know how annoying it is to have an unsolved problem running around in the back of your mind....

 

[sfx2000]

I have four routers to test it with

there ya go - cheap cluster... 35 bucks a slice....

 

[tooandrew]

Also make sure your devices DNS Server points to that router: (Windows: ipconfig/all)
And you flush the DNS cache (ipconfig/flushdns).

strangely, neither my device nor the main router are still configured to use this router as a dns server, but just setting it to do so for a moment made it work despite my changing the settings back to my vpn provider's smart dns service

 

[Beherit]

5 is Friday for cron, so your update really runs at that day.

My apologies, that was after I changed the scheduled update from Sunday to Friday. The output when Sunday is set is identical, except for the 5 becoming a 0.

I noticed now when changing it, that Sunday is the first day on the list and not the last one?? I could had sworn that Monday was listed first last time I checked.

0. Sunday
1. Monday
2. Tuesday
3. Wednesday
4. Thursday
5. Friday
6. Saturday
7. Today's day

....aaaaaand yet another new error, after the Large lists (4) have been downloaded, it says

sort: out of memory
your RT-AC87U did it in 12 seconds

8 hosts are now blocked <----- those 8 are from my custom blacklist

I tried it again, whilst ssh'ing from another terminal and running the free command as it was sorting the whitelist. Free memory decreased steadily by the second and when it was down to around 20k it said "your RT-AC87U did it in 40 seconds" with no error message this time. But it still lists only 8 hosts as blocked.

AB-Solution message:
Error(s) found. Type 22 and hit [Enter]

The erronous part of the output from 22 is:

hosts-adblock is NOT OK, missing entries
Run [ u] to see if the error goes away

Anyway, I wonder, maybe you should set that new and get new files. Although the manual run works (u) it could be an error in the update-hosts.add.
Enter (cu) and then enter 1 for Redownload all add-on scripts.
Then enter (h) and select the same hosts file you use at the moment, then select to change the day (and time) when the update should run.

It works, for whatever reason it didn't ran out of memory this time around. I monitored the whitelist sorting process and free memory got down to around 32k which I guess is enough for ABS to finish it.

According to htop, "dnsmasq --log-async" is the process that stands out by using 30.9% memory. Which is a lot more than any other. Watchdog02, data_colld, tftpd and lighthttpd use 1.0% memory and all the others process < 1%.

Is dnsmasq using so much because of ABS or is this something I should report to Merlin?

Thanks for your time and help!

 

[thelonelycoder]

I noticed now when changing it, that Sunday is the first day on the list and not the last one?? I could had sworn that Monday was listed first last time I checked.

Sunday was always 0 and at the top of the list.

8 hosts are now blocked <----- those 8 are from my custom blacklist
I tried it again, whilst ssh'ing from another terminal and running the free command as it was sorting the whitelist. Free memory decreased steadily by the second and when it was down to around 20k it said "your RT-AC87U did it in 40 seconds" with no error message this time. But it still lists only 8 hosts as blocked.

There's your error, if another task uses more RAM than usual, it will fail as the sorting uses a lot of memory. Watch this and use a custom hosts list to make it smaller if necessary.

Is dnsmasq using so much because of ABS or is this something I should report to Merlin?

That is normal, no need to involve RMerlin, dnsmasq loads the hosts list into memory.

 

[tomsk]

I don't get it, the cron job still needs to be written with a day. And that is my problem. If I write 0 for sunday it works for some TZ's and not for others.
But there seems to be no reliable way to find out what day format cron actually runs on, and ot throws an error if it's set wrong (0-6 or 1-7).

I certainly learnt something here... i'd always assumed that cron would accept a 0 or 7 as sunday irrespective of the time zone ... now all have to do is work out what on earth the thinking was behind the 0-6 or 1-7 scheme that needs them to be exclusive....arrgghh my brain!

There's your error, if another task uses more RAM than usual, it will fail as the sorting uses a lot of memory. Watch this and use a custom hosts list to make it smaller if necessary.

possible use case for a swap if going for large host files?...would it help?

 

[sfx2000]

According to htop, "dnsmasq --log-async" is the process that stands out by using 30.9% memory. Which is a lot more than any other. Watchdog02, data_colld, tftpd and lighthttpd use 1.0% memory and all the others process < 1%.

Is dnsmasq using so much because of ABS or is this something I should report to Merlin?

I would give it some time and let it settle - but I don't believe this would be cause for alarm - dnsmasq can do this, and it's not unexpected..

 

[thelonelycoder]

I certainly learnt something here... i'd always assumed that cron would accept a 0 or 7 as sunday irrespective of the time zone ... now all have to do is work out what on earth the thinking was behind the 0-6 or 1-7 scheme that needs them to be exclusive....arrgghh my brain!

possible use case for a swap if going for large host files?...would it help?

Stackexchange with a most useful answer why. Hindsight from an American point of view, as with many things they did early on, never thinking it would spread further than their shores.
As it happens, BusyBox only accepts TZ relevant values.
As for the out of Memory error, this only occasionally happens for some. I have no plans to build in checks or add a swap file in AB.

 

[john9527]

I certainly learnt something here... i'd always assumed that cron would accept a 0 or 7 as sunday irrespective of the time zone ...

I agree....my best guess is that there are various interpretations between different linux distributions. The crontab man includes these lines. Sounds like something else to keep in mind to write truly portable code.

When specifying day of week, both day 0 and day 7 will be considered
Sunday. BSD and ATT seem to disagree about this.

Lists and ranges are allowed to co-exist in the same field.
"1-3,7-9" would be rejected by ATT or BSD cron -- they want to see "1-3" or "7,8,9" ONLY.

 

[thelonelycoder]

I agree....my best guess is that there are various interpretations between different linux distributions. The crontab man includes these lines. Sounds like something else to keep in mind to write truly portable code.

When specifying day of week, both day 0 and day 7 will be considered
Sunday. BSD and ATT seem to disagree about this.

Lists and ranges are allowed to co-exist in the same field.
"1-3,7-9" would be rejected by ATT or BSD cron -- they want to see "1-3" or "7,8,9" ONLY.

I hope that at least, sun or fri is understood in all of them to represent Sunday and Friday...

 

[RMerlin]

And don't forget you're not dealing with crond here, but with Busybox. It's not 100% compatible with the regular crond package.

 

[thelonelycoder]

And don't forget you're not dealing with crond here, but with Busybox. It's not 100% compatible with the regular crond package.

I will test this as throrouhly as I can, no more silly errors.
This is 2016, and we still deal with minor stuff like that. Developers, Developers, Developers, the beardy sort.