AB-Solution 1.06 is out!

[thelonelycoder]

should i upgrade to 1.06 or stay at 1.05?

This is likely the last version (exept bug fixes, under the same version number) to come out.
It's up to you, if you have no use of the new functions then stay on 1.05.

 

[thelonelycoder]

Hi Lonely, I upgraded both 1.05 and 1.06 today, and the only problem I got was about the services-start script deleted on my /jffs partition. [/CODE]

I see why it happened, the shell type/version on Asuswrt-Merlin is not as capable as I thought.

The file /jffs/scripts/services-start now gets copied into the backup directory before unneeded entries by my script are removed.
If services-start is not empty after that, it is left in place.

The fix is already pushed to version v1.06 on GitHub.
Re-downloading the file with curl as in post #1 will fetch the fixed version.

Be aware that this bug is still present in v1.05 and will not be corrected.

 

[GGavan]

I upgraded to 1.06 and I used "sh" - Sort hosts.
After it finished I observed that in my case the file "hosts-adblock" has ~half of the size after sorting.
It is ok? Maybe it is removing some duplicates during sort?
I have to mention that in the "update-hosts.sh" I added some more hosts lists.

 

[thelonelycoder]

I upgraded to 1.06 and I used "sh" - Sort hosts.
After it finished I observed that in my case the file "hosts-adblock" has ~half of the size after sorting.
It is ok? Maybe it is removing some duplicates during sort?
I have to mention that in the "update-hosts.sh" I added some more hosts lists.

Post your update file here, so I can check it.

 

[GGavan]

The added part below...
When I updated I had ~24MB and after sort the file arrived at ~12MB
I tried to obtain a merge between option 3 and 4 of host file type + few others.

wget -qO- \
"http://winhelp2002.mvps.org/hosts.txt" \
"http://someonewhocares.org/hosts/zero/hosts" \
"http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" \
"http://www.malwaredomainlist.com/hostslist/hosts.txt" \
"http://support.it-mate.co.uk/downloads/hosts.txt" \
"http://hosts-file.net/hphosts-partial.txt" \
"http://www.hosts-file.net/emd.txt" \
"http://hosts-file.net/grm.txt" \
"http://hosts-file.net/mmt.txt" \
"http://hosts-file.net/ad_servers.txt" \
"http://adaway.org/hosts.txt" \
"http://hosts-file.net/exp.txt" \
"http://hosts-file.net/fsa.txt" \
"http://hosts-file.net/hfs.txt" \
"http://hosts-file.net/hjk.txt" \
"http://hosts-file.net/pha.txt" \
"http://hosts-file.net/psh.txt" \
> $dir/temp1

 

[GGavan]

Maybe the sort is done only for the original list from original installation with 6 hosts, ignoring the lists added by me. I saw that option 4 has ~ 12MB.

 

[thelonelycoder]

The added part below...
When I updated I had ~24MB and after sort the file arrived at ~12MB
I tried to obtain a merge between option 3 and 4 of host file type + few others.

wget -qO- \
"http://winhelp2002.mvps.org/hosts.txt" \
"http://someonewhocares.org/hosts/zero/hosts" \
"http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" \
"http://www.malwaredomainlist.com/hostslist/hosts.txt" \
"http://support.it-mate.co.uk/downloads/hosts.txt" \
"http://hosts-file.net/hphosts-partial.txt" \
"http://www.hosts-file.net/emd.txt" \
"http://hosts-file.net/grm.txt" \
"http://hosts-file.net/mmt.txt" \
"http://hosts-file.net/ad_servers.txt" \
"http://adaway.org/hosts.txt" \
"http://hosts-file.net/exp.txt" \
"http://hosts-file.net/fsa.txt" \
"http://hosts-file.net/hfs.txt" \
"http://hosts-file.net/hjk.txt" \
"http://hosts-file.net/pha.txt" \
"http://hosts-file.net/psh.txt" \
> $dir/temp1

This file is full of duplicates (as in it's full of stars...). Hang on...

 

[elorimer]

Maybe the sort is done only for the original list from original installation with 6 hosts, ignoring the lists added by me. I saw that option 4 has ~ 12MB.

The sort uses the -u option to remove duplicates.

 

[thelonelycoder]

The sort uses the -u option to remove duplicates.

I know that, still the update-hosts.sh should take care of the dups.

 

[GGavan]

Maybe it helps... on version 1.05 with the same list I had ~24MB

 

[thelonelycoder]

Maybe it helps... on version 1.05 with the same list I had ~24MB

This version did not have a sort function. I may be onto something here.

 

[thelonelycoder]

What's that sed command doing again?

sed $'s/\r$//'

Grabs and replaces it with nothing, but what is it really doing?

 

[GGavan]

I have no idea if you ask me. I'm just a windows user. On Linux I'm only executing the given instructions or copy/paste

 

[thelonelycoder]

Haha, this <sed $'s/\r$//'> is doing bloody well nothing.

Replace this in /adblocking/scripts/update-hosts.sh

cat $dir/temp1 | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' > $dir/temp2

with:

cat $dir/temp1 | sed s/127.0.0.1/0.0.0.0/g | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u > $dir/temp2

As it turns out, the duplicates were never removed.
Pushing a bugfix up to GitHub sometime later.

 

[GGavan]

Just checked in Excel before seeing your answer , and yes a lot of duplicates were there after update hosts.
I'm trying now with your new recommendation.
It seems that is already sorted after "update hosts", probably because of "| sort -u > $dir/temp2" from new line.
I think that is much better now to have it already sorted. It doesn't take much time on my router.
Maybe "sh" - sort option has no use now because is already sorted?
Thx

 

[thelonelycoder]

Just checked in Excel before seeing your answer , and yes a lot of duplicates were there after update hosts.
I'm trying now with your new recommendation.
It seems that is already sorted after "update hosts", probably because of "| sort -u > $dir/temp2" from new line.
I thing that is much better now to have it already sorted. It doesn't take much time on my router.
Thx

I've had this code running on my routers for close to a year without realising it never removed the duplicates.
I have to sit down now and consider my life choices. Need a sandwitch or something to recover...

 

[GGavan]

It worked anyway before, now it is better
Great tool!

 

[elorimer]

What's that sed command doing again?

sed $'s/\r$//'

Grabs and replaces it with nothing, but what is it really doing?

I thought it was there to guard against importing a dos formatted text file, to convert return/newline to newline.

 

[elorimer]

I know that, still the update-hosts.sh should take care of the dups.

Sorry. I only meant to confirm for ggavan that the sort is what shrunk his file. And his additional sources would have had a lot of duplicates.

I see from the original combined hosts scripts (standard and advanced), the wget commands themselves used a sort -u in the piping. So the advanced method wouldn't have done a global sort or remove duplicates globally, just from within the wget sources.

I guess removing the duplicates gives you the sort for free.

Also: 1. The curl -O fetch method works fine, completely painless way to upgrade. 2. The Sunday switch to 0 works fine. 3. I manually edited the script to insert the sort in the piping and that works fine. Cut the middle level from 200,000 lines to 190,000 lines, around 5.8 MB.

 

[RMerlin]

I thought it was there to guard against importing a dos formatted text file, to convert return/newline to newline.

That's correct. It strips any "/r" from DOS files that used "/r/n" as line terminations.