Unbound unbound_manager (Manager/Installer utility for unbound - Recursive DNS Server)

[joe scian]

Far from my expertise level, and maybe it is just a formatting, cut n' paste issue, but wouldn't all of those '@' symbols under the DoT section wreak some havoc??

yes they should all be # instead of @ or 1 missing # infront of the first @

 

[juched]

Far from my expertise level, and maybe it is just a formatting, cut n' paste issue, but wouldn't all of those '@' symbols under the DoT section wreak some havoc??

Agree, that is the issue. Now, why did it get there is the question.

 

[Khadanja]

Is it possible to view Adblock logs?

 

[Martineau]

Is it possible to view Adblock logs?

There is no separate 'Adblock' log.

If you enable unbound logging (I have enabled scribe) you can see the Ad Block blocked domains as 'always nxdomain' entries

e.g.

e  = Exit Script

A:Option ==> l

/opt/var/log/unbound.log (syslog-ng)        Press CTRL-C to stop

RT-AC68U unbound: [22284:0] info: device-metrics-us.amazon.com. always_nxdomain 127.0.0.1@37217 device-metrics-us.amazon.com. A IN
RT-AC68U unbound: [22284:0] info: device-metrics-us.amazon.com. always_nxdomain 127.0.0.1@56789 device-metrics-us.amazon.com. AAAA IN
RT-AC68U unbound: [22284:0] info: zemanta.com. always_nxdomain 127.0.0.1@63769 b1-chidc2.zemanta.com. A IN
RT-AC68U unbound: [22284:0] info: zemanta.com. always_nxdomain 127.0.0.1@31480 b1t-chidc2.zemanta.com. A IN

 

[juched]

Thanks to @SomeWhereOverTheRainBow there have been some enhancements, performance and cleanup made to the gen_adblock script. @Martineau will be releasing a new version soon to enable it, so once 2.17 comes out hopefully people can benefit from it.

To make things more clean the "sites" file has been split into two (both located in /opt/

1. blocksites - these are URLs for the domain lists or hosts file format for sites to adblock.
2. allowsites - these are URLs for domain lists or hosts file format for sites to ensure are not adblocked.

NOTE: You no longer need to prepend a <type> to the list. This list is now easier to use since it is just a list of URLs, one per line. Now the 2 files are the difference types.

For individual hosts you want to allow or block:

1. allowhost - domain list of hosts to ensure are not blocked by adblock
2. blockhost - additional domain list of hosts you want to block with adblock

You may want to back your old "sites" file as it will be moved and modified, or possibly ignored during this update.

Thank you @SomeWhereOverTheRainBow and @Martineau !!

--- edit ---
version is now v1.0.4

 

[rgnldo]

version is now v1.0.4

There is no need

destinationIP="0.0.0.0"

 

[Martineau]

Thanks to @SomeWhereOverTheRainBow there have been some enhancements, performance and cleanup made to the gen_adblock script. @Martineau will be releasing a new version soon to enable it, so once 2.17 comes out hopefully people can benefit from it.

To make things more clean the "sites" file has been split into two (both located in /opt/

1. blocksites - these are URLs for the domain lists or hosts file format for sites to adblock.
2. allowsites - these are URLs for domain lists or hosts file format for sites to ensure are not adblocked.

NOTE: You no longer need to prepend a <type> to the list. This list is now easier to use since it is just a list of URLs, one per line. Now the 2 files are the difference types.

For individual hosts you want to allow or block:

1. allowhost - domain list of hosts to ensure are not blocked by adblock
2. blockhost - additional domain list of hosts you want to block with adblock

You may want to back your old "sites" file as it will be moved and modified, or possibly ignored during this update.

Thank you @SomeWhereOverTheRainBow and @Martineau !!

--- edit ---
version is now v1.0.4

@juched I have just tried unbound_manager (unreleased v2.17) to install Ad Block

e  = Exit Script

A:Option ==> i 3

and it has detected 'gen_adblock v1.0.4' and downloaded the new files

Restarting dnsmasq.....
Done.
Option Auto Reply 'y'    Installing Ads and Tracker Blocking.....
    adblock/gen_adblock.sh downloaded successfully
    adblock/permlist downloaded successfully
    adblock/blocksites downloaded successfully
    adblock/allowsites downloaded successfully
    adblock/blockhost downloaded successfully
    adblock/allowhost downloaded successfully
Adding Ad and Tracker 'include: /opt/var/lib/unbound/adblock/adservers'
Creating Daily cron job for Ad and Tracker update
Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
Removing possible temporary files..
Attempting to Download https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
######################################################################## 100.0%
Downloading User Allow List...
Combining User Custom block host...
Filtering user requested domains from adblock list...
Filtering required domains from adblock list...
Removing unnecessary formatting from the domain list...
0 domains compiled
Generating Unbound adlist.....
Removing temporary files...
Restarting Unbound DNS server...

but your script doesn't appear to correctly populate '/opt/var/lib/unbound/adblock/adservers' ​

0 domains compiled

 

[SuperDuke]

Hello all.....and just a quick question......but what are benefits of moving to the Unbound resident adblock vs. the tried and true Diversion? It may be splitting hairs but Diversion does it's job exceptionally well as does Unbound.....

And as for block lists....I'm on Large+ which it sounds like is a non-starter for import into Unbound.....thanks!

 

[SomeWhereOverTheRainBow]

@juched I have just tried unbound_manager (unreleased v2.17) to install Ad Block

e  = Exit Script

A:Option ==> i 3

and it has detected 'gen_adblock v1.0.4' and downloaded the new files

Restarting dnsmasq.....
Done.
Option Auto Reply 'y'    Installing Ads and Tracker Blocking.....
    adblock/gen_adblock.sh downloaded successfully
    adblock/permlist downloaded successfully
    adblock/blocksites downloaded successfully
    adblock/allowsites downloaded successfully
    adblock/blockhost downloaded successfully
    adblock/allowhost downloaded successfully
Adding Ad and Tracker 'include: /opt/var/lib/unbound/adblock/adservers'
Creating Daily cron job for Ad and Tracker update
Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
Removing possible temporary files..
Attempting to Download https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
######################################################################## 100.0%
Downloading User Allow List...
Combining User Custom block host...
Filtering user requested domains from adblock list...
Filtering required domains from adblock list...
Removing unnecessary formatting from the domain list...
0 domains compiled
Generating Unbound adlist.....
Removing temporary files...
Restarting Unbound DNS server...

but your script doesn't appear to correctly populate '/opt/var/lib/unbound/adblock/adservers'

0 domains compiled

It is the domains counter mechanism I bet, check to see if the file is physically there.

 

[dave14305]

It is the domains counter mechanism I bet, check to see if the file is physically there.

I think there's an extra variable being passed to the cleanup function. It doesn't do anything with $3.

cleanup $tempoutlist $outlist $finalist

I didn't scroll right far enough.

 

[Martineau]

It is the domains counter mechanism I bet, check to see if the file is physically there.

So whilst the script reports 0 (using the wrong entity) to the user, the script uses the correct entity to create 'stats.txt' which is also 0

 Number of adblocked (ads/malware/tracker) and blacklisted domains: 0
 Last updated: Thu Mar 12 18:34:12 2020

and of course if the file is physically empty.....

 ls -lah /opt/var/lib/unbound/adblock/adservers
 
-rw-rw-rw-    1 admin    root           0 Mar 12 18:35 /opt/var/lib/unbound/adblock/adservers

then there is clearly a major fault in the code... i.e. the new cleanup() function?

 

[SomeWhereOverTheRainBow]

I think it may need something like an awk 'print $1' in the one line cat command something along those lines I will test shortly

 

[Martineau]

I think it may need something like an awk 'print $1' in the one line cat command something along those lines I will test shortly

FYI,

Use of 'cat' in code such as

cat $unclean | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | sort -u

is inefficient, when

grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" $unclean | sort -u

works just as well. (The same applies to 'sed'/'awk' i.e. use of using 'cat' to pipe a file into these utilities isn't necessary)

 

[SomeWhereOverTheRainBow]

FYI,

Use of 'cat' in code such as

cat $unclean | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | sort -u

is inefficient, when

grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" $unclean | sort -u

works just as well. (The same applies to 'sed'/'awk' i.e. use of using 'cat' to pipe a file into these utilities isn't necessary)

I am unable to produce the same results as you in regards to not producing a list, try a different list please , as I want to see if that list is causing issues.

 

[Martineau]

I am unable to produce the same results as you in regards to not producing a list, try a different list please , as I want to see if that list is causing issues.

I did a fresh install of unbound, then opted for the 'i 3' Ad Block option, and it downloads the relevant 'gen_adblock.sh' v1.04 files from GitHub.

At this point the empty file is created, using the default list(s).

I'll wait for @juched to respond.

 

[SomeWhereOverTheRainBow]

I did a fresh install of unbound, then opted for the 'i 3' Ad Block option, and it downloads the relevant 'gen_adblock.sh' v1.04 files from GitHub.

At this point the empty file is created, using the default list(s).

I'll wait for @juched to respond.

I got a fix
https://github.com/juched78/Unbound-Asuswrt-Merlin/pull/2

awk 'NR==FNR{a[$0];next} !($0 in a) {print $NF}' $output $unclean | sort -u > ${output}.tmp

this actually relies on their being a word count provided by

grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" $unclean | sort -u > ${output}.tmp

so basically if the file has no IP addresses then the whole

awk 'NR==FNR{a[$0];next} !($0 in a) {print $NF}' $output $unclean | sort -u > ${output}.tmp

outputs a blank file.

 

[SomeWhereOverTheRainBow]

thanks @Martineau for testing that out. my setup would have never found it as some of my list used have hidden IP addresses in them, so i would have never known.

 

[Martineau]

thanks @Martineau for testing that out. my setup would have never found it as some of my files used have hidden IP addresses in them, so i would have never known.

- Clearly the default GitHub file(s) do not contain any IP addresses.

P.S. @juched informed me that he was considering adding more formats perhaps AdBlockPlus ? etc., so I suspected it may have been a premature work-in-progress release.

It's late here so I'll delay releasing 'unbound_manager v2.17' until @juched has had an opportunity to review your pull-request etc.

 

[juched]

I got a fix
https://github.com/juched78/Unbound-Asuswrt-Merlin/pull/2

awk 'NR==FNR{a[$0];next} !($0 in a) {print $NF}' $output $unclean | sort -u > ${output}.tmp

this actually relies on their being a word count provided by

grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" $unclean | sort -u > ${output}.tmp

so basically if the file has no IP addresses then the whole

awk 'NR==FNR{a[$0];next} !($0 in a) {print $NF}' $output $unclean | sort -u > ${output}.tmp

outputs a blank file.

Shoot. I see that now. I reviewed and accepted the merge. Thanks for testing. I just got home after rushing out this morning. Soft update, still version v1.0.4.

---- edit ---

tested with the default list and the new code BTW.

 

[SomeWhereOverTheRainBow]

@Martineau
from these list https://v.firebog.net/hosts/lists.php?type=tick
(i use more lists than what is listed here by the way)

I generate a separate IP.list @ https://github.com/jumpsmm7/GeneratedAdblock/blob/master/IPlist.list

my current list are hosted here

https://github.com/jumpsmm7/GeneratedAdblock