Unbound unbound_manager (Manager/Installer utility for unbound - Recursive DNS Server)

[Martineau]

Pushed an update for the Adblock and YouTube Adblock scripts.
- Added command to YT script "force_newip" to pick a new IP from the DNS cache and update the YT file.

Version=3.13
Github md5=88e48deea3afb4ef38f3d4399dacae1d​

e  = Exit Script [?]

A:Option ==> youtube newip

Y88b   d88P 88888888888     d8888      888 888888b.   888                   888
 Y88b d88P      888        d88888      888 888  "88b  888                   888
  Y88o88P       888       d88P888      888 888  .88P  888                   888
   Y888P        888      d88P 888  .d88888 8888888K.  888  .d88b.   .d8888b 888  888
    888         888     d88P  888 d88" 888 888  "Y88b 888 d88""88b d88P"    888 .88P
    888         888    d88P   888 888  888 888    888 888 888  888 888      888888K
    888         888   d8888888888 Y88b 888 888   d88P 888 Y88..88P Y88b.    888 "88b
    888         888  d88P     888  "Y88888 8888888P"  888  "Y88P"   "Y8888P 888  888
## by @juched - dynamically block YT ads - v1.2                    
gen_ytadblock.sh
Forgetting old IP...

P.S. Not sure why you specify 'bash' rather than 'sh' ?

'gen_adblock.sh'

#!/bin/bash

# AdBlock Script to download from customizable lists and merge into unbound rules file.
# Use permlist file in same folder to add per line domains you want to whitelist.
# Use blocklist file in same folder to block per line domains you want to block.

 

[juched]

P.S. Not sure why you specify 'bash' rather than 'sh' ?​

'gen_adblock.sh'

#!/bin/bash

# AdBlock Script to download from customizable lists and merge into unbound rules file.
# Use permlist file in same folder to add per line domains you want to whitelist.
# Use blocklist file in same folder to block per line domains you want to block.

Yes, good catch, this was inheritied from the original script, and will change it.

 

[penguin22]

Version=3.13
Github md5=88e48deea3afb4ef38f3d4399dacae1d​

Following update, is it necessary to remove and re-add YouTube Adblock to apply the updates, or does that occur with the update to 3.13?

 

[here1310]

? after each update option 1
Execute "1 = Update unbound files and configuration" or not ??
adjusted settings are then always lost and must be readjusted in unbound.conf ...

 

[Slawek P]

Hi,

I have just installed unboud and love its speed and Youtube blocking.
Hit cache ratio shows around 60-70% on WebUI.

A few things where I am stuck with and I need pointers for dummies please.

1/ Is there a link comparing pros/cons of ad-blocking using unbound vs diversion/pixelserv. Has anybody switched to unboud ad-blocking?
2/ While analysing port traffic with tcpdump from Entware (for the first time actually) I discovered none of parameters filtering ports or src/dst works for me. This is really annoying.
I am able to do see all traffic OK with tcpdump -i eth0. Any suggestions?
tcpdump version 4.9.3
libpcap version 1.9.1 (with TPACKET_V3)
3/ Need to figure out what do about IPv6 settings - I've read by default they are not setup after installation.
4/ tcpdump showed something strange - getting lots of Cloudflare calls
[my address] > 1.1.1.3.domain: 19160+ A? dns.msftncsi.com. (34)
[my address]> 1.1.1.3.domain: 28895+ PTR? [my addres].in-addr.arpa. (43)
[my address] > 1.1.1.3.domain: 24792+ AAAA? dns.msftncsi.com. (34)
To me that is strange and confusing. Does it imply forwarding to recursive CloudFlare DNS?
I thought standard install without Stubby should leave me with Unbound setup as Authoritative DNS.

Many thanks in advance for hints.

 

[dave14305]

4/ tcpdump showed something strange - getting lots of Cloudflare calls
[my address] > 1.1.1.3.domain: 19160+ A? dns.msftncsi.com. (34)
[my address]> 1.1.1.3.domain: 28895+ PTR? [my addres].in-addr.arpa. (43)
[my address] > 1.1.1.3.domain: 24792+ AAAA? dns.msftncsi.com. (34)
To me that is strange and confusing. Does it imply forwarding to recursive CloudFlare DNS?

Is 1.1.1.3 your WAN DNS setting? You can disable Network Monitoring in the Administration / System page to stop these queries.

2/ While analysing port traffic with tcpdump from Entware (for the first time actually) I discovered none of parameters filtering ports or src/dst works for me. This is really annoying.
I am able to do see all traffic OK with tcpdump -i eth0. Any suggestions?

What command options are you trying? I can run:

tcpdump -i eth0 -n port 53 and dst host 9.9.9.11

 

[Slawek P]

Is 1.1.1.3 your WAN DNS setting? You can disable Network Monitoring in the Administration / System page to stop these queries.

What command options are you trying? I can run:

tcpdump -i eth0 -n port 53 and dst host 9.9.9.11

Thanks. It is not networking monitoring - both are disabled. And indeed - common generates some traffic when I chose name rather than IP for pings. But that's not what it is. I had DoT DNS to 1.1.1.3 before Unbound indeed. Perhaps Diversion generates something, not Unbound.
In tcpdump anything related to filters does not apear to work. So first one dumps data, but others do not
tcpdump -i eth0 -n
tcpdump -i eth0 -n port 53
tcpdump -i eth0 -n -X icp
tcpdump -i eth0 -n host 1.1.1.3

 

[joe scian]

Hi Martineau
I have 19 hosts in /opt/var/lib/unbound/permlist
as shown below.

Is /opt/share/unbound/configs/allowhost the place where we can also add whitelisted hosts. Because I have over 210 there and the whitelist output below is still counting only 19 in permlist. When I move the 210 from allowlist to permlist the counter is updated. There may also be an issue with opt/share/unbound/configs/blockhost - I have 15 domains there and at least two of these domains eg ngw.dvr163.com and p2plog.dvr163.com have a huge number of hits during the day ( my NVR camera ). They appear prominently in DIVERSION stats ( in top 10 blocked domains) but NEVER appear in Top Ad-Blocked domains or even in top reply Domains for ADBLOCK.

  Options: Auto Reply='y' for User Selectable Options ('1 3 4 5') unbound Logging,Ad Block,Performance Tweaks,Firefox DoH

        [✔] unbound Logging
        [✔] Ad and Tracker Blocking (No. of Adblock domains=89116,Blocked Hosts=15,Whitelist=19)
        [✔] unbound CPU/Memory Performance tweaks
        [✔] Firefox DNS-over-HTTPS (DoH) DISABLE/Blocker
        [✔] Router Graphical GUI statistics TAB installed
        [✔] unbound-control FAST response ENABLED
        [✔] DNS Firewall ENABLED
        [✔] Unbound is the Primary DNS for ALL LAN Clients (dnsmaq DNS features DISABLED e.g. IPSET auto-populate)
        [✔] YouTube Ad Blocking (Forcing to use YT IP 220.233.204.109, No. of YouTube Video Ad domains=103)

 

[Twiglets]

Hi Martineau
I have 19 hosts in /opt/var/lib/unbound/permlist
as shown below.

Is /opt/share/unbound/configs/allowhost the place where we can also add whitelisted hosts. Because I have over 210 there and the output below is still 19. When I move them to permlist the counter is updated. There also may be an issue with opt/share/unbound/configs/blockhost - I have 15 domains there and at least two of these domains eg ngw.dvr163.com and p2plog.dvr163.com have a huge number of hits during the day ( my NVR camera ). They appear prominently in DIVERSION stats ( in top 10 blocked domains) but NEVER appear in Top Ad-Blocked domains or even in top reply Domains for ADBLOCK.

  Options: Auto Reply='y' for User Selectable Options ('1 3 4 5') unbound Logging,Ad Block,Performance Tweaks,Firefox DoH

        [✔] unbound Logging
        [✔] Ad and Tracker Blocking (No. of Adblock domains=89116,Blocked Hosts=15,Whitelist=19)
        [✔] unbound CPU/Memory Performance tweaks
        [✔] Firefox DNS-over-HTTPS (DoH) DISABLE/Blocker
        [✔] Router Graphical GUI statistics TAB installed
        [✔] unbound-control FAST response ENABLED
        [✔] DNS Firewall ENABLED
        [✔] Unbound is the Primary DNS for ALL LAN Clients (dnsmaq DNS features DISABLED e.g. IPSET auto-populate)
        [✔] YouTube Ad Blocking (Forcing to use YT IP 220.233.204.109, No. of YouTube Video Ad domains=103)

The 'whitelist' does appear to be the 'permlist' file because if I run the 'ad' command (tweaked to perform 'merge' not 'analyse') the Diversion whitelist entries end up in the 'permlist' file.

Is there any doc which explains what all the files are for/file format ?
Might it be useful to have some example files to learn from ...... much like the example files that are available with dnscrypt-proxy ?
(I know there is a commented .conf for unbound in the /opt/share/unbound/configs/doc/example.conf.in file.)

 

[SomeWhereOverTheRainBow]

P.S. Not sure why you specify 'bash' rather than 'sh' ?

SH is limited with some of its metrics for running shell .

 

[Twiglets]

Hi Martineau
I have 19 hosts in /opt/var/lib/unbound/permlist
as shown below.

Is /opt/share/unbound/configs/allowhost the place where we can also add whitelisted hosts.
[/CODE]

Answer at top of page !!!
https://www.snbforums.com/threads/r...cursive-dns-server.61669/page-110#post-584325

Read Header of 'gen_adblock.sh'.

 

[juched]

Hi Martineau
I have 19 hosts in /opt/var/lib/unbound/permlist
as shown below.

Is /opt/share/unbound/configs/allowhost the place where we can also add whitelisted hosts. Because I have over 210 there and the output below is still 19. When I move them to permlist the counter is updated. There also may be an issue with opt/share/unbound/configs/blockhost - I have 15 domains there and at least two of these domains eg ngw.dvr163.com and p2plog.dvr163.com have a huge number of hits during the day ( my NVR camera ). They appear prominently in DIVERSION stats ( in top 10 blocked domains) but NEVER appear in Top Ad-Blocked domains or even in top reply Domains for ADBLOCK.

  Options: Auto Reply='y' for User Selectable Options ('1 3 4 5') unbound Logging,Ad Block,Performance Tweaks,Firefox DoH

        [[emoji818]] unbound Logging
        [[emoji818]] Ad and Tracker Blocking (No. of Adblock domains=89116,Blocked Hosts=15,Whitelist=19)
        [[emoji818]] unbound CPU/Memory Performance tweaks
        [[emoji818]] Firefox DNS-over-HTTPS (DoH) DISABLE/Blocker
        [[emoji818]] Router Graphical GUI statistics TAB installed
        [[emoji818]] unbound-control FAST response ENABLED
        [[emoji818]] DNS Firewall ENABLED
        [[emoji818]] Unbound is the Primary DNS for ALL LAN Clients (dnsmaq DNS features DISABLED e.g. IPSET auto-populate)
        [[emoji818]] YouTube Ad Blocking (Forcing to use YT IP 220.233.204.109, No. of YouTube Video Ad domains=103)

I need to change the comment at the top of the gen_adblock file, that is old.

permlist is an internal hard coded list of sites needed to keep Merlin update checks etc working.

All files you should be editing are in the /opt/share/unbound/configs folder.

allowsites and blocksites are for your URLs to download block or allow files.

allowhosts and blockhosts are where you enter one per line domain hosts you want to explcitly allow or block.

You really shouldn’t edit permlist but should move them to allowhosts.

 

[Twiglets]

I need to change the comment at the top of the gen_adblock file, that is old.

permlist is an internal hard coded list of sites needed to keep Merlin update checks etc working.

All files you should be editing are in the /opt/share/unbound/configs folder.

allowsites and blocksites are for your URLs to download block or allow files.

allowhosts and blockhosts are where you enter one per line domain hosts you want to explcitly allow or block.

You really shouldn’t edit permlist but should move them to allowhosts.

Thanks for the clarification.
One more question, what is the pixelserv file format ?

 

[joe scian]

I need to change the comment at the top of the gen_adblock file, that is old.

permlist is an internal hard coded list of sites needed to keep Merlin update checks etc working.

All files you should be editing are in the /opt/share/unbound/configs folder.

allowsites and blocksites are for your URLs to download block or allow files.

allowhosts and blockhosts are where you enter one per line domain hosts you want to explcitly allow or block.

You really shouldn’t edit permlist but should move them to allowhosts.

But is Blockhost actually blocking hosts??? Read my post above>

 

[Martineau]

Hi Martineau
I have 19 hosts in /opt/var/lib/unbound/permlist
as shown below.

Is /opt/share/unbound/configs/allowhost the place where we can also add whitelisted hosts. Because I have over 210 there and the whitelist output below is still counting only 19 in permlist. When I move the 210 from allowlist to permlist the counter is updated. .

  Options: Auto Reply='y' for User Selectable Options ('1 3 4 5') unbound Logging,Ad Block,Performance Tweaks,Firefox DoH

        [✔] unbound Logging
        [✔] Ad and Tracker Blocking (No. of Adblock domains=89116,Blocked Hosts=15,Whitelist=19)
        [✔] unbound CPU/Memory Performance tweaks
        [✔] Firefox DNS-over-HTTPS (DoH) DISABLE/Blocker
        [✔] Router Graphical GUI statistics TAB installed
        [✔] unbound-control FAST response ENABLED
        [✔] DNS Firewall ENABLED
        [✔] Unbound is the Primary DNS for ALL LAN Clients (dnsmaq DNS features DISABLED e.g. IPSET auto-populate)
        [✔] YouTube Ad Blocking (Forcing to use YT IP 220.233.204.109, No. of YouTube Video Ad domains=103)

Hmm, there appears to be an issue when first displaying '/opt/share/unbound/configs/blockhost' count 'Blocked Hosts=0' as I know I have 1 entry.

[✔] Ad and Tracker Blocking (No. of Adblock domains=89965,Blocked Hosts=0,Whitelist=19)

There may also be an issue with opt/share/unbound/configs/blockhost - I have 15 domains there and at least two of these domains eg ngw.dvr163.com and p2plog.dvr163.com have a huge number of hits during the day ( my NVR camera ). They appear prominently in DIVERSION stats ( in top 10 blocked domains) but NEVER appear in Top Ad-Blocked domains or even in top reply Domains for ADBLOCK

I tested as follows.....

e  = Exit Script [?]

A:Option ==> eb          # <<--basically I added your 'p2plog.dvr163.com'

 Ad Block file '/opt/share/unbound/configs/blockhost' changed....updating Ad Block
                            
 _____   _ _   _         _  
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 14869 @juched - v1.0.8 - Thanks to @SomeWhereOverTheRainBow

[✔] Ad and Tracker Blocking (No. of Adblock domains=89993,Blocked Hosts=2,Whitelist=19)

So the count jumped to 2, as it now includes my single static entry

e  = Exit Script [?]

A:Option ==> eb          # <<--basically I added your 'ngw.dvr163.com'

 Ad Block file '/opt/share/unbound/configs/blockhost' changed....updating Ad Block
                            
 _____   _ _   _         _  
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 14869 @juched - v1.0.8 - Thanks to @SomeWhereOverTheRainBow

[✔] Ad and Tracker Blocking (No. of Adblock domains=89994,Blocked Hosts=3,Whitelist=19)

So the displayed 'Blocked Hosts=3' count is correctly incrementing and displayed as expected and the two new entries are in

grep -E "dvr163"  /opt/var/lib/unbound/adblock/adservers

local-zone: "ngw.dvr163.com" always_nxdomain
local-zone: "p2plog.dvr163.com" always_nxdomain

Seems to work as designed?

P.S. I'll investigate why the initial display count is seemingly incorrectly displayed as 0 despite there being 1 static entry.

 

[Martineau]

The 'whitelist' does appear to be the 'permlist' file because if I run the 'ad' command (tweaked to perform 'merge' not 'analyse') the Diversion whitelist entries end up in the 'permlist' file.

Whoah, unsupported software.....call the copyright police! ….poking around in my code and subsequent unauthorised activation 'tweak' of hidden features!!!

P.S. Absolutely flabbergasted - firstly, I didn't think anyone would find it useful , and secondly, it actually works?

Is there any doc which explains what all the files are for/file format ?
Might it be useful to have some example files to learn from ...…

I did suggest it here back at the beginning of March

 

[Martineau]

Thanks for the clarification.
One more question, what is the pixelserv file format ?

@dave14305 found that a single 'local-zone' record for pixelserv-tls redirection uses less memory than the original formal 'redirect/local-data' pair of records.

You can see the commented code in my script! (just a few lines below where you found the 'merge' feature)

 

[Huey11]

Hi, great stuff, all of this!

I now use adguardhome for the "normal" adblocking, but this YT adblock is smart, and I do not want to revert to my previously in use pihole for that (and atguardhome people seem reluctant to implement it).
I do presume it is ok to use only the YT adblocker without the general adblock?
Because I noticed that when I install unbound with ONLY the YT adblock, the update of unbound and configuration gives an error on the general adblock script missing... This is when using the simple install.

Spoiler: error

/opt/var/lib/unbound/unbound.conf:142: error: cannot open include file '/opt/var/lib/unbound/adblock/adservers': No such file or directory
read /opt/var/lib/unbound/unbound.conf failed: 1 errors in configuration file

***ERROR INVALID unbound configuration - use option 'vx' to correct 'unbound.conf' or 'rl' to load a valid configuration file

When using the advanced update (i) it goes well though. So I guess the simple install presumes the YT adblock to always accompany the general, but the advanced not?


All seems to work ok using only YT and not the general...

Regards

Edit: thinking of reverting to adblock. Hopefully it is not to restrictive. Using adguardhome instead of pihole with the abp.oisd.nl list cause it seems snappier, easier to use and update in my docker cluster.
And primarily using an external adblocker as I then can easily switch with dnsfilter and keep some devices ad blocking free (aka defending myself for the waf). The YT adblock is an easy win with no (known) side effects, so easy for every user.

 

[Twiglets]

Whoah, unsupported software.....call the copyright police! ….poking around in my code and subsequent unauthorised activation 'tweak' of hidden features!!!

P.S. Absolutely flabbergasted - firstly, I didn't think anyone would find it useful , and secondly, it actually works?



I did suggest it here back at the beginning of March

View attachment 23504

Martineau,

Mea culpa, mea culpa, mea maxima culpa !!! .....

Abject apologies for 'running amok' in your code !!!
(Hair shirt is on order from Amazon .. as we speak !!!)

I learn so much looking at other peoples code ....... and hacking it 'badly' !!!!
(Do usually keep quiet about the 'hacks' so as to avoid creating problems for the original authors with 'unsupported software', as you noted )

I can understand most things in most languages .... if the maths does not get beyond me .... (which is quite easy to do) !!!

Re: Your gast being flabbered ..... [Careful now !!!]
It is useful if you are trying to replicate the blocking etc that you have in Diversion.
It does appear to work ..... yes really
I am trying to run unbound with the same blocking/whitelist/etc as Diversion, hence the 'unauthorised' hack to activate 'merge'.
I want to contrast 'unbound with Diversion' vs 'Ad-Blocking with unbound ONLY'.

When I deactivate 'Diversion' I use a hacked version of the code in /opt/etc/init.d/S80pixelserv-tls [yet another apology required to LonelyCoder ] to manually start 'pixelserv-tls'.
[I have noted your own 'hack' 'code evolution' to enable pixelserv-tls to be started if Diversion is down, which I found later]

I update adblocking for unbound which contains 2 of the lists I use in Diversion then use 'ad'[hacked] to merge in the latest from Diversion.
The only thing I have not yet worked out is how to handle the 'wildcard' blocking list from Diversion.

Finally, many many thanks for the huge effort made by you for creating unbound_manager which is amazing just as a script never mind the actual job it does
Between yourself and LonelyCoder et al and my noseyness, I should be able to write code that works ....... 'should' is a very good word must check its meaning one day !!!

 

[Martineau]

? after each update option 1
Execute "1 = Update unbound files and configuration" or not ??
adjusted settings are then always lost and must be readjusted in unbound.conf ...

As explained in previous posts, it is rare when pushing a new version of 'unbound_manager.sh' (using 'u' command) that it is necessary to also issue '1 or i' to update the unbound installation.

However, if you choose to issue '1 or i' then, like most update procedures, it is prudent/wise to backup your existing configuration to '/opt/share/unbound/configs/' so you can restore your custom 'unbound.conf' using the 'rl' command - although any new features that rely on the new 'unbound.conf' contents will be unavailable.