Unbound unbound_manager (Manager/Installer utility for unbound - Recursive DNS Server)

[dave14305]

kasper@RT-AC88U-C440:/tmp# cat /jffs/scripts/dnsmasq.postconf sh /jffs/addons/unbound/unbound.postconf "$1"

You need this as the first line of dnsmasq.postconf:

#!/bin/sh

Run these 3 commands:

sed -i '1s~^~#!/bin/sh\n~' /jffs/scripts/dnsmasq.postconf
chmod 755 /jffs/scripts/dnsmasq.postconf
service restart_dnsmasq

 

[krgck]

You need this as the first line of dnsmasq.postconf:

#!/bin/sh

Run these 3 commands:

sed -i '1s~^~#!/bin/sh\n~' /jffs/scripts/dnsmasq.postconf
chmod 755 /jffs/scripts/dnsmasq.postconf
service restart_dnsmasq

That did the trick. Thanks for saving my time and sleep

 

[tomsk]

You need this as the first line of dnsmasq.postconf:

#!/bin/sh

Run these 3 commands:

sed -i '1s~^~#!/bin/sh\n~' /jffs/scripts/dnsmasq.postconf
chmod 755 /jffs/scripts/dnsmasq.postconf
service restart_dnsmasq

Oh the shebang is there for me because the dnsmasq.postconf was added by diversion ... but this install was written by the manager script?

 

[dave14305]

Oh the shebang is there for me because the dnsmasq.postconf was added by diversion ... but this unbound install was written by the manager script?

unbound.postconf is from Martineau. krgck must never have had a dnsmasq.postconf and unbound_manager doesn’t (yet) account for that scenario.

 

[tomsk]

Oh well
https://www.snbforums.com/threads/nextdns-installer.61002/page-20#post-582558

 

[Twiglets]

Yes fairly close depending on that else you might have bolted onto diversion

A:Option ==> ad

Analysed Diversion file: 'blockinglist'     Type=pixelserv, (Adblock Domains=55204) would add 620 entries
Analysed Diversion file: 'blacklist'     Type=pixelserv, (Adblock Domains=55204) would add 2 entries
Analysed Diversion file: 'whitelist'     Type=URL, (Adblock URLs=19) would add 22 entries

I guess the ultimate if you wanted identical lists would to be to import them from diversion... the code is in the script to do the dnsmasq to unbound type lists conversion already ( the steven black one downloads in dnsmasq form and needs conversion)

The script already has a 'merge' option that is not tied to a menu option.
A small 'edit' can run the 'merge' code which does appear to work !!!???

Not sure why it has been left out ?

 

[tomsk]

The script already has a 'merge' option that is not tied to a menu option.
A small 'edit' can run the 'merge' code which does appear to work !!!???

Not sure why it has been left out ?

Thats not just the icing... that's the cherry on top too.

 

[joe scian]

Hi Martineau

when I issue the command below I get

A:Option ==> ad


cp: write error: No space left on device
sed: write error

and now my syslog is filled up with. A router reboot fixes the syslog spam below but I wont issue that ad command again. !

May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:14 RT-AC5300-0680 syslog: Error locking /var/lock/cfg_mnt.lock: 28 No space left on device

 

[Martineau]

I just had a look at my unbound.conf just now ... With dnsmasq enabled shouldn't the unbound interface on 127.0.0.1 for port 53 be commented out?

#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
port: 53535                                 # v1.08 If 53, requires 'port=0' in '/etc/dnsmasq.conf' to 'disable' dnsmasq to answer queries direct from LAN clients
interface: 127.0.0.1@53535                  # v1.01 As per @dave14305 minimal config; Will be overwritten by $(nvram get lan_ipaddr_rt) if dnsmasq 'disabled'
interface: 127.0.0.1@53                    # v1.10 Required by router if dnsmasq 'disabled'
#access-control: 0.0.0.0/0 allow            # v1.10 Will be overwritten by LAN subnet "${lan_ip_addr_rt}/24" if 'dnsmasq disabled' aka bypassed
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Whoops!

In trying to tidy up 'unbound.conf' v1.10, I 'helpfully' added a comment....see this commit

Clearly the original code

[ -n "$(grep "^interface: 127.0.0.1@53$" /opt/var/lib/unbound/unbound.conf)" ] && sed   's/^interface: 127\.0\.0\.1@53$/#interface: 127\.0\.0\.1@53/' /opt/var/lib/unbound/unbound.conf

will now never match/find an uncommented line containing ONLY the text 'interface: 127.0.0.1@53'

I'll push a Hotfix later.

 

[tomsk]

I was having a look around for other organisations suppling RPZ data other than spamhaus . seems the other main player is SURBL .... the SURBL combined list is interesting in the way they use ip to show what list its from http://www.surbl.org/lists

multi.surbl.org - Combined SURBL list
All of the SURBL data sources are combined into a single, bitmasked list: multi.surbl.org.
Bitmasking means that there is only one entry per domain name or IP address, but that entry will resolve into an address (DNS A record) whose last octet indicates which lists it belongs to. The bit positions in that last octet for membership in the different lists are:
8 = listed on PH
16 = listed on MW
64 = listed on ABUSE
128 = listed on CR
If an entry belongs to just one list it will have an address where the last octet has that value. For example 127.0.0.8 means it's on the phishing list, while 127.0.0.64 means it's listed on the ABUSE list. An entry on multiple lists gets the sum of those list numbers as the last octet, so 127.0.0.80 means a record is on both MW and ABUSE (comes from: 16 + 64 = 80). In this way, membership in multiple lists is encoded into a single response. Octets other than the first and last one are reserved for future use and should be ignored.
/QUOTE]

 

[Martineau]

Hi Martineau

when I issue the command below I get

A:Option ==> ad


cp: write error: No space left on device
sed: write error

and now my syslog is filled up with. A router reboot fixes the syslog spam below but I wont issue that ad command again. !

May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:09 RT-AC5300-0680 syslog: Error locking /var/lock/networkmap.lock: 28 No space left on device
May 11 21:08:09 RT-AC5300-0680 syslog: Error unlocking -1: 9 Bad file descriptor
May 11 21:08:14 RT-AC5300-0680 syslog: Error locking /var/lock/cfg_mnt.lock: 28 No space left on device

OK, what size blocking list do you use?

 

[juched]

@juched

FYI,,,

 _____   _ _   _         _ 
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 28116 @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Attempting to Download 4 of 4 from https://raw.githubusercontent.com/llacb47/mischosts/master/tiktok-hosts.
######################################################################## 100.0%
Attempting to Download 7 of 4 from https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list.
######################################################################## 100.0%
Attempting to Download 8 of 4 from https://blocklist.cyberthreatcoalition.org/vetted/domain.txt.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...

should be

Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Attempting to Download 2 of 4 from https://raw.githubusercontent.com/llacb47/mischosts/master/tiktok-hosts.
######################################################################## 100.0%
Attempting to Download 3 of 4 from https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list.
######################################################################## 100.0%
Attempting to Download 4 of 4 from https://blocklist.cyberthreatcoalition.org/vetted/domain.txt.
######################################################################## 100.0%

'gen_adblock.sh' needs a minor tweak

    for url in $(echo $line); do
      [ "${url:0:1}" == "#" ] && continue # skip commented out lines - Thanks @Martineau
      echo "Attempting to Download $count of $(awk 'NF && !/^[:space:]*#/' $sites | wc -l) from $url."
      curl --progress-bar $url | grep -o '^[^#]*' | grep -v "::1" | grep -v "0.0.0.0 0.0.0.0" | sed '/^$/d' | sed 's/\ /\\ /g' | awk '{print $NF}' | grep -o '^[^\\]*' | grep -o '^[^\\$]*' | sort >> $list
      dos2unix $list
      count=$((count + 1)) 
    done

Thanks, fixed and pushed to master. I also pushed YT ad block script to master as well.

 

[Martineau]

yes it should be perhaps an introduced "feature" since last hotfix.

What?

Also noted that

# log-local-actions - yes

remains commented out during adblock installation when going back and forth enabling Dnsmasq and subsequently choosing Dnsmasq disable and Adblock install. Had to manually uncomment this.

The option is only enabled if explicitly requested, and logging to 'syslog-ng/scribe' is also explicitly ENABLED.

The design/reasoning was that if unbound logging was ENABLED, then previously as there was no housekeeping performed on the size of the log, it could silently fill the disk a lot quicker.

e.g. explicitly ENABLE the setting

e  = Exit Script [?]

A:Option ==> adblock track

Option Auto Reply 'y'    Installing Ads and Tracker (Ad Block) Blocking.....
    adblock/gen_adblock.sh downloaded successfully
    adblock/permlist downloaded successfully
Custom '/opt/share/unbound/configs/blocksites' already exists - 'adblock/blocksites' download skipped
Custom '/opt/share/unbound/configs/allowsites' already exists - 'adblock/allowsites' download skipped
Custom '/opt/share/unbound/configs/blockhost' already exists - 'adblock/blockhost' download skipped
Custom '/opt/share/unbound/configs/allowhost' already exists - 'adblock/allowhost' download skipped
Creating Daily cron job for Ad and Tracker update
Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
                       
 _____   _ _   _         _
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 3200 @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow


Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Attempting to Download 4 of 4 from https://raw.githubusercontent.com/llacb47/mischosts/master/tiktok-hosts.
######################################################################## 100.0%
Attempting to Download 7 of 4 from https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list.
######################################################################## 100.0%
Attempting to Download 8 of 4 from https://blocklist.cyberthreatcoalition.org/vetted/domain.txt.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
Adding user requested hosts to list...
Removing user requested hosts from list...
Removing required hosts from list...
Removing unnecessary formatting from the domain list...
Generating Unbound adservers file...
(gen_adblock.sh): 3200 Number of adblocked hosts: 77143

Generating Unbound unload/load lists...
Loading/Unload Unbound local-zones to take effect...
removed 77143 zones
added 77143 zones
Removing temporary files...
Adblock update complete!

Logging Ad Block BLOCKED domains to scribe

I suppose now that logging housekeeping should no longer be an issue, as @juched's GUI stats can use the metrics, I suppose I could change the criteria to auto-enable if 'sgui' is ENABLED?

 

[juched]

What?

The option is only enabled if explicitly requested, and logging to 'syslog-ng/scribe' is also explicitly ENABLED.

The design/reasoning was that if unbound logging was ENABLED, then previously as there was no housekeeping performed on the size of the log, it could silently fill the disk a lot quicker.

e.g. explicitly ENABLE the setting

e  = Exit Script [?]

A:Option ==> adblock track

Option Auto Reply 'y'    Installing Ads and Tracker (Ad Block) Blocking.....
    adblock/gen_adblock.sh downloaded successfully
    adblock/permlist downloaded successfully
Custom '/opt/share/unbound/configs/blocksites' already exists - 'adblock/blocksites' download skipped
Custom '/opt/share/unbound/configs/allowsites' already exists - 'adblock/allowsites' download skipped
Custom '/opt/share/unbound/configs/blockhost' already exists - 'adblock/blockhost' download skipped
Custom '/opt/share/unbound/configs/allowhost' already exists - 'adblock/allowhost' download skipped
Creating Daily cron job for Ad and Tracker update
Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
                        
 _____   _ _   _         _
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 3200 @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow


Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 4 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Attempting to Download 4 of 4 from https://raw.githubusercontent.com/llacb47/mischosts/master/tiktok-hosts.
######################################################################## 100.0%
Attempting to Download 7 of 4 from https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list.
######################################################################## 100.0%
Attempting to Download 8 of 4 from https://blocklist.cyberthreatcoalition.org/vetted/domain.txt.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
Adding user requested hosts to list...
Removing user requested hosts from list...
Removing required hosts from list...
Removing unnecessary formatting from the domain list...
Generating Unbound adservers file...
(gen_adblock.sh): 3200 Number of adblocked hosts: 77143

Generating Unbound unload/load lists...
Loading/Unload Unbound local-zones to take effect...
removed 77143 zones
added 77143 zones
Removing temporary files...
Adblock update complete!

Logging Ad Block BLOCKED domains to scribe

I suppose now that logging housekeeping should no longer be an issue, as @juched's GUI stats can use the metrics, I suppose I could change the criteria to auto-enable if 'sgui' is enabled?

I think so, since every hour I flush those message as well, and support both scribe and non-scribe logging.

Would be good to have an "option" to use "sgui extended" to enable log-replies as well, so you can get extended stats showing all requested queries. Works even if dnsmasq disable is not used. No need to enable log-queries for extended stats, and my script will flush those lines from the log as well, so the disk should not fill up.

Thoughts?

 

[Martineau]

I've uploaded v3.12

Version=3.12
Github md5=4a207524e455366859549c3cce137e95​

use 'u' to update when prompted on screen

Use of the 'i = Update unbound Installation' ** not required **

FIX:    If '/jffs/scripts/dnsmasq.postconf doesn't exist then create it with '#!/bin/sh' (and execute attributes) before adding 'sh /jffs/scripts/addons/unbound/unbound.postconf' - @dave14305
FIX:    dnsmasq bypass, 'unbound.conf' v1.10 added a comment, so switching back to dnsmasq as Primary DNS, unbound now correctly releases '127.0.0.1@53' - @tomsk
FIX:    'ad' to use '/opt/tmp/' rather than '/tmp/' - @joe scian
CHANGE: If 'sgui' ENABLED and 'adblock' requested then ENABLE tracking of blocked domains 'log-local-actions: yes' - @joe scian
CHANGE: 'youtube' YT Video Ad blocking will now retrieve script from @juched's Github master branch rather than his dev branch.
CHANGE: Enhance 'debug' diagnostics
ADD:    'youtube [view | edit]' to allow access (for the curious) to the current YT blocked domains
CHANGE: '?' command, include clickable URL link to unbound v1.10.0 manual

    About unbound: https://nlnetlabs.nl/projects/unbound/about/ , Manual https://nlnetlabs.nl/documentation/unbound/unbound.conf/

Thanks again to the usual suspects for proof reading/bug reporting.

 

[Martineau]

I think so, since every hour I flush those message as well, and support both scribe and non-scribe logging.

Would be good to have an "option" to use "sgui extended" to enable log-replies as well, so you can get extended stats showing all requested queries. Works even if dnsmasq disable is not used. No need to enable log-queries for extended stats, and my script will flush those lines from the log as well, so the disk should not fill up.

Thoughts?

'lo = Enable Logging' currently ENABLEs both queries and replies, and 'lx Disable Logging' DISABLEs them.

I have added 'sgui all' which will now enable tracking of the Ad Blocking ('log-local-actions: yes') but if logging is DISABLED then it is obviously irrelevant until logging is ENABLED.

 

[juched]

'lo = Enable Logging' currently ENABLEs both queries and replies, and 'lx Disable Logging' DISABLEs them.

I have added 'sgui all' which will now enable tracking of the Ad Blocking ('log-local-actions: yes') but if logging is DISABLED then it is obviously irrelevant until logging is ENABLED.

Log local actions should always work, even if log replies or log queries is disabled. So what do you mean that logging needs to be enabled? What does logging enabled or disabled mean to you? Verbosity?

 

[Martineau]

Log local actions should always work, even if log replies or log queries is disabled. So what do you mean that logging needs to be enabled?

What does logging enabled or disabled mean to you? Verbosity?

Yes

 

[glehel]

how to add another rpz list? can I use multiple lists?

 

[Martineau]

how to add another rpz list? can I use multiple lists?

Add them to

'/opt/share/unbound/configs/rpzsites'