Unbound unbound_manager (Manager/Installer utility for unbound - Recursive DNS Server)

[juched]

I have not tried the huge list, and I do not recommend it. As I was creating my message I did further evaluation and saw how big it was and that is why I added the message to not change.

Am considering making the script read from a file with URL and format (host vs. domain list) and then the script just needs to go through the cfg txt file and do one line at a time.... I will see what I can do.

Created a pull request for the branch JackYaz is hosting.
https://github.com/jackyaz/Unbound-Asuswrt-Merlin/pull/5

Updated script to not be hard coded with fixed lists, but now you can edit a new file called "sites" which can handle 3 file types.

The file is of the format:
type url
type url
...

1. hosts -> These are host type files, and the processing is done to prepare them for merge.
2. domains -> These are lists of domains to block, they are simply appended.
3. whitelist-domains -> These are lists of domains to whitelist (remove) from the merged ad-block list.

I also added some protection for Asus Merlin and entware URL from accidently being blocked.

 

[SomeWhereOverTheRainBow]

Created a pull request for the branch JackYaz is hosting.
https://github.com/jackyaz/Unbound-Asuswrt-Merlin/pull/5

Updated script to not be hard coded with fixed lists, but now you can edit a new file called "sites" which can handle 3 file types.

The file is of the format:
type url
type url
...

1. hosts -> These are host type files, and the processing is done to prepare them for merge.
2. domains -> These are lists of domains to block, they are simply appended.
3. whitelist-domains -> These are lists of domains to whitelist (remove) from the merged ad-block list.

I also added some protection for Asus Merlin and entware URL from accidently being blocked.

I left you some feedback...

 

[juched]

I left you some feedback...

Thank you, incorporated it.

 

[SomeWhereOverTheRainBow]

yea that fgrep was killing me on my generated list , i noticed it was removing 10,000 domains and i only had a whitelist with 500 domains

 

[Martineau]

If I choose to enable Stubby it throws me an error which says "DNS Privacy is not enabled in the GUI" something like this. What do I need to enable? Firmware's DoT and fill the DoT servers in GUI?

Thanks for the feedback.

I've added (unreleased v2.13) a more descriptive message

  = Exit Script

A:Option ==> i 2

<snip>

Retrieving Custom unbound configuration
           unbound.conf downloaded successfully
Checking IPv6.....
Customising unbound IPv6 configuration.....
Customising unbound configuration Options:
unbound-checkconf: no errors in /opt/var/lib/unbound/unbound.conf
Option Auto Reply 'y' Integrating Stubby with unbound.....

           ERROR: DNS Privacy (DoT) not enabled in GUI. see http://192.168.1.1:80/Advanced_WAN_Content.asp WAN->DNS Privacy Protocol

Restarting dnsmasq.....
Done.

 Shutting down unbound...              done.
 Starting unbound...              done.

Auto install unbound Customisation complete 0 minutes and 9 seconds elapsed - Please wait for up to 10 seconds for status.....

 Installation of unbound completed

 

[rgnldo]

Updated script to not be hard coded with fixed lists, but now you can edit a new file called "sites" which can handle 3 file types.

The file is of the format:
type url
type url

Tested. In this way, it will be possible via script to add lists by the user.
I recommend not using lists of allowed domains online. The user must add manually.
You can maintain and update the adblock list for the unbound script. I have other blocking projects.

By the way, this link redirects to the Merlin Alpha repository. However, it is an ad tracker link:
https://1drv.ms/f/s!AuCcWdNeYuXMixNmNrWhl9Np1Zlg

 

[Martineau]

Created a pull request for the branch JackYaz is hosting.
https://github.com/jackyaz/Unbound-Asuswrt-Merlin/pull/5

Updated script to not be hard coded with fixed lists, but now you can edit a new file called "sites" which can handle 3 file types.

The file is of the format:
type url
type url
...

1. hosts -> These are host type files, and the processing is done to prepare them for merge.
2. domains -> These are lists of domains to block, they are simply appended.
3. whitelist-domains -> These are lists of domains to whitelist (remove) from the merged ad-block list.

I also added some protection for Asus Merlin and entware URL from accidently being blocked.

FYI,

To pre-empt the PULL request being accepted, I've pushed a hotfix to Github

if [ -n "$(grep -F "adblock/sites" ${CONFIG_DIR}adblock/gen_adblock.sh)" ];then  # v2.13 @juched rewrote 'gen_adblock.sh' to reference this file
   download_file ${CONFIG_DIR} adblock/sites jackyaz   
fi

so unbound_manager will now download the new 'adblock/sites' file if 'gen_adblock.sh' requires it.

Hope you have added version control to your new script?

Also, a couple of code suggestions....

numberOfAdsBlocked=$(cat $outlist | wc -l | sed 's/^[ \t]*//')

change to

numberOfAdsBlocked=$(wc -l < $outlist)

and

cat $finalist | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" always_nxdomain"}' > $adlist

change to

awk '/^0.0.0.0/ {print "local-zone: \""$2"\" always_nxdomain"}' $finalist > $adlist

Since the script is running via cru (cron), you should use logger to ensure the ERROR message appears in Syslog - just in case.

echo "Missing $sites file"

change to

logger -st "($(basename $0))" "Missing $sites file"

 

[rgnldo]

As a suggestion, instead of creating another file, there is this option, see if it is interesting.

Spoiler: gen_unbound.sh

#!/bin/bash
destinationIP="0.0.0.0"
tempoutlist="/opt/var/lib/unbound/adblock/adlist.tmp"
outlist='/opt/var/lib/unbound/adblock/tmp.host'
finalist='/opt/var/lib/unbound/adblock/tmp.finalhost'
permlist='/opt/var/lib/unbound/adblock/permlist'
adlist='/opt/var/lib/unbound/adblock/adservers'

echo "Removing possible temporary files..."
[ -f $tempoutlist ] && rm -f $tempoutlist
[ -f $outlist ] && rm -f $outlist
[ -f $finalist ] && rm -f $finalist

hosts='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'

domains='https://raw.githubusercontent.com/dnswarden/blocklist/master/blacklist-full.txt
         https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list'

echo "Process sites list ..."
echo "Get hosts list ..."
curl --progress-bar $hosts | grep -v "#" | grep -v "::1" | grep -v "0.0.0.0 0.0.0.0" | sed '/^$/d' | sed 's/\ /\\ /g' | awk '{print $2}' | grep -v '^\\' | grep -v '\\$'| sort >> $tempoutlist
echo "Get domains list ..."
curl --progress-bar $domains >> $tempoutlist

echo "Combining User Custom block host..."
cat /opt/var/lib/unbound/adblock/blockhost >> $tempoutlist

 

[juched]

FYI,

To pre-empt the PULL request being accepted, I've pushed a hotfix to Github

if [ -n "$(grep -F "adblock/sites" ${CONFIG_DIR}adblock/gen_adblock.sh)" ];then  # v2.13 @juched rewrote 'gen_adblock.sh' to reference this file
   download_file ${CONFIG_DIR} adblock/sites jackyaz 
fi

so unbound_manager will now download the new 'adblock/sites' file if 'gen_adblock.sh' requires it.

Hope you have added version control to your new script?

Also, a couple of code suggestions....

numberOfAdsBlocked=$(cat $outlist | wc -l | sed 's/^[ \t]*//')

change to

numberOfAdsBlocked=$(wc -l < $outlist)

and

cat $finalist | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" always_nxdomain"}' > $adlist

change to

awk '/^0.0.0.0/ {print "local-zone: \""$2"\" always_nxdomain"}' $finalist > $adlist

Since the script is running via cru (cron), you should use logger to ensure the ERROR message appears in Syslog - just in case.

echo "Missing $sites file"

change to

logger -st "($(basename $0))" "Missing $sites file"

Reply, thanks for the feedback. Not familiar with awk so all the suggestions from everyone has been appreciated. I have submitted the changes.

What do you mean by version control? a header? or just checked into github? The latter is done.

 

[juched]

Tested. In this way, it will be possible via script to add lists by the user.
I recommend not using lists of allowed domains online. The user must add manually.
You can maintain and update the adblock list for the unbound script. I have other blocking projects.

By the way, this link redirects to the Merlin Alpha repository. However, it is an ad tracker link:
https://1drv.ms/f/s!AuCcWdNeYuXMixNmNrWhl9Np1Zlg

Ya, good point about the whitelist... what should the defaults be for a regular user? Seems from diversion that Small is good for most, and it uses Steven's List, maybe it should just be that for a default.

--edit--
Modified to just Steven's List for now.

 

[rgnldo]

what should the defaults be for a regular user? Seems from diversion that Small is good for most

Small list of allowed domains is enough. The user is adding as needed.

 

[rgnldo]

good for most, and it uses Steven's List, maybe it should just be that for a default.

Spoiler: gen_unbound

#!/bin/bash
destinationIP="0.0.0.0"
tempoutlist="/opt/var/lib/unbound/adblock/adlist.tmp"
outlist='/opt/var/lib/unbound/adblock/tmp.host'
finalist='/opt/var/lib/unbound/adblock/tmp.finalhost'
permlist='/opt/var/lib/unbound/adblock/permlist'
adlist='/opt/var/lib/unbound/adblock/adservers'

echo "Removing possible temporary files..."
[ -f $tempoutlist ] && rm -f $tempoutlist
[ -f $outlist ] && rm -f $outlist
[ -f $finalist ] && rm -f $finalist

hosts='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'

domains='https://github.com/jackyaz/Unbound-Asuswrt-Merlin/blob/master/adblock/host
         https://raw.githubusercontent.com/dnswarden/blocklist/master/blacklist-full.txt
         https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list'

echo "Process sites list ..."
echo "Get hosts list ..."
curl --progress-bar $hosts | grep -v "#" | grep -v "::1" | grep -v "0.0.0.0 0.0.0.0" | sed '/^$/d' | sed 's/\ /\\ /g' | awk '{print $2}' | grep -v '^\\' | grep -v '\\$'| sort >> $tempoutlist
echo "Get domains list ..."
curl --progress-bar $domains >> $tempoutlist

echo "Combining User Custom block host..."
cat /opt/var/lib/unbound/adblock/blockhost >> $tempoutlist
numberOfAdsBlocked=$(cat $tempoutlist | wc -l | sed 's/^[ \t]*//')
echo "$numberOfAdsBlocked domains temporary compiled"

echo "Edit User Custon list of allowed domains..."
awk 'NR==FNR{a[$0];next} !($0 in a) {print $NF}' $permlist $tempoutlist > $outlist

echo "Removing duplicate formatting from the domain list..."
cat $outlist | sed -r -e 's/[[:space:]]+/\t/g' | sed -e 's/\t*#.*$//g' | sed -e 's/[^a-zA-Z0-9\.\_\t\-]//g' | sed -e 's/\t$//g' | sed -e '/^#/d' | sort -u | sed '/^$/d' | awk -v "IP=$destinationIP" '{sub(/\r$/,""); print IP" "$0}' > $finalist

echo "Generating Unbound adlist....."
awk '/^0.0.0.0/ {print "local-zone: \""$2"\" always_nxdomain"}' $finalist > $adlist
numberOfAdsBlocked=$(cat $adlist | wc -l | sed 's/^[ \t]*//')
echo "$numberOfAdsBlocked suspicious and blocked domains"

echo "Removing temporary files..."
[ -f $tempoutlist ] && rm -f $tempoutlist
[ -f $outlist ] && rm -f $outlist
[ -f $finalist ] && rm -f $finalist

Look there. I put together a little of each suggestion. Disregard what you think is unnecessary. You can be the maintainer of the script.

 

[rgnldo]

--edit--
Modified to just Steven's List for now.

I recommend that you use at least one wildcard domain list.
https://github.com/jackyaz/Unbound-Asuswrt-Merlin/blob/master/adblock/host

 

[crandy2]

Small list of allowed domains is enough. The user is adding as needed.

Thanks to all that are making this happen. Be aware that the Amazon shopping app for Android (big deal at our house, YMMV) will constantly stop without the current permlist file. Steven's list alone renders it useless. It took me days (and an earfull of complaints) to figure that out.

Sent from my SM-T597W using Tapatalk

 

[SomeWhereOverTheRainBow]

Thanks to all that are making this happen. Be aware that the Amazon shopping app for Android (big deal at our house, YMMV) will constantly stop without the current permlist file. Steven's list alone renders it useless. It took me days (and an earfull of complaints) to figure that out.

Sent from my SM-T597W using Tapatalk

that is because Amazon is one Giant Advertisement. Even the Amazon Alexa uses numerous blocked advertisement domains just to function.

 

[rgnldo]

will constantly stop without the current permlist file

Spoiler: permilist

asuswrt.lostrealm.ca
bin.entware.net
codeload.github.com
fwupdate.asuswrt-merlin.net
pkg.entware-backports.tk
pkg.entware.net
raw.githubusercontent.com
sourceforge.net
www.asuswrt-merlin.net
ipinfo.io
aax-eu.amazon-adsystem.com
aax-us-east.amazon-adsystem.com
ad2.netshelter.net
assets.omidoo.com
flashtalking.com
fls-na.amazon-adsystem.com
images-na.ssl-images-amazon.com
ir-na.amazon-adsystem.com
ir-uk.amazon-adsystem.com
pagead2.googlesyndication.com
servedby.flashtalking.com
tgdaily.com
tgdaily.net
vma.tgdaily.com
vma.tgdaily.net
wms-eu.amazon-adsystem.com
wms-na.amazon-adsystem.com
wms-na.assoc-amazon.com
ws-eu.amazon-adsystem.com
ws-na.amazon-adsystem.com
z-na.amazon-adsystem.com

 

[Martineau]

What do you mean by version control? a header?

Yes.

Whilst it would be apparent that your mods to the original script improves its functionality, besides allowing you to publicly take the credit in the header, it also means that if it needs tweaking in the future, you could quickly ask 'which version'? and then instruct/guide someone to retrieve the 'latest' patched version etc.

 

[Skeptical.me]

At least qname minimization makes their work a little bit harder. You always have the option of sharing that data with a VPN provider instead, but either way, you will have to "trust" somebody, because (I think) DoT support for root servers are not coming in the near future (perhaps dprive will bring some change into this field?).

I'm now using DNSCrypt again with Anon relays, not sure how reliable their claims of anonymity are, however.

 

[SomeWhereOverTheRainBow]

Yes.

Whilst it would be apparent that your mods to the original script improves its functionality, besides allowing you to publicly take the credit in the header, it also means that if it needs tweaking in the future, you could quickly ask 'which version'? and then instruct/guide someone to retrieve the 'latest' patched version etc.

could just incorporate md5 check that downloads new gen_adblock.sh upon changes during the update process, I know it is nice to curl | grep for a version as well, but it isn't necessary if the gen_adblock.sh gets considered to be apart of the package as a whole instead of a side script.


Other wise i concur with @Martineau

It should have a version.

 

[rgnldo]

See if it's feasible. Patch for the script:
on /jffs/scripts/post-mount

before:

logger -t "(dnsmasq.postconf)" "Updating $CONFIG for unbound..."                                      
if [ -n "$(pidof unbound)" ];then
    service restart_dnsmasq

add:

logger -t "(dnsmasq.postconf)" "Updating $CONFIG for unbound..."                                      
if [ -n "$(pidof unbound)" ];then
    service restart_dnsmasq
fi

There will be no need to restart dnsmasq with each Unbound restart action, on the trigger S61unbound.

Spoiler: S61unbound

#!/bin/sh
logger -t S61unbound "Starting Unbound DNS server $0"
# set environment PATH to system binaries
export PATH=/sbin:/bin:/usr/sbin:/usr/bin:$PATH
export TZ=$(cat /etc/TZ)
ENABLED=yes
PROCS=unbound
ARGS="-c /opt/var/lib/unbound/unbound.conf"
PREARGS="nohup"
DESC=$PROCS
PATH=/opt/sbin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

. /opt/etc/init.d/rc.func