Unbound unbound_manager (Manager/Installer utility for unbound - Recursive DNS Server)

[juched]

@juched Did you see @ika's reported issue here?

I updated to v384.16 Beta this morning.....

Spoiler: Fresh unbound install 'z';amtm;i,7 then Ad Block 'i 3' and all h*ll breaks loose....

Restarting dnsmasq.....
Done.
Option Auto Reply 'y' Installing Ads and Tracker Blocking.....
 adblock/gen_adblock.sh downloaded successfully
 adblock/permlist downloaded successfully
Custom '/opt/share/unbound/configs/blocksites' already exists - 'adblock/blocksites' download skipped
Custom '/opt/share/unbound/configs/allowsites' already exists - 'adblock/allowsites' download skipped
Custom '/opt/share/unbound/configs/blockhost' already exists - 'adblock/blockhost' download skipped
Custom '/opt/share/unbound/configs/allowhost' already exists - 'adblock/allowhost' download skipped
Adding Ad and Tracker 'include: /opt/var/lib/unbound/adblock/adservers'
Creating Daily cron job for Ad and Tracker update
Executing '/opt/var/lib/unbound/adblock/gen_adblock.sh'.....
                          
 _____   _ _   _         _
|  _  |_| | |_| |___ ___| |_
|     | . | . | | . |  _| '_|
|__|__|___|___|_|___|___|_,_|
(gen_adblock.sh): 17451 @juched - v1.0.6 - Thanks to @SomeWhereOverTheRainBow

Removing possible temporary files..
Downloading list(s) from block site(s) configured...
Attempting to Download 1 of 1 from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts.
######################################################################## 100.0%
Downloading list(s) from allow site(s) configured...
Adding user requested hosts to list...
Removing user requested hosts from list...
Removing required hosts from list...
Removing unnecessary formatting from the domain list...
Generating Unbound adservers file...
(gen_adblock.sh): 17451 Number of adblocked hosts: 52826
Generating Unbound unload/load lists...
[1584288202] unbound-control[17678:0] error: connect: Connection refused for 127.0.0.1 port 953
Loading/Unload Unbound local-zones to take effect...
(gen_adblock.sh): 17451 Warning unbound NOT running!
Removing temporary files...
Adblock update complete!

Auto install unbound Customisation complete 1 minutes and 47 seconds elapsed - Please wait for up to 10 seconds for status.....

 ***ERROR unbound went AWOL after 1 seconds.....

 ***ERROR Unsuccessful installation of unbound detected

Mar 15 15:59:21 RT-AC68U (dnsmasq.postconf): Updating /etc/dnsmasq.conf for unbound.....
Mar 15 15:59:56 RT-AC68U (dnsmasq.postconf): Updating /etc/dnsmasq.conf for unbound.....
Mar 15 16:00:55 RT-AC68U (unbound_manager.sh): 16260 Starting Script Execution (menu)
Mar 15 16:02:58 RT-AC68U (dnsmasq.postconf): Updating /etc/dnsmasq.conf for unbound.....
Mar 15 16:03:25 RT-AC68U (gen_adblock.sh): 17451 Warning unbound NOT running!
[1584288206] unbound[17723:0] notice: Start of unbound 1.9.6.
Mar 15 16:03:26 unbound[17723:0] debug: increased limit(open files) from 1024 to 1684
Mar 15 16:03:26 unbound[17723:0] debug: creating udp4 socket 127.0.0.1 53535
Mar 15 16:03:26 unbound[17723:0] debug: creating tcp4 socket 127.0.0.1 53535
Mar 15 16:03:26 unbound[17723:0] error: Setting TCP Fast Open as server failed: Protocol not available
Mar 15 16:03:26 unbound[17723:0] debug: creating tcp4 socket 127.0.0.1 953
Mar 15 16:03:26 unbound[17723:0] error: Setting TCP Fast Open as server failed: Protocol not available
Mar 15 16:03:26 unbound[17723:0] debug: setup SSL certificates
Mar 15 16:03:27 unbound[17723:0] debug: chdir to /opt/var/lib/unbound
Mar 15 16:03:27 unbound[17723:0] debug: chroot to /opt/var/lib/unbound
Mar 15 16:03:27 unbound[17723:0] debug: drop user privileges, run as nobody
Mar 15 16:03:27 unbound[17723:0] debug: switching log to /opt/var/lib/unbound/unbound.log

Anyone else experience similar?

The error about

[1584288202] unbound-control[17678:0] error: connect: Connection refused for 127.0.0.1 port 953

is becasue I am trying to run unbound-control and during install unbound isn't running. I have pushed a v1.0.6 hotfix which stops this error message but that should have no impact on the install, it is just an extra error.


Why is our unbound AWOL... that shouldn't be impacted by gen_adblock error message.

 

[juched]

Reporting an issue: Now I can't enable addblock anymore with the script. The whole script is full of the "SSL handshake failed" errors, and if I select adblock to be enabled, the scripts always fails with an"ERROR unbound-control - failed?" message at the end (after answering the redownload/keep config question).

I cannot quite read the screenshot. How many hosts were added? 52K or 92K?

I know there is a limit to the size of the adservers file before unbound fails to load the file. Separate files at a size may fix this, but with the default lists it isn't close to the limit. What is your allowsites and blocksites file showing?
Did you customize them?

 

[Martineau]

The error about

[1584288202] unbound-control[17678:0] error: connect: Connection refused for 127.0.0.1 port 953

is becasue I am trying to run unbound-control and during install unbound isn't running. I have pushed a v1.0.6 hotfix which stops this error message but that should have no impact on the install, it is just an extra error.


Why is our unbound AWOL... that shouldn't be impacted by gen_adblock error message.

I suspect we need to patch both of our scripts....so that during the initial install phase of unbound (prior to unbound actually running), ensure that Ad Block reverts to the original v1.0.5 restart behaviour, but allow the cron invocation (or the 'adblock' command) to use the 'local zone' remove/load process.

EDIT: 'unbound_manager.sh' v2.18 contains fix.

 

[juched]

I suspect we need to patch both of our scripts....so that during the initial install phase of unbound (prior to unbound actually running), ensure that Ad Block reverts to the original v1.0.5 restart behaviour, but allow the cron invocation (or the 'adblock' command) to use the 'local zone' remove/load process.

e.g. proposed 'gen_adblock.sh' v1.0.6 patch

if [ "$1" != "forcerestart" ];then                # v1.0.6 Martineau Hotfix
    echo "Generating Unbound unload/load lists..."
    unbound-control list_local_zones | grep "always_nxdomain" | grep -v "use-application-dns.net" | awk '{print ""$1""}' > $unloadlist
    awk '{print ""$1" always_nxdomain"}' $finalist > $loadlist
    echo "Loading/Unload Unbound local-zones to take effect..."
    if [ -n "$(pidof unbound)" ];then
      [ -f $unloadlist ] && unbound-control local_zones_remove < $unloadlist
      [ -f $loadlist ] && unbound-control local_zones < $loadlist
    else
      SayT "Warning unbound NOT running"
    fi
else
    /jffs/addons/unbound/unbound_manager.sh restart
fi

Could you not simply run "restart" after the gen_adblock.sh completes during install? Then there is no special changes to coordinate?

 

[ika]

I cannot quite read the screenshot. How many hosts were added? 52K or 92K?

I know there is a limit to the size of the adservers file before unbound fails to load the file. Separate files at a size may fix this, but with the default lists it isn't close to the limit. What is your allowsites and blocksites file showing?
Did you customize them?

Hello!

No customization at all. There are 52826 adblocked domains, 0 hosts and 19 whitelist entries. But as I mentioned here and here, stopping the script and restarting unbound makes installation possible.

 

[rgnldo]

Just looking at the metrics and properties of the shell script encoder. Nothing open source and more privateer. I want to see solve problems.

 

[Martineau]

I've uploaded v2.18

Version=2.18
Github md5=4ae7db8bc247621e2fe05424771ddc84
​

Use of the 'i = Update unbound Installation' **n/a**

FIX: Reinstate deleted Restart_unbound() call/check during initial unbound installation.
ADD: Create 'gen_adblck.sh' filter clause when creating the scribe syslog-ng configuration
CHANGE: 'i' command will now remember which options are currently implemented and will auto reply to the existing options.

e.g. 'i 4' may be used, then if you exit unbound_manger, a subsequent 'i 3' would ensure both options o3 & o4 are auto-reapplied.​

CHANGE: '?' command will now additionally display a description of the numeric values

e  = Exit Script

A:Option ==> ?

           Version=2.18
           Github      md5=4ae7db8bc247621e2fe05424771ddc84

<snip>

           Options: Auto Reply='y' for User Selectable Options ('3 4') Ad Block,Performance Tweaks

ADD: 'adblock [uninstall]' command (rather than use 'i 3') as a convenience which only retrieves the relevant Ad Block files from the Github repository during the install/update.

 

[Martineau]

Hello!

No customization at all. There are 52826 adblocked domains, 0 hosts and 19 whitelist entries. But as I mentioned here and here, stopping the script and restarting unbound makes installation possible.

Hopefully v2.18 will fix your issue.

 

[L&LD]

As promised, I did a more controlled test when updating from v2.17 to v2.18.

From amtm, using '7', and immediately afterward issuing an 's', I see a 77% hit rate for the cache.

Using 'u' and allowing it to finish and then issuing an 's' again, I see the same statistics. Perfect.

Using 'i' and only replying 'y' to CPU and Memory Optimizations, and 'n' to everything else (i.e. 'Enter') and then a final 'Enter' when asked to keep current unbound configuration to fully implement the new features in v2.18.

Now, when I run 's', the stats are at 'zero' for all. Am I misunderstanding this should be still at (or close) to the 77% above?

To be sure, I am not that worried about the cache resetting. Just trying to understand how it is 'supposed' to work.

Thank you @Martineau for your hard work here and I hope the above helps a little.

 

[Kingp1n]

As promised, I did a more controlled test when updating from v2.17 to v2.18.

From amtm, using '7', and immediately afterward issuing an 's', I see a 77% hit rate for the cache.

Using 'u' and allowing it to finish and then issuing an 's' again, I see the same statistics. Perfect.

Using 'i' and only replying 'y' to CPU and Memory Optimizations, and 'n' to everything else (i.e. 'Enter') and then a final 'Enter' when asked to keep current unbound configuration to fully implement the new features in v2.18.

Now, when I run 's', the stats are at 'zero' for all. Am I misunderstanding this should be still at (or close) to the 77% above?

To be sure, I am not that worried about the cache resetting. Just trying to understand how it is 'supposed' to work.

Thank you @Martineau for your hard work here and I hope the above helps a little.

L&LD are you still using the mods you posted previously?

 

[L&LD]

@Kingp1n, not lately.

https://www.snbforums.com/threads/r...ecursive-dns-server.61669/page-51#post-560213

But I do feel the network a little slower without them. When the fast pace of updates decreases for this script, I will be testing their merits again.

 

[Martineau]

Now, when I run 's', the stats are at 'zero' for all.

Precisely which 'stats' metrics are you referring to?

I suggest you spend some time studying my metrics I posted here.

In summary, the left-most column will always be reset to zero, whereas the right-most column should remain quite close to the pre-restart values (although it appears some internal housekeeping occurs, to presumably flush stale cache entries).

 

[SomeWhereOverTheRainBow]

What is the best way to view the status of unbound as it is not showing up in the scribe log is there away to enable it?

 

[Deleted member 62525]

With the new release update the process hang for me. This is the first time. I had abort the process. When I restarted unbound_manager I found that
1. My unbound.conf was overridden with the default
2. AdBlock was disabled

Then I proceeded with "i" to install again and this is where things got complicated. The new process did not ask me if I want to install AdBlock or any other questions. I kept searching when and how to display the advanced menu but each time I tried it failed to display.Then I remembered to invoke 2 4 to do it. So finally I was able to restore my config but only after many trials.

I think the whole process and menu with so many letters and selection is getting cumbersome. Not like for example Skynet which is simple and easy. I think we trying too hard to satisfy 100 % cases. I feel it would be best if we do 80% most common and necessary cases and leave the rest outer cases be left for advanced users using manual config. I appreciate all the work that @Martineau has done. Unbound is a great addition to Merlin firmware but maybe we are doing too many small updates and the menu is getting more and more complex. With complexity comes more testing and issues.

 

[SomeWhereOverTheRainBow]

With the new release update the process hang for me. This is the first time. I had abort the process. When I restarted unbound_manager I found that
1. My unbound.conf was overridden with the default
2. AdBlock was disabled

Then I proceeded with "i" to install again and this is where things got complicated. The new process did not ask me if I want to install AdBlock or any other questions. I kept searching when and how to display the advanced menu but each time I tried it failed to display.Then I remembered to invoke 2 4 to do it. So finally I was able to restore my config but only after many trials.

I think the whole process and menu with so many letters and selection is getting cumbersome. Not like for example Skynet which is simple and easy. I think we trying too hard to satisfy 100 % cases. I feel it would be best if we do 80% most common and necessary cases and leave the rest outer cases be left for advanced users using manual config. I appreciate all the work that @Martineau has done. Unbound is a great addition to Merlin firmware but maybe we are doing too many small updates and the menu is getting more and more complex. With complexity comes more testing and issues.

No issues here, update went smooth. For those advanced users needing to be free of menu options, it is recommended to try @rgnldo install method here at this link https://www.snbforums.com/threads/unbound-authoritative-recursive-caching-dns-server.58967/

This is for users who need a complete manual config.

**edit** it appears the instructions have been removed.

 

[L&LD]

Precisely which 'stats' metrics are you referring to?

I suggest you spend some time studying my metrics I posted here.

In summary, the left-most column will always be reset to zero, whereas the right-most column should remain quite close to the pre-restart values (although it appears some internal housekeeping occurs, to presumably flush stale cache entries).

Everything is zero. But like I said, I'm not too concerned (and I did compare to that post you linked to).

Fyi, this has now happened on RT-AX88U's, RT-AC68U's, RT-AC66U_B1's, RT-AC86U's and RT-AC3100's, each variously running either 384.15_0 or 384.16 Beta1.

Just data points for you. Again, many thanks for what you offer.

 

[Kingp1n]

Everything is zero. But like I said, I'm not too concerned (and I did compare to that post you linked to).

Fyi, this has now happened on RT-AX88U's, RT-AC68U's, RT-AC66U_B1's, RT-AC86U's and RT-AC3100's, each variously running either 384.15_0 or 384.16 Beta1.

Just data points for you. Again, many thanks for what you offer.

I'm seeing alot of great updates compared to when I 1st tried unbound. What's the difference in using the new adblocker vs Diversion. Is it supposed to be less stressful in the CPU?

 

[SomeWhereOverTheRainBow]

I'm seeing alot of great updates compared to when I 1st tried unbound. What's the difference in using the new adblocker vs Diversion. Is it supposed to be less stressful in the CPU?

The biggest difference is the response, unbound is doing nxdomain for ads, where diversion has a standard blocking mode and a lite blocking mode. Where the major difference is unbound nxdomain blocking versus diversion standard pixelservtls blocking. If you are only going to block a standard to small amount of domains, then unbound blocking is comparable to diversion lite.

 

[ika]

Hopefully v2.18 will fix your issue.

The error did went away, but I still can't install the adblock option with the script. There was a new message stating that I have Diversion installed, but I never had (I don't see anything in dnsmasq.conf either).
The script ran without user interaction (I had no option to do advanced setup with the ability to choose components) and even "2 4" or other kind of attempts did not change that. After it was done I had no adblock. Using "adblock install" installed the adblock components, but now it says that I have Diversion isntalled, which I do not have.

edit: Tried "adblock uninstall" which does not uninstall adblock.

 

[Kingp1n]

The biggest difference is the response, unbound is doing nxdomain for ads, where diversion has a standard blocking mode and a light blocking mode. Where the major difference is unbound nxdomain blocking versus diversion standard pixelservtls blocking. If you are only going to block a standard to small amount of domains, then unbound blocking is comparable to diversion lite.

Thanks for the info, so if I decide to use unbound nxdomain, is the recommended way to go is to turn off diversion and pixelserv tls correct?