Unbound unbound_manager (Manager/Installer utility for unbound - Recursive DNS Server)

[juched]

Playing around:

 

[martinr]

Playing around:

View attachment 21665

Any stat to give some sort of indication of the size of the cache and/or its rate of growth?

 

[Martineau]

Playing around:

View attachment 21665

Not sure if the '*.exceeded' variables etc. would need to be monitored in the GUI?

e.g. if the value is non-Zero this would imply that unbound is actually dropping queries

unbound.thread.requestlist.exceeded=0

 

[Billy Chaney]

Playing around:

View attachment 21665

How do I get that GUI tab?

 

[martinr]

How do I get that GUI tab?

How do we all get it?

 

[BlueiPhone]

How do we all get it?

+1

 

[juched]

How do we all get it?

Working on it, just so much spare time

 

[martinr]

Working on it, just so much spare time

Seriously, like all the coders/developers, it’s far better you take as long as necessary, and only release anything when and if you are happy with it. One wonders if Adamm, thelonelycoder, RMerlin, Jack Yaz, Martineau ..... had any idea, when they started out, of just how much of a drain on their free their projects would become as they evolved, answering forum queries and maintaining their code and scripts.

 

[juched]

This is an update to the previous post about optimizing unbound.conf. Please read at the link below for further information.

https://www.snbforums.com/threads/r...recursive-dns-server.61669/page-3#post-548469

@Martineau are you up for this one?

These settings are specifically with my RT-AX88U and my 1Gbps up/down symmetrical Fibre connection to my ISP.

Proceed at your own risk!

While these settings give me an exceptionally fast and responsive network experience, test it thoroughly before implementing it into your normal workflows.

# no threads and no memory slabs for threads
num-threads: 8                    # L&LDv1.09 (Orig 1) RT-AX88U For RT-AC86U use (2)
msg-cache-slabs: 16                # L&LDv1.09 (Orig 1) RT-AX88U For RT-AC86U use (2)
rrset-cache-slabs: 16                # L&LDv1.09 (Orig 1) RT-AX88U For RT-AC86U use (2)
infra-cache-slabs: 16                # L&LDv1.09 (Orig 1) RT-AX88U For RT-AC86U use (2)
key-cache-slabs: 16                # L&LDv1.09 (Orig 1) RT-AX88U For RT-AC86U use (2)

# tiny memory cache
key-cache-size: 32m                # L&LDv1.09 (Orig 8m) RT-AX88U For RT-AC86U use (8m)
msg-cache-size: 32m                # L&LDv1.09 (Orig 8m) RT-AX88U For RT-AC86U use (8m)
rrset-cache-size: 64m                # L&LDv1.09 (Orig 16m) RT-AX88U For RT-AC86U use (16m)
cache-max-ttl: 21600             
cache-min-ttl: 0                # L&LDv1.09 (Orig 5) RT-AX88U or RT-AC86U
prefetch: yes
prefetch-key: yes
serve-expired: yes
serve-expired-ttl: 3600
incoming-num-tcp: 1024                # L&LDv1.09 (Orig 600) RT-AX88U For RT-AC86U use (1024)
outgoing-num-tcp: 128                # L&LDv1.09 (Orig 100) RT-AX88U For RT-AC86U use (512)
ip-ratelimit: 0                    # L&LDv1.09 (Orig 100) RT-AX88U For RT-AC86U use (0)
edns-buffer-size: 4096                # L&LDv1.09 RT-AX88U (Orig (1472) v1.01 as per @dave14305 minimal config)

I've upgraded to v2.09 and even with a 'zero' % hit ratio initially, this still is faster than the 'safe' defaults for me.

The 1GB RAM on the RT-AX88U has disappeared, there is between 16MB and 36MB depending on what I'm using the network for at the time. Asus, give us more RAM!

The 2GB swap file is at about 29.57 / 2048 MB currently (have seen it to almost 490MB).

Note the limitations that @skeal points out:



For the RT-AC86U? Someone else (who has the router in use) needs to test the limits and report back to us.

For anything below the two HND models (RT-AX88U and the RT-AC86U)? Who wants to be a pioneer?

Opening up a shortcut folder (Open all) of 25 or 40 links in Edge Chromium has never happened so quickly before.

Metrics on these kinds of improvements? Like driving a fine car and trying to explain it to someone who only drives a horse and buggy.

You just have to get behind the wheel to find out.

I spent some time thinking about this, as I wanted to optimize my RT-AX88U, and after some digging, not sure bigger is better.

First of all, from the URL (https://nlnetlabs.nl/documentation/unbound/howto-optimise/) if you scroll down to the bottom, they indicate if you are on a forking solution (which we are, we get 1 unbound instance for each num-threads) then it recommends setting the number of threads to the number of CPUs, and it recommends using 1 for all the slab settings.

However, it also says "This means that much more memory is used, as every core uses its own cache. Because every core has its own cache, if one gets cache poisoned, the others are not affected.". This means that each instance has a separate cache, meaning all your DNS requests are distributed and meaning you will have to recursively resolve more URLs as now it needs to do it once for each instance of unbound running. This means slower lookups as each cache needs to build up (as others have indicated and seen on this thread).

The reason to have more instances is to handle higher levels of load then 1 process (thread) can handle. In my case looking at total.requestlist.avg I see:

total.requestlist.avg = 0.82111

From the unbound site I see:

The average number of requests in the internal recursive pro-cessing request list on insert of a new incoming recursive pro-cessing query.

This means my 1 process is enough to keep the requestlist below 1, which means 1 is more than enough for me.

So, the defaults of 1 thread and 2 slabs seems ok for me. Perhaps the slabs can be set to 1, or 2 to be a multiple of two, there seem to be some debates online, but if you listen to the official documentation for forked operations it recommends 1 so perhaps we should use as the default:

    msg-cache-slabs: 1
    rrset-cache-slabs: 1
    infra-cache-slabs: 1
    key-cache-slabs: 1

About the memory size for the rst and mem cache, the default of 8mb and 16 mb see to be more than enough for my family. Looking at the extended stats I see:

mem.cache.rrset=1934308
mem.cache.message=763110

meaning I am using ~763K of my 8MB message cache, and ~1.9MB of my 16MB rrset cache.


So, my recommendation is to not try to run more than 1 thread (process) to optimize your lookups, and use the stats to see if/when you should expand to 2. Same for memory size.

Bigger isn't always better


--- edit ----

I may have jumped the gun on requestlist.avg. That is the list of requests for recursive lookups, not the load on incoming queries. Hmm... need to think some more.

 

[Jack Yaz]

Seriously, like all the coders/developers, it’s far better you take as long as necessary, and only release anything when and if you are happy with it. One wonders if Adamm, thelonelycoder, RMerlin, Jack Yaz, Martineau ..... had any idea, when they started out, of just how much of a drain on their free their projects would become as they evolved, answering forum queries and maintaining their code and scripts.

Unbound stats with charts was on my to-do, but it looks like I can cross that one off

 

[Maverickcdn]

Seriously, like all the coders/developers.....

Dont forget @john9527 Maintaining the older model firmware..... thank you to everyone who contributes here.

 

[rgnldo]

Bigger isn't always better

I agree

1 thread and 2 slabs

Try to omit this information in the configuration file. Let the unbound do it on its own.

 

[bluzfanmr1]

I spent some time thinking about this, as I wanted to optimize my RT-AX88U, and after some digging, not sure bigger is better.

First of all, from the URL (https://nlnetlabs.nl/documentation/unbound/howto-optimise/) if you scroll down to the bottom, they indicate if you are on a forking solution (which we are, we get 1 unbound instance for each num-threads) then it recommends setting the number of threads to the number of CPUs, and it recommends using 1 for all the slab settings.

However, it also says "This means that much more memory is used, as every core uses its own cache. Because every core has its own cache, if one gets cache poisoned, the others are not affected.". This means that each instance has a separate cache, meaning all your DNS requests are distributed and meaning you will have to recursively resolve more URLs as now it needs to do it once for each instance of unbound running. This means slower lookups as each cache needs to build up (as others have indicated and seen on this thread).

The reason to have more instances is to handle higher levels of load then 1 process (thread) can handle. In my case looking at total.requestlist.avg I see:

total.requestlist.avg = 0.82111

From the unbound site I see:


This means my 1 process is enough to keep the requestlist below 1, which means 1 is more than enough for me.

So, the defaults of 1 thread and 2 slabs seems ok for me. Perhaps the slabs can be set to 1, or 2 to be a multiple of two, there seem to be some debates online, but if you listen to the official documentation for forked operations it recommends 1 so perhaps we should use as the default:

    msg-cache-slabs: 1
    rrset-cache-slabs: 1
    infra-cache-slabs: 1
    key-cache-slabs: 1

About the memory size for the rst and mem cache, the default of 8mb and 16 mb see to be more than enough for my family. Looking at the extended stats I see:

mem.cache.rrset=1934308
mem.cache.message=763110

meaning I am using ~763K of my 8MB message cache, and ~1.9MB of my 16MB rrset cache.


So, my recommendation is to not try to run more than 1 thread (process) to optimize your lookups, and use the stats to see if/when you should expand to 2. Same for memory size.

Bigger isn't always better


--- edit ----

I may have jumped the gun on requestlist.avg. That is the list of requests for recursive lookups, not the load on incoming queries. Hmm... need to think some more.

I too have been reading and tweaking my settings to try and get the best results. I am using 2 threads since my router is dual core.

One thing I added this morning was a setting from my Pi-hole servers:

use-caps-for-id: no

Just about everything I've run across uses this setting. I immediately noticed that everything was loading noticeably faster, though I have no hard numbers to prove it.

I did vaguely recall that these numbers below were lower than what I noticed before:

thread0.recursion.time.avg=0.314983
thread0.recursion.time.median=0.175796
thread1.recursion.time.avg=0.231887
thread1.recursion.time.median=0.175082
total.recursion.time.avg=0.273310
total.recursion.time.median=0.175439

I don't have anything to compare these numbers to but the documentation from NLnet Labs seem to me to be a good measure overall, though I'm sure someone will correct me if I'm way off here:

https://nlnetlabs.nl/documentation/unbound/unbound-control/

threadX.recursion.time.avg
Average time it took to answer queries that needed recursive
processing. Note that queries that were answered from the cache
are not in this average.

threadX.recursion.time.median
The median of the time it took to answer queries that needed
recursive processing. The median means that 50% of the user
queries were answered in less than this time. Because of big
outliers (usually queries to non responsive servers), the aver-
age can be bigger than the median. This median has been calcu-
lated by interpolation from a histogram.

Does anyone else want to compare your numbers to these? If this isn't a good comparison, what stats are best?

 

[juched]

I too have been reading and tweaking my settings to try and get the best results. I am using 2 threads since my router is dual core.

One thing I added this morning was a setting from my Pi-hole servers:

use-caps-for-id: no

Just about everything I've run across uses this setting. I immediately noticed that everything was loading noticeably faster, though I have no hard numbers to prove it.

I did vaguely recall that these numbers below were lower than what I noticed before:

thread0.recursion.time.avg=0.314983
thread0.recursion.time.median=0.175796
thread1.recursion.time.avg=0.231887
thread1.recursion.time.median=0.175082
total.recursion.time.avg=0.273310
total.recursion.time.median=0.175439

I don't have anything to compare these numbers to but the documentation from NLnet Labs seem to me to be a good measure overall, though I'm sure someone will correct me if I'm way off here:

https://nlnetlabs.nl/documentation/unbound/unbound-control/

threadX.recursion.time.avg
Average time it took to answer queries that needed recursive
processing. Note that queries that were answered from the cache
are not in this average.

threadX.recursion.time.median
The median of the time it took to answer queries that needed
recursive processing. The median means that 50% of the user
queries were answered in less than this time. Because of big
outliers (usually queries to non responsive servers), the aver-
age can be bigger than the median. This median has been calcu-
lated by interpolation from a histogram.

Does anyone else want to compare your numbers to these? If this isn't a good comparison, what stats are best?

I don't see this one in my conf file, and lookup it up from my running instance it seems to default to no. So seems we are already running it this way.

 

[Treadler]

After a fair bit of experimentation,
IMHO, Unbound works best with the installed config. defaults.

 

[jsbeddow]

After a fair bit of experimentation,
IMHO, Unbound works best with the installed config. defaults.

Would that be the absolute "plain vanilla" defaults, or do you mean with the optional "memory optimized" tweaks? Just curious, not a criticism. I have been running with the optional tweaks, but wonder if I really should.

 

[Treadler]

Would that be the absolute "plain vanilla" defaults, or do you mean with the optional "memory optimized" tweaks? Just curious, not a criticism. I have been running with the optional tweaks, but wonder if I really should.

I enable the cpu optimisation, (I don’t think it actually does anything for me, but it doesn’t hurt anything). Also enable the ad blocking & prevent Firefox DoH.

I don’t opt for the logging, or stubby integration. (Stubby integration kinda defeats the purpose if you ask me).

It ‘just works’.

 

[Treadler]

Created a pull request for the branch JackYaz is hosting.
https://github.com/jackyaz/Unbound-Asuswrt-Merlin/pull/5

Updated script to not be hard coded with fixed lists, but now you can edit a new file called "sites" which can handle 3 file types.

The file is of the format:
type url
type url
...

1. hosts -> These are host type files, and the processing is done to prepare them for merge.
2. domains -> These are lists of domains to block, they are simply appended.
3. whitelist-domains -> These are lists of domains to whitelist (remove) from the merged ad-block list.

I also added some protection for Asus Merlin and entware URL from accidently being blocked.

Ok, here’s where I approach (once again), showing my lack of knowledge.

I have gone to the relevant folder (sites) using winscp, & amended & saved the existing (Steven Black) list.
Then run ‘i’ to update Unbound, & my change gets wiped & put back to the default.
So, 2nd attempt, left the default list there, & added the additional Steven Black list I would like to use.
Run ‘i’, everything put back to default.
So, both my amendments & changes were ignored.

What glaringly obvious step am I missing to alter my ad blocking list?

 

[SomeWhereOverTheRainBow]

Ok, here’s where I approach (once again), showing my lack of knowledge.

I have gone to the relevant folder (sites) using winscp, & amended & saved the existing (Steven Black) list.
Then run ‘i’ to update Unbound, & my change gets wiped & put back to the default.
So, 2nd attempt, left the default list there, & added the additional Steven Black list I would like to use.
Run ‘i’, everything put back to default.
So, both my amendments & changes were ignored.

What glaringly obvious step am I missing to alter my ad blocking list?

it isn't you brother.

when i try and run the script manually i get

if [ "$2" != "" ]; then

i get

sh: 1: [: !=: unexpected operator

also

this is failing to pass the arguments of $1 and $2 for example "hosts" $1 on the read line and "url" for $2 on the read line.
if [ "$1" == "hosts" ]; then
elif [ "$1" == "domains" ]; then
elif [ "$1" == "whitelist-domains" ]; then

it is telling me unknown arguement

 

[SomeWhereOverTheRainBow]

[ "x$var2" == "x" ]

or [ -z "$2"] may bet better for
if [ "$2" != "" ]

and i don't understand why "$1" isn't getting passed off properly.


"$1" is representing host or domains or whitelist whatever you have defined it as.
"$1" "$2"
hosts url
domains url
whitelist-domains url

in the sites file