uiDivStats uiDivStats - WebUI for Diversion statistics

[grifo]

Mine has just one ext2 partition.

 

[Jack Yaz]

I'm not sure exactly why but uiDivStats increases the CPU usage dramatically during high usage of dnsmasq, which means during heavy web surfing between multiple web pages but I guess this is expected but I'm not sure this is caused by uiDivstats directly.

This is with uiDivStats;

View attachment 24096

This is without;
View attachment 24097

maybe this is because of USB Stick write processes of uiDivStats I'm not sure but this is what I see.

On the other side; I can't feel any problems like web pages stuck or any other slowness.

Please provide screenshots/outputs of:

atop #install from entware opkg install atop
htop #install from entware opkg install htop
df -Th | grep mnt | grep dev

 

[SomeWhereOverTheRainBow]

I'm not sure exactly why but uiDivStats increases the CPU usage dramatically during high usage of dnsmasq, which means during heavy web surfing between multiple web pages but I guess this is expected but I'm not sure this is caused by uiDivstats directly.

This is with uiDivStats;

View attachment 24096

This is without;
View attachment 24097

maybe this is because of USB Stick write processes of uiDivStats I'm not sure but this is what I see.

On the other side; I can't feel any problems like web pages stuck or any other slowness.

I have security systems constantly chatting and i do not experience cpu issues. how do you have your diversion setup?

 

[pattiri]

Please provide screenshots/outputs of:

atop #install from entware opkg install atop
htop #install from entware opkg install htop
df -Th | grep mnt | grep dev

admin@FaTiii:/tmp/home/root# df -Th | grep mnt | grep dev
/dev/sda1            ext2           14.7G      1.3G     12.6G   9% /tmp/mnt/entware

https://imgur.com/a/Iy8jmHd

atop is too complex for me but as I check htop I saw taildns and pixelserv-tls at the top most of the time, and this is when I try to access many pages in a couple of seconds.

 

[SomeWhereOverTheRainBow]

admin@FaTiii:/tmp/home/root# df -Th | grep mnt | grep dev
/dev/sda1            ext2           14.7G      1.3G     12.6G   9% /tmp/mnt/entware

https://imgur.com/a/Iy8jmHd

atop is too complex for me but as I check htop I saw taildns and pixelserv-tls at the top most of the time, and this is when I try to access many pages in a couple of seconds.

how many domains does your diversion blockfile block?

 

[pattiri]

how do you have your diversion setup?

Diversion 4.1.12                 by thelonelycoder

 RT-AC88U (armv7l) FW-384.17 @ 172.24.5.1

 965,296  blocked domains by  2  hosts file(s)
 356,841 t  14,528 w  10 n ads since Jun 17 08:26

 /mnt/entware | Size 14.7G | Used 1.3G (9%)         

 d   Diversion Standard   enabled
 c   communication        DivUn stats backup FWun

 a   ad-blocking          to IP 172.24.5.254
 l   logging              dnsmasq.log 25.5M

 ep  pixelserv-tls        172.24.5.254 v2.3.1

 b   blocking list        Large Mon @ 1:00
 el  edit lists            210 w  22 b  0 wb  936 yt

 f   follow dnsmasq.log

 ac  update ads counter   Auto-update  au  off
 ds  dnsmasq settings     on
 sw  manage swap file     /mnt/entware 1.0G
 ct  change theme colors   green

 sh  shell history        on
 cj  show cron job(s)     
 sf  show file content   

 e   exit  u  update                less options  o

Dnsmasq settings:

 See the Dnsmasq man page what these options do:
 http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

 1. Disable Dnsmasq settings
 2. log-async         5
 3. cache-size        1500
 4. bogus-priv        on
 5. domain-needed     on
 6. log-queries=extra

 Enter selection [1-6 e=Exit]

Disabling Dnsmasq settings does not change anything. I'm using Sandisk Extreme 16GB USB3.0 in USB 2.0 port.

 

[SomeWhereOverTheRainBow]

Diversion 4.1.12                 by thelonelycoder

 RT-AC88U (armv7l) FW-384.17 @ 172.24.5.1

 965,296  blocked domains by  2  hosts file(s)
 356,841 t  14,528 w  10 n ads since Jun 17 08:26

 /mnt/entware | Size 14.7G | Used 1.3G (9%)   

 d   Diversion Standard   enabled
 c   communication        DivUn stats backup FWun

 a   ad-blocking          to IP 172.24.5.254
 l   logging              dnsmasq.log 25.5M

 ep  pixelserv-tls        172.24.5.254 v2.3.1

 b   blocking list        Large Mon @ 1:00
 el  edit lists            210 w  22 b  0 wb  936 yt

 f   follow dnsmasq.log

 ac  update ads counter   Auto-update  au  off
 ds  dnsmasq settings     on
 sw  manage swap file     /mnt/entware 1.0G
 ct  change theme colors   green

 sh  shell history        on
 cj  show cron job(s)
 sf  show file content

 e   exit  u  update                less options  o

Dnsmasq settings:

 See the Dnsmasq man page what these options do:
 http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

 1. Disable Dnsmasq settings
 2. log-async         5
 3. cache-size        1500
 4. bogus-priv        on
 5. domain-needed     on
 6. log-queries=extra

 Enter selection [1-6 e=Exit]

Disabling Dnsmasq settings does not change anything. I'm using Sandisk Extreme 16GB USB3.0 in USB 2.0 port.

turn off the custom DNSMasq options inside diversion, your issues should go away. those options are default in dnsmasq.

ds dnsmasq settings on

turn the ds feature off.

Also: You could dump your dnsmasq logfile. see if that helps.

the size of your log file...
dnsmasq.log 25.5M

 

[Jack Yaz]

admin@FaTiii:/tmp/home/root# df -Th | grep mnt | grep dev
/dev/sda1            ext2           14.7G      1.3G     12.6G   9% /tmp/mnt/entware

https://imgur.com/a/Iy8jmHd

atop is too complex for me but as I check htop I saw taildns and pixelserv-tls at the top most of the time, and this is when I try to access many pages in a couple of seconds.

atop screenshot is fine. htop shows that network map is using all of your CPU. try navigating to a static WebUI page (or logging out) and repeat the tests.

 

[pattiri]

atop screenshot is fine. htop shows that network map is using all of your CPU. try navigating to a static WebUI page (or logging out) and repeat the tests.

Logged out and opened many web pages in about 15-20 seconds, all loaded fine with no error and I got this;

Then I've uninstalled uiDivStats; flushed DNS of my PC and deleted databases of firefox and restarted.

this time (after uninstalling uiDivStats) I've tried to open 40 tabs as soon as possible and for your reference here my 9900K CPU usage during this;

I mean second test should be more intense for router but it didn't even sweat.

 

[SomeWhereOverTheRainBow]

Logged out and opened many web pages in about 15-20 seconds, all loaded fine with no error and I got this;
View attachment 24100

Then I've uninstalled uiDivStats; flushed DNS of my PC and deleted databases of firefox and restarted.

View attachment 24101

this time (after uninstalling uiDivStats) I've tried to open 40 tabs as soon as possible and for your reference here my 9900K CPU usage during this;

View attachment 24102

I mean second test should be more intense for router but it didn't even sweat.

I experienced this same issue with the DS setting in diversion enabled along time ago before UiDivstats was around. I disabled it and rebooted the router and the problem went away.

 

[pattiri]

I experienced this same issue with the DS setting in diversion enabled along time ago before UiDivstats was around. I disabled it and rebooted the router and the problem went away.

Could you please send your current DS settings? Even I disable settings I think I can change bogus-priv and domain-needed.

Let me change the settings as same of yours reboot and check

 

[SomeWhereOverTheRainBow]

Could you please send your current DS settings? Even I disable settings I think I can change bogus-priv and domain-needed.

Let me change the settings as same of yours reboot and check

Mine are complety disabled meaning I do not have that diversion "experimental" feature enabled.

 

[pattiri]

Mine are complety disabled meaning I do not have that diversion "experimental" feature enabled.

OK; disabled and rebooted the router, installed iuDivStats and waited for a while for a cool down.

And did the same test with 40 tabs and here

additional notes I would like to mention.

- When I did the test with 40 tabs when iuDivStats is not installed; the CPU Load average reached its max just after all tabs are loaded
- When I did the test with 40 tabs with iuDivStats is installed when all tabs are loaded CPU load average for 1 min was 0.78 after about 10-15 seconds it reached 1.45.

I guess this shows this is not because of iuDivStats it maybe because of database or something else being written into USB drive.

I've never seen good write speeds with my router, I've tested again. it changes between 2 MB/S and 45 MB/s but this is USB 2.0 and I'm not sure how it's 45 MB/S

Maybe it's time for me to reset router to factory defaults and start over I haven't done this for more than one year I guess

 

[Makaveli]

That's a point. What filesystem and number of partitions do people have on their USB drive?

admin@RT-AC86U:/tmp/home/root# df -Th | grep mnt | grep dev
/dev/sda3            ext4            6.3G      2.7G      3.2G  46% /tmp/mnt/entware
/dev/sda1            tfat            7.9G      1.3M      7.9G   0% /tmp/mnt/data

I've had no issue on this version aswell.

on a 512GB SSD attached via USB 3.0 port.

ASUSWRT-Merlin RT-AX88U 384.17_0 Sun Apr 26 02:29:04 UTC 2020
Makaveli@RT-AX88U-0B10:/tmp/home/root# df -Th | grep mnt | grep dev
/dev/sda1            ext4          436.6G      1.3G    412.0G   0% /tmp/mnt/Scripts

Cpu usage normal.

 

[doczenith1]

Here we go.
Uninstalled 2.1.0
Deleted dnsqueries.db
Installed 2.1.0

Diversion and scMerlin are the the only scripts I currently have installed on 384.17. It has been at least a year since a factory reset and looking at an old screen shot I took of Diversion after first installing the version is 4.0.6.

Now seeing the cpu spiking to 100% and alternating between cores. It's been going at it for about 20 minutes now.

/dev/sda1 ext2 7.2G 2.1G 4.8G 30% /tmp/mnt/Diversion

 

[dave14305]

Has anyone tested putting the database on a separate USB drive from the dnsmasq logfiles? Split the contention, so-to-speak.

 

[Jack Yaz]

Here we go.
Uninstalled 2.1.0
Deleted dnsqueries.db
Installed 2.1.0

Diversion and scMerlin are the the only scripts I currently have installed on 384.17. It has been at least a year since a factory reset and looking at an old screen shot I took of Diversion after first installing the version is 4.0.6.

Now seeing the cpu spiking to 100% and alternating between cores. It's been going at it for about 20 minutes now.

/dev/sda1 ext2 7.2G 2.1G 4.8G 30% /tmp/mnt/Diversion

View attachment 24109

View attachment 24110


View attachment 24111

htop is showing hardly any cpu usage - do you see the 100% there or on network map? honestly, im starting to think network map is a steaming pile of s**t that causes the router to work overtime to show anything

 

[doczenith1]

htop is showing hardly any cpu usage - do you see the 100% there or on network map? honestly, im starting to think network map is a steaming pile of s**t that causes the router to work overtime to show anything

Good catch. htop is not showing the spikes when I monitor it and it is showing around 1-4% usage. But...the last log entry from uiDivStats is:
Jun 17 11:05:23 uiDivStats: Starting first run of stat generation...
Should there be a log entry stating that the generation has completed? It's been running for over three hours now.

I am seeing values in the uiDivStats tab. On the DNS Queries graph the x-axis jives with the time that I installed 2.1.0. It's showing roughly 300 total DNS queries and ~50 blocked.

Just in case it might make a difference my stats config in Diversion is:

My USB stick does have an activity light and it seems way more active that it used to be. Pretty much going nonstop with a pause here and there.

 

[Jack Yaz]

Good catch. htop is not showing the spikes when I monitor it and it is showing around 1-4% usage. But...the last log entry from uiDivStats is:
Jun 17 11:05:23 uiDivStats: Starting first run of stat generation...
Should there be a log entry stating that the generation has completed? It's been running for over three hours now.

I am seeing values in the uiDivStats tab. On the DNS Queries graph the x-axis jives with the time that I installed 2.1.0. It's showing roughly 300 total DNS queries and ~50 blocked.

Just in case it might make a difference my stats config in Diversion is:
View attachment 24114

My USB stick does have an activity light and it seems way more active that it used to be. Pretty much going nonstop with a pause here and there.

Looks like the "complete" message only prints to the console at the moment, and wont appear in the log file. I assume that's where you're looking? I'll fix that for the next release

 

[doczenith1]

Yes, I am looking in the system log for the "complete" message.

My initial atop didn't show this but I'm seeing it now.

It's hard to see but it's showing the dsk busy 100%.

After uninstalling uiDivStats the GUI cpu graph settles down and my USB stick light stops flashing non stop. And now dsk only shows up intermittently in atop.