uiDivStats uiDivStats - WebUI for Diversion statistics

[Jack Yaz]

Yes, I am looking in the system log for the "complete" message.

My initial atop didn't show this but I'm seeing it now.
View attachment 24115
It's hard to see but it's showing the dsk busy 100%.

After uninstalling uiDivStats the GUI cpu graph settles down and my USB stick light stops flashing non stop. And now dsk only shows up intermittently in atop.

Sit tight, I'm finalising testing on a caching based approach. Once I'm happy I'll add it to the develop branch.

The cache approach, rather than write dns query records directly to the db on the usb, stores them in tmpfs (memory). Then, every 5 minutes (starting at 4 past the hour), the cache is written to the usb. This means you should not see 100% disk ongoing.

 

[Jack Yaz]

I re-installed my entire router again, and have this :

ATOP - RT-AC5300-5880 2020/06/17  21:03:0 2020/06/17  21:05:07                                 --                       10
PRC | sys    0.32s  | user   0.21s |  #proc    141 |  #trun      1 | #tslpi   134  | #tslpu     6 |  #zombie    1 |  no  procacct |
CPU | sys       6%  | user     12% |  irq       2% |  idle     88% | wait     92%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
cpu | sys       2%  | user      9% |  irq       0% |  idle     41% | cpu001 w 48%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
cpu | sys       4%  | user      3% |  irq       2% |  idle     46% | cpu000 w 45%  | ipc notavail |  curf    ?MHz |  curscal   ?% |
CPL | avg1    2.35  | avg5    1.36 |  avg15   0.99 |               | csw    28010  | intr    8853 |               |  numcpu     2 |
MEM | tot   503.1M  | free  187.6M |  cache  30.5M |  buff    1.9M | slab  116.6M  | shmem   2.4M |  vmbal   0.0M |  hptot   0.0M |
SWP | tot     2.0G  | free    2.0G |               |               |               |              |  vmcom  63.2M |  vmlim   2.2G |
DSK |  sda  | busy     98% |  read      41 |  write     41e     88 | KiB/w      7  | MBr/s    1.2 |  MBw/s    0.0 |  avio  120 ms |
NET | transport     | tcpi     136 |  tcpo      93 |  udpi     372 | udpo     391  | tcpao      0 |  tcppo     20 |  tcprs      0 |
NET | network       | ipi      816 |  ipo      794 |  ipfrw    306 | deliv    508  |              |  icmpi      0 |  icmpo      6 |
NET | eth0     20%  | pcki    3118 |  pcko    3023 |  sp   10 Mbps | si 2023 Kbps  | so 2030 Kbps |  erri       0 |  erro       0 |
NET | vlan1     1%  | pcki    1192 |  pcko     189 |  sp   10 Mbps | si  109 Kbps  | so   48 Kbps |  erri       0 |  erro       0 |
NET | fwd1      1%  | pcki    1872 |  pcko    1134 |  sp   10 Mbps | si    0 Kbps  | so  102 Kbps |  erri       0 |  erro       0 |
NET | fwd0      0%  | pcki      85 |  pcko      58 |  sp   10 Mbps | si    0 Kbps  | so    7 Kbps |  erri       0 |  erro       0 |
NET | eth3    ----  | pcki       0 |  pcko    1873 |  sp    0 Mbps | si    0 Kbps  | so 1910 Kbps |  erri       0 |  erro       0 |
NET | lo      ----  | pcki     358 |  pcko     358 |  sp    0 Mbps | si   60 Kbps  | so   60 Kbps |  erri       0 |  erro       0 |
NET | br0     ----  | pcki     403 |  pcko     183 |  sp    0 Mbps | si   58 Kbps  | so   47 Kbps |  erri       0 |  erro       0 |
NET | eth2    ----  | pcki       0 |  pcko      69 |  sp    0 Mbps | si    0 Kbps  | so   19 Kbps |  erri       0 |  erro       0 |
NET | eth1    ----  | pcki       0 |  pcko      54 |  sp    0 Mbps | si    0 Kbps  | so   13 Kbps |  erri       0 |  erro       0 |
NET | wl1.2   ----  | pcki       0 |  pcko     100 |  sp    0 Mbps | si    0 Kbps  | so   12 Kbps |  erri       0 |  erro       0 |
NET | wl2.2   ----  | pcki       0 |  pcko      37 |  sp    0 Mbps | si    0 Kbps  | so    9 Kbps |  erri       0 |  erro       0 |
  PID    SYSCPU    USRCPU     VGROW     RGROW    RUID EUID ST    EXC     THR    S    CPUNR     CPU    CMD 1/3
  318     0.02s     0.09s 0K 0K    francop     francop     --      - 1    S 0      1%    nt_center

View attachment 24117

View attachment 24118

Do you have a different usb drive just to rule it out being a usb 3 issue? Or try it in the 2.0 port. Are you using samba, ftp server or dlna on the router at all?

 

[doczenith1]

@TheMorpN is showing similar DSK values and GUI cpu activity that I experienced. To add more info on my setup I am using an old USB 2 stick on the USB 3 port that is set to USB 2 mode. I'm not using samba, ftp server or dlna on the router nor Qos or AiProtection.

 

[Jack Yaz]

For those experiencing high DSK and/or CPU usage with 2.X

Please can you test the "cache" mode available on the develop branch? You can switch by running:

uiDivStats develop

and using option c in the CLI menu.

The cache approach, rather than write dns query records directly to the db on the usb, stores them in tmpfs (memory). Then, every 5 minutes (starting at 4 past the hour), the cache is written to the usb. This means you should not see 100% disk ongoing.

 

[doczenith1]

So far so good.
USB stick light not flashing non stop....check
cpu graph in GUI calm....check
atop dsk usage not pegged at 100%....check

 

[QuikSilver]

@Jack Yaz , installed on a secondary router and its bogging down. Installed atop and see this for DSK.

DSK |   sda |  busy    102%  | read 7  |  write   2 16  write 108225 |  KiB/r     40  | KiB/w     23  | MBr/s    0.0  |  MBw/s    0.1 |  avq    11.34  | avio  370 ms  |

This is running the cache version you have above and turning on using "C".

 

[QuikSilver]

@Jack Yaz , installed on a secondary router and its bogging down. Installed atop and see this for DSK.

DSK |   sda |  busy    102%  | read 7  |  write   2 16  write 108225 |  KiB/r     40  | KiB/w     23  | MBr/s    0.0  |  MBw/s    0.1 |  avq    11.34  | avio  370 ms  |

This is running the cache version you have above and turning on using "C".

 

[SomeWhereOverTheRainBow]

For those experiencing high DSK and/or CPU usage with 2.X

Please can you test the "cache" mode available on the develop branch? You can switch by running:

uiDivStats develop

and using option c in the CLI menu.

The cache approach, rather than write dns query records directly to the db on the usb, stores them in tmpfs (memory). Then, every 5 minutes (starting at 4 past the hour), the cache is written to the usb. This means you should not see 100% disk ongoing.

no issues here on new test branch

 

[LimJK]

For those experiencing high DSK and/or CPU usage with 2.X

Please can you test the "cache" mode available on the develop branch? You can switch by running:

uiDivStats develop

and using option c in the CLI menu.

The cache approach, rather than write dns query records directly to the db on the usb, stores them in tmpfs (memory). Then, every 5 minutes (starting at 4 past the hour), the cache is written to the usb. This means you should not see 100% disk ongoing.

Jack,

All seems to be working very well with the development version 2.2.0 with and without caching dns query records:

PS: I experienced high CPU spike continuously last few weeks. Yesterday I decided to changed my old (no brand) USB ThumbDrive to a newer SanDisk Ultra USB 3.0 (32GB) using ext4 format with Journaling. After that, no more CPU spikes . So for my case, the CPU spikes issue was my old USB ThumbDrive
Henceforth, my new SOP gleaned from discussions in this forum (I think from L&LD): I will always try to remember to Remove (unmount) my USB Thumbdrive via GUI first, before doing my Firmware Update, Factory Reset or Reboot. Hoping to reduce any risks of corrupting my USB Thumbdrive (I must confess that I don't really know if it is really necessary).

 

[SomeWhereOverTheRainBow]

Jack,

All seems to be working very well with the development version 2.2.0 with and without caching dns query records:

View attachment 24121

PS: I experienced high CPU spike continuously last few weeks. Yesterday I decided to changed my old (no brand) USB ThumbDrive to a newer SanDisk Ultra USB 3.0 (32GB) using ext4 format with Journaling. After that, no more CPU spikes . So for my case, the CPU spikes issue was my old USB ThumbDrive
Henceforth, my new SOP gleaned from discussions in this forum (I think from LD&D): I will always try to remember to Remove (unmount) my USB Thumbdrive via GUI first, before doing my Firmware Update, Factory Reset or Reboot. Hoping to reduce any risks of corrupting my USB Thumbdrive (I must confess that I don't really know if it is really necessary).

The important factor is to use unmount because it tells processes runned by entware and other scripts"hey stop hogging all the resources, I am about to flash."

 

[Jack Yaz]

View attachment 24120

That's weird. Stop taildns :

/opt/etc/init.d/S90taildns stop

Does disk usage drop? If so, start it again then check again. If disk usage increases again please send output of

ps | grep taildns

 

[Jack Yaz]

I'm running out of ideas if the cache approach doesn't help

 

[L&LD]

Jack,

All seems to be working very well with the development version 2.2.0 with and without caching dns query records:

View attachment 24121

PS: I experienced high CPU spike continuously last few weeks. Yesterday I decided to changed my old (no brand) USB ThumbDrive to a newer SanDisk Ultra USB 3.0 (32GB) using ext4 format with Journaling. After that, no more CPU spikes . So for my case, the CPU spikes issue was my old USB ThumbDrive
Henceforth, my new SOP gleaned from discussions in this forum (I think from LD&D): I will always try to remember to Remove (unmount) my USB Thumbdrive via GUI first, before doing my Firmware Update, Factory Reset or Reboot. Hoping to reduce any risks of corrupting my USB Thumbdrive (I must confess that I don't really know if it is really necessary).

The important factor is to use unmount because it tells processes runned by entware and other scripts"hey stop hogging all the resources, I am about to flash."

More specifically, it releases as much RAM as possible so that the firmware can be uploaded to the router, verified by the router, and then properly flashed.

 

[L&LD]

I have updated a few of each of the following routers to their respective/latest 384.18 Alpha 1 RMerlin firmware and there are no issues with CPU usage with uiDivStats v2.1 with any of them.

RT-AC66U_B1, RT-AC68U, RT-AC3100, RT-AC86U, RT-AX58U, RT-AX88U. (About 22 routers in all).

My now standard update procedure:

• updated all scripts, including amtm, if needed.
  • if uiDivStats v1.3x was installed, ran 'u' twice (the second 'u' isn't really needed within the uiDivStats script, just accessing uiDivStats via 'j5' downloaded and updated a few files if the script was on v2.0, first.
• Safely removed all USB connected devices.
• I waited a couple of minutes for the CPUs to settle before flashing the firmware for the router model in question.
• After the router rebooted and I checked the firmware was successfully flashed, I rebooted the router again just after 15 minutes. A final reboot of the routers was done after a further hour had passed.

Note that at least one of each of the router models above was flashed via OpenVPN (remotely) with the customer enabling the OpenVPN server for me and staying on the phone until they confirmed they had internet again.

Thank you @Jack Yaz for this upgrade. I'm sure the cache approach will make things even better when it goes past developer mode and is a stable release.

 

[Jack Yaz]

I have updated a few of each of the following routers to their respective/latest 384.18 Alpha 1 RMerlin firmware and there are no issues with CPU usage with uiDivStats v2.1 with any of them.

RT-AC66U_B1, RT-AC68U, RT-AC3100, RT-AC86U, RT-AX58U, RT-AX88U. (About 22 routers in all).

My now standard update procedure:

• updated all scripts, including amtm, if needed.
  • if uiDivStats v1.3x was installed, ran 'u' twice (the second 'u' isn't really needed within the uiDivStats script, just accessing uiDivStats via 'j5' downloaded and updated a few files if the script was on v2.0, first.
• Safely removed all USB connected devices.
• I waited a couple of minutes for the CPUs to settle before flashing the firmware for the router model in question.
• After the router rebooted and I checked the firmware was successfully flashed, I rebooted the router again just after 15 minutes. A final reboot of the routers was done after a further hour had passed.

Note that at least one of each of the router models above was flashed via OpenVPN (remotely) with the customer enabling the OpenVPN server for me and staying on the phone until they confirmed they had internet again.

Thank you @Jack Yaz for this upgrade. I'm sure the cache approach will make things even better when it goes past developer mode and is a stable release.

I'm happy to hear its working on such a wide variety of routers!

 

[SomeWhereOverTheRainBow]

I'm happy to hear its working on such a wide variety of routers!

I am not quite sure it is all the fault of the script or the router that is causing these issues.from what I can tell so far most the issues seem to stem from performance of the flash devices being used, which is silly though because we are not transferring huge amounts of data.

 

[Chris0815]

most the issues seem to stem from performance of the flash devices being used

I am also dealing with these CPU-Spikes on AC-86U with uidivstats 2.1.0 - also had to uninstall. Since the beginning I use an old USB2-Stick, 8 GB capacity, 2GB SWAP (ext4), plugged in the USB3-Port but downgraded to USB2 in the GUI - no problems before.
Perfekt time to try an upgrade. But it will take me some days until I see the Router physically again... (managed by VPN-Server). Any recommendations/hints or is a simple USB3-Stick enough?

 

[Jack Yaz]

I am not quite sure it is all the fault of the script or the router that is causing these issues.from what I can tell so far most the issues seem to stem from performance of the flash devices being used, which is silly though because we are not transferring huge amounts of data.

The cache mode should mitigate this, since writes are limited to once every 5 minutes for the cache flush, and once every minute for the "query log" - which is a small file (300kb)

 

[Chris0815]

The cache mode should mitigate this, since writes are limited to once every 5 minutes for the cache flush, and once every minute for the "query log" - which is a small file (300kb)

Running v2.2.0 with the toggled cache-option since 5 hours without any issues (RT-86U). And it seems to work! Thanks @Jack Yaz!

 

[andresmorago]

For those experiencing high DSK and/or CPU usage with 2.X

Please can you test the "cache" mode available on the develop branch? You can switch by running:

uiDivStats develop

and using option c in the CLI menu.

The cache approach, rather than write dns query records directly to the db on the usb, stores them in tmpfs (memory). Then, every 5 minutes (starting at 4 past the hour), the cache is written to the usb. This means you should not see 100% disk ongoing.

Hello Jack
I’m running your developer version and works ok. I noticed that the client drop down menu under the query log is now gone