uiDivStats uiDivStats - WebUI for Diversion statistics

[ugandy]

fresh new install:

cromo@RT-AX88U-8158:/tmp/home/root# /usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/jackyaz/uiDivStats/maste
r/uiDivStats.sh" -o "/jffs/scripts/uiDivStats" && chmod 0755 /jffs/scripts/uiDivStats && /jffs/scripts/uiDivStats instal
l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 43273 100 43273 0 0 355k 0 --:--:-- --:--:-- --:--:-- 449k
uiDivStats: Welcome to uiDivStats v2.0.0, a script by JackYaz

uiDivStats: Checking your router meets the requirements for uiDivStats

Downloading http://bin.entware.net/aarch64-k3.10/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Package grep (3.4-1) installed in root is up to date.
Package sqlite3-cli (3310100-1) installed in root is up to date.
Package procps-ng-pkill (3.3.15-4) installed in root is up to date.
uiDivStats: New version of uidivstats_www.asp downloaded

uiDivStats: Mounting uiDivStats WebUI page as user3.asp

uiDivStats: New version of taildns.tar.gz downloaded

Downloading http://bin.entware.net/aarch64-k3.10/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Package grep (3.4-1) installed in root is up to date.
Package sqlite3-cli (3310100-1) installed in root is up to date.
Package procps-ng-pkill (3.3.15-4) installed in root is up to date.
uiDivStats: Creating database table and enabling write-ahead logging...

uiDivStats: Creating database table indexes...

Error: near line 1: index idx_dns_domains already exists
Error: near line 1: index idx_dns_time already exists
Error: near line 1: index idx_dns_clients already exists
uiDivStats: Database ready, starting services...

uiDivStats: Starting first run of stat generation...

uiDivStats: Stats updated successfully



After this, the UI page is missing. even after a reboot

Fresh install did work!
i just missed that the page locaton moved!
stats generated.
all good. thanks!

 

[Jack Yaz]

I thought Diversion loaded itself prior to launching Entware services? I could be wrong

@dave14305 it is probably sensible for me to hook into dnsmasq restarts, I'll work on that now

 

[dave14305]

@dave14305 it is probably sensible for me to hook into dnsmasq restarts, I'll work on that now

Or trust only the diversion.conf “loqQueriesExtra=on” presence. But still you’d want to check on dnsmasq restart if that setting changed.

 

[Jack Yaz]

Or trust only the diversion.conf “loqQueriesExtra=on” presence. But still you’d want to check on dnsmasq restart if that setting changed.

I've hooked into dnsmasq.postconf https://github.com/jackyaz/uiDivStats/commit/78b49993ee0bcf97e145f4b204dea7827eaea38d

I would use service-event but I have logic to discard non-uiDivStats service calls, and that would be a lot more complicated to fix. All taildns needs to do is check log-queries in dnsmasq.conf so this should work!

 

[SomeWhereOverTheRainBow]

I've hooked into dnsmasq.postconf https://github.com/jackyaz/uiDivStats/commit/78b49993ee0bcf97e145f4b204dea7827eaea38d

I would use service-event but I have logic to discard non-uiDivStats service calls, and that would be a lot more complicated to fix. All taildns needs to do is check log-queries in dnsmasq.conf so this should work!

Nice!

 

[Jack Yaz]

This is a great upgrade! A lot of complexity behind the scenes to make it look so cool.

Noticed this today that when dnssec is enabled there are extra lines in the log that probably want to be filtered out. I had dnssec disabled during beta testing.

View attachment 23936

Jun  6 12:08:13 dnsmasq[9727]: dnssec-query[DS] asuswrt-merlin.net to 9.9.9.11
Jun  6 12:08:13 dnsmasq[9727]: dnssec-query[DNSKEY] asuswrt-merlin.net to 9.9.9.11

I assume you get query[A] lines as normal? Can you provide the output of a full query please, from query to reply?

 

[dave14305]

I assume you get query[A] lines as normal? Can you provide the output of a full query please, from query to reply?

Two examples before and after enabling extra.

Jun  6 12:08:13 dnsmasq[9727]: query[A] www.asuswrt-merlin.net from 192.168.1.86
Jun  6 12:08:13 dnsmasq[9727]: forwarded www.asuswrt-merlin.net to 9.9.9.11
Jun  6 12:08:13 dnsmasq[9727]: dnssec-query[DS] asuswrt-merlin.net to 9.9.9.11
Jun  6 12:08:13 dnsmasq[9727]: reply asuswrt-merlin.net is DS keytag 2371, algo 13, digest 2
Jun  6 12:08:13 dnsmasq[9727]: dnssec-query[DNSKEY] asuswrt-merlin.net to 9.9.9.11
Jun  6 12:08:13 dnsmasq[9727]: reply asuswrt-merlin.net is DNSKEY keytag 2371, algo 13
Jun  6 12:08:13 dnsmasq[9727]: reply asuswrt-merlin.net is DNSKEY keytag 34505, algo 13
Jun  6 12:08:13 dnsmasq[9727]: reply www.asuswrt-merlin.net is 104.18.41.167
Jun  6 12:08:13 dnsmasq[9727]: reply www.asuswrt-merlin.net is 104.18.40.167
Jun  6 12:08:13 dnsmasq[9727]: reply www.asuswrt-merlin.net is 172.67.183.16
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 query[A] www.asuswrt-merlin.net from 192.168.1.86
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 forwarded www.asuswrt-merlin.net to 9.9.9.11
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 dnssec-query[DS] asuswrt-merlin.net to 9.9.9.11
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 reply asuswrt-merlin.net is DS keytag 2371, algo 13, digest 2
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 dnssec-query[DNSKEY] asuswrt-merlin.net to 9.9.9.11
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 reply asuswrt-merlin.net is DNSKEY keytag 2371, algo 13
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 reply asuswrt-merlin.net is DNSKEY keytag 34505, algo 13
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 reply www.asuswrt-merlin.net is 172.67.183.16
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 reply www.asuswrt-merlin.net is 104.18.41.167
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 reply www.asuswrt-merlin.net is 104.18.40.167

 

[Jack Yaz]

Two examples before and after enabling extra.

Jun  6 12:08:13 dnsmasq[9727]: query[A] www.asuswrt-merlin.net from 192.168.1.86
Jun  6 12:08:13 dnsmasq[9727]: forwarded www.asuswrt-merlin.net to 9.9.9.11
Jun  6 12:08:13 dnsmasq[9727]: dnssec-query[DS] asuswrt-merlin.net to 9.9.9.11
Jun  6 12:08:13 dnsmasq[9727]: reply asuswrt-merlin.net is DS keytag 2371, algo 13, digest 2
Jun  6 12:08:13 dnsmasq[9727]: dnssec-query[DNSKEY] asuswrt-merlin.net to 9.9.9.11
Jun  6 12:08:13 dnsmasq[9727]: reply asuswrt-merlin.net is DNSKEY keytag 2371, algo 13
Jun  6 12:08:13 dnsmasq[9727]: reply asuswrt-merlin.net is DNSKEY keytag 34505, algo 13
Jun  6 12:08:13 dnsmasq[9727]: reply www.asuswrt-merlin.net is 104.18.41.167
Jun  6 12:08:13 dnsmasq[9727]: reply www.asuswrt-merlin.net is 104.18.40.167
Jun  6 12:08:13 dnsmasq[9727]: reply www.asuswrt-merlin.net is 172.67.183.16
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 query[A] www.asuswrt-merlin.net from 192.168.1.86
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 forwarded www.asuswrt-merlin.net to 9.9.9.11
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 dnssec-query[DS] asuswrt-merlin.net to 9.9.9.11
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 reply asuswrt-merlin.net is DS keytag 2371, algo 13, digest 2
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 dnssec-query[DNSKEY] asuswrt-merlin.net to 9.9.9.11
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 reply asuswrt-merlin.net is DNSKEY keytag 2371, algo 13
Jun  6 12:53:02 dnsmasq[29185]: * 192.168.1.86/54635 reply asuswrt-merlin.net is DNSKEY keytag 34505, algo 13
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 reply www.asuswrt-merlin.net is 172.67.183.16
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 reply www.asuswrt-merlin.net is 104.18.41.167
Jun  6 12:53:02 dnsmasq[29185]: 100 192.168.1.86/54635 reply www.asuswrt-merlin.net is 104.18.40.167

OK so in theory, I think I can adjust the logic to only process a line if "query" is matched, and not "dnssec"

 

[dave14305]

I assume you get query[A] lines as normal? Can you provide the output of a full query please, from query to reply?

Line 7 of dnsmasqtotalextra.awk needs incrementing to field 7.

OK so in theory, I think I can adjust the logic to only process a line if "query" is matched, and not "dnssec"

Or “ query” with a leading space?

 

[Jack Yaz]

Line 7 of dnsmasqtotalextra.awk needs incrementing to field 7.

Or “ query” with a leading space?

Yes spotted that while fixing dnssec....that probably fixes the problems that @Dan Ljungberg was having. Damnit.

 

[Jack Yaz]

@dave14305 new version of taildns uploaded, please update uiDivStats and retry with dnssec

 

[Dan Ljungberg]

at this stage, would you be up for a screensharing session using Teamviewer or VNC or something?

at this stage, would you be up for a screensharing session using Teamviewer or VNC or something?

Yes, tomorrow fits, because now the family and Sweden's national day applies.

 

[dave14305]

@dave14305 new version of taildns uploaded, please update uiDivStats and retry with dnssec

Looking clean!

 

[Jack Yaz]

Yes, tomorrow fits, because now the family and Sweden's national day applies.

I've just pushed an update that may well resolve the issue. While fixing another issue I (and @dave14305) spotted I'd goofed a positional argument in the "extra" version of the log parsing script

 

[Chris0815]

I also had the topic that update to version 2.0.0 got stuck this morning. After solving this problem I was not able to see any stats on the Tab since hours. But since the update it works fine here (AC86U).
I Cant wait to look at the graph in 30 days...
Jack Yaz - Thank you for the effort and the possibility for detailed analysis of block/unblock domains!

 

[JemTheWire]

Since I updated to v2, I have noticed that each of my 4 CPU cores hit 100%, but not all at the same time. Also, the general GUI ‘seems’ a bit sluggish.

When I uninstall, CPU cores just have the usual chatter. Reinstall v2, and the cores peek again.

Is this normal?

 

[Jack Yaz]

Since I updated to v2, I have noticed that each of my 4 CPU cores hit 100%, but not all at the same time. Also, the general GUI ‘seems’ a bit sluggish.

When I uninstall, CPU cores just have the usual chatter. Reinstall v2, and the cores peek again.

Is this normal?

Not normal at all. Check what is actually using cpu in top or htop

 

[JT Strickland]

After updating to v2, After a Reboot, I lost my NtpMerlin tab, as well as the Diversion stats tab. The diversion tab in the LAN looks the same.
Anybody else? Before Reboot I still had the old diversion stats tab with nothing on it.
Reboot some more?
I forgot to mention I did a force update afterwards, same results.

Edit: 2nd Reboot was weird, didn't ever shut off, or the led's never went off, usually does it 3 times if it don't go to sleep (86U). But I've got the NTP daemon tab back, so it did something.

jts
RT-AC86U w/ 384.17, RT-AC68U Aimesh node w/ same, 250/10 cable w/ Netgear CM-1000, Diversion, UiDivstats, Skynet, AiProtection, DoT, Scribe, UiScribe, Connmon, SpdMerlin, ScMerlin, Nsrum, NtpMerlin

 

[AntonK]

I've uninstalled Diversion and uiDivDtats, rebooted the router (twice), but their two tabs remain under LAN. Is there something else I need to do, or do they just stay there?

 

[Stephen Harrington]

Love your work @Jack Yaz

Wondering if it would be possible for the "Client To Display" selection for Top Blocked and Top Requested to show friendly/host names rather than IP addresses, rather like what happens in Network Tools | Wake On LAN when you go to add an entry, shows active and even inactive clients. Don't know if you can access that data somehow to use in uiDivStats?

Likewise, for the Query Log filtering, could the Client field of the filter similarly be a dropdown list of hostnames rather than having to enter an IP perhaps? Maybe keep a separate IP field as well so it goes Domain | IP | Client | Type | Result ?

What do others think?