uiDivStats uiDivStats - WebUI for Diversion statistics

[thelonelycoder]

Once i started adding menus it became user-focused!

True. Also true for all of us are honest mistakes and errors in reasoning.

 

[Jack Yaz]

@Jack Yaz could you not start taildns when logging is disabled?
I modified my files thus:
/jffs/scripts/uiDivStats, line 1288:

    dnsmasq)
       if grep 'log-facility' /etc/dnsmasq.conf; then
           Print_Output "true" "dnsmasq has restarted, restarting taildns" "$PASS"
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           sleep 5
           /opt/etc/init.d/S90taildns start >/dev/null 2>&1
       else
           /opt/etc/init.d/S90taildns stop >/dev/null 2>&1
           Print_Output "true" "dnsmasq has restarted, Dnsmasq logging is disabled, stopped taildns" "$PASS"
       fi
       exit 0
   ;;

And /opt/bin/taildns:

#!/bin/sh
#shellcheck disable=SC2039
trap '' SIGHUP

if [ ! -f /opt/bin/pkill ]; then
    opkg update
    opkg install procps-ng-pkill
fi

if grep 'log-facility' /etc/dnsmasq.conf; then
    /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &

    while true; do
        sleep 5
        if [ "$(pidof taildnstotal | wc -w)" -lt 2 ]; then
            logger -t "uiDivStats" "taildns dead, restarting..."
            killall -q taildnstotal
            /opt/bin/pkill -f "tail -F /opt/var/log/dnsmasq.log /opt/var/log/dnsmasq.log3"
            sleep 5
            /jffs/addons/uiDivStats.d/taildns.d/taildnstotal &
            logger -t "uiDivStats" "taildns restarted"
        fi
    done
else
    logger -t "uiDivStats" "taildns not started, Dnsmasq logging is disabled"
fi

Edit: You may have to add more code elsewhere, I just changed the obvious files to me

I've opted to add the below to S90taildns - this way it covers all calls including those if the user tries to manually manipulate taildns

if ! grep -q 'log-facility' /etc/dnsmasq.conf; then
    logger -st "uiDivStats" "taildns not started: dnsmasq logging is disabled"
    exit 1
fi

 

[joe scian]

I suppose I also need to test for whether the alternate instance of dnsmasq is enabled, and include the blocklist_fs if it is enabled...

OR just leave out "Domains currently on blocklist" since its a known quantity if you are using Diversion!!

 

[Jack Yaz]

OR just leave out "Domains currently on blocklist" since its a known quantity if you are using Diversion!!

Code already added on develop branch. I'm quietly confident it will work (famous last words) now.

 

[Jack Yaz]

OK so I think I'm ready for the 2.1.0 release now. Have I missed anything I promised I'd do? I've lost track.

 

[joe scian]

verif

Code already added on develop branch. I'm quietly confident it will work (famous last words) now.

verified working fine Jack

 

[thelonelycoder]

OK so I think I'm ready for the 2.1.0 release now. Have I missed anything I promised I'd do? I've lost track.

Go ahead, I'm waiting patiently.
When the full Diversion WebUI comes out eventually, it'll have some additional variables of interest for you in custom_settings.txt or the Diversion config file to include in the stats.
More on that when I'm ready to release.

 

[Treadler]

OK so I think I'm ready for the 2.1.0 release now. Have I missed anything I promised I'd do? I've lost track.

drum roll............!!

 

[Jack Yaz]

drum roll............!!

It will be live shortly.

 

[Jack Yaz]

v2.1.0 is now available
Changelog:

NEW: Hostnames are shown along with the IP Address in the client dropdown. These are determined by using dig for reverse DNS lookups
NEW: Add option to log only A/AAAA DNS queries instead of all types (useful if you have noisy clients firing PTR queries to resolve IPs)
CHANGED: "Last 24 hours" now explicitly collects data for all 15 minute periods. The previous "Group By" method used by 2.0.0 would exclude any points where no queried/blocked domains were found in a 15 minute period, which would leave gaps in data plots.
CHANGED: Lengthy domain names on the Y-Axis are now split to multiple lines
FIXED: Only start taildns if dnsmasq logging is enabled

If you run into any problems, or have any questions, please let me know.

 

[thelonelycoder]

v2.1.0 is now available
Changelog:

NEW: Hostnames are shown along with the IP Address in the client dropdown. These are determined by using dig for reverse DNS lookups
NEW: Add option to log only A/AAAA DNS queries instead of all types (useful if you have noisy clients firing PTR queries to resolve IPs)
CHANGED: "Last 24 hours" now explicitly collects data for all 15 minute periods. The previous "Group By" method used by 2.0.0 would exclude any points where no queried/blocked domains were found in a 15 minute period, which would leave gaps in data plots.
CHANGED: Lengthy domain names on the Y-Axis are now split to multiple lines
FIXED: Only start taildns if dnsmasq logging is enabled

If you run into any problems, or have any questions, please let me know.

Looking good. Thanks.
Just a minor inconsistency that is caused by not checking the log file status in uiDivStats as I proposed:

uiDivStats: dnsmasq has restarted, restarting taildns
(...)
uiDivStats: taildns not started: dnsmasq logging is disabled

 

[heslo]

Just updated, so far so good! Thanks Jack!

 

[5stringdeath]

Did this move somewhere? I updated via AMTM and then the "extras" tab in my webUI only showed the help tab. So I rebooted the router and now the entire tab is missing. Only update via AMTM for me was the uiDivStas so nothing else changed. I feel like I missed something. AMTM still says everything is installed and AOK.

 

[CriticJay]

v2.1.0 is now available
Changelog:

NEW: Hostnames are shown along with the IP Address in the client dropdown. These are determined by using dig for reverse DNS lookups
NEW: Add option to log only A/AAAA DNS queries instead of all types (useful if you have noisy clients firing PTR queries to resolve IPs)
CHANGED: "Last 24 hours" now explicitly collects data for all 15 minute periods. The previous "Group By" method used by 2.0.0 would exclude any points where no queried/blocked domains were found in a 15 minute period, which would leave gaps in data plots.
CHANGED: Lengthy domain names on the Y-Axis are now split to multiple lines
FIXED: Only start taildns if dnsmasq logging is enabled

If you run into any problems, or have any questions, please let me know.

Hi Jack

While installing 2.1.0 I saw the following in amtm:

uiDivStats: Creating database table indexes...
Error: near line 1: index idx_dns_domains already exists
Error: near line 1: index idx_dns_time already exists
Error: near line 1: index idx_dns_clients already exists
uiDivStats: Database ready, starting services...
uiDivStats: Starting first run of stat generation...
uiDivStats: Stats updated successfully

should i be concerned?

I previously had 2.0.0 installed a week ago. I uninstalled it due to the "high cpu usage" issue.

 

[Jack Yaz]

Hi Jack

While installing 2.1.0 I saw the following in amtm:



should i be concerned?

I previously had 2.0.0 installed a week ago. I uninstalled it due to the "high cpu usage" issue.

No that's expected if the db wasn't deleted. Looks like I missed that in the uninstaller.

 

[Jack Yaz]

Did this move somewhere? I updated via AMTM and then the "extras" tab in my webUI only showed the help tab. So I rebooted the router and now the entire tab is missing. Only update via AMTM for me was the uiDivStas so nothing else changed. I feel like I missed something. AMTM still says everything is installed and AOK.

It's under LAN now.

 

[5stringdeath]

It's under LAN now.

ROFL, thanks, its like the ONE tab I didn't look in

 

[CriticJay]

So I still have the high CPU usage (alternating cores spiking to 100%, load average throwing numbers like: 5.13, 4.55, 3.83) on v2.1.0

maybe the problem is my USB drive? atop reports "DSK" at busy: 100% and colors it red

I think I need to uninstall this for now and wait until I change my USB drive to something else!

 

[CriticJay]

Regarding PTR queries:

I ran this command suggested by Dave earlier:

grep -E " query\[PTR]" /opt/var/log/dnsmasq.log | awk ' { print $NF } ' | sort | uniq -c | sort -rn | head -10

my top culprit is a machine with only 56 PTR queries so I guess I don't have the same issue as JemTheWire?
my router uptime is currently 2.5 days

 

[Jack Yaz]

So I still have the high CPU usage (alternating cores spiking to 100%, load average throwing numbers like: 5.13, 4.55, 3.83) on v2.1.0

maybe the problem is my USB drive? atop reports "DSK" at busy: 100% and colors it red

I think I need to uninstall this for now and wait until I change my USB drive to something else!

you could try toggling to A/AAAA only mode to see if this makes a difference