Diversion Diversion and a crashing dnsmasq

[wpte]

Hi, a new user of Diversion here.

I've some issues I have not managed to fix myself, or find any information about here on the forums (or I missed it in my search).

First let's start with my setup:
RT-AX88U with merlin 384.19
Storage is a sandisk extreme pro 128GB USB 3.1 (read/write is around 250MB/s) for now, obviously ext4 with 2gb swap
Skynet - default settings
FlexQoS - default settings
Diversion standard - default settings + large list + youtube blocking + logging on
uiDivstats - default settings

Now this seems to run fine, recently I only added Diversion, so the setup normally works.
But after about 2 days of running Diversion it seems like dnsmasq crashes and diversion keeps looping into (I think updating lists) with some grep and sed commands which show in top in high load

What could be the cause of this?
RAM is relatively full, but swap is not even used and CPU load normally is pretty low

 

[dave14305]

uiDivStats had been known to put a higher load during its constant log parsing. If it is dnsmasq though, try running with the medium or standard list for a few days.

 

[wpte]

uiDivStats had been known to put a higher load during its constant log parsing. If it is dnsmasq though, try running with the medium or standard list for a few days.

Hi Dave, I've kept the list on large now but put uidivstats in caching mode.
That way it only writes every 5 minutes to the usb storage, which seems to help since it has been running stable since your reply.
It's nice to see the statistics, but as you said, it does seem to put a higher load on the system

 

[Jack Yaz]

Hi Dave, I've kept the list on large now but put uidivstats in caching mode.
That way it only writes every 5 minutes to the usb storage, which seems to help since it has been running stable since your reply.
It's nice to see the statistics, but as you said, it does seem to put a higher load on the system

which is why caching mode was implemented, for slower read/write speed USB drives

 

[wpte]

which is why caching mode was implemented, for slower read/write speed USB drives

Interesting argument, considering I'm using a usb flash drive which is among the fastest.
Sandisk specs say the following:
420MB/s read and 380MB/s write

Further, benchmarks online for 4k random acces say: 12.65MB/s read 9.644MB/s write
That's all SSD-like performance

On the router:

/dev/sda1:
 Timing buffered disk reads: 930 MB in  3.01 seconds = 309.41 MB/sec

My guess is something ended up in a deadlock, as diversion was constantly updating, again and again.
Sounds to me like there's a crash, things start again and then somehow it crashes again.
Perhaps the constant reading of the logfile isn't that stable? Picking it up every 5min reduces the chance of conflicts

 

[SomeWhereOverTheRainBow]

Interesting argument, considering I'm using a usb flash drive which is among the fastest.
Sandisk specs say the following:
420MB/s read and 380MB/s write

Further, benchmarks online for 4k random acces say: 12.65MB/s read 9.644MB/s write
That's all SSD-like performance

On the router:

/dev/sda1:
 Timing buffered disk reads: 930 MB in  3.01 seconds = 309.41 MB/sec

My guess is something ended up in a deadlock, as diversion was constantly updating, again and again.
Sounds to me like there's a crash, things start again and then somehow it crashes again.
Perhaps the constant reading of the logfile isn't that stable? Picking it up every 5min reduces the chance of conflicts

It definitely has to do with the way flash drives behave, in particular, read and write. The speed does not seem to be the issue as much as sustained read and write that uidivistats demands. If it cannot sustain read and write, then the flash takes a clobbering.

 

[wpte]

I'm starting to think it doesn't have anything to do with uiDivstats .
Today I just got the same problem again, after running stable for days.

Diversion keeps grepping things, but then it perhaps takes too long and the router tries to restart dnsmasq again, and then again it starts grepping...

As a result, at least 2 greps are busy processing the blacklist

In the log, these 2 lines appear a lot:

Nov 20 12:07:47 rc_service: watchdog 1311:notify_rc start_dnsmasq
Nov 20 12:07:47 rc_service: waitting "start_dnsmasq" via watchdog ...

In the end, killing these greps, and consecutive seds seems to at least get me out of this situation.
Then afterwards, I can update the blocklist from the diversion menu without any problems.

Also, when loading the diversion menu, this command seems to take a long time to process (for stats?)

time /opt/bin/grep ^[^#] /opt/share/diversion/list/blacklist
Command exited with non-zero status 1
real 2m 48.91s
user 2m 48.58s
sys 0m 0.23s

This file is 269MB big and completely empty (filled with spaces)

I also have diversion running at my parent's place, also a AX88U with a pretty slow usb stick btw (10MB/s write and 20MB/s read)
I compared the file sizes, and please note I use the same diversion settings (large list on both, yt blocking, logging):

@thelonelycoder I'm confused, how can this be so different?
Perhaps I need to do a reset of some sorts?

 

[dave14305]

For troubleshooting, run head /opt/share/diversion/list/blacklist to provide a sample of what’s messed up, then I’d delete it, or move it elsewhere in case thelonelycoder wants to analyze it.

Is IPv6 even enabled?

 

[Jack Yaz]

I'm starting to think it doesn't have anything to do with uiDivstats .
Today I just got the same problem again, after running stable for days.

Diversion keeps grepping things, but then it perhaps takes too long and the router tries to restart dnsmasq again, and then again it starts grepping...
View attachment 27802
As a result, at least 2 greps are busy processing the blacklist

In the log, these 2 lines appear a lot:


In the end, killing these greps, and consecutive seds seems to at least get me out of this situation.
Then afterwards, I can update the blocklist from the diversion menu without any problems.

Also, when loading the diversion menu, this command seems to take a long time to process (for stats?)

This file is 269MB big and completely empty (filled with spaces)

I also have diversion running at my parent's place, also a AX88U with a pretty slow usb stick btw (10MB/s write and 20MB/s read)
I compared the file sizes, and please note I use the same diversion settings (large list on both, yt blocking, logging):
View attachment 27803

@thelonelycoder I'm confused, how can this be so different?
Perhaps I need to do a reset of some sorts?

I'm off the hook! Yay my amateur scripting skills survive another day

 

[wpte]

For troubleshooting, run head /opt/share/diversion/list/blacklist to provide a sample of what’s messed up, then I’d delete it, or move it elsewhere in case thelonelycoder wants to analyze it.

Is IPv6 even enabled?

Like I said, it's filled with spaces, so nothing to show with head.
I did move it away now and just touch-ed a new file for diversion to use, everything feels super snappy again.
I haven't added any blacklisted domains myself, ever

I'm off the hook! Yay my amateur scripting skills survive another day

Haha, you being an amateur never came to mind

 

[storkinsj]

Hi- I know this is an old thread but I just want to say Diversion will take your router out if you aren't careful. And it was behaving Very strangely.

I had network up but DNS failures 2 days in a row in the morning. I assume maybe it was triggered by diversion updating its blacklist but it's not clear. Setting local DNS on the computer bypassed the issue but of course all the iot and phones stop working right. One of the 4 cores on the AX 11000 was pegged during the outage.

Diversion had been workign with no problem for months. Potentially a problem was brewing on the flash drive.

Finding this thread because I guessed diversion was tied into DNS, I tried to disable Diversion in the WebUI but the screen would refresh and it would still be enabled. I decided to uninstall it.

I could not find the diversion install/option script. At one point I found a ghost version of it and then it disappeared rapidly:

admin@myrouter:/tmp/mnt/8GBUSB/entware/bin# ls
ash               diversion         fgrep             locale.new        opkg
base64            egrep             find              localedef.new     sh
column            entware-services  grep              netstat           xargs
admin@myrouter:/tmp/mnt/8GBUSB/entware/bin# ./diversion
./diversion: exec: line 53: /opt/bin/diversion: not found
admin@myrouter:/tmp/mnt/8GBUSB/entware/bin# cd diversion
-sh: cd: can't cd to diversion
admin@myrouter:/tmp/mnt/8GBUSB/entware/bin# ls -l
lrwxrwxrwx    1 admin    root             8 Dec 29  2022 ash -> /bin/ash
lrwxrwxrwx    1 admin    root            29 Dec 29  2022 base64 -> /opt/libexec/base64-coreutils
-rwxr-xr-x    1 admin    root         51792 Jul 23  2022 column
## ???!!!!!!

lrwxrwxrwx    1 admin    root            22 Dec 29  2022 egrep -> /opt/libexec/egrep-gnu
lrwxrwxrwx    1 admin    root            26 Dec 29  2022 entware-services -> /opt/etc/init.d/rc.unslung
lrwxrwxrwx    1 admin    root            22 Dec 29  2022 fgrep -> /opt/libexec/fgrep-gnu
-rwxr-xr-x    1 admin    root        278992 Apr  3  2022 find
lrwxrwxrwx    1 admin    root            21 Dec 29  2022 grep -> /opt/libexec/grep-gnu
-rwxr-xr-x    1 admin    root         42872 Mar 19  2021 locale.new
-rwxr-xr-x    1 admin    root        261696 Mar 19  2021 localedef.new
lrwxrwxrwx    1 admin    root            12 Dec 29  2022 netstat -> /bin/netstat
-rwxr-xr-x    1 admin    root        860208 Dec  4  2022 opkg
lrwxrwxrwx    1 admin    root             7 Dec 29  2022 sh -> /bin/sh
-rwxr-xr-x    1 admin    root         72432 Apr  3  2022 xargs
admin@myrouter:/tmp/mnt/8GBUSB/entware/bin# ls
ash               egrep             find              localedef.new     sh
base64            entware-services  grep              netstat           xargs
column            fgrep             locale.new        opkg

REALLY strange. And sort of suspicious.

I was able to restore service but only by rm -rf /opt/* which whacked entware.

It is almost as though it were up to something and was trying very hard to fight me.

Besides the suspicious behavior, I would say that if you are using usb for jffs, don't spare the money. I have been similarly using usb to boot ESX servers. Cheap flash sticks no doubt last much less long than good ones. Do your research and go for a high quality high speed sandisk or other brand.

 

[Saus]

Any link with this…? https://www.snbforums.com/threads/diversion-dnsmaq-logging-hangs-slowdowns.84110/