Solved LAN client running qBittorrent crashes gt-ax6000 (running Merlin 3004.388.4)

[ika]

Hello!

I swapped my ac86u to a gt-ax6000 a few weeks ago, and everything was fine.
I'm running unrogged latest Merlin on it, swap file, Diversion, Skynet, an OpenVPN client which provides VPN network for one LAN IP only (I had some other scripts like syslog-ng, unbound, but I eventually removed them as I was troubleshooting this issue). I do not run Trendmicro, SMB (etc.) or any other built in services except the DLNA server. Diversion can be uninstalled, it makes no difference.
The device was hard reset to the factory defaults after installing Merlin, the USB storage was formatted with AMTM and everything was set up from scratch.
I attached my DNS setup. DNS director is “ON” with global redirection to the “Router”.

Here is my problem:

In the first few weeks I had no issues, but I realized that my uptimes were kinda short and I did not reboot the router. I found nothing in the logs, but it was obvious that the router restarts itself from time to time (it is on a UPS, so power outage would be something I would notice).
Long story short, with lots of investigation I figured out that if I run qBittorrent on my PC, it somehow poisons the router. I know this is a weird wording, but I cannot describe it better. If I start qBittorent (nothing heavy, it has like 20 torrents completed and seeding and concurrent connections globally capped at 256, only TCP allowed with forced encryption). It must be noted that torrents have many trackers, some of them are in countries which I block with Skynet (I block 7 countries with Skynet).

As soon as I start qBittorrent on the PC the router starts to struggle. DNS queries on the LAN are considerably slower. I do not see much in HTOP, but SSH and the GUI immediately becomes very slow (see attached pic which I took under the “load”; note the pic shows syslog-ng running, but I uninstalled it since, it made no difference, I just removed it because of its RAM usage, and I do not really need it).
Typing AMTM and pressing enter takes 15-20 seconds to register (usually many seconds between the letters). The GUI is even slower, it is very hard to get to the logs. There is no visible load on the network map page, the only visual thing what can be seen is a ton of warning about possible DNS rebind attacks about the blocked tracker URLs in the log page.

The slowdown last as long as I run the torrent client on the PC, and it stops if I close qBittorent down, but the damage is already done. The router is not slow anymore, there is no detectable sign that something is wrong (free memory not decreasing, CPU is idling, temps are fine, etc.), but in 5-10-15 minutes, the router will crash 10 out of 10 times (there is nothing in the log until the crash, zero information about anything except the usual router things like STAs leaving and entering the range of the antennas, so only normal stuff). The router does not crash if I do not use the torrent client on the PC. Maybe there are some other programs which would lead to the same issue, but I could not find one.

It doesn’t matter how long I run the torrent client on the PC, I can run it for 30 sec or until the router crashes, the damage is done in seconds and I couldn’t even find what is going on, let alone “defuse” the issue.

Not really useful info, but I had no such issue with the ac86u which was doing the same things, I only swapped to the new unit because it was on sale and because of the two 2,5Gbps ports.

I really would like to know why is this happening. Any help or suggestion would be much appreciated.

Noble regards,
ika

 

[ColinTaylor]

Disable Skynet and try again.

 

[ika]

Disable Skynet and try again.

Thank you for the answer.

I'm sorry if it was not clear from my post, but the issue is not happening without Skynet. Disabling /Uninstalling Diversion makes no difference, but doing the same thing with Skynet does.
I just would like to understand why this did not happen to me with the previous router, where I was doing the very same thing and had no such issues. What could be the difference, what is causing this crash? Maybe it is something in the closed source part which is different for each model?

 

[ColinTaylor]

Possibly just a bug/incompatibility between Skynet and the 388.x branch vs. the 386.x used by the RT-AC86U. Probably best to post your question in the addons forum as a Skynet question.

 

[ika]

Possibly just a bug/incompatibility between Skynet and the 388.x branch vs. the 386.x used by the RT-AC86U. Probably best to post your question in the addons forum as a Skynet question.

Yes, it might be just that. I will try to investigate a little more. I'm setting up the ac86u again with a USB stick, and I will make sure they have exactly the same settings everywhere before I "blame" Skynet.
Thank you very much for the replies and the help.

 

[Adamm]

Possibly just a bug/incompatibility between Skynet and the 388.x branch vs. the 386.x used by the RT-AC86U. Probably best to post your question in the addons forum as a Skynet question.

The issue is most likely with what he is blocking not skynet. OP mentions he blocks 7 countries… id wipe your blacklist and start there.

 

[ika]

The issue is most likely with what he is blocking not skynet. OP mentions he blocks 7 countries… id wipe your blacklist and start there.

Thank you for the reply Adamm.

The issue is definitely not what I'm blocking (edit: it's countries like North Korea), but indeed I'm also not sure it is Skynet (maybe Skynet only "allows" the issue to happen). The blacklist only strengthens the signs, and yes, I tried to remove it, that was one of the first things I tried.
Without the country blocks, the router does not slow down, but the poisoning still happens the same way, the router will crash.
Last night - before I went to bed - I set up the ac86u the same way as my ax6000 is running (well, the ISP is a different one, but I assume that doesn't matter) and started running the torrent client on that network. No issues so far on that side.

What puzzles me the most is what could qBittorrent do to the router which - sooner or later - ends in a crash without any warning or error in the logs, and why doesn't it happening without it?

​

 

[Morris]

Torrents have been crashing and/or overloading routers since invented. This is because they can create so many connections and this is particularly bad as the NAT and connection table can fill. If you are going to run a Torrent application, limit the number of hosts it can connect to to a rather small number, say 50 or 100.

 

[ika]

Torrents have been crashing and/or overloading routers since invented. This is because they can create so many connections and this is particularly bad as the NAT and connection table can fill. If you are going to run a Torrent application, limit the number of hosts it can connect to to a rather small number, say 50 or 100.

Thank you for the reply.

Could you please elaborate why would a heavy load crash a router? One of the very main function of a router is to deal with those kind of loads, so I don't see how would it be ok, if it would made it crash.
I agree that tables storing connection data can be quickly filled/overflown, especially with a lots of stateless active connections on routers without a proper RFC4787 implementation, but I think a single client with simultaneous TCP connections capped at 256, all that in 2023, should only lead to connection drops, hangs, slowdowns and definitely not a crash without warnings (in my opinion), and especially not minutes after the causing load happened.

It should be also noted that the 256 limit has zero issues on the ac86u, and it also causes zero issues if I set it to 1000, none whatsoever.



Something else is going on here, me thinks.

 

[Adamm]

Thank you for the reply.

Could you please elaborate why would a heavy load crash a router? One of the very main function of a router is to deal with those kind of loads, so I don't see how would it be ok, if it would made it crash.
I agree that tables storing connection data can be quickly filled/overflown, especially with a lots of stateless active connections on routers without a proper RFC4787 implementation, but I think a single client with simultaneous TCP connections capped at 256, all that in 2023, should only lead to connection drops, hangs, slowdowns and definitely not a crash without warnings (in my opinion), and especially not minutes after the causing load happened.

It should be also noted that the 256 limit has zero issues on the ac86u, and it also causes zero issues if I set it to 1000, none whatsoever.

View attachment 53428

Something else is going on here, me thinks.

I can tell you from personal experience that qBittorrent runs fine on LAN devices with Skynet installed on the router, I have a very busy Linux iso download server of my own running on my NAS.

Skynet logs every connection it blocks assuming logging is enabled, there is never an exception to this rule. I'd wipe your blacklist and start from there with 0 blocked IP's.

 

[Morris]

Thank you for the reply.

Could you please elaborate why would a heavy load crash a router? One of the very main function of a router is to deal with those kind of loads, so I don't see how would it be ok, if it would made it crash.
I agree that tables storing connection data can be quickly filled/overflown, especially with a lots of stateless active connections on routers without a proper RFC4787 implementation, but I think a single client with simultaneous TCP connections capped at 256, all that in 2023, should only lead to connection drops, hangs, slowdowns and definitely not a crash without warnings (in my opinion), and especially not minutes after the causing load happened.

It should be also noted that the 256 limit has zero issues on the ac86u, and it also causes zero issues if I set it to 1000, none whatsoever.

View attachment 53428

Something else is going on here, me thinks.

Routers are designed to support a limited number of hosts doing average things. One heavy user will not crash a router though everyone on the network may suffer. Add many heavy hitters and the router's design goals can be exceeded. This is when bad things happen. Add things like hosts that generate random MAC addresses every time they connect add to the adventure. On little networks with only a few users one may not have issues, yet add lots of horsts and little routers like ours can be overloaded. Adamm's statement "I'd wipe your blacklist and start from there with 0 blocked IP's." is a great example of how tables can exceed design goals. Every feature we turn on and/or add-on we install increases demand for resources.

 

[Morris]

If swapping the router is easy, it's a great test and can avoid a lot of work if you are dealing with a hardware issue. Since the firmware the routers can run is deferent, it's not a perfect test yet most people seem to have a stable environment with the recent Merlin firmware so I encourage the router swap.

Good luck!

 

[ika]

I wrote a reply here and somehow I deleted(?) it (not sure how that happened, I was trying to post in a different thread). I won't write it again, but I will do an update on the weekend, I won't give up

 

[ika]

Update:

So after many hours of testing, disabling DNSSEC, DNS director, and all kind of settings in the router, and also in the torrent client. Settings like local peer discovery (creates lots of multicast packets), DHT, and everything you name it, I'm a little bit closer to the cause perhaps. I managed to make the router crash without Skynet, so I'm glad I did not blame that awesome addon at the first place, I knew it was innocent, it just makes the problem a lot more apparent.

I also managed to observe that the router crashes after 10 minutes from the moment I quit the torrent client on the PC.
I have a note here on my desk that once it crashed when I was just letting it run, but I cannot replicate that scenario anymore sadly. I have to quit the client to make it crash, and it will crash after 10 minutes.

Any idea or suggestion where to go from here?

ps.: This not happening with the ac86u

Thank you very much in advance
ika

 

[Morris]

Update:

So after many hours of testing, disabling DNSSEC, DNS director, and all kind of settings in the router, and also in the torrent client. Settings like local peer discovery (creates lots of multicast packets), DHT, and everything you name it, I'm a little bit closer to the cause perhaps. I managed to make the router crash without Skynet, so I'm glad I did not blame that awesome addon at the first place, I knew it was innocent, it just makes the problem a lot more apparent.

I also managed to observe that the router crashes after 10 minutes from the moment I quit the torrent client on the PC.
I have a note here on my desk that once it crashed when I was just letting it run, but I cannot replicate that scenario anymore sadly. I have to quit the client to make it crash, and it will crash after 10 minutes.

Any idea or suggestion where to go from here?

ps.: This not happening with the ac86u

Thank you very much in advance
ika

Try a deferent torrent app or none

 

[ika]

Try a deferent torrent app or none

- Hey doc, it really hurts to pee after I use the bike. What's wrong?
- I have no clue and also not interested in finding out, just don't use the bike.

 

[Tech9]

Any idea or suggestion where to go from here?

I would test with stock Asuswrt as well.

 

[ika]

I would test with stock Asuswrt as well.

Thank you for the reply and the suggestion. Yes, that is something what I have on the list, and will test.

 

[ika]

Update:

I reset the router today, and also reflashed Merlin 3004.388.4. Userscipts were not enabled, nor did I enable VPN, TrendMicro, SMB or any of the built in apps, it was as simple as possible (even left DOT disabled).

The issue is still happening; exactly 10 minutes after I quit the qBittorrent client on the PC, the router chrases.
During the testing, there was nothing in the logs, nor did I see anything using top (htop or other utilities were not available because of the lack of Entware)

I was trying to take a quick search in the source code, what could have a 10 minutes time interval set, but did not find anything (did not look for long).
I'm guessing that after 10 minutes, the router is trying to close zombie "things" in some tables and it messes itself up doing that on the gt-ax6000, but this is only a baseless guess at this moment.

The next three thing I will try (whenever I will have time):
- Force client PC to 1Gbps instead of 2.5Gbps and test
- Replace client PC (install and move the torrents to a different PC) and test
- Reflash latest Asuswrt instead of Asuswrt-Merlin and test

 

[Tech9]

Test with Asuswrt and report your findings in Feedback or Asuswrt-Merlin release thread depending on the results.