Firewall/Skynet Question

[mongodb]

Hi guys,

So starting approx. last Thursday I've noticed my Asus Merlin router become unresponsive for a few seconds before coming back to life. This has happened roughly 6-8 hours apart but hasn't been consistent. I have constant pings going to it and modem beyond Merlin; which both stopped responding at the same time. Luckily, I was at the desktop when it happened about an hour ago and as soon as Merlin was responsive again I looked under system logs to find tons of external port scans.

Now, Skynet is doing its job and thwarting these scans but I noticed 2-3 separate port scans, all from Digital Ocean's ASN, which makes be believe that the router gets inundated with blocking these scans and stops processing legit outbound traffic.

My question is- has anyone seen this happen to them and is there any thing I can do to help avoid Merlin falling over?

Thanks for looking.

 

[cptnoblivious]

First, skynet is really the thing that prevents _outbound_ traffic. Inbound traffic to non-forwarded ports doesn't need skynet.
So if you're concerned with inbound scans, you can disable skynet to see if that makes a different to the router processing your packets

Second, when you say port scans, how many packets inbound of what amount of time? I'd be surprised if a scan was so processing heavy for the router that it couldn't keep up. The biggest issue I could imagine is that it had a ton to write to the log files. So, I'd reduce logging (if turning off skynet isn't an option) if possible. Honestly I've never looked into this because I've never experienced any issues with it.

Hope you figure out what's causing your issues!

 

[mongodb]

Hi @cptnoblivious, thanks for taking the time. Cheers for providing insight into Skynet processing outbound traffic and not non-open inbound. This is indeed correct as the logs suggest these are being logged by the kernel and not Skynet itself.

You are bang on the money about logs though. I currently have Merlin set to "all" logs; I'll drop this down to "critical" and see how it behaves. I literally just had one an hour ago which was particularly at the wrong time (last few laps of the F1 race!!). In terms of how much scan traffic there is- I'm currently seeing about a dozen or so logs per second because of port scans from two different IP addresses.

I'll keep an eye on how Merlin behaves after dropping the logs down to critical only.

Thanks a bunch.

 

[mongodb]

Here's a summary of my findings so far:

- erratic pings from LAN to Merlin interface and vice versa reported (>200ms intermittently)
- suspected LAN cable from router to switch replaced; behaviour did not go away
- unplugged WAN interface on Merlin and pings came back down to normal levels (<1ms)
- reconnected WAN interface and pings went erratic again; confirming something happening on the router itself
- keeping WAN cable connected switched off Skynet and pings are now normal again (<1ms)

Not sure why but it appears that Skynet is chewing up CPU cycles on the router sometimes making is unable to forward traffic out.

Merlin is running 10/05/2021 - v7.2.5 which appears to be the latest version and nothing has changed from a config perspective on the router at all. I've given it multiple restarts between last night and now, and the problem always appears to come back.

Is it worth blowing away Skynet and re-installing it? Is there a way to backup Skynet config before removing it?

 

[ColinTaylor]

Not sure why but it appears that Skynet is chewing up CPU cycles on the router sometimes making is unable to forward traffic out.

I suspect it is because Skynet is performing large amounts of IO to the USB drive (probably logging) causing the processor to go into iowait states. The easy answer is to stop logging all this information, i.e. turn off Skynet. If you really need Skynet either reduce the amount of log writing that it does (I don't know if there's an option for that) or use a USB device with a very fast response time (e.g. an SSD not a flash drive).

 

[mongodb]

Hi @ColinTaylor, thanks. Yep, you are defo right in suggesting that. Here's what I did just now- re-enabled Skynet and noticed erratic ping times. Went into firewall settings and disabled logging completely:

[3] --> Logging | [Disabled]

So, end result is that Skynet is running minus the logging and pings are now normal (<1ms) again:

206 packets transmitted, 206 received, 0% packet loss, time 209920ms
rtt min/avg/max/mdev = 0.219/0.366/0.580/0.066 ms

I don't understand why this kicked off last week though. As Skynet blocks outbound connections only, I'm now a little worried for its need to write so much to the USB stick to start dropping legit packets.

 

[mongodb]

Here's a weekly view of CPU usage for the last week; notice the ramp up Thursday onwards:

 

[dave14305]

As Skynet blocks outbound connections only, I'm now a little worried for its need to write so much to the USB stick to start dropping legit packets.

Only if you configured Skynet to block only Outbound traffic. It wants to block both inbound and outbound by default.

 

[mongodb]

Only if you configured Skynet to block only Outbound traffic. It wants to block both inbound and outbound by default.

Thanks Dave. Apologies, you're right. I have config'd Skynet to block both in and out:

[4] --> Filter Traffic | [all]

This makes more sense and I have seen an increase in port scans from Digital Ocean's ASN recently; as mentioned in my original post.

 

[mongodb]

Follow-up question- does anyone know how Skynet implements DoS protection if enabled?

It's currently no enabled but I want understand what it does if I do choose to enable it.

 

[mongodb]

Nevermind, read up another thread where this was discussed not too long ago and RMerlin suggested enabling DoS protection but also noted that it is fairly useless against a DDoS attack which, in all fairness, is a different beast in itself.

 

[mongodb]

One thing to note for anyone reading this- when running "top" keep a close watch on the sirq values:

CPU: 1.9% usr 1.9% sys 0.0% nic 95.3% idle 0.0% io 0.0% irq 0.6% sirq

When running top before solving my issue, whilst the usr and sys values were very low the sirq value was reading circa 25~30%.