DOSed by ssh connection attempts. what to do

[finite9]

I regularly see ssh connection attempts to my home server in my server logs and have done for years. I used to use denyhosts which worked well. It's never been an issue before on 24mbit adsl with a pfsense router.

Since I got fibre, and an Asus RT-AC68U and put Merlin fw on it, there have been a couple of incidents where my router has hung, so I cannot access it when on the LAN and had no internet access, and ive had to reboot the asus, but I can see the traffic light on WAN device and LAN port of the router going berserk.

When I log dropped packets in the Asus router, I see lots of entries for ssh from various IP addresses. At the moment, I cannot send magic packet to my server from work and im assuming it could be because the traffic to the router seems to be so high that my packets arent getting through. This morning when I did get through, the ssh console was very slow to respond.

Is there a realt-time way to monitor what's going on? Can you get nethogs for Entware (which I have installed) or nettop to see what's going on?

And what is the issue? Is the fw unable to handle the amount of [attack] requests, or has upgrading to 500mbit synchronous fibre amplified the issue?

 

[RMerlin]

You can enable the SSH brute force protection on the System page, which will throttle connection attempts. The DOS protection on the Firewall page will also take care of throttling other types of packets such as connection attempts and pings.

 

[finite9]

i had both settings enabled already.

this is what my log looks like for today with dropped packet logging:
http://pastebin.com/NZKPhxp1

 

[RMerlin]

I see only one connection dropped per second as the brute force protection does its job, which won't have any performance impact on your router. Note that enabling logging of dropped packets WILL have an impact however.

If you are still worried about security, disable password authentication, and switch to a keypair for logging into your SSH server.

 

[finite9]

aha, is that how the brute force protection works? one a second. Ok looks like it is working then--I did sort of notice that there weren't _that_ many dropped connections. When my router hung the first time, I did suspect that maybe the log had grown massive or had some other negative effect, so after reboot I disabled logging of dropped packets. I only enabled it now to demonstrate what it looks like, but I experienced the slow ssh session with logging dropped packets disabled. I agree that it does not look like it's that big of an attack that would crash my router or slow down a remote ssh connection from work... maybe it is something else causing the hangs/slowdowns. I did not notice this since before installing 380.57 so I will have to do further digging.

My instinct was that a) router is hung and I have no Internet connection. b) the LAN1 on the router (connected to fibre box) is going crazy and WAN LED on fibre box is also blinking it's heart out, so I concluded that something was flooding my router.

Thanks for the tip, but I already use key pairs for ssh, not password auth.

 

[martinr]

....
If you are still worried about security, disable password authentication, and switch to a keypair for logging into your SSH server.

And would you say move the ssh port from 22 (via the gui)? I've seen heated arguments on the Internet about this. Nevertheless, I did move mine from 22 to a 5-figure port number and see no attempts to connect.

 

[finite9]

ive always been in the camp of "secure through obscurity is the wrong road to follow", but then again, even though I trust the security of my Fedora server, it's a non-trivial problem to have a flooded router (if thats the issue at all), so I might change the port. i always thought that attackers just did port scans/port knocking with nmap so changing port would be irrelevant, but maybe they all just try 22 by default. who knows.

found nethogs in the entware repos. top line shows im getting15KB/s on LAN from my Fedora server which is serving Plex to the Internet:


NetHogs version 0.8.1-SNAPSHOT

PID USER PROGRAM DEV SENT RECEIVED
? admin ..372-192.168.1.109:32400 0.000 15.595 KB/sec
? admin ..0-212.181.171.172:46795 0.000 1.087 KB/sec
? admin ..795-192.168.1.109:32400 0.000 0.530 KB/sec
? admin ..770-192.168.1.109:32400 0.000 0.513 KB/sec
26730 admin dropbear br0 2.907 0.090 KB/sec
481 admin networkmap br0 0.034 0.048 KB/sec
? admin ..:32858-52.24.198.95:443 0.000 0.013 KB/sec
? admin ..443-192.168.1.109:32858 0.000 0.013 KB/sec
? admin ..927-54.235.134.171:6861 0.000 0.000 KB/sec
? admin ..34504-83.233.164.156:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.109:34504 0.000 0.000 KB/sec
? admin ..50754-83.233.164.143:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.109:50754 0.000 0.000 KB/sec
? admin ..443-192.168.1.219:54719 0.000 0.000 KB/sec
? admin ..1587-82.209.181.253:443 0.000 0.000 KB/sec
? admin ..:38210-52.27.138.29:443 0.000 0.000 KB/sec
? admin ..443-192.168.1.109:38210 0.000 0.000 KB/sec
? admin ..:35424-192.168.1.141:80 0.000 0.000 KB/sec
? admin ..4719-82.209.181.242:443 0.000 0.000 KB/sec
? admin ..50048-83.233.164.154:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.109:50048 0.000 0.000 KB/sec
? admin ..46304-83.233.164.152:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.109:46304 0.000 0.000 KB/sec
? admin ..54214-83.233.164.144:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.233:54214 0.000 0.000 KB/sec
? admin ..54491-83.233.164.146:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.233:54491 0.000 0.000 KB/sec
? admin ..55062-83.233.164.148:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.109:55062 0.000 0.000 KB/sec
? admin ..54320-83.233.164.145:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.109:54320 0.000 0.000 KB/sec
? admin ..39673-83.233.164.149:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.233:39673 0.000 0.000 KB/sec
? admin ..45106-83.233.164.153:80 0.000 0.000 KB/sec
? admin ..:80-192.168.1.233:45106 0.000 0.000 KB/sec
? admin unknown TCP 0.000 0.000 KB/sec

TOTAL 2.941 17.889 KB/sec

 

[RMerlin]

And would you say move the ssh port from 22 (via the gui)? I've seen heated arguments on the Internet about this. Nevertheless, I did move mine from 22 to a 5-figure port number and see no attempts to connect.

The purists will dislike having a service hosted on an ephemeral port, but that's something I've done myself quite frequently when exposing a SSH server to the Internet. It resolves 99% of potential security issues - only if someone were to specifically target YOU and went through the trouble of a full range portscan could they become a problem.

 

[Nullity]

Aren't passwords just "security through obscurity"? Brute-forcing only takes time.

After changing from port 22, I had zero malicious connection attempts, compared to the hundreds of attempts on the default port.

 

[cockytrumpet]

Aren't passwords just "security through obscurity"? Brute-forcing only takes time.

If my math is right, one guess per second with only mixed case alphanumeric would take 6,923,519 years to 8 characters. So on average about half that.

Nothing wrong with changing the port though.

 

[Nullity]

If my math is right, one guess per second with only mixed case alphanumeric would take 6,923,519 years to 8 characters. So on average about half that.

Nothing wrong with changing the port though.

Exactly (except word-lists are the norm for mass scans), which was the point I was trying to make. "Security through obscurity" is a type of security which can decrease your chances of being infiltrated.

 

[sfx2000]

Well - get used to folks rattling the doorknob on Port 22 (and many other ports) - don't expose the ssh daemon on the router to the outside world - dropbear is good, but the router itself will eventually run out of memory with the default config on that daemon...

Thing is, folks running small devices - the recommendation perhaps is to implement some IPTables rulesets, which just moves the problem from the ssh application (dropbear) to the kernel - and then the kernel crashes as it runs out of memory...

One can rate limit on the application level with OpenSSH if you're port forwarding - and then you can protect that host will tools like Fail2Ban...

on OpenSSH - go to /etc/ssh/sshd_config - check/modify the below

Protocol 2
LoginGraceTime 30
PermitRootLogin no
StrictModes yes
MaxAuthTries 3
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
Wostbasedauthentication no
PermitEmptyPasswords no
MaxStartups 3:50:10

In fail2Ban... enable SSH filtering, and in jail.local...

bantime = 7200 # or some other value, all in seconds
findtime = 300
maxretry = 3

Whether it's a VM or a RaspPI - good ways to put the ssh doorknockers away - and again, don't expose the Router's sshd to the public internet - just doesn't make sense...

 

[cockytrumpet]

I don't expose dropbear, but if I needed to, fwknop or a similar implementation would be my choice.

 

[sfx2000]

The purists will dislike having a service hosted on an ephemeral port, but that's something I've done myself quite frequently when exposing a SSH server to the Internet. It resolves 99% of potential security issues - only if someone were to specifically target YOU and went through the trouble of a full range portscan could they become a problem.

I'm likely not a purist, but just setting a secure key transaction is more than enough for most folks... and dropbear supports keys...

Dropbear and IPTables - they _might_ get overwhelmed, but I highly doubt that on a residential connection - seriously...

 

[Nullity]

I'm likely not a purist, but just setting a secure key transaction is more than enough for most folks... and dropbear supports keys...

Dropbear and IPTables - they _might_ get overwhelmed, but I highly doubt that on a residential connection - seriously...

Even during my more adversarial days, my ~200Mhz ~128Mbyte RAM PC withstood any layer-7 ddos with ease (though my ADSL link did not), including SSH, HTTP, IDENTd, etc.

Although link-rates are climbing, an application ddos seems unlikely on residential connections.

 

[RMerlin]

Exactly (except word-lists are the norm for mass scans), which was the point I was trying to make. "Security through obscurity" is a type of security which can decrease your chances of being infiltrated.

The "security through obscurity" mantra applies if you rely on it to PROVIDE security. Moving your SSH port isn't the same thing - you are merely moving it outside of the path taken by all worms and scanning bots to reduce its visibility. Your security remains a safe password/key pair.

Kinda like not leaving your safe on the front porch, but moving it inside in a closet. The actual security is the safe's lock, not the fact it's hidden out of view to reduce temptation.

 

[RMerlin]

I'm likely not a purist, but just setting a secure key transaction is more than enough for most folks... and dropbear supports keys...

Securing the login will secure things, but it won't prevent worms/bots from potentially spending a lot of time hammering it with brute force attempts, slowing things down, and filling up log. That is the rationale behind moving SSH to a non-standard port.

Dropbear and IPTables - they _might_ get overwhelmed, but I highly doubt that on a residential connection - seriously...

DDoS through a bot farm all hitting your SSH port from different IP will bring down your bandwidth to its knees.

 

[Nullity]

DDoS through a bot farm all hitting your SSH port from different IP will bring down your bandwidth to its knees.

This is possible, but is is likely?

In my limited experience, if distributed bots are attacking a certain layer-7 service, they are trying to brute-force, rather than DDoS.

DoS is easy when attacking consumer-grade bandwidth. You (RMerlin) could probably bandiwdth DoS me right now.

 

[kvic]

If people have VPN server running on their router, I don't see the need for them to also open up ports e.g. to WebUI and SSH. Securing one door is way easier than a few more..

Dropped packets can also be rate limited in iptables to prevent swamping the log. My router is harvesting 3MiB everyday. No longer interested in looking at the content on a daily basis..

 

[finite9]

I think I changed my mind about keeping ssh on port 22 I think i'll move it. But just to clarify, I do not allow ssh to the _router_ from the internet and do not allow GUI access either. I NAT port 22 in the router config to my home server, and connect to the sshd on the server. (But I configure Putty at work for tunneling so that I can set the proxy in Firefox once i've got an ssh session to my server and access the router gui and local LAN resources through Firefox as if I was on my local LAN).

In the past I used denyhosts on the server to stop attempts from a single IP address after 3 failures. This worked very well and my hosts.deny file was quite large, but these days it is not available/does not work/I have not figured out what the issue is yet. I blame it all on systemd cause it all worked great before every one migrated to systemd.

But maybe I should just allow all the ssh traffic to my server without restrictions in the router? and then let Fedora handle the ssh attacks? Because then the router won't be spending resources on rate limiting with the brute force protections; it will just be packet shifting which requires less resources I suppose. My first thought when I saw the brute force protection was that I would be keeping my subnet clean of garbage packets if they got stopped at the door, but if the 'door' is getting all busted up in the process, then maybe it's better to just let them through? asuswrt-merlin FTW but maybe the router hardware/firmware core design has trouble handling the load.