But just to clarify, I do not allow ssh to the _router_ from the internet and do not allow GUI access either. I NAT port 22 in the router config to my home server, and connect to the sshd on the server.
That's port forwarding. I didn't read the whole thread before drop off of my previous post
You can have two layers of defense to secure against pokes on your sshd. On the router, set up ipset to allow certain networks (being it /16 or /24 or something else) to reach 22 or the new port you end up with (search ryzhov_al's examples on ipset on this forum). On the server itself follow sfx's suggestion for OpenSSH. With both you shall be pretty secure IMO.