Port Forwarding and Skynet firewall ruleset

[Phantomski]

Hi everyone,

playing a bit with log management / analysis and potential feedback loop to live firewall rules. Call it a poor man tinkerer IDS/IPS
I have a fairly simple setup with RT-AC88U (with Skynet) using port forwarding rules on ports 80 and 443, redirecting to a Docker-ised rPi4 running a reverse proxy container, which then splits my subdomains to various HTTP(S) server containers running various services.

The idea is that I basically look at my webserver logs for fingerprints of malicious activity (WordPress and Router GUI scans are fairly common and easily identified for example) and based on the severity I'd then add an IP ban to the Skynet rules on my router, potentially report the offender to OTX like AlienVault, etc.

Anyway.

Now one thing left me head scratching a bit, as I could not find a definite and concise documentation for a standardised way of doing these things and that's the order of firewall rules.
Looking into my current (more or less factory default + OVPN and few scripts) iptables/IPSet it seems to be the case, that the firewall rules take the precedence over the port forwarding rules. Indeed if I ban an IP, the packets get dropped by firewall first, before having a chance to get forwarded to my local webserver, unlike in the case of let's say DMZ setup. However, that seems to be entirely by chance (uncomfortable feeling) order in iptables, rather than a general rule or a design practice that would apply to routers/firewalls. In other words if it wasn't for Skynet, the port forwarding would happen before the DROP, quite obviously as there's no per-IP IPSet filtering added on top. And if Skynet wasn't a Skynet, one could easily make a mistake of port forwarding first, firewall second. Ouch.

So here comes a question (tada) - is this a standard design practice and I'm just missing something obvious, or was Adamm00 just diligent enough to make it a robust universal solution (inbound is simple, but outbound and special rules are not) that kinda works well because it's always nr.1 rule in the flow? Banning lots of wide IP ranges outright certainly adds to a peace of mind, just want to make sure it is robust enough.

Cheers.

 

[eibgrad]

And if Skynet wasn't a Skynet, one could easily make a mistake of port forwarding first, firewall second. Ouch.

Port forwarding is implemented in the firewall, since by default, unsolicited incoming connections are denied on the WAN. The port forward is, essentially, an firewall exception applied to the WAN, that includes a redirection. You can also limit access to a port forward itself using the Source field.

Given the above, I don't know what you mean about it being a mistake to port forward first, firewall second. They are intimately related. You typically have to evaluate ALLOW rules first, then followup with DENY rules. That's exactly what happens w/ port forwarding.

Now if you decide to preempt that process w/ another process (e.g., Skynet), one that chooses to block access FIRST, that's just a matter of personal preference by either the author or user.

Also, the DMZ is nothing more than the option to bypass the WAN's firewall for anything that would otherwise be blocked or port forwarded, and passed to another device, presumably one w/ its own firewall (e.g., another router). Given its intent, it only makes sense to have the DMZ evaluated *after* port forwarding, but *before* the firewall attempts to block it.

IOW, as far as I can tell, putting Skynet aside for the moment, everything works as intended. But again, once you introduce another process like Skynet, all bets are off. The author or user are free to apply their own changes as they sit fit, for better or worse.

 

[Phantomski]

Given the above, I don't know what you mean about it being a mistake to port forward first, firewall second. They are intimately related. You typically have to evaluate ALLOW rules first, then followup with DENY rules. That's exactly what happens w/ port forwarding.

Thanks very much for the great writeup.

Mistake was a bad choice of words. What I was trying to say, that in non-standard case of extended IP address based ruleset or when using a process like Skynet, an author might easily make a mistake of introducing this step in iptables after port forwarding, which would obviously let all the forwarded packets to pass through unfiltered. Now in this case it doesn't - and that makes things a lot simpler. I guess I'm constantly pleasantly surprised by the skill of the people here designing great tools with capabilities well beyond the original router. Thank you all for that!

Now if you decide to preempt that process w/ another process (e.g., Skynet), one that chooses to block access FIRST, that's just a matter of personal preference by either the author or user.

That's what I was afraid of. Not that it is a massive issue, it just makes any firewall change a little bit more time consuming to crosscheck and evaluate. As you'd do anyway of course, just an extra step to think through carefully I was almost ignoring before.

IOW, as far as I can tell, putting Skynet aside for the moment, everything works as intended. But again, once you introduce another process like Skynet, all bets are off. The author or user are free to apply their own changes as they sit fit, for better or worse.

Yes, you're right, it does. From the user perspective, I'd expect everything to be "protected" behind the firewall, port forwarding or not. Skynet adds another layer that works just like that and for that I am grateful (phew). Kudos to adamm. But I of course understand that on the more fundamental basic level, port forwarding is just a basic ALLOW and I as an admin should deal with the consequences downstream and not expect magic where magic is not due.

Thanks for the insight. I'm in the middle of a yes/no/worth it thought process to split my current router into component parts in the future (PFSense, L3 switch, individual APs) and the more I look into it, more I realise the drawing board is really necessary The simplicity of one Asus, despite the limitations, thanks to Merlin and others makes it really hard to abandon completely

 

[Maverickcdn]

My understanding is port forward rules are processed first in the WAN (nat) side of IPTABLES, but then traffic is also processed by the LAN side (FORWARD,INPUT, OUTPUT chains).

If you have a port forward, the WAN side tables will allow it but it still traverses the LAN side of IPTABLES, so if you have a rule blocking traffic to your web server in the FORWARD chain it will enter the WAN but gets dropped before being forwarded to your web server if its matching a rule in the FORWARD chain.

You should look at running something like Fail2ban on your webserver to handle blocking/traffic scanning etc. Its far more suited to identifying bot/probe/malicious traffic (especially wordpress).

I have it setup on my webserver where if an IP is a persistent abuser I have a script which permanently bans them by adding a DROP rule in the FORWARD table of the router so traffic never makes it to the web server.

A script could be written to push IPs found in Fail2ban on the web server to manually add them to Skynets manual blacklist on the router automating the whole process. My Fail2ban blocks about 10-12 IPs/day and my automated script adds about 4-5 IPs/month to the FORWARD chain on the router firewall for a permanent ban. I dont use Skynet but I assume this would be feasible, if not you can duplicate my process of simply adding a IPTABLES rules to firewall-start which would accomplish the same.

This is a rather convoluted setup and is not for the faint of heart as there is lots of configuration involved on both the web server and the router

 

[Phantomski]

Thanks for insight and tips @Maverickcdn, will check Fail2ban out.

Looking at the traffic manually at the moment, my number of bans would be roughly 5-10 daily and that's me not even being on a fixed IP and Skynet blocking plenty of countries altogether. I guess some countries / cloud service providers really don't care what customers do as long as they pay.

 

[dave14305]

You might want to familiarize yourself with the flow of packets through the netfilter tables.

https://stuffphilwrites.com/wp-content/uploads/2014/09/FW-IDS-iptables-Flowchart-v2019-04-30-1.png

Skynet works in the RAW table, while port forwarding starts in NAT, and then other tables, etc.

 

[Phantomski]

Thanks @dave14305, the pic leads to plenty of great stuff. I have lots to learn
https://www.digitalocean.com/commun...ables-and-netfilter-architecture#introduction
https://lewestech.com/mirrors/www.iptables.info/en/structure-of-iptables.html
https://www.sans.org/cyber-security...k-forensics-threat-hunting-incident-response/
https://www.sans.org/uk/

 

[Maverickcdn]

I think once you see/understand Fail2ban you'll like it for what you're after... heres my stats since Jan 2022

Jail - dropbear
Jail - nginx-auth
Jail - nginx-badbots
|  |- Total failed:     508
   |- Currently banned: 55
Jail - nginx-botsearch
Jail - nginx-debug
|  |- Total failed:     3
Jail - nginx-errorlog
Jail - nginx-hp4
Jail - nginx-login
Jail - nginx-noscript
|  |- Total failed:     390
   |- Currently banned: 48
Jail - nginx-proxy
|  |- Total failed:     11
Jail - nginx-status400
|  |- Currently failed: 56
|  |- Total failed:     801
   |- Currently banned: 20
Jail - nginx-status404
|  |- Currently failed: 50
|  |- Total failed:     294
   |- Currently banned: 3
Jail - nginx-wordadmin
Jail - nginx-wordpress
|  |- Currently failed: 25
|  |- Total failed:     140
   |- Currently banned: 1
Jail - nginx-x00
|  |- Currently failed: 17
|  |- Total failed:     195
   |- Currently banned: 3
Jail - obfsproxy
|  |- Currently failed: 4
|  |- Total failed:     1630
Jail - php-url-fopen
Jail - sshd
Jail - manualipban
Jail - permabanned
   |- Currently banned: 68

And a sample of a user created filter list I use... slightly overkill for my needs but it works great.
Fail2ban wordpress filter