One mirroring solution for IDS (my approach and solution)

[HELLO_wORLD]

Hello all,

As you may know if you read my recent posts in this forum, I was setting up an IDS system (Suricata), outside of the router (that is not strong enough to handle the IDS).

So what I needed to do was to send a copy of all the packets/frames I want to monitor to the IDS device.

I am skipping here the IDS system setup itself, and just precise that it has two ethernet interfaces: one for normal LAN interconnection, and the other strictly to receive the packets to process through the IDS.

Now, my first thought was naturally to simply use the port mirroring that the router offers (from the debug page). It works, but I noticed it impacted the performance of the router.
I found that not acceptable and decided to find another way.

After many trials, tests, experiments and all, I found what is working the best for me:

What is mirrored?
What is mirrored is all the IPv4 and IPv6 packets that are transiting through WAN, and reaching either the LAN (forwarded/routed packets), or the router itself (user space for local process), both ways.
Anything that is discarded by the firewall before it reaches either the router or the LAN is not mirrored (it can be set up differently, but I just want to see what made it through my firewall and is still a threat).

How is it mirrored?
Well, I use two techniques. The result of the two techniques are that the mirrored frames are identical to the original ones except the ethernet header on which the destination and source MAC are changed. I don't care about that as the IDS is working on the layers above ethernet.

First, to mirror the packets forwarded (LAN <-> WAN) by the router, and the ones going to the router itself (WAN -> router) I use iptables rules this way:

/usr/sbin/iptables -t filter -w -N bolemo_ids
/usr/sbin/iptables -t filter -w -A bolemo_ids -m mark --mark 0x01 -j RETURN
/usr/sbin/iptables -t filter -w -A bolemo_ids -j MARK --set-mark 0x01
/usr/sbin/iptables -t filter -w -A bolemo_ids -j TEE --gateway IDS-IP

/usr/sbin/iptables -t filter -w -I FORWARD -o brwan -j bolemo_ids
/usr/sbin/iptables -t filter -w -I FORWARD -i brwan -j bolemo_ids
/usr/sbin/iptables -t filter -w -I INPUT -i brwan -j bolemo_ids

So here, I first create a chain (bolemo_ids) with 3 rules, the first one is to ignore packets marked with 0x01; the second to mark the packets with 0x01, and the third one to tee (mirror) the packets to the IDS device with the IP "IDS-IP"

Same can be done for IPv6:

/usr/sbin/ip6tables -t filter -w -N bolemo_ids
/usr/sbin/ip6tables -t filter -w -A bolemo_ids -m mark --mark 0x01 -j RETURN
/usr/sbin/ip6tables -t filter -w -A bolemo_ids -j MARK --set-mark 0x01
/usr/sbin/ip6tables -t filter -w -A bolemo_ids -j TEE --gateway IDS-LL

/usr/sbin/ip6tables -t filter -w -I FORWARD -o brwan -j bolemo_ids
/usr/sbin/ip6tables -t filter -w -I FORWARD -i brwan -j bolemo_ids
/usr/sbin/ip6tables -t filter -w -I INPUT -i brwan -j bolemo_ids

Same exact thing but mirrored to the IPv6 link local of the IDS; "IDS-LL" (can also be the IPv6 global address if there is on set).

The big plus of this technique, is that first it uses the hardware acceleration (little to no impact on performances), and second it mirrors the routed packets from the FORWARD table, meaning the source or destination in the LAN is known (very useful for IDS monitoring to know which device is involved in sending or receiving a suspicious packet).

Now, we have this : LAN <-> WAN and WAN -> router
We are missing the WAN <- router

One could just add a rule in iptables and ip6tables in the OUTPUT chain to tee the packets…
That would work, but it would be missing any packets that would not go through iptables (in particular locally generated raw socket packets).

So my solution is to use tc this way:

/sbin/insmod /lib/modules/3.4.103/act_pedit.ko
/sbin/insmod /lib/modules/3.4.103/act_skbedit.ko
/sbin/insmod /lib/modules/3.4.103/act_mirred.ko
/sbin/insmod /lib/modules/3.4.103/act_gact.ko
/sbin/insmod /lib/modules/3.4.103/cls_u32.ko

/usr/sbin/tc qdisc add dev ifb0 handle 1: root prio
/usr/sbin/tc filter add dev ifb0 parent 1: prio 2 protocol all u32 match u32 0 0 \
  action pedit munge offset -14 u16 set 0x0011 munge offset -12 u16 set 0x2233 munge offset -10 u16 set 0x4455 pipe \
  action mirred egress redirect dev ethlan
/usr/sbin/ip l s dev ifb0 up

/usr/sbin/tc qdisc add dev ethwan handle 1: root prio
/usr/sbin/tc filter add dev ethwan parent 1: prio 1 protocol all u32 match mark 0x1 0xFFF action pass
/usr/sbin/tc filter add dev ethwan parent 1: prio 2 protocol arp u32 match u32 0 0 action pass
/usr/sbin/tc filter add dev ethwan parent 1: prio 3 protocol all u32 match u32 0 0 action skbedit mark 0x1 pipe action mirred egress mirror dev ifb0

So here, I first enable a few kernel modules (not all are natively available in the firmware, it requires for some of them to be compiled in it first).

Then I create a filter for the device ifb0 that first changes the MAC destination (on ether layer) for all the packets to the one of the IDS device (assuming here it is 00:11:22:33:44:55), then it redirect the packets to the interface ethlan (since the IDS is plugged on one of the LAN ports of the router).

And finally, I create a chain of filters for the interface ethwan (wan interface): any packets about to leave the router and 1) marked with 0x1 are ignored; 2) then any ARP packets are ignored; 3) then the remaining packets are marked with 0x1, then mirrored to ifb0.

And now, I do have this being mirrored: LAN <-> WAN and WAN <-> router, including raw socket packets
All with hardware acceleration, and information about LAN devices involved.

PS/EDIT: to come to that solution, I first asked questions in other threads, here: IP source and destination not in LAN or public ip… and here: Need help/advice with advanced networking interfaces needs

 

[kamoj]

This is very interesting!
Thank you very much for your postings on this subject.
I like how you have described the object and your way forward.
Very educating!

 

[sfx2000]

keep a copy of this handy...

https://people.netfilter.org/pablo/netdev0.1/slides/IPQ806x-Hardware-acceleration_v2.pdf

 

[HELLO_wORLD]

keep a copy of this handy...

https://people.netfilter.org/pablo/netdev0.1/slides/IPQ806x-Hardware-acceleration_v2.pdf

This is very cool

 

[sfx2000]

Couple of things...

1) please don't fragment the thread - you had good stuff in the other, and by splitting it, others will lose context of posts...

You can ask @RMerlin or @thiggins to merge them

2) as I said before - think frames, my friend, not packets - keep it in the switch, lots of bandwidth there, much more than what you have on the IPQ80xx GMAC - copy the frames, move them

In OpenWRT speak - this is done with a couple of checkboxes.... (in the example, port0 is WAN, port 1 is LAN1)

root@thumper:~# uci show | grep mirror
network.@switch[0].enable_mirror_rx='1'
network.@switch[0].enable_mirror_tx='1'
network.@switch[0].mirror_source_port='0'
network.@switch[0].mirror_monitor_port='1'

and done..

The path you are taking - yes, it's possible, it's just a lot of work for the kernel inside netfilter, and you'll be thrashing memory - at least that's how I'm reading your approach as every packet needs to go thru the chain and be evaluated for routing of those packets.

Not to mention that it could double bandwidth usage on the switch and GMAC's on the IPQ - performance likely won't be good, but that's just my opinion.

Good luck - I'm sure you're tired of hearing me rant on this...

 

[HELLO_wORLD]

Couple of things...

1) please don't fragment the thread - you had good stuff in the other, and by splitting it, others will lose context of posts...

You can ask @RMerlin or @thiggins to merge them

2) as I said before - think frames, my friend, not packets - keep it in the switch, lots of bandwidth there, much more than what you have on the IPQ80xx GMAC - copy the frames, move them

In OpenWRT speak - this is done with a couple of checkboxes.... (in the example, port0 is WAN, port 1 is LAN1)

root@thumper:~# uci show | grep mirror
network.@switch[0].enable_mirror_rx='1'
network.@switch[0].enable_mirror_tx='1'
network.@switch[0].mirror_source_port='0'
network.@switch[0].mirror_monitor_port='1'

and done..

The path you are taking - yes, it's possible, it's just a lot of work for the kernel inside netfilter, and you'll be thrashing memory - at least that's how I'm reading your approach as every packet needs to go thru the chain and be evaluated for routing of those packets.

Not to mention that it could double bandwidth usage on the switch and GMAC's on the IPQ - performance likely won't be good, but that's just my opinion.

Good luck - I'm sure you're tired of hearing me rant on this...

1) About fragmenting the posts, I try to keep on topic.

In the other one, I was asking a question. Here, I post my working solution.
Sure, both are related, and if a moderator decides it should be merged, it is ok by me.


2) I 90% agree with you, but unfortunately, the Netgear firmware on which @Voxel's one is based does not give much access to the switch:
Not much control by uci or nvram over its settings, particularly mirroring. Only mirroring setting is the same as from debug page (mirror WAN to port 1 LAN), and it is very inefficient (performances are impacted). Other settings are similar to what can be seen in swconfig, and the version in the Netgear firmware is very limited (no mirroring functions).
I have seen with @Voxel about compiling a newer version of swconfig into his firmware, but this would be extremely complicated and difficult as we are stuck with an old kernel (3.4) because of proprietary Netgear binaries (the ones allowing hardware acceleration), while newer swconfig relies on a more recent kernel.
This is why OpenWRT is not an option, because it is not accelerated for R7800.

And I don't see any other way to work at the switch level in my situation, it is not that I don't want, but that I simply cannot.

Now, why 90%
Because going through iptables allows a) to know about the routing decision and b) mirror only the significant packets.


Remarks:
As I see it, all the packets being mirrored are already going through netfilter, as they go through the firewall rules (from NG and custom ones), and this with or without the TEE rule.
I don't notice any effect on performances: I can run a speedtest (from LAN) and reach full gigabit both ways, not see much of an impact on the CPU (proving it is going through NSS), and see the packets being mirrored on the IDS. I don't see either an impact on memory of latency.

Netdata info (test ran between 12:54:30 and 12:55:00:

And the speedtest:

root@hestia:~# speedtest -s 2231

   Speedtest by Ookla

     Server: LaFibre.info BBR - Massy (id = 2231)
        ISP: K-NET SARL
    Latency:    31.33 ms   (0.14 ms jitter)
   Download:   974.50 Mbps (data used: 1.2 GB )                               
     Upload:   936.46 Mbps (data used: 1.1 GB )                               
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/35b8c683-053e-47c0-995e-b3b3ed7825d5

Now, about you "ranting", I am not tired at all. I think it is a very interesting discussion.