Need help/advice with advanced networking interfaces needs

[HELLO_wORLD]

Hello,

I hope everyone is having a great Sunday

I am looking for help from the master gurus out here.

My goal is simple: I want to be able to capture in iptables some packets generated locally by raw sockets to tee them to my IDS (I am able to tee everything coming through the router, to the router and from the router, except those few packets generated in the router via raw sockets like dhcpd does for example).

Raw socket packets are bypassing iptables, and can only be seen in the OUTPUT chain of ebtables.
Unfortunately, it is too late there to redirect it to iptables.

My initial solution was to force the packets to go through the BROUTING chain and to use the -j redirect --redirect-target DROP rule to force it to go through iptables.
So far, after a lot of experiments and failures, I was able to see the packet in BROUTING using veth interfaces, one in brwan, etc. But unfortunately, it appears to be a lot more complicated as this solution is changing the destination MAC address, and therefore the packets (DHCP are broadcast ones), hit iptables, but not anymore with the MAC of the veth in brwan instead of ff:ff:ff:ff:ff:ff. This is however normal behavior for the BROUTING redirect rule.

This way seems a lot too complicated, and now, I am thinking of a totally different route: programmatically catch all packets leaving the router with a specific mark.
tcpdump does not seem to allow this (it does not deal at all with the eth/ip stack where the marking is taking place).

I can do a program in C, using sockets or pcap, but is it possible to catch only asked packets?

Or I may go a totally different way again and use tc mirror to a virtual interface on which destination MAC is changed of the one of the IDS device…

 

[HELLO_wORLD]

Trying the tc way…

That should be enough to mirror traffic:

tc qdisc add dev IFACE_LISTENED handle ffff: ingress
tc filter add dev IFACE_LISTENED parent ffff: protocol all u32 match u32 0 0 action mirred egress mirror dev MIRROR_IFACE

But I got a RTNETLINK answers: No such file or directory error.

I was able to get the first line to work by enabling the ingress module:

insmod /lib/modules/3.4.103/sch_ingress.ko

But I am unable to get the second line to work, it constantly says:

RTNETLINK answers: No such file or directory
We have an error talking to the kernel

I tried to enable all sch_ modules available:

sch_teql 4493 0 - Live 0xbf191000
sch_sfq 8889 0 - Live 0xbf172000
sch_red 4691 0 - Live 0xbf160000
sch_htb 12832 0 - Live 0xbf159000
sch_hfsc 14429 0 - Live 0xbf152000
sch_gred 10535 0 - Live 0xbf14c000
sch_dsmark 4089 0 - Live 0xbf00a000
sch_codel 4683 0 - Live 0xbf142000
sch_ingress 1279 1 - Live 0xbf002000

But no change…

In fact, I get this error with any tc filter … command

I am probably missing a kernel module, but which one? Any clue?

 

[HELLO_wORLD]

Made some progress here… found the kernel modules to activate:
cls_u32 6343 1 - Live 0xbf147000
act_mirred 2966 0 - Live 0xbf002000

And no error

Now I will experiment and report my progress here (for posterity and helping anyone looking to do the same).

 

[HELLO_wORLD]

So, here is some feedback.

I am finally finding a solution, a little convoluted but that will likely do it.

I am keeping all my tee rules as is, and to catch the generated raw sockets, I will do this (here for ethwan):

1) mark the packets generated in the router (including using raw sockets) in ebtables:
ebtables -A OUTPUT -o ethwan -j mark --mark-set 0x1 --mark-target CONTINUE

2) mirror these packets
tc qdisc add dev ethwan handle 1: root prio
tc filter add dev ethwan parent 1: protocol all prio 99 u32 match mark 0x1 0xFFF action mirred egress mirror dev veth0

This is not using ingress, just root prio, so we mirror only outgoing packets (this is perfect as it is what we want).
Here I use veth0 - veth1 peered interfaces.

What I need to do know is to add veth1 into br0 and make a bridge rule to forward all incoming traffic in veth1 to the MAC of my IDS device (that is in br0 [ethlan]).

I will keep you posted…

 

[HELLO_wORLD]

It is becoming very complicated for a simple traffic mirroring…

So after this:
ebtables -A OUTPUT -o ethwan -j mark --mark-set 0x2 --mark-target CONTINUE
tc qdisc add dev ethwan handle 1: root prio
tc filter add dev ethwan parent 1: protocol all prio 99 u32 match mark 0x2 0xFFF action mirred egress mirror dev veth0

And then I add this to inject the packet to iptables (after adding veth1 into its own self bridge):
ebtables -t broute -A BROUTING -i veth1 -p IPv4 -j redirect --redirect-target DROP

Then:
iptables -t raw -I PREROUTING 1 -i veth1 -m comment --comment IDS -j TEE --gateway IDS-IP
iptables -t raw -I PREROUTING 2 -i veth1 -m comment --comment IDS -j DROP

And with all of that, I can see raw socket generated packets sent to brwan in my IDS…

But then I tried to do the same for ethlan, ath0 and ath1 (ebtables rules, tc rules to mirror to veth0), but it crashes the router… Probably too much for it or some king of looping happening…

So for now, I will give up as the raw socket packets are very few (mostly DHCP server and client on the router), and this is a lot of effort and ressources for just catching these.
The IDS is working quite well just with regular TEE rules in iptables. I might try again in the future when I have time for this.

Here I am also teeing the packets that were already mirrored, then dropping them… The tee is convenient to send the packets exactly where I want them.
Ideally, I would redirect them directly to the right place, but I was not successful with dnat rules either in ebtables for dest MAC or iptables for dest IP.

 

[Atko]

Possibly an unpopular opinon, but wouldn't it be easier to put a small managed SoHo switch, that has port mirroring capabilities, between the router and the internet? For example, a Netgear GS305E could be purchased new for under $40USD or maybe a TP-Link TL-SG108E which can be had even cheaper.

 

[HELLO_wORLD]

Possibly an unpopular opinon, but wouldn't it be easier to put a small managed SoHo switch, that has port mirroring capabilities, between the router and the internet? For example, a Netgear GS305E could be purchased new for under $40USD or maybe a TP-Link TL-SG108E which can be had even cheaper.

I don't see why this would be unpopular. It would work fine, I am sure.

There are two major schools: tapping the WAN or tapping the LAN.

I really like the tee solution, because it is hybrid: you can mirror from wherever you want in iptables, meaning after all the first DROP rules (Aegis for example), and after the routing decision.
This way, you know where the packets are going in the LAN (to which specific devices).

A tap before the router would get all the traffic between the router and internet (including what is blocked by the firewall), but you would not see where it would go (and where it would be coming from) in the LAN.

To finish, I like the challenge of this, and I learn a lot this way

 

[HELLO_wORLD]

Getting to an elegant solution, and learning a lot in the process

First, the required kernel modules, all to use with tc:
act_pedit.ko
act_mirred.ko
cls_u32.ko

Then I use the already existing ifb0 interface (kind of what it is for); arp and multicast is already off, but to be sure I added it to my commands.

Basically:
1) In ebtables, I mark the traffic generated on the router itself, about to leave from the interface ethwan and not already teed (mark 0x1) with the mark 0x2.
2) With tc, I mirror the marked (0x2) packets to ifb0.
3) With tc, to all the packets mirrored to ifb0, I change their destination MAC to the one of my IDS device (in this example, it would be 00:11:22:33:44:55).
4) Then I redirect all the packets to ethlan (it could be br0, but this way I skip the bridge).

ip l s ifb0 arp off
ip l s ifb0 multicast off
tc qdisc add dev ifb0 handle 1: root prio
tc filter add dev ifb0 parent 1: protocol all prio 1 u32 match u32 0 0 \
    action pedit munge offset -14 u16 set 0x0011 munge offset -12 u16 set 0x2233 munge offset -10 u16 set 0x4455 pipe \
    action mirred egress redirect dev ethlan
ip l s ifb0 up
ebtables -t nat -A OUTPUT -o ethwan --mark ! 0x1 -j mark --mark-set 0x2 --mark-target CONTINUE
tc qdisc add dev ethwan handle 1: root prio
tc filter add dev ethwan parent 1: protocol all prio 1 u32 match mark 0x2 0xFFF action mirred egress mirror dev ifb0

And I do see the packets in the IDS side

 

[Voxel]

Funnily. Please continue.

Voxel.

 

[HELLO_wORLD]

So…

Here where I am at:

Using these modules:

insmod /lib/modules/3.4.103/act_pedit.ko
insmod /lib/modules/3.4.103/act_skbedit.ko
insmod /lib/modules/3.4.103/act_mirred.ko
insmod /lib/modules/3.4.103/act_gact.ko
insmod /lib/modules/3.4.103/cls_u32.ko

Then setting the ifb0 tc qdisc and filter to change the MAC destination, then inject the packets into ethlan:

tc qdisc add dev ifb0 handle 1: root prio
tc filter add dev ifb0 parent 1: prio 2 protocol all u32 match u32 0 0 \
action pedit munge offset -14 u16 set 0x0011 munge offset -12 u16 set 0x2233 munge offset -10 u16 set 0x4455 pipe \
action mirred egress redirect dev ethlan

Then setting the mirroring for ethwan, ath0 and ath1, skipping ARP packets and the ones marked, and marking the ones being mirrored:

tc qdisc add dev ethwan handle 1: root prio
tc filter add dev ethwan parent 1: prio 1 protocol all u32 match mark 0x1 0xFFF action pass
tc filter add dev ethwan parent 1: prio 2 protocol arp u32 match u32 0 0 action pass
tc filter add dev ethwan parent 1: prio 3 protocol all u32 match u32 0 0 action skbedit mark 0x1 pipe action mirred egress mirror dev ifb0

tc qdisc add dev ath0 handle 1: root prio
tc filter add dev ath0 parent 1: prio 1 protocol all u32 match mark 0x1 0xFFF action pass
tc filter add dev ath0 parent 1: prio 2 protocol arp u32 match u32 0 0 action pass
tc filter add dev ath0 parent 1: prio 3 protocol all u32 match u32 0 0 action skbedit mark 0x1 pipe action mirred egress mirror dev ifb0

tc qdisc add dev ath1 handle 1: root prio
tc filter add dev ath1 parent 1: prio 1 protocol all u32 match mark 0x1 0xFFF action pass
tc filter add dev ath1 parent 1: prio 2 protocol arp u32 match u32 0 0 action pass
tc filter add dev ath1 parent 1: prio 3 protocol all u32 match u32 0 0 action skbedit mark 0x1 pipe action mirred egress mirror dev ifb0

All of that works very fine, but I cannot mirror anything coming from ethlan; using this, it crashes the router almost instantly:

tc qdisc add dev ethlan handle 1: root prio
tc filter add dev ethlan parent 1: prio 1 protocol all u32 match ether dst 00:11:22:33:44:55 action pass
tc filter add dev ethlan parent 1: prio 2 protocol all u32 match mark 0x1 0xFFF action pass
tc filter add dev ethlan parent 1: prio 3 protocol arp u32 match u32 0 0 action pass
tc filter add dev ethlan parent 1: prio 4 protocol ip u32 match u32 0 0 action skbedit mark 0x1 pipe action mirred egress mirror dev ifb0

If I mirror it to ifb1 for example (or any other nummy interface), it works (because not redirected to ethlan) to see what is being mirrored with tcpdump, I don't see anything that would cause a loop.
The packets I see are packets generated by the router and sent to the LAN (but not to the IDS device).

I start to think the bast way would be to add a physical ethernet port to the router… Anyone knows if USB adapters like an RTL8153 Gigabit Ethernet Adapter (usb ref 0bda:8153) or another one similar is supported by the router/firmware (@Voxel)?
I suppose there is no way from the command line to access the internal switch rules…

To be continued…

 

[sfx2000]

I suppose there is no way from the command line to access the internal switch rules…

swconfig list

swconfig dev switch0 show

swconfig dev switch0 help <-- shows interesting items

For example

# swconfig dev switch0 help
switch0: QCA HPPE(QCA HPPE), ports: 8 (cpu @ 0), vlans: 128
     --switch
    Attribute 1 (int): enable_vlan (Enable 8021q VLAN)
    Attribute 2 (int): max_frame_size (Set Max frame Size Of Mac)
    Attribute 3 (none): reset_mibs (Reset All MIB Counters)
    Attribute 4 (none): flush_arl (Flush All ARL table)
    Attribute 5 (string): dump_arl (Dump All ARL table)
    Attribute 6 (unknown): switch_ext (Switch extended configuration)
    Attribute 7 (none): apply (Activate changes in the hardware)
    Attribute 8 (none): reset (Reset the switch)
     --vlan
    Attribute 1 (int): vid (Configure Vlan Id)
    Attribute 2 (ports): ports (VLAN port mapping)
     --port
    Attribute 1 (none): reset_mib (Reset Mib Counters)
    Attribute 2 (string): mib (Get Mib Counters)
    Attribute 3 (int): enable_eee (Enable EEE)
    Attribute 4 (int): pvid (Primary VLAN ID)
    Attribute 5 (string): link (Get port link information)

Give those a try - you didn't mention which Netgear AC device you're working with, but QCA is built on QSDK, and swconfig should be there...

 

[HELLO_wORLD]

Thank you @sfx2000
I have a R7800, and I do have swconfig:

root@HERMES:~$ swconfig dev switch0 help
switch0: ethlan(QCA AR8327 AR8337), ports: 7 (cpu @ 0), vlans: 128
     --switch
    Attribute 1 (int): enable_vlan (Enable 8021q VLAN)
    Attribute 2 (int): max_frame_size (Set Max frame Size Of Mac)
    Attribute 3 (none): reset_mibs (Reset All MIB Counters)
    Attribute 4 (none): flush_arl (Flush All ARL table)
    Attribute 5 (string): dump_arl (Dump All ARL table)
    Attribute 6 (unknown): switch_ext (Switch extended configuration)
    Attribute 7 (none): apply (Activate changes in the hardware)
    Attribute 8 (none): reset (Reset the switch)
     --vlan
    Attribute 1 (int): vid (Configure Vlan Id)
    Attribute 2 (ports): ports (VLAN port mapping)
     --port
    Attribute 1 (none): reset_mib (Reset Mib Counters)
    Attribute 2 (string): mib (Get Mib Counters)
    Attribute 3 (int): pvid (Primary VLAN ID)
    Attribute 4 (string): link (Get port link information)

And:

root@HERMES:~$ swconfig dev ethlan show
Global attributes:
    enable_vlan: 1
    max_frame_size: 1518
    dump_arl: MAC: SWITCH1-LAN PORTMAP: 0x08 VID: 0x1 STATUS: 0x0
              MAC: DIRECT-IDS  PORTMAP: 0x10 VID: 0x1 STATUS: 0x0
              MAC: SWITCH1-LAN PORTMAP: 0x08 VID: 0x1 STATUS: 0x0
              MAC: DIRECT-WAN  PORTMAP: 0x20 VID: 0x2 STATUS: 0x0
              MAC: WIFI5G-LAN  PORTMAP: 0x40 VID: 0x1 STATUS: 0x0
              MAC: ETHLAN      PORTMAP: 0x40 VID: 0x1 STATUS: 0x0
              MAC: SWITCH2     PORTMAP: 0x04 VID: 0x1 STATUS: 0x0
              MAC: DIRECT-WAN  PORTMAP: 0x20 VID: 0x2 STATUS: 0x0
              MAC: WIFI2G-LAN  PORTMAP: 0x40 VID: 0x1 STATUS: 0x0
              MAC: SWITCH2-LAN PORTMAP: 0x04 VID: 0x1 STATUS: 0x0
              MAC: WIFI5G-LAN  PORTMAP: 0x40 VID: 0x1 STATUS: 0x0
              MAC: SWITCH1-LAN PORTMAP: 0x08 VID: 0x1 STATUS: 0x0
              MAC: SWITCH2-LAN PORTMAP: 0x04 VID: 0x1 STATUS: 0x0
              MAC: WIFI2G-LAN  PORTMAP: 0x40 VID: 0x1 STATUS: 0x0
              MAC: WIFI5G-LAN  PORTMAP: 0x40 VID: 0x1 STATUS: 0x0
              MAC: SWITCH1     PORTMAP: 0x08 VID: 0x1 STATUS: 0x0
              MAC: WIFI2G-LAN  PORTMAP: 0x40 VID: 0x1 STATUS: 0x0
              MAC: ETHWAN      PORTMAP: 0x01 VID: 0x2 STATUS: 0x0

    switch_ext: ???
    
Port 0:
    (…)
    pvid: 2
    link: port:0 link:up speed:1000baseT full-duplex txflow rxflow
    
Port 1:
    (…)
    pvid: 1
    link: port:1 link:down
    
Port 2:
    (…)
    pvid: 1
    link: port:2 link:up speed:1000baseT full-duplex txflow rxflow
    
Port 3:
    (…)
    pvid: 1
    link: port:3 link:up speed:1000baseT full-duplex txflow rxflow
    
Port 4:
    (…)
    pvid: 1
    link: port:4 link:up speed:1000baseT full-duplex txflow rxflow
    
Port 5:
    (…)
    pvid: 2
    link: port:5 link:up speed:1000baseT full-duplex txflow rxflow
    
Port 6:
    (…)
    pvid: 1
    link: port:6 link:up speed:1000baseT full-duplex txflow rxflow
    
VLAN 1:
    vid: 1
    ports: 1 2 3 4 6
    
VLAN 2:
    vid: 2
    ports: 0 5

So I deduct this:

VLAN 2 is dedicated to the WAN, and has:

• Port 0 has portmap 0x01 and is internal to the router (interface ethwan)
• Port 5 has portmap 0x20 and is most likely the physical WAN plug interface

VLAN 1 is dedicated to the LAN, and has (had to guess from stats):

• Port 1 (unused)
• Port 2 has portmap 0x04 and is linked to my switch #2
• Port 3 has portmap 0x08 and is linked to my switch #1
• Port 4 has portmap 0x10 and is linked to my IDS device
• Port 6 has portmap 0x40 and is internal to the router (interface ethlan)

Ideally, my goal is to:

1. create a VLAN 3, vid 3
2. put port 4 (portmap 0x10) into VLAN 3
3. have VLAN 3 matching some interface in the router, BUT there is no more internal port available (port 0 is used by ethwan, port 6 by ethlan)

So point 3 will likely be a problem…
However, I can observe what happens in swconfig output with

• Settings -> Advanced -> VLAN / Bridge Settings -> Enable VLAN / Bridge group
• Debug -> WAN Port mirror to LAN port1 (that is matching port4 in swconfig)

 

[HELLO_wORLD]

root@HERMES:~$ swconfig dev switch0 show
Global attributes:
    enable_vlan: 1
    max_frame_size: 1518
    dump_arl: (…)
MAC: IDS MAC PORTMAP: 0x10 VID: 0x3 STATUS: 0x0
(…)
Port 4:
(…)
    pvid: 3
    link: port:4 link:up speed:1000baseT full-duplex txflow rxflow
(…)
VLAN 1:
    vid: 1
    ports: 1 2 3 6
VLAN 2:
    vid: 2
    ports: 0 5
VLAN 3:
    vid: 3
    ports: 4

So I can put the port where my IDS is connected into its own VLAN, separated from ethwan and ethlan, but… I have no internal device to connect it to.
arp table does not show IDS MAC link anymore. So this might be a dead end.

 

[HELLO_wORLD]

Follow up:

The tc way is still causing problems with ethlan…

I tried to exclude all packets already mirrored (mark 0x1), exclude packets destined to the IDS itself (to prevent loop), prevent broadcast and multicast packets (to prevent loop), but this instantly crashes the router, while similar code is working perfectly fine to mirror ethwan, ath0 and ath1 this way:

tc qdisc add dev ethlan handle 1: root prio
tc filter add dev ethlan parent 1: prio 1 protocol all u32 match mark 0x1 0xFFF action pass
tc filter add dev ethlan parent 1: prio 2 protocol all u32 match ether dst IDS MAC action pass
tc filter add dev ethlan parent 1: prio 3 protocol all u32 match ether dst ff:ff:ff:ff:ff:ff action pass
tc filter add dev ethlan parent 1: prio 4 protocol all u32 \
    match u32 0x00000100 0x0000ffff at -16 \
    match u32 0x5e000000 0xff800000 at -12 \
    action pass
tc filter add dev ethlan parent 1: prio 5 protocol all u32 \
    match u32 0x00003333 0x0000ffff at -16 \
    action pass
tc filter add dev ethlan parent 1: prio 6 protocol ip u32 match u32 0 0 action skbedit mark 0x1 pipe action mirred egress mirror dev ifb0

Filter prio 4 is to bypass IPv4 multicast: dst MAC 01:00:5e:YX:XX:XX (Y being in bits 0XXX, and X being anything).
Filter prio 5 is to bypass IPv6 multicast: dst MAC 33:33:XX:XX:XX:XX (X being anything).

And to be sure I even limited the mirrored traffic to IPv4 only in prio 6.

But no can do.

The weirdest thing is if I only mirror something destined to a specific MAC on my LAN, it does not crash.
Also sending the mirroring to ifb1 for example (and ifb1 not redirecting to IDS), it does not crash, and studying the packets with tcpdump does not suggest any loop.

So I am definitely giving up with the tc way for now.

I decided to totally exclude packets generated in the router in my IDS.

There might be a solution with swconfig, but only if there is a way to compile a more advanced version allowing port mirroring.

The best solution would definitely to add a brand new port via USB, totally separated from ethlan, but I don't know if there are drivers for such USB/Ethernet interfaces.

 

[sfx2000]

The tc way is still causing problems with ethlan…

tc is not routing, it's not switching...

It's QoS, and with QSDK based devices, it directly interacts with QCA's nss subsystem - Don't waste your time here, as traffic can either go down the fast path, or the slow path...

"these are not the droids you are looking for"

netflow, implemented as softflowd is probably the better choice, but I'm not sure if that lives in the third party packages or not - I know it's available in openwrt as a package. Worst case I suppose, is one could build it and add it to the firmware image...

Last hint here - keep the focus on frames, and how to push those around - you're just making a copy at that layer, and sending it somewhere else for post-processing - don't need iptables for that.

 

[HELLO_wORLD]

tc is not routing, it's not switching...

It's QoS, and with QSDK based devices, it directly interacts with QCA's nss subsystem - Don't waste your time here, as traffic can either go down the fast path, or the slow path...

"these are not the droids you are looking for"

netflow, implemented as softflowd is probably the better choice, but I'm not sure if that lives in the third party packages or not - I know it's available in openwrt as a package. Worst case I suppose, is one could build it and add it to the firmware image...

Last hint here - keep the focus on frames, and how to push those around - you're just making a copy at that layer, and sending it somewhere else for post-processing - don't need iptables for that.

Well, the router has a port mirroring feature (in debug page), but it reduces significantly the performances of the router.
There is no way to see how this is done as swconfig does show exactly the same settings with or without the port mirroring.
Si this option is not acceptable.

The iptables way might not seem the best at first, but if offers two major advantages:

1. using the TEE module, I can duplicate the frames and send them to the gateway of my choice (aka, to the device I want). What it does is it only changes the ether layer of the frame, src and dst MAC; all the rest is identical. I don't care that the ether layer is changed. Being in iptables, it uses the QCA'nss subsystem, therefore the fast way (no impact on performances!)
2. At the iptables level, I can decide where to TEE the packets, and doing it in the FORWARD chain, I gain information by knowing to (or from) which device in the LAN the packet is related to. A simple port mirroring on the WAN side would not give me this information. I need to also add the TEE in the INPUT chain to catch packets destined to the router itself.

Note that the TEE module works only with iptables, not ebtables. I would gladly use a similar module in ebtables (so layer 2), but this option does not exist, and the iptables way has no impact on performance, so it is fine.
This way, I might lose all packets going nowhere, but I prefer my IDS to catch what is actually reaching my LAN (real threats) than what is being already firewalled or dismissed. Again, a port mirroring in WAN would show all the packets that I block with Aegis and any other blocking rule.

Now, the only packets I am missing with iptables are the ones generated by the router in the user space by raw sockets (there are not many of these). This is where tc is being used, as I can catch these packets (where iptables cannot).
I decided to only mirror the packets generated by the router going to the WAN, and this is working perfectly fine, again without performance impact as it is using as you mentioned the fast QCA's nss subsystem.

So I have a perfectly complete IDS system for anything coming from and going to the WAN and is either hitting the router or the LAN.

Anything LAN/LAN or LAN/router is not monitored. It is easy to add LAN/router both ways by tweaking the iptables rules, and it works fine, but where I had problems is catching raw sockets packets generated by the router and going towards the LAN. Again, there are rare (mainly udhcpd)


Now, I will look into softflowd, but it will be running in user space, and not going through the NSS acceleration, so likely to impact the router CPU and saturate it with high traffic.