Upgraded Opnsense Firewall Router hardware

[Maverick009]

So I have been talking about it for a while now and finally did a fully upgrade revamp of my Opnsense Firewall. Major upgrade to the firewall that will last for a very long time. I have the Intel NICs also setup in Bridge mode as it made complete sense with my home network and the CPU is more than capable of running it that way. I have 1 port of the 10G NIC running directly into the 10G Aquantia card on my gaming and NAS server that also host my Lancache. Currently have the main feed of my devices running from one of the I350-T4 gigabit ports to a 24-port L3 Switch that route out to the various hardwired devices and to 2 Asus Routers in AP mode for the wireless devices. I will probably be making some changes to the way devices are connected at some point and possibly look at adding multigig 2.5G/5G/10G Switches to the mix within the next year. The difference in hardware alone and specifically the CPUs, sees a nice power reduction and with that a huge heat difference, which is a good thing. As for the OS, Opnsense has been working quite well and really gets the job done and for anyone on the fence, this has been a great project I can speak too as well and wanted to share.

Previous hardware
Intel Q6600 2.4Ghz CPU (95W)
4GB DDR3 Dual-Channel Memory
240G SATA SSD (OS)
Gigabyte G41MT-USB3 motherboard
Intel I350-T4 Quad Gigabit NIC
Dual 2.5G Realtek NIC

Upgraded hardware
AMD Ryzen 5700G 3.8Ghz Base/4.6Ghz (Single Core)/4.2Ghz (All Core) (65W)
Asrock B550M Steel Legend Motherboard
16GB DDR4-3200Mhz Dual-Channel Memory
512GB M.2 NVMe SSD (OS)
2.5G Onboard Realtek NIC (Modem/Internet connection)
1G Intel I350-T4 Quad Gigabit NIC
10G Intel X540-T2 Dual NIC
Opnsense 23.7.6

 

[coxhaus]

What L3 switch?

I had a Intel Q6600 CPU many years ago. It was my first quad processor. They did run hot.

 

[Adooni]

I am using PCI_E 4xLAN cards as LAN bridge with OPNsense and it is working very well (LAN Bridge do not work OK with pfSense). I did not needed additional switches sry.

I am planning to change HW to miniPC with 6LANs and i3 N305 CPU

 

[coxhaus]

I don't really care about bridging as it is slower than layer 3 switching. There is too much baggage with bridging. And if that was not true then we would all be bridging and layer 3 switching would not exist.

Bridging does work. And maybe that is a reason to run OPNsense. Me I am always going to use layer 3 switching.

 

[sfx2000]

I would agree - nice to have a managed L3 switch downstream...

 

[sfx2000]

Upgraded hardware
AMD Ryzen 5700G 3.8Ghz Base/4.6Ghz (Single Core)/4.2Ghz (All Core) (65W)
Asrock B550M Steel Legend Motherboard
16GB DDR4-3200Mhz Dual-Channel Memory
512GB M.2 NVMe SSD (OS)
2.5G Onboard Realtek NIC (Modem/Internet connection)
1G Intel I350-T4 Quad Gigabit NIC
10G Intel X540-T2 Dual NIC
Opnsense 23.7.6

Another answer...

New 4x 2.5GbE and 2x 10GbE Intel Core Firewall and Virtualization Appliance

We take a look at a 2x 10GbE and 4x 2.5GbE Intel Core firewall and virtualization appliance and see why it is one of the craziest out there

www.servethehome.com

 

[coxhaus]

Another answer...

New 4x 2.5GbE and 2x 10GbE Intel Core Firewall and Virtualization Appliance

We take a look at a 2x 10GbE and 4x 2.5GbE Intel Core firewall and virtualization appliance and see why it is one of the craziest out there

www.servethehome.com

It looks interesting but I am going to stick with a stripped down small form factor Dell PC using a 30 or 35 watt CPU. Since the cases are built for much higher powered CPUs they have plenty of cooling capacity and the fans are quiet for low power CPUs. You also gain better natural cooling in a larger case.

Router software is still kind of stuck on single core processor since udp packets get out of sequence multi-tasking them. The other cores can do other things, but I don't think they have routing multi-tasking.

 

[sfx2000]

It looks interesting but I am going to stick with a stripped down small form factor Dell PC using a 30 or 35 watt CPU. Since the cases are built for much higher powered CPUs they have plenty of cooling capacity and the fans are quiet for low power CPUs. You also gain better natural cooling in a larger case.

And more importantly - with the Dell, one can easily get replacement parts...

 

[coxhaus]

Yes. I live 20 miles from Dell.

 

[coxhaus]

I think as long as the single core routing issue exists with udp packets getting out of sequence multi-tasking then using a lower core CPU with a higher clock rate will be best for router software.

 

[Maverick009]

What L3 switch?

I had a Intel Q6600 CPU many years ago. It was my first quad processor. They did run hot.

It is a TP-Link T1600G-28TS L3 Managed Gigabit Switch. Works good and does the job well, but I will at some point had a couple upgraded switches including a 10G multigig switch. Currently contempt with a working setup.

 

[Maverick009]

I don't really care about bridging as it is slower than layer 3 switching. There is too much baggage with bridging. And if that was not true then we would all be bridging and layer 3 switching would not exist.

Bridging does work. And maybe that is a reason to run OPNsense. Me I am always going to use layer 3 switching.

Using Pfsense or Opnsense, in some cases just a switch is needed and does not need to be necessarily an L3 switch, as a lot of the functionality part of an L3 is baked into Pfsense and Opnsense, but if the price is right, L3 is the way to go even if not using all features out of the gate. As for bridging, it can be faster than an L3 switch, but it all depends on the internal hardware of the firewall router server. The problem, is bridging brings other headaches with it, I was using it, and recently for now disabled it as I ran into some issues. I may revisit it later.

 

[Maverick009]

Another answer...

New 4x 2.5GbE and 2x 10GbE Intel Core Firewall and Virtualization Appliance

We take a look at a 2x 10GbE and 4x 2.5GbE Intel Core firewall and virtualization appliance and see why it is one of the craziest out there

www.servethehome.com

Nah, not worth it. Limited expandability. For the system I have, it is in a 2U Rack server case in my rack. The 5700G can be tuned for low wattage and with default eco-mode it could go to 45W. With further tuning and locking clock speeds, it is even possible to get it down to about 15W. As for components, I have more freedom to upgrade and add as needed.

 

[Maverick009]

I think as long as the single core routing issue exists with udp packets getting out of sequence multi-tasking then using a lower core CPU with a higher clock rate will be best for router software.

I am not sure of Pfsense, as it has been a moment since I played with it, but Opnsense, has been moving more and more to multi-core/multi-tasking design phase. You can also actually enable UDP firewall request to be handled by multiple cores now with a simple setting. That is just another advantage over prebuilt consumer routers.

 

[coxhaus]

You can also actually enable UDP firewall request to be handled by multiple cores now with a simple setting. That is just another advantage over prebuilt consumer routers.

The problem with multi-tasking UDP is when you have out of packet sequence errors the whole packet string needs to be resent which causes slowdowns. UDP is not like TCP.

 

[coxhaus]

Using Pfsense or Opnsense, in some cases just a switch is needed and does not need to be necessarily an L3 switch, as a lot of the functionality part of an L3 is baked into Pfsense and Opnsense, but if the price is right, L3 is the way to go even if not using all features out of the gate. As for bridging, it can be faster than an L3 switch, but it all depends on the internal hardware of the firewall router server. The problem, is bridging brings other headaches with it, I was using it, and recently for now disabled it as I ran into some issues. I may revisit it later.

If you bridge 2 networks together it will be slower than a L3 switch routing at LAN speeds.

The reason you are paying more for an L3 is to use it for L3 switching, routing, at LAN speeds not using it as an L2 switch. If you are using it as a L2 switch you are not routing, the router is routing networks.

 

[Maverick009]

If you bridge 2 networks together it will be slower than a L3 switch routing at LAN speeds.

The reason you are paying more for an L3 is to use it for L3 switching, routing, at LAN speeds not using it as an L2 switch. If you are using it as a L2 switch you are not routing, the router is routing networks.

Not exactly as it also depends on the hardware backbone too. A dedicated L3 switch helps alleviate some of the stress on the system as you expand without the need to upgrade the server/firewall/router hardware alone. It also supports some server functionality such as DHCP. However all ports are bridged to a certain extent. For me bridging did not hurt or slow my network down even under stress with the Opnsense Firewall. There was a few other quirks that backed me out for now as I did not have time to play with it.

As mentioned with the hardware, it too was significantly updated and even beats most routers and switches out under the hood. If it wasn't for price of the components, I would of probably used lowered hardware. For home use, I have more advantages than not.

 

[coxhaus]

Not exactly as it also depends on the hardware backbone too. A dedicated L3 switch helps alleviate some of the stress on the system as you expand without the need to upgrade the server/firewall/router hardware alone. It also supports some server functionality such as DHCP. However all ports are bridged to a certain extent. For me bridging did not hurt or slow my network down even under stress with the Opnsense Firewall. There was a few other quirks that backed me out for now as I did not have time to play with it.

It sounds like you are mixing up L2 bridging and L3 bridging. They are completely different and working at different OSI layers.

 

[Maverick009]

It sounds like you are mixing up L2 bridging and L3 bridging. They are completely different and working at different OSI layers.

No real confusion over the two. Just knowing the difference between them and hardware level support you can build into a router at the cost of expense to reach the same or very close to similar levels. Now at lower cost of hardware or if there is a lot of resources being used on the firewall router, the L3 could/would be the better option, but again I beefed the hardware up with an 8C/16T Ryzen 7 5700G APU and 16GB DDR4 Memory. The Intel NICS are all hardware multi-queue NICS. For a home network, this Firewall Router Server is more than capable of the same duties. The main features of the L3 after all is the router features and if you have a lot of VLAN cross-traffic at the same time, the ASIC in the L3 is better at handling. Otherwise an L2 Switch with Opnsense works very well. Let me also say, I have a TP-Link T1600-28ts L3 Gigabit Switch now, so I did not lower the hardware, but I am also at the same time not using it fully and with prices my next switch purchase may be the Tplink multigig 2.5G 24 Port L2 switch with 4 10G SFP+ ports. Again great for a home network. Business Networking would always be an L3 switch no matter the cost due to long term usage and support of business growing needs.

 

[coxhaus]

L3 switching also comes in to play when your outside router LAN port bandwidth becomes saturated. Using L2 slows your network down regardless of processor speed on the router as it is a bandwidth issue that does not happen using a L3 switch.

If you lightly use your local network then L2 will work fine.

I don't think I would trust TP-Link trying L3 switching as their programing code is weak.

Comparing Layer 3 and Layer 2 Switches

This article discusses the difference between layer 2 and layer 3 switches and the appropriate use cases for each.

documentation.meraki.com