Is there anything 1 level above pfSense?

[avtella]

@Maverick009, yeah it’s not surprising to have such issues with Realtek cards. However since you already seem to have one, I have seen in pfSense forums that some people avoided issues with the newer Realtek cards by compiling a newer driver themselves in FreeBSD and uploading it, could do the same for your OPNSense setup.

EDIT: Found one of the “fix” threads:

[Guide] Resolve Realtek NIC Stability Issues on FreeBSD - FreeNAS/TrueNAS, pfSense (2.4.4, 2.4.5, 2.5.0) + OPNSense, use 2.5Gb Realtek

If you are running a pfSense/any FreeBSD machine with a Realtek NIC, you may have noticed it randomly shirts the bed when under load: re0: watchdog timeout re0: link state changed to DOWN Well, the default drivers in the FreeBSD kernel are older than dirt and god awful (if you mention this to...

forums.serverbuilds.net

 

[Maverick009]

@Maverick009, yeah it’s not surprising to have such issues with Realtek cards. However since you already seem to have one, I have seen in pfSense forums that some people avoided issues with the newer Realtek cards by compiling a newer driver themselves in FreeBSD and uploading it, could do the same for your OPNSense setup.

EDIT: Found one of the “fix” threads:

[Guide] Resolve Realtek NIC Stability Issues on FreeBSD - FreeNAS/TrueNAS, pfSense (2.4.4, 2.4.5, 2.5.0) + OPNSense, use 2.5Gb Realtek

If you are running a pfSense/any FreeBSD machine with a Realtek NIC, you may have noticed it randomly shirts the bed when under load: re0: watchdog timeout re0: link state changed to DOWN Well, the default drivers in the FreeBSD kernel are older than dirt and god awful (if you mention this to...

forums.serverbuilds.net

Thanks but I am on Opnsense and not Pfsense and Opnsense uses Harden BSD Distro. Also OpnSense added recent driver for the newer Realtek cards as of Legendary Lion 27.1.6/7 update. The out errors issue seems to be cosmetic too and not affecting the network and only really happens when a speed test was done. It may be due to the fact I am running that NIC plus 2 others as a bridge.

It looks like there is a fix coming with the next major update due sometime this month from the road map.

 

[avtella]

The updated driver to fix the watchdog timer issue was actually for OPNSense and FreeNAS as well. One of the users in that thread were on OPNSense 20.7. Good to hear it’s not affecting you connection.

 

[Maverick009]

No connection seems to be quite stable and I am monitoring it for now, but I think the errors only happened with the speed test coming from the Realtek 8125 2.5G port and speed test on other ports in that bridge from the Intel NIC not causing the error out counter to grow. Also during normal everyday usage, I am not seeing errors from that port, so that is what is having me think it is a cosmetic issue with the specific driver/realtek card, as real world usage, shows no issues. Also from the roadmap, it may be an issue addressed as the Base updates include Fix stability and reliability issues with regard to vmx(4), vtnet(4), ixl(4), ix(4) and em(4) ethernet drivers. We should see soon....

 

[thecheapseats]

<snip>One thing to note about pfsense and layer 3,... pfSense comes somewhat pre-biased as the single L3 control point.... That could potentially be quite a bit of effort there, on its own.

and as a suggestion for anyone thinking of moving to pfsense or opensense - seriously consider getting layer3 under your fingers first... reviewing and even setting up a prelim mini-lab (using old-school cisco switch/stacking topology as a model) for L3, L2 and VLAN segregation with whatever mfg's switch/switches you might deploy is a good strategy... it will save you a lot of hassle down the line... and pcie for Intel nics gives you swap modularity as lan speed-demands increase in the future...

 

[L&LD]

@avtella, @Maverick009, @Trip, thank you for your input. I have calmed down and have (rightfully) given up on the Protectli line of action. How bad an idea that was is evident in all your responses!

You are all correct that I want something that is 'enough' for today but can grow if needed (modularity!). Protectli doesn't offer that at all. Nor does any other similar product.

@thecheapseats, can you expand on your thoughts about Layer 3? Would I need it in the scenario outlined below?

Since I have recently 'cleaned house' of all old parts/computers, I am now looking at the THINKCENTRE M70S with an Intel I350-T4 Quad Port Gigabit Ethernet Card for $600 (with 4GB RAM, i3-10100, and 128GB SSD). This should be enough to start with. And I should be easily able to build up from there. This seems reasonable to me as the Intel I350-T4 is $500 by itself!

I think I would like to give pfSense a try once more and see where I go next.

@Trip, is that Intel I350-T4 networking card equivalent or better than the I210AT you recommended?

I haven't looked at SuperMicro yet and I am never going to consider eBay, but is this ThinkCentre a candidate with no glaring flaws?


Here is a look into the network layout in my home:

Basement:

• Currently, no connectivity needed except for any stray Wi-Fi that makes it down there right now.
• Future wired AiMesh/AP (Wi-Fi only).

Main floor:

• ISP connection.
• On one end of the long side of the building (was also my original router location, (anywhere within) sixteen feet or so from the building's edge).
• Only a single Cat5e cable (100' run) going upstairs to main RT-AX86U directly from ONT w/1Gbps symmetrical ISP service (possibly going to 1.5Gbps 'soon').

Top floor:

• Main router (currently RT-AX86U).
• Connected to second RT-AX86U via Cat5e (100' run) on the 2.5GbE Ports between both routers for backhaul for AiMesh v2.0

Connected equipment: 3 NAS boxes (QNAP) directly to one or the other RT-AX86U via 1GbE Ports (considering LAGG for them though).


And here is a rough outline:

Top floor: ---------------------RT-AX86U--------------------------------------------------------------------------------------RT-AX86U-----------------------

Main floor --------------------------------------------------------------------------------------------------------------------------------------------ISP/ONT---

Basement ---------------------------------------------------------------------------------------------------------------------------------------------------------


I would like to add the pfSense router at/near the ISP/ONT location and have a LAN port available for the future basement AP. And a 'spare' one too, for the main floor.

What I would like to do with adding the pfSense box is to make the wired LAN independent of the Wi-Fi (routers). I will need a switch on the top floor to do this properly, I know.

Another thing the pfSense box may offer in the future is NAS-like storage of the most important files (as a backup of the NAS').



The current layout (rough outline above) provides fast Wi-Fi everywhere I need (main and top floors) and I can hit 940Mbps or more with my AX capable laptop. The AX phone, less so.

I occasionally run into a used tower computer that I may consider using here too, but the 4 Port Intel card would still be the biggest cost by itself if it is recommended and I bought it separately.

Any further thoughts and pointing me towards the right path gratefully accepted. Thank you for all the input so far.

 

[ddaenen1]

@avtella, @Maverick009, @Trip, thank you for your input. I have calmed down and have (rightfully) given up on the Protectli line of action. How bad an idea that was is evident in all your responses!

You are all correct that I want something that is 'enough' for today but can grow if needed (modularity!). Protectli doesn't offer that at all. Nor does any other similar product.

@thecheapseats, can you expand on your thoughts about Layer 3? Would I need it in the scenario outlined below?

Since I have recently 'cleaned house' of all old parts/computers, I am now looking at the THINKCENTRE M70S with an Intel I350-T4 Quad Port Gigabit Ethernet Card for $600 (with 4GB RAM, i3-10100, and 128GB SSD). This should be enough to start with. And I should be easily able to build up from there. This seems reasonable to me as the Intel I350-T4 is $500 by itself!

I think I would like to give pfSense a try once more and see where I go next.

@Trip, is that Intel I350-T4 networking card equivalent or better than the I210AT you recommended?

I haven't looked at SuperMicro yet and I am never going to consider eBay, but is this ThinkCentre a candidate with no glaring flaws?


Here is a look into the network layout in my home:

Basement:

• Currently, no connectivity needed except for any stray Wi-Fi that makes it down there right now.
• Future wired AiMesh/AP (Wi-Fi only).

Main floor:

• ISP connection.
• On one end of the long side of the building (was also my original router location, (anywhere within) sixteen feet or so from the building's edge).
• Only a single Cat5e cable (100' run) going upstairs to main RT-AX86U directly from ONT w/1Gbps symmetrical ISP service (possibly going to 1.5Gbps 'soon').

Top floor:

• Main router (currently RT-AX86U).
• Connected to second RT-AX86U via Cat5e (100' run) on the 2.5GbE Ports between both routers for backhaul for AiMesh v2.0

Connected equipment: 3 NAS boxes (QNAP) directly to one or the other RT-AX86U via 1GbE Ports (considering LAGG for them though).


And here is a rough outline:

Top floor: ---------------------RT-AX86U--------------------------------------------------------------------------------------RT-AX86U-----------------------

Main floor --------------------------------------------------------------------------------------------------------------------------------------------ISP/ONT---

Basement ---------------------------------------------------------------------------------------------------------------------------------------------------------


I would like to add the pfSense router at/near the ISP/ONT location and have a LAN port available for the future basement AP. And a 'spare' one too, for the main floor.

What I would like to do with adding the pfSense box is to make the wired LAN independent of the Wi-Fi (routers). I will need a switch on the top floor to do this properly, I know.

Another thing the pfSense box may offer in the future is NAS-like storage of the most important files (as a backup of the NAS').



The current layout (rough outline above) provides fast Wi-Fi everywhere I need (main and top floors) and I can hit 940Mbps or more with my AX capable laptop. The AX phone, less so.

I occasionally run into a used tower computer that I may consider using here too, but the 4 Port Intel card would still be the biggest cost by itself if it is recommended and I bought it separately.

Any further thoughts and pointing me towards the right path gratefully accepted. Thank you for all the input so far.

 

[avtella]

Your hardware choice should be more than good enough. I merely mentioned SuperMicro because I see their boards and appliances used/recommended a lot and Netgate's own highest end models are SuperMicro rebrands ( XG1541 and XG1537). Definitely did not intend for you to limit yourself to them, there are wide variety of options and brands that are pretty good, even maybe your old PCs, just try get something with an open PCIE slot. An old PC might be a power guzzler though.

As for the switch situation, a temporary/stop gap solution till you eventually get a switch, you can either bridge multiple ports on the ethernet adapter into a larger LAN or just run the APs on different sub nets. Bridged interfaces would not be as efficient as an actual switch which will have dedicated ASICs for the purpose of switching, but with modern hardware it shouldn't be to bad for home use, you should still be able to get line rate for gigabit or below but the price being a decent latency penalty.

In regards to ethernet adapters look at Newegg, Amazon or the likes for third party vendors selling on those sites, some sell server pulled adapters for a decent discount. I'd focus on Intel ethernet adapters, like @Trip mentioned. The Intel 210 should be fine as well. Chelsio is also often recommended as well and as for Broadcom I think it should be ok but have no experience myself.

EDIT: Also wait where are you seeing the i350-T4 for $500 (Canada?) it should be like $80-150. $400-500 would get you the much newer Intel X710-T4 or even newer T4L quad port 10Gbe adapter here in the states. My X710-T2L itself was like $265-270 back in March-April when I bought it and the quad port version: X710-T4L was like $465 (on Provantage) and T4 was like $400 on Newegg, prices have gone up a since then on the T4 (~$480-500 now) and T4L (Can't find outside of special orders). The X550-T2 already has a peak power draw close to the quad port X710-T4L, and I haven't seen an X550-T4 variant outside of custom slot/daughterboard implementations I'd guess it would need some decent cooling/airflow if used to full capability.

i350-T4
https://www.amazon.com/dp/B005ATA17I/?tag=snbforums-20

X710-T2L

PROVANTAGE: Intel X710T2LBLK Black X710 10GBE 2 Port RJ45 PCIE LP

Get Fast Service & Low Prices on X710T2LBLK Intel Black X710 10GBE 2 Port RJ45 PCIE LP - 100% Satisfaction Guaranteed at PROVANTAGE.

www.provantage.com

X710-T4 (T4L variant is like $20-50 more depending on site but uses 1/2 the power)

PROVANTAGE: Intel X710T4G1P5 MM 943101 Networking Card

Get Fast Service & Low Prices on X710T4G1P5 Intel MM 943101 Networking Card - 100% Satisfaction Guaranteed at PROVANTAGE.

www.provantage.com

Edit: $460 at server supply for the T4.

>Intel X710T4G1P5 Ethernet Converged Network Adptrer at ServerSupply.com

Buy Intel X710t4g1p5 Ethernet Converged Network Adptrer X710-t4. Refurbished..Great customer service.

www.serversupply.com

 

[ddaenen1]

@L&LD i am positive you will find pfsense a different experience nowadays. I would stick with Intel NIC's too as their reliability is unmatched but if you are thinking about the future i would consider an X550-T2 or X550-T4 NIC instead of an i350-T4.

This is what i like about my setup. Intel Xeon with 16GB ECC RAM, 2 100Gb SSD's in ZRAID1 and a full height PCI-e slot to plug in a 10Gbe card when needed. I am now on 1000/40 so i have no use for an upgrade but as soon as my ISP offers another bandwidth upgrade, i'll throw in an X550-T2 and i am good to go.

 

[L&LD]

@avtella, thank you for those links. I must have mistyped, the I350-T4's from Lenovo are in the lower $400 range.

@ddaenen1, the X550-T2 or T4 looks like a good upgrade when I'll need it in the future too (but it seems like they're $500 for the T4 right now). About the same price as the X710-T4 suggested by @avtella. Maybe the latter is the better buy (particularly the 'L' versions with half the power requirements)?

Right now, though, is the I350-T4 a bad idea for 4x 1GbE Ports? This will give me the ports I need to have a wired AP/AiMesh node on every level (if I so wish).

 

[Maverick009]

@L&LD The unit you are looking at may be good, but I worry more about the Intel Core i3 CPU in it, as we are now seeing some of the Broadcom/ARM based 4-6 core CPUs match or beat that processor in performance on the mid highend to top highend home routers. Only advantage that system would then offer, would be in memory and I/O performance at that $600 price. If truly looking for modularity, and starting out but with future expandability that requires minimum effort, I provided a list of a build I plan going with my firewall upgrade and the price in not really that different, but with a significant uplift in performance out of the box.

AMD Ryzen 5 4650G 6C/12T APU with Radeon Graphics (Save $100 and go to the last gen 3100G 4C/8T APU)
16GB Corsair Vengeance LPX Dual-Channel Memory
ASUS B550M TUF mATX motherboard with built-in 2.5G LAN
240GB Kingston NVME M.2 SSD
Intel based I350-T4 Quad 1GB Ethernet Card
500W EVGA Power Supply
Rackmount Case

Price: $855 ($755 with Ryzen 3100G)

Packs a lot more power and punch for very little difference is price (especially when you take long term effect into account) and it will give you everything you really one and still be significantly cheaper then what you were originally looking at. You can also add the X550-T2/T4 out of the gate if want to, but with what you currently have you do not need that out of the gate and can always add it later.

Edit: The I350-T4 is a good start with 4x 1GB ports, and you can invest in a Smart Switch (I have a TP-Link T1600G-28TS) and link at least 1 port or 2 ports in LAGG config and from that you will have a decent amount of ports to plug all your devices into including your access point.

Edit 2: SuperMicro boards can also be good as @avtella mentioned, but I have noticed that some of their boards are specially designed to only go into certain cases they design or partnered with someone on and why I have not gone that route entirely.

 

[avtella]

Most of the SuperMicro boards I’ve used/seen are actually pretty standard ie ATX/ITX/mATX and come with a standard IO shield. Unless your going for some those odd shaped or dimensioned boards which can be proprietary fits but you would easily know by also looking at the board description on their site.


Another option If attempting to build a NAS/Firewall combo I’d look at a small NAS oriented case (can be any brand) plus a SuperMicro 8CT board (AMD EPYC 3201 8 Cores / 8 Threads 30W TDP) or 4C/CT board (EPYC 3151/3101 4 Cores / 4 or 8 Threads depending on C/CT). The Asrock version of the EPYC 3151 comes with 2 built in 10GBe ports via X550 chipset. The SuperMicro models come with 4x Gbe ports using the Intel i350 chipset. I’d get ECC DIMMs for FreeNAS.

Asrock EPYC 3101 board:

ASRock Rack > EPYC3101D4I-2T

www.asrockrack.com

SuperMicro EPYC 3000 Embedded boards:
EPYC 3201:

M11SDV-8CT-LN4F | Motherboards | Super Micro Computer, Inc.

EPYC 3101:

M11SDV-4CT-LN4F | Motherboards | Super Micro Computer, Inc.

There’s also the EPYC 3251 and higher models but it’s 55W TDP and above from there.

I’m currently using the SuperMicro AMD EPYC 3251 board in a 1U chassis. Entire unit with 2x Crucial MX100s in RAID1 Mirror and a Samsung 970 Pro 1TB, Intel X710-T2L and 3x 40mm 8500 RPM fans uses around 31 watts at the wall idle/low load according to my Kill-a-watt, fans are at ~1,500-1,800 RPM. CPU Temps at ~38-40C. Full load on all 8 cores results in 80-85C temps and fans at 6k+ RPM.

 

[L&LD]

Thanks, @Maverick009, and @avtella.

I don't care about the power used. But 3x 40mm fans @8500RPM would drive me bananas. This is one HUGE reason I was looking at the silent/passively cooled Protectli and now a Tower or SFF system (I want 120MM fans or larger, quiet is heaven).

1U Chassis' are also off the list. Not looking for a server closet, rack, or set up. Not only no room for one, but hardly justified in my use case.

 

[avtella]

I actually was gonna say don’t get 1U lol if you intend to use it put some heavy loads on it, though still not very loud (rated max is 35 decibels) the fans can still be mildly annoying at 6K RPM at full load. For pFsense/OPNSense alone the 3251/3201 are overkill as my CPU usage is rarely above idle with IDS/IPS and pfBlockerng active. I went 1U for 55W TDP system not realizing how loud 40mm fans can be, the original 11000 RPM fans that I took out at peak were like 50+ decibels.... The Intel Xeon D-1541 board that was in the chassis previously would idle at 50C with fans at like 3-4K RPM... just imagine what mild load was like with the old 11K RPM fans . The EPYC 3201 and C3000 boards can even get away with a Noctua 5K RPM fan rated at less than 25 decibels.

 

[Maverick009]

Most of the SuperMicro boards I’ve used/seen are actually pretty standard ie ATX/ITX/mATX and come with a standard IO shield. Unless your going for some those odd shaped or dimensioned boards which can be proprietary fits but you would easily know by also looking at the board description on their site.


Another option If attempting to build a NAS/Firewall combo I’d look at a small NAS oriented case (can be any brand) plus a SuperMicro 8CT board (AMD EPYC 3201 8 Cores / 8 Threads 30W TDP) or 4C/CT board (EPYC 3151/3101 4 Cores / 4 or 8 Threads depending on C/CT). The Asrock version of the EPYC 3151 comes with 2 built in 10GBe ports via X550 chipset. The SuperMicro models come with 4x Gbe ports using the Intel i350 chipset. I’d get ECC DIMMs for FreeNAS.

Asrock EPYC 3101 board:

ASRock Rack > EPYC3101D4I-2T

www.asrockrack.com

SuperMicro EPYC 3000 Embedded boards:
EPYC 3201:

M11SDV-8CT-LN4F | Motherboards | Super Micro Computer, Inc.

EPYC 3101:

M11SDV-4CT-LN4F | Motherboards | Super Micro Computer, Inc.

There’s also the EPYC 3251 and higher models but it’s 55W TDP and above from there.

I’m currently using the SuperMicro AMD EPYC 3251 board in a 1U chassis. Entire unit with 2x Crucial MX100s in RAID1 Mirror and a Samsung 970 Pro 1TB, Intel X710-T2L and 3x 40mm 8500 RPM fans uses around 31 watts at the wall idle/low load according to my Kill-a-watt, fans are at ~1,500-1,800 RPM. CPU Temps at ~38-40C. Full load on all 8 cores results in 80-85C temps and fans at 6k+ RPM.

There is pluses and minuses to that approach. Those boards are all ITX and embedded boards. Plus is on the one board, the Intel X550-T2 card is integrated, but your minuses begin to creep up pretty quickly. ITX offers limited expandability and on those boards do not utilize the I/O abilities of EPYCC to the full extent. Also you loose some of the modularity and left with one PCIe X16 slot for any future expandability.

By using at least a Micro ATX motherboard instead of the ITX, you gain at least one more physical PCIe X16 port plus 1-2 X1 ports for expandability on the PCI Express I/O alone and the B550 chipset would be more then enough. The socket is expandable as the CPU will not be embedded and the Ryzen 5 4650G offers a couple more cores and/or threads and is based on ZEN2 where EPYCC 3000 processors are based off of ZEN so you gain IPC alone. When Ryzen 5000 APUs hit you can get ZEN3 and PCIe 4.0 feature set. Also due to more PCIe slots, you can expand as needed and more then likely have room for most new networking tech for quite some time to come.

As also mentioned, the Ryzen CPU/APUs can have ECO mode enabled in the BIOS, and for instance the 4650G 65W APU could be switched to the 35W 4650GE APU. Modularity and ECO mode can be worth it as embedded would be more suited to specific tasks/builds where expandability is more of an after thought and due to buying new, I could not recommend the older gen EPYCC based on the original ZEN core, as there are significant benefits jumping to ZEN2/3 cores. If you already had that hardware and was repurposing it, then I can see that route being feasible.

 

[avtella]

The Lenovo i3 model he originally picked was actually pretty decent. Some i3s support ECC not sure of that one though.

Nothing wrong with the Ryzen for a firewall. Main reason I’d recommend the EPYC 3000 instead is for a Firewall/NAS combo due to ECC support. I don’t think the IPC difference between Zen/Zen2 to will be a major player for the home firewall aspect other than looking nice on paper. Even my old Intel Xeon D-1541 barely had more than 5% percent CPU use, most of the time showing 0-1%, so we are already pretty overpowered at this point once we get past the Intel Atom C2/3000 series, for home use. However multiple OpenVPN tunnels and IPSEC can up the CPU usage significantly though. Also only the standard non-APU Ryzen variants support Unbuffered ECC but it's up to board manufacturers to support it or not, which can be a hit or miss.

The nice thing about the boards I mentioned earlier is that they have a BMC so you can do stuff like management and installations via virtual KVM so no need to even hook up a monitor/keyboard, probably a moot point and the IPMI ethernet port might be rarely used though in a home environment. Overall I guess it depends on if he wants a large unit or something compact but still silent. If he's ok going big yeah mATX/ATX would be more worthwhile with additional PCI slots. As for PCIE 3.0 x16, even a quad port 10Gbe adapter would not saturate it you'd need like a dual port 50 GbE NIC to go past that.

 

[ddaenen1]

the X550-T2 or T4 looks like a good upgrade when I'll need it in the future too (but it seems like they're $500 for the T4 right now). About the same price as the X710-T4 suggested by @avtella.

500 USD is hefty. I can pick up a 2nd hand X550-T2 for about 170 USD. I buy 2nd hand on local sites most of the time. There is quite an extensive market overhere in 2nd hand hardware. My Dell R210 was about 120 USD and i spend another 100 USD to expand the memory to 16GB ECC and 2 OCZ Deneva R 100GB SSD's.

 

[Maverick009]

The Lenovo i3 model he originally picked was actually pretty decent. Some i3s support ECC not sure of that one though.

Nothing wrong with the Ryzen for a firewall. Main reason I’d recommend the EPYC 3000 instead is for a Firewall/NAS combo due to ECC support. I don’t think the IPC difference between Zen/Zen2 to will be a major player for the home firewall aspect other than looking nice on paper. Even my old Intel Xeon D-1541 barely had more than 5% percent CPU use, most of the time showing 0-1%, so we are already pretty overpowered at this point once we get past the Intel Atom C2/3000 series, for home use. However multiple OpenVPN tunnels and IPSEC can up the CPU usage significantly though. Also only the standard non-APU Ryzen variants support Unbuffered ECC but it's up to board manufacturers to support it or not, which can be a hit or miss.

The nice thing about the boards I mentioned earlier is that they have a BMC so you can do stuff like management and installations via virtual KVM so no need to even hook up a monitor/keyboard, probably a moot point and the IPMI ethernet port might be rarely used though in a home environment. Overall I guess it depends on if he wants a large unit or something compact but still silent. If he's ok going big yeah mATX/ATX would be more worthwhile with additional PCI slots. As for PCIE 3.0 x16, even a quad port 10Gbe adapter would not saturate it you'd need like a dual port 50 GbE NIC to go past that.

Actually Ryzen supports full ECC just like its EPYCC variants and the Asus B550 TUF board I mentioned has full ECC support as well and the APU has ECC, but unless you are doing some heavy constant memory constrained operations or enterprise based tasks, ECC is not completely needed, but there optionally if needed. I mentioned Pcie 4.0 as a nice expandable feature and not the main go to as the B550 chipset supports only Pcie 4 on the one x16 slot and the rest are still 3.0 version, so it gives room to expand to that bandwidth as needed. Now Pcie 4.0 is available also on the M.2 storage subsystem and may be worth it just for that depending on the case and if for NAS, the SATA and M.2 performance matched with 2.5Gbps or 5G/10Gbps NICs would be a win/win.

For me more cores and/or IPC matched with sufficient memory is worth it, due to having several devices both wired and wireless ultimately connecting and split amongst IOTs devices, including 2 Apple TVs, Apple TV 4K, Nvidia Shield Pro 2019 model, Google Nest and Google Speaker, mobile Phones, Gaming Consoles, Business PCs, a few laptops, 2 Gaming PCs with 1G/2.5G/5G/10G Ethernet, a Gaming/NAS Multimedia Windows 2019 Server with a 10G and 1G ethernet connection, a Laptop configured Ubuntu Server that also will be running Asterisk VOIP, a couple VOIP phones, and a Smart Switch. May of missed a few, but depending on what is happening, and the bandwidth saturation, it can cause nice steady spikes hitting 28-60% at times. I have already reviewed the hardware and path, and since OpnSense can take advantage of multiple threads as most current software is doing, it makes sense to have more then 4 cores in todays equipment including a firewall router.

The reason I would also go with ZEN2 at minimum is not just IPC, but also some underlying enhancements including security. If buying new and price difference is not that much different, why would you jump on the older architecture? Plus support is already being killed off little by little from AMD and vendors, as AMD has moved EPYCC and even Threadripper to newer sockets with same pins but different layout to prepare for future expandability and the older chips are incompatible and vice versa. I am just thinking log term and with the way AMD is now moving, I would recommend the Ryzen APU and pair it up with a B550 chipset that has ECC support and PCIe 4.0, as now you have a long term probability with features of home and business, and you can expand on as needed down the road knowing there is support. I also have never had major issues dealing with ASUS for repairs and/or assistance.

Also some of the ASUS boards believe it or not support headless OS installs even on the gaming side. I just don't think ASUS advertises it in the open. Found that out thanks to Linus Tech Tips. I agree KVM can be good, but also if there is integrated graphics, it can sometimes be better for a centralized screen/keyboard/mouse that could even fold down and slide in and out of the rack through a drawer, which can be good for troubleshooting at the source or for direct access to certain features as you build out your network and configuring settings.

 

[L&LD]

I'm trying to get Lenovo to sell me a desktop system without an OS, but it doesn't seem possible.

I do have a quote for a ThinkCentre M920q (1L 'Tiny' desktop) with an i7-9700T processor and the Intel I350-T4 Quad Port Gigabit Ethernet Card for $930 including taxes (about 50% savings).

I'll see if I can configure anything like you're suggesting @Maverick009, thank you.

@Trip are you seeing any obvious pitfalls in the path I'm taking? Do you think a full-size Tower or a SFF tower would be a better option instead of the 'Tiny' I'm looking at now?

 

[Maverick009]

I actually was gonna say don’t get 1U lol if you intend to use it put some heavy loads on it, though still not very loud (rated max is 35 decibels) the fans can still be mildly annoying at 6K RPM at full load. For pFsense/OPNSense alone the 3251/3201 are overkill as my CPU usage is rarely above idle with IDS/IPS and pfBlockerng active. I went 1U for 55W TDP system not realizing how loud 40mm fans can be, the original 11000 RPM fans that I took out at peak were like 50+ decibels.... The Intel Xeon D-1541 board that was in the chassis previously would idle at 50C with fans at like 3-4K RPM... just imagine what mild load was like with the old 11K RPM fans . The EPYC 3201 and C3000 boards can even get away with a Noctua 5K RPM fan rated at less than 25 decibels.

I thought about 1U at the time for giving me room for something else in the rack, and did not take into account, smaller, may be louder lol. In the end I went with a 2U Rack, but have not added any fans to it as of yet, as it still temp wise is low enough, and I am starting by just upgrading the heatsink/fan for the CPU. Another reason for no further fans besides noise, was that I ultimately plan on upgrading the internals from this aging Core 2 Quad/Gigabyte build with 4GB DDR3 memory to a newer more robust AMD Ryzen APU/ASUS B550 build with 16GB DDR4 memory, and will plan accordingly from that. The Rack with the servers may ultimately be in a closet, but it is loud enough with the 2 internal rack fans, and I don't want to hear the sound beyond the closet doors lol.