Is there anything 1 level above pfSense?

[Budgeter]

Maybe many of you have experienced this before. From integrated modem provided by ISP, you move on to a separate router. From a router, you learn about third-party firmware (Voxel, Merlin, etc) and so on. For me, it's like this:

ISP integrated modem
-> ISP modem + dedicated router
-> dedicated modem + router w/ third-party firmware
-> router + consumer-grade access points
-> router + business-grade APs
-> [pfSense (firewall w/ routing capabilities) + business-grade APs] (current state, still learning on this). I don't really pay attention to switches side though.

As you see, the level of complexity, flexibility and performance increases over time. The more I dive into this hobby, the more technology I know, and this makes me wonder what will be the one after my current level of knowledge (pfSense). From what I know, on a normal consumer side (home lab?), this is an endgame. After this is al about real network engineering (enterprise-level), not what a non-expert like me can do, not sure if this's true though.

With that said, what's your opinion on this matter? Is there anything I should know about after pfSense setup?

 

[L&LD]

If you have the budget, just put in an enterprise system, and let others manage it for you (on contract, of course, for full responsibility on their end).

 

[Budgeter]

If you have the budget, just put in an enterprise system, and let others manage it for you (on contract, of course, for full responsibility on their end).

As this is my DIY hobby, letting the other do it would be out of my consideration

 

[L&LD]

I would think that anything above where you're at already, that is the only way 'up'. Even if you think you have control, you won't.

@Trip has some good recommendations for 'install and forget' network infrastructure that doesn't need a monthly service contract or 'cloud' connection to the manufacturer.

But then, the DIY part would be gone after that initial install too.

You're at a good place to be my friend. Enjoy it.

 

[ddaenen1]

Maybe many of you have experienced this before. From integrated modem provided by ISP, you move on to a separate router. From a router, you learn about third-party firmware (Voxel, Merlin, etc) and so on. For me, it's like this:

ISP integrated modem
-> ISP modem + dedicated router
-> dedicated modem + router w/ third-party firmware
-> router + consumer-grade access points
-> router + business-grade APs
-> [pfSense (firewall w/ routing capabilities) + business-grade APs] (current state, still learning on this). I don't really pay attention to switches side though.

As you see, the level of complexity, flexibility and performance increases over time. The more I dive into this hobby, the more technology I know, and this makes me wonder what will be the one after my current level of knowledge (pfSense). From what I know, on a normal consumer side (home lab?), this is an endgame. After this is al about real network engineering (enterprise-level), not what a non-expert like me can do, not sure if this's true though.

With that said, what's your opinion on this matter? Is there anything I should know about after pfSense setup?

I Guess i am in a similar place as yourself. I am very happy with pfsense though and in general with my setup. The only downside of moving to Cisco WAP AP's is that my signal outside on the patio is not as good as it used to be with an RT-AC88u as AP but i am going to put an additional one outside to fix that. I did get into FreeNAS (TrueNAS nowadays) and have my own cloud (Nextcloud) running now which i am seeing more and more advantages over time.

You could try other firewall software like Untangle or OPNsense but if pfsense serves your purpose, it will only keep you busy for a short time. In my case, i don't want to change as pfsense does exactly what i want it to do and does it very well and secondly, the whole family depends on the network with remote working and home school and all in these times so i refrain from too much tinkering at the moment

For the future, i still want to configure a separate wifi guest network with bandwidth limitations and all equipment i have in place can support this but i have started that yet. I am not so familiar with VLANs so i need to do homework first.

 

[Trip]

@Budgeter - Your presumptions are mostly correct. In the grand scheme of things, the place you're at now is more or less at the diminishing end of returns as far as noticeable changes to your day-to-day network. Rather, a growing bulk of what you do from here forward would be more about increasing your knowledge base and tweaking the "what" and "how", less so the end result. I'm talking about fancier, more robust, albeit somewhat functionally redundant hardware, coupled with CCNA/CWNA-level knowledge, and beyond.

As to where you are, exactly? You're one or two degrees of convergence away from full hardware modularity (including virtualization) and one "level" away from being fully SDN-capable, ie. being able to control every part of your network programatically, as opposed to manually administrated. The former piece can be achieved by simply breaking out switching and possibly firewall to discrete hardware or virtual appliances, if you so desired, and the latter would require CLI/API-capability across your entire stack.

If you're that motivated, then the best next step, IMHO, would be to grab a used, working-pull enterprise switch off eBay. Juniper would be a great fit here, as it's all Layer-3 capable and JunOS is built on BSD (just like pfSense). EX2200's are cheap, can be easily quieted by mod'ing the fans, and the CLI is wickedly good. Beyond that, the next step up elsewhere would be enterprise wireless; used Aruba Instant is awesome for its configurability, Ruckus Unleashed for its amazing RF performance. Another route, of course, would be staying inside the Cisco walled garden with Catalyst/Aironet; there's more support, more materials, more everything out there for Cisco. Also good if you're perhaps thinking of a career track in networking.

One thing to note about pfsense and layer 3, as @coxhaus has mentioned several times; pfSense comes somewhat pre-biased as the single L3 control point. If you ever want to offload local L3 to your switching, and/or certain gateway-level services to other boxes, you have to be mindful to de-configure those components in the proper way on the pfSense box, such that it plays well enough with the other kids in the sandbox. That could potentially be quite a bit of effort there, on its own.

Considering all of the above, and the day-to-day stability of your network in general, I would look into standing up a completely separate (as in, physically separate) lab environment at home. That way you can tinker till your heart's content without having to worry about compromising the baseline connectivity of your home's actual network.

Hope that helps to give you some perspective as to where you are, and where you could go.

 

[Budgeter]

Much appreciate for all the advice here. I have a background in Software Engineering so I also have some level of knowledge in term of networking, albeit no where near a Network Engineer. Although I did ask the question, I've just recently learned to use pfSense setup and there is still a long way to go Might be at some point in the future, I will start doing what @Trip suggests, but for now, I will keep the advice as a reference.

@ddaenen1 I thought about trying TrueNAS a while ago, but since I don't really need a NAS, I haven't tried it yet. However, I have never thought/heard about a local cloud. I will definitely look into it

 

[L&LD]

@Budgeter, once the bug is in our ear, it is only a matter of time before we do what it's telling us.

I look forward to reading about your experiences and successes as you travel the pathway outlined by @Trip in his post above. Please do keep us informed! There is no such thing as giving 'too much' information.

Wishing everyone all the best in the new year!

 

[Maverick009]

I am also one that is using PfSense and after you get it up and running and synced correctly to your cable modem (mine was a slight fight as I have 802.3ad link aggregation on for full bandwidth and throughput from my Netgear modem) seems to be quite stable and a beast. There isn't necessarily anything above, unless you are going into full enterprise equipment/software. However, if you are looking for a little more flexibility and 100% opensource, there is OpnSense, which is a branch off PfSense, due to a falling out with some of the developers. OpnSense pretty much offers the same feature set, with the difference being in updates. You may see more major/minor updates, and there is weekly security updates. There will be slight differences in the interface and where to locate features. I have been looking at it and may switch over as the support forums are more friendly, but when I recommend do not rush through as I had done the first go. Due to slight naming and firewall changes, I could not figure out for the life of me how to get internet too all local devices. I did like the fact that it seems to better support LAGG with my modem immediately connecting and no packet loss. Just be prepared with some tinkering and set time for yourself to configure it correctly. If you have another system, or on your computer if it supports, virtualization, you could create an environment to play and tinker with it more, without causing your main network to be down due to a configuration issue.

 

[ddaenen1]

OpnSense pretty much offers the same feature set, with the difference being in updates. You may see more major/minor updates, and there is weekly security updates. There will be slight differences in the interface and where to locate features. I have been looking at it and may switch over as the support forums are more friendly, but when I recommend do not rush through as I had done the first go.

I have to admit, this is something i read often as a differentiator between pfsense and OPNsense. Honestly, i am unsure why i would want more updates? pfsense runs stable as a rock and i update the packages (ACME, pfblockerNG, HAproxy) as they become available. For the rest, i just want things to run and be reliable. About the Netgate forum, i read this also before. I have to admit, i have never actively used the forum. Read up on some stuff but not registered and no account. I have read the same about the TrueNAS community but i have been an active member since i started tinkering with FreeNAS and i have always found the members really helpful.

Pfsense is pretty easy to use and there are plenty of resources if you want to get into a specific functionality. Nothing compared to RouterOS or worse, Ubiquity Edge OS.

 

[Maverick009]

I have to admit, this is something i read often as a differentiator between pfsense and OPNsense. Honestly, i am unsure why i would want more updates? pfsense runs stable as a rock and i update the packages (ACME, pfblockerNG, HAproxy) as they become available. For the rest, i just want things to run and be reliable. About the Netgate forum, i read this also before. I have to admit, i have never actively used the forum. Read up on some stuff but not registered and no account. I have read the same about the TrueNAS community but i have been an active member since i started tinkering with FreeNAS and i have always found the members really helpful.

Pfsense is pretty easy to use and there are plenty of resources if you want to get into a specific functionality. Nothing compared to RouterOS or worse, Ubiquity Edge OS.

The frequent weekly updates are more about security to plug up holes in the code or adjust to security changes that needed to be made in general. Not that either has had any major issues, but Opnsense gives you a little more piece of mind. As for the major updates, there is at least 2 a year and they usually add new drivers, features, and major OS updates. This can be real helpful for buying newer NIC cards or upgrading your hardware, as Pfsense requires in most cases a new revision of the software that included the driver set or you manually updating the OS to add the drivers hoping you don't break the code. I see the big difference between the two being pfsense with slightly easier install, but limited new hardware support, until a new revision comes out that may add support. Opnsense on the other hand has slightly higher security but a slightly more user involved install. Once up it is rock solid, and is quicker to support security updates and new hardware due to the open source community dedication at large.

 

[Maverick009]

So now that I have had time and made a proper switch from Pfsense to OpnSense, I can say once you get past some of the beginner jitters and initial setup, Opnsense seems to be a slightly better firewall and due to the updates, my Dual 2.5Gb Ethernet NIC now is detected and works after the out of the box update I ran, which in Pfsense was still not supported and would of required a tricky install process to add the Realtek RTL8125 drivers to it. Very stable and detected my cable modem in multigig/LAGG mode immediately with little to no issues. Only hard issue was using some old hardware with a socket 775 motherboard and Q6600 2.4Ghz Core 2 Quad as I had no headless mode to work from when I started the install.

Overall it is running well, and the software is rock solid (Pfsense also was rock solid). Major differences I see, is Opnsense has better support for hardware out of the gate, and is quicker to support new hardware. The support forum is better, especially for those who are completely new to DIY Firewall software/hardware builds. Opnsense LAGG functions do not seem to be broken like in Pfsense. The weekly security updates can come in handy, especially if new threats/vulnerabilities are found that need immediate patching. There are some other minor/major differences, but so far the move for me was worth it after all the research and it meets my needs and supports all my NICS. I also setup a Bridge combining some of my ports, so they can all share internet and same IP pool. I have an Intel I350-T4 NIC (2 Ports are in LAGG mode with the Netgear CM1200 Modem plugged in. Other 2 ports are currently bridged with my Dual 2.5G NIC using the RTL8125 chipset.) and Syba Dual 2.5G Ethernet card, with a dedicated connection from the Intel NIC to a 24 port Switch that also has my Asus RT-AX11000 plugged into it as an Access point, amongst other devices, and one of the 2.5G ports is connected right to my main gaming multimedia computers 10G Ethernet port. I have done the speed test and hitting between 935Mbps to around 1150Mbps on a 1Gbps connection from Comcast.

Overall either software is good, but after playing with both, I venture to say Opnsense may be the better software in the long term and it is completely open source, with no tricks or questions involved in the licensing agreement. I may give another update after its been up and running for some time.

 

[L&LD]

I teased @Budgeter about 'once the bug gets in your ear'... now, I got it too.

I'm looking at something like this.

FW6C - 6 Port Intel® i5 - Protectli

I am thinking I would want it with 64GB of RAM and a 1TB mSATA SSD (like I've always wanted my Asus' routers to be configured with).

The only two things holding me back is the fact that it 'only' has 6x 1GbE Ports (2.5GbE would be my minimum today, with the RT-AX86U I would use with this wired router).

The other is that this is a processor from 2016 (i5-7200U)! Even an i5-8250U from 6 months later is a much better long term buy.

I know, I know, this is 'too much hardware' for a 1Gbps up/down (symmetrical) ISP connection. But I don't want to buy this again for the next half-decade either.

Does anyone know of a fanless unit like this Protectli 6 Port upgradable unit with 2.5GbE ports or better? And a newer processor than one from over 4 years ago?

This configuration costs well north of $1,6K USD with a 4TB SATA SSD (I'm assuming this unit would have no issues being a small 'nas', too) plus taxes and shipping.

But if a new platform with upgraded features and components is available, I could be persuaded to consider (or wait) for that instead.

Additionally, what are people's views on the coreboot BIOS with the SeaBIOS payload?

@Maverick009 I would love to see your network diagram and core settings for your Opnsense install.

Looking forward to all your responses!

 

[avtella]

64GB RAM? Overkill might be an understatement especially for home use. Even 4 GB would be fine with things like pfblockerng with lots of DNSBL lists with TLD Analysis active, especially now with python mode. Even with IDS/IPS like Suricata or Snort on top of that with lots of rules on multiple interfaces I can see maybe 8-16GB at most in a home setup. In my already overkill setup with 16GB, RAM usage currently is only 10-15% and I think 20-25% peak. I have pfblocker enabled with 2 Million DNSBL entries and TLD Analysis enabled.

Rather than a protectli, I d get a used SuperMicro appliance based on an Intel Atom C3000/Xeon-D 15XX series (2-4 Core) setup and add a multigig card like the Intel X550-T2 (Multigig works fine but UI just shows unknown for link rate, ~$150 on eBay) or an Intel X710-T2L (UI shows Multi-Gig rates on this properly, but expensive at $300). Of course a more powerful but still low wattage Ryzen/EPYC 3000 Embedded or Core series setup would be fine too. Also try to avoid Realtek based cards when possible.

Can find used SuperMicro and other appliances with PCIE slots for like $300-500 on eBay, saw one with a Xeon D-1518 in that range, with dual 10Gbe built in (X550 Ethernet chipset) and an open PCIE slot. You probably don’t need high wattage high core setups unless you’re doing some enterprise level stuff. Same with more than 8-16+ GB RAM...

If you want to double it as a NAS and install FreeNAS alongside pfSense/OPNSense using ESXi/Proxmox I’d definitely go the SuperMicro appliance route rather than the protectli. I’d lean towards 16GB if going for a Firewall/NAS combo setup.

As for Opensense vs pfSense, for basic use either should be fine but from at least one person I know who tried OPNSense a while back they said while OPNsense had more frequent updates they also tended to break things more often... Main reason I personally stay on pfSense is pfBlockerng.

 

[Maverick009]

I teased @Budgeter about 'once the bug gets in your ear'... now, I got it too.

I'm looking at something like this.

FW6C - 6 Port Intel® i5 - Protectli

I am thinking I would want it with 64GB of RAM and a 1TB mSATA SSD (like I've always wanted my Asus' routers to be configured with).

The only two things holding me back is the fact that it 'only' has 6x 1GbE Ports (2.5GbE would be my minimum today, with the RT-AX86U I would use with this wired router).

The other is that this is a processor from 2016 (i5-7200U)! Even an i5-8250U from 6 months later is a much better long term buy.

I know, I know, this is 'too much hardware' for a 1Gbps up/down (symmetrical) ISP connection. But I don't want to buy this again for the next half-decade either.

Does anyone know of a fanless unit like this Protectli 6 Port upgradable unit with 2.5GbE ports or better? And a newer processor than one from over 4 years ago?

This configuration costs well north of $1,6K USD with a 4TB SATA SSD (I'm assuming this unit would have no issues being a small 'nas', too) plus taxes and shipping.

But if a new platform with upgraded features and components is available, I could be persuaded to consider (or wait) for that instead.

Additionally, what are people's views on the coreboot BIOS with the SeaBIOS payload?

@Maverick009 I would love to see your network diagram and core settings for your Opnsense install.

Looking forward to all your responses!

What you are looking at and to max it out is overkill....

Currently I have Opnsense running on an old 775 socket Core 2 Quad Q6600 2.4Ghz Processor with Hyper-Threading and 4GB DDR3 Dual-Channel Memory. It all sits on Gigabyte Motherboard. CPU with a light load hovers between 3-10% with spikes getting to 26-31%. Now if I do a full bandwidth saturation of just the internet with a speed test, it can spike the CPU between 35-51% on the 2.5G direct connection, but even with all the devices I have connected, in real world testing, I have not once come close to a full saturation of the network. The memory also only has around 8-16% utilized depending on my settings and usage and again that is with 4GB.

Now with that said, unless you are running some corporate network or thousands to hundreds of thousand machines, 64GB of memory would be overkill. I do plan on actually upgrading my firewall hardware eventually, and going the complete DIY custom build way for a couple reasons. One I already have a network server case that my current firewall is using that fits up to a mATX motherboard. The 2nd is that I like to have full control of my hardware and the ability to know what is in it and expand as needed, including the NIC cards. I have priced out hardware to build my new firewall, using an ASUS TUF B550 mATX motherboard, that already has a 2.5Gb Ethernet connection built-in and at least a Ryzen 5 3500G Quad-Core with SMT and Radeon Graphics (Depending on when I do this and the pricing of the new Ryzen 5000 APUs, I may go straight to them for a 6C/12T APU). I plan on installing 16GB Dual-Channel Memory as it is cheap enough and reasonable (still may be a little overkill, but price is right), and putting a M.2 500GB-1TB SSD in as prices are dropping, even though a 240GB would still be plenty. I already have an Intel I350-T4 Quad 1GB Ethernet card I would put in it along with my Syba Dual 2.5G Ethernet. I am thinking at that time I may had at least an Intel Dual 10G Card as well, as my main Gaming multimedia computer has a 10GB ethernet, and my Windows 2019 Server also has a 10GB ethernet card.

Prebuilt can be good because it was built for that one specific function and support is there for the hardware, but I like custom building as I have control of what goes in and knowing I can expand or even repurpose the equipment, and I am getting more. Without adding the NIC cards into the equation, my new hardware upgrade will be around $625-700 using all current gen tech.

 

[L&LD]

@Maverick009, thank you for your input.

I have already tried pfSense some time ago, but it was a very unsatisfying experience. An i5 6500 with 16GB RAM and an SSD along with the Intel I350-T4 card too (if I'm remembering the specs correctly).

Agreed that it didn't get anywhere near being limited by those resources, but it also never worked for me with a 1Gbps symmetrical connection either back then.

When I buy, I will not be buying this for today. I want to run this 'router' for as long as possible. I can see 2.5GbE and even 10GbE ISP connections within that time. Spending the money today and just having this not be part of any foreseeable future network problem/bottleneck is what I hope for.

Fyi, my previous experience with pfSense was that even with that hardware, the online experience would be slower after just a day or two, no matter how many times I re-installed it. The hardware is faultless (it is still running years later with Windows 10).

Too many (possible) incompatibilities for me to mess around with parts I may have laying around.

Even the Protectli website warns about RAM issues with some of the vaults.

As @Trip states all the time, if I do this, I don't want a toy anymore, I want an appliance that just works.

Looking for more input! I'm in no rush to buy this today.

 

[Maverick009]

So was also looking around and happen to find that soon at least one AMD Ryzen 4000 series APU will be available as OEM. It is the 4600G, which is a using Zen2 cores as a 6C/12T CPU with Radeon Graphics at 65W. If you wanted to bring the wattage down, you could change an AMD setting to Eco-Mode and bring it down to 35W basically giving you the 4600GE version of the chip with maximum boosting still being the same at 4.2Ghz. Price was slightly more then the Ryzen 5 3500G but worth it with more cores/threads to work with. Now I will see between this APU and the newer 5000 APUs, and price may be the final determination, as the Zen3 core in the 5000 series, is overall much better, but for a firewall, the Zen2 cores are still no slouch and will perform well for this purpose.

 

[Maverick009]

@Maverick009, thank you for your input.

I have already tried pfSense some time ago, but it was a very unsatisfying experience. An i5 6500 with 16GB RAM and an SSD along with the Intel I350-T4 card too (if I'm remembering the specs correctly).

Agreed that it didn't get anywhere near being limited by those resources, but it also never worked for me with a 1Gbps symmetrical connection either back then.

When I buy, I will not be buying this for today. I want to run this 'router' for as long as possible. I can see 2.5GbE and even 10GbE ISP connections within that time. Spending the money today and just having this not be part of any foreseeable future network problem/bottleneck is what I hope for.

Fyi, my previous experience with pfSense was that even with that hardware, the online experience would be slower after just a day or two, no matter how many times I re-installed it. The hardware is faultless (it is still running years later with Windows 10).

Too many (possible) incompatibilities for me to mess around with parts I may have laying around.

Even the Protectli website warns about RAM issues with some of the vaults.

As @Trip states all the time, if I do this, I don't want a toy anymore, I want an appliance that just works.

Looking for more input! I'm in no rush to buy this today.

I definitely get what you are saying. At first this was only a side project with me to play with and I had old hardware lying around collecting dust. I did some research and decided to go the custom built way, as I have done that with all my hardware, except the wireless router/cable modem stuff. Obviously I had a bigger plan in store, and began what would be a complete overhaul of my home network, but at the time still wanted to keep it cheap enough as I was not sure how it would all pan out. Bought a Network Server Rack that had at least a 20in depth for clearance. Had a Silverstone G08 HTPC case that could optionally be rack mounted, and transformed it into a Windows Server running Windows Server 2019, powered on a Gigabyte Aorus X470 Motherboard, with a Ryzen 7 2700 and 16GB DDR4 Dual-Channel memory. I have the OS installed onto an Samsung 960 Evo M.2 SSD, and currently a few older hard drives used for storage. I eventually plan on replacing the drives with Seagate Ironwolf 10TB+ SATA drives, with at least 2 sets of RAID, and will use a small 240GB-500GB SATA SSD for caching. I have both a 1GB Ethernet NIC + Wireless 9260 802.11AC (both onboard), and a 10GBase-T Aquantia card. That server I am working making it a Streaming Multimedia and Gaming Server.

For the Firewall, I purchased an iStarUSA Group 2U compact Rackmount case, 500W EVGA Power Supply, Intel I350-T4 Quad 1Gb NIC and Syba Dual 2.5G Realtek RTL8125 NIC, and a small Kingston 240GB SATA SSD for the OS install. Installed the old 775 socket motherboard/Q6600 CPU with 4GB DDR3. As I mentioned earlier, since it was older hardware, that was my biggest issue with the install, as I had no headless install mode to work with and I had to do the main install/configuration before placing the Intel I350-T4 card in the PCIe X16 slot as once I do install it, I lose the screen image from the onboard GPU. I guess since it was first gen PCIe, it expects the only card in the X16 slot to be a GPU. Once I have access to the web interface, it is for the most part a breeze, as long as I do not lock myself out.

Now that I have played with the most recent production versions of Pfsense and OpnSense, they are both good, and stable once up and running, but I noticed OpnSense was more friendly with my cable modem in LAGG mode so I can get full multigig throughput, and via its quicker updates, it supports newer hardware sooner, which to me is the best case scenario especially for network hardware as I upgrade networking equipment a little more frequent to some would. The weekly updates from Opnsense are another added bonus and piece of mind. I have had the chance to play with them both and getting use to OpnSense a little quicker then expected, to a point that I feel ok with full hardware upgrades to the firewall and rollout. The current hardware is already in production running my home network. I have the cable modem coming in with 2 ethernet cables connected to 2 ports on I350-T4 in LAGG mode. Currently have the other 2 ports of the I350 and the Syba Realtek RTL8125 2.5G card bridged together, and 1 2.5G port is connected directly to my Multimedia/Gaming computer with CAT8 cabling, while one of the I350 bridged ports is connected to a TD-Link 24 Port Smart Switch (I may connect the other I350 bridged port to the switch too in LAGG, providing double bandwidth to the switch, and allowing me to add LAGG to any other devices I need to via the Switch. I currently do have 2 work computers wired in, as both my Wife and I are working from home, a Silicon Dust HDHomerun Prime3 Cable card, the ASUS GT-AX11000 802.11AX Tri-Band Wireless Router in Access Point mode, the Windows Server connected with both ethernet connections, and a repurposed HP Laptop turned Server running Ubuntu Server 20LTS and may house a variation of Asterisk on it for my VOIP system.

Opnsense is handling everything even on that older hardware fairly well, and with less resources, but more ethernet muscle then when I was just using my ASUS Router to do everything. If looking long term, I would recommend going DIY with a custom built system. You do not need to max it out all at once and it still can last you years to come and room to upgrade as needed, especially in the ethernet card area, as that seems to be your biggest concern. Even if you go a prebuilt box, you may not get everything you want, and tech can always change drastically, to where all that money you spent for years may go out the door quickly. It has already happened a couple times in the past 4yrs. Even with a custom built system that can always be a chance, but usually last longer, and unless software stops supporting it, you can get further.

The way I am looking at it, with the upgrade and DIY path I am taking, I have an appliance that doesn't only just work, but more or less works in the manner I want it to work, which is what matters to me most.

 

[Trip]

@avtella and @Maverick009 - Thanks for your input on right-sizing.

@L&LD - Remember, going the x86 route opens up the possibility not just for pfSense (or OPNsense), but also any x86-compatible distro, physical or virtualized, including OpenWRT, IPFire, Endian, Untangle, Sophos UTM Community, VyOS, etc. -- ie. the freedom to find the exact distro and software stack you want. Perfomance-wise, provided you have the proper-quality baseline hardware and cabling infrastructure in place, you should end up with delay-free packet processing that doesn't degrade from day-to-day (contrary to your initial experience, which I would respectfully view as an outlier case).

For the actual hardware itself, considering your goals, I would err on the side of modularity, including PCIe-based upgrade-able networking cards, as opposed to an embedded box with fixed ports. If you do go embedded anyways, I would stick to all-Intel NICs, and try for at least the I210AT or better -- as opposed to the 211 or lower, which may not have enough queues per port to satisfy the amount of imix traffic you hope to be able to push through. Also, instead of trying to cram as many ports and packet processing as possible on the gateway hardware, items such as core switching may be best served by a discrete SMB or enterprise-grade hardware switch. This depends perhaps on the amount and type of physical cabling you have in place, as well as if you want things like centralized PoE and similar items that are often only found on dedicated switching hardware.

If I can help give any more feedback here I will.

 

[Maverick009]

@L&LD what @Trip is stating is what I was trying to convey but he also mentioned another benefit of being able to use just about any software distro you want due to architecture is X86. Modularity has a lot of benefits over embedded devices. As you mentioned earlier, you want expandability, and doing a custom build would be more or less relegated to a rack but gives you flexibility and modularity.

I also would say even with modularity, I would stick with Intel NICs as much as possible. Aquantia may also be another option as the are now largely owned by Broadcom I believe. Realtek NICs are common place on hardware but can also be more problematic. The Syba Dual 2.5G card I have works well but currently in Bridged mode with the other shared ports, when I saturate the card with a speed test, it gives me a lot of out errors on the LAN. Now I have troubleshooted and can a test it only happens when I saturate that one port. No issues on the Intel NICs with same test. Now real world not seeing any issues, but that is why I agree with @Tripp on the NICs to use as Realtek may not always be a problem, but they can also make you think there is something wrong due to some compatibility issue and drivers.

Overall your biggest advantage is that of future expandability and control of what goes in your setup. If you didn't account for something now and want it later, embedded may not be able to give you that. Just food for thought. Always plan before doing where you can.