Is there anything 1 level above pfSense?

[Maverick009]

Sorry late to the party. From looking at your post and being concerned with power draw and what process Intel chips are on, why even bother? Intel at this moment has fallen behind. I would recommend an AMD chip. There is a couple ways you can go. My first recommendation as it is the current gen available, is to go with a Ryzen Pro 4650G 6C/12T APU and pair it with an ASUS TUF Gaming B550M-Plus motherboard, which carries a nice feature set including an onboard Intel 2.5Gb LAN, PCIe 4.0 x16 and PCIe 3.0 X16 and X1 slots, along with other up to date board features. Pair with 16GB Dual-Channel Memory and either a 240G NVMe or SATA SSD (Drive will not matter as much if using as a firewall router). You can add a Quad 1GB NIC for instance I350-T4 and a Dual 10Gb NIC. You can then route ports to Switches from there. Multigig 1G/2.5G/5G/10G switches are starting to get more comfortable in price and affordable. Also you can start out in small steps and not splurge all at once in hardware as you mentioned price was a problem with rack mountable hardware. For the few bits of hardware you said were not rack mountable, you can get a shelf for the rack to accommodate that hardware. Also unless you are doing some heavy NAS or Gaming Network, you will not see a lot of demand from the NIC interfaces, as internet speeds for home use are capped at about 2Gbps at a premium price or 1.2Gbps for everyday enthusiast.

 

[Hikari]

Tnx for the tip!

Indeed, I stopped following AMD when Intel Core arrived and should restart learning about them. This ASUS TUF Gaming B550M-Plus is a very nice board, the 2.5Gbps onboard is awesome and it has a nice pair of x16 for a great price. I guess AMD also has better CPU at 35W TDP, and respecting that TDP!

I looked on some stores and sadly it seems that AMD doesn't have the same supply as Intel. I couldn't find 4650G anywhere, nor any 35W CPU, and found a unique unit of ASUS TUF Gaming B550M-Plus. All store owners are afraid of keeping stock while COVID-19 is killing so many ppl, unemployment is skyrocketing and currency is undervalued :/

A few days ago I decided to take a cheaper path. I had found an EdgeRouter X on sale for R$410 (20% cheaper than next unit of the same model) and decided to buy it. I had found more ppl having same hang issues I am, but it's popular on OpenWRT users and other 256MB RAM routers are at least 50% more expensive, while 512MB are inexistent and 1024MB are the same price of an AMD64 PC. Also, it's less troubling to have 2 units of the same model and be able to replicate everything on them than having 2 different models to learn on.

I'm gonna stay on ER-X for at least a couple more years, then hopefully market will get better and we'll have better prices and availability for 35W CPUs and mobos with 3GIO 4.0.

 

[Maverick009]

Tnx for the tip!

Indeed, I stopped following AMD when Intel Core arrived and should restart learning about them. This ASUS TUF Gaming B550M-Plus is a very nice board, the 2.5Gbps onboard is awesome and it has a nice pair of x16 for a great price. I guess AMD also has better CPU at 35W TDP, and respecting that TDP!

I looked on some stores and sadly it seems that AMD doesn't have the same supply as Intel. I couldn't find 4650G anywhere, nor any 35W CPU, and found a unique unit of ASUS TUF Gaming B550M-Plus. All store owners are afraid of keeping stock while COVID-19 is killing so many ppl, unemployment is skyrocketing and currency is undervalued :/

A few days ago I decided to take a cheaper path. I had found an EdgeRouter X on sale for R$410 (20% cheaper than next unit of the same model) and decided to buy it. I had found more ppl having same hang issues I am, but it's popular on OpenWRT users and other 256MB RAM routers are at least 50% more expensive, while 512MB are inexistent and 1024MB are the same price of an AMD64 PC. Also, it's less troubling to have 2 units of the same model and be able to replicate everything on them than having 2 different models to learn on.

I'm gonna stay on ER-X for at least a couple more years, then hopefully market will get better and we'll have better prices and availability for 35W CPUs and mobos with 3GIO 4.0.

If looking for the 4650G, you can find it on Amazon along with that Asus board. The processor is OEM as originally AMD was selling their latest APUs to OEMs. It also has a nice feature with the Bios and Ryzen Master software to adjust its wattage and I believe it can be configured to mirror its 35w 4650GE counterpart. That is one of the nice features of the Ryzen product line.

 

[avtella]

Good advice but I just want to add, regardless of AMD/Intel I think power draw is a very valid question to ask, I mean for example no need to have a power guzzling chip not adding any benefit to your usage. Ryzen at the moment is better power draw wise to equivalent Intel parts. As for gaming you wouldn't really need anything powerful NIC or CPU wise, gaming actually in most cases uses like a few hundred kbps of bandwidth and a decent QoS is probably more important for it. As for Ryzen Master or the Intel utility, they can not be run on a firewall or BSD in general, they would be limited to bios settings. I would probably disable Turbo Boost (Intel) or Core Performance Boost (AMD) to save power if needs are relatively low, that's what I do. One quirk of the Ryzen is that if you disable SMT it also knocks out S3 Sleep State (S3 not really applicable to a firewall), additionally temps actually went higher, so if you get an AMD chip make sure you don't disable SMT, as it's expected behavior according to AMD.

@Hikari also take a look at pre-built small form factor PCs from Lenovo, HP, Dell and others sometimes they come with decent i3 low wattage "T" series or Quad Core Ryzens with 4-8GB Ram, an open PCI slot and cost as low as around $300-400 in the states. Unless you get a great deal, I personally wouldn't worry too much about more than 4 cores or 8GB RAM unless your are dual purposing the machine to run also as a NAS or doing something enterprise level. As for storage regardless of OPNSense/pfSense/OpenWRT you don't really need much even a 64-128GB drive would more than suffice unless your are doing some really heavy logging, or caching using something like squid. You might get pretty good deals on used SFF PCs or appliances so I'd look through eBay, Amazon etc.

 

[Maverick009]

Good advice but I just want to add, regardless of AMD/Intel I think power draw is a very valid question to ask, I mean for example no need to have a power guzzling chip not adding any benefit to your usage. Ryzen at the moment is better power draw wise to equivalent Intel parts. As for gaming you wouldn't really need anything powerful NIC or CPU wise, gaming actually in most cases uses like a few hundred kbps of bandwidth and a decent QoS is probably more important for it. As for Ryzen Master or the Intel utility, they can not be run on a firewall or BSD in general, they would be limited to bios settings. I would probably disable Turbo Boost (Intel) or Core Performance Boost (AMD) to save power if needs are relatively low, that's what I do. One quirk of the Ryzen is that if you disable SMT it also knocks out S3 Sleep State (S3 not really applicable to a firewall), additionally temps actually went higher, so if you get an AMD chip make sure you don't disable SMT, as it's expected behavior according to AMD.

@Hikari also take a look at pre-built small form factor PCs from Lenovo, HP, Dell and others sometimes they come with decent i3 low wattage "T" series or Quad Core Ryzens with 4-8GB Ram, an open PCI slot and cost as low as around $300-400 in the states. Unless you get a great deal, I personally wouldn't worry too much about more than 4 cores or 8GB RAM unless your are dual purposing the machine to run also as a NAS or doing something enterprise level. As for storage regardless of OPNSense/pfSense/OpenWRT you don't really need much even a 64-128GB drive would more than suffice unless your are doing some really heavy logging, or caching using something like squid. You might get pretty good deals on used SFF PCs or appliances so I'd look through eBay, Amazon etc.

You are correct to an extent about the Ryzen Master software on a firewall, but the latest Bios of many motherboards, supports the feature set at the hardware level from within the Bios. Also with the Bios update and Zen2/Zen3 processors, power draw and heat levels will now be nearly identical with or without SMT support on.

IMO I would also now recommend a 6C/12T CPU/APU with 16GB Ram as long term they will be more cost effective and due to even some minor features being memory and in some case CPU intensive such as Sensai, it makes sense. Plus the fact that it looks like AMD and even Intel are preparing to Jettison their 4 core processors going forward.

 

[avtella]

Anandtech tech recently with the 5950X (December 2020) got similar slightly higher temps with SMT off, reason was the CPU would boost higher with SMT Off, my assumption is some of this is due to freed up power head room with disabled aspects of the CPU and power draw was still overall a little higher as well though.

Additionally in FreeBSD and in turn pfSense/OPNSense (HardenedBSD is still FreeBSD at its core) doing a combo of disabling CPB and SMT seems to have an odd effect at times where CPB is still active unless SMT is enabled again. SuperMicro looked into this and after engaging with AMD seemed to agree that the OS was overriding the CPB disable, not sure how that was a thing. Windows Enterprise however didn’t have this issue.

 

[Maverick009]

Anandtech tech recently with the 5950X (December 2020) got similar slightly higher temps with SMT off, reason was the CPU would boost higher with SMT Off, my assumption is some of this is due to freed up power head room with disabled aspects of the CPU and power draw was still overall a little higher as well though.

Additionally in FreeBSD and in turn pfSense/OPNSense (HardenedBSD is still FreeBSD at its core) doing a combo of disabling CPB and SMT seems to have an odd effect at times where CPB is still active unless SMT is enabled again. SuperMicro looked into this and after engaging with AMD seemed to agree that the OS was overriding the CPB disable, not sure how that was a thing. Windows Enterprise however didn’t have this issue.

As far as CPB and SMT, It would not be AMD, but the board Vendors. Jayztwocents, Steve of Gamer Nexus, and Linus of LTT all have reviewed these processors left and right, including with more recent Bios updates. Jayztwocents I believe was the one that also explained that Motherboard vendors have put their own spin on the CPB/SMT features to an extent too and you have to disable those core features. Also SMT off seems to draw more heat due to it boosting higher frequencies then when SMT is on. Best option is to keep those settings on. There is no major harm. You still can reduce power. With the ASUS boards, you can turn their full EPU (Energy Processing Unit) features on to reduce board and certain component power draws down. The processor can also has a configurable cTDP that I believe is to lows of 35w/45w/65w depending on needs.

Just noticed too what you are looking at was the Ryzen 5950X which is the top end processor for gaming and multimedia. However it is also the top power sipper when compared to the rest of the 5000 series. I have the Ryzen 5900X in my main system, and it still much nicer on power draw and performance. Barely sucking up power with what I am doing now which still nicely boosting speeds. Now when I turn on something intensive, it powers up and begins generating heat and power, but still better controlled then the 5950x. The 5950X was really intended to only be liquid cooled and it has higher clocks out the gate for a 16C/32T CPU. As far as the Zen3 core, it is recommended to keep SMT on as there is more to be had from it both in the fact the core is already efficient and that you still can gain some performance benefits up to about 22% with SMT on and no major change in power/heat draw.

 

[avtella]

I only mentioned the 5950X to reinforce my point as it was the test unit on Anandtech seeing a similar pattern, but yeah for desktop use it’s an awesome chip. As for the SMT & CPB issue, SuperMicro is not using any non standard changes, and they pretty much worked with AMD in regards to the issue report I filed especially in regards to S3 sleep, So I doubt AMD itself is lying. Mine is a server grade product not consumer so they’re a bit more focused on stability at the get go. As for cTDP it is only workable on some SKUs, even my board offers it in the bios but it doesn’t have any affect on my CPU. The Epyc 3255 which is similar to my CPU allows cTDP down to 35 Watts. My entire unit with 3x Fans 2x SSDs and 16GB RAM and including the ARM based BMC chip/RAM uses around 31-32 Watts on average having used a Kill-A-Watt to see usage over a month, it easily handles IDS/IPS, pfBlockerng, VPN etc and all thrown on it, pretty overkill so it rarely hits peak power use of ~70 Watts unless benchmarking like a 16 thread SSL benchmark.

Edit: As for SMT/HT I’m relatively well versed in how they function , yes in most cases best left alone.

AMD has come far since my time using the Opteron 170 back in 05, but still some minor platform issues here and there noting huge though.

Anyway sorry for derailing the thread we can get back to topic.

 

[Maverick009]

I only mentioned the 5950X to reinforce my point as it was the test unit on Anandtech seeing a similar pattern, but yeah for desktop use it’s an awesome chip. As for the SMT & CPB issue, SuperMicro is not using any non standard changes, and they pretty much worked with AMD in regards to the issue report I filed especially in regards to S3 sleep, So I doubt AMD itself is lying. Mine is a server grade product not consumer so they’re a bit more focused on stability at the get go. As for cTDP it is only workable on some SKUs, even my board offers it in the bios but it doesn’t have any affect on my CPU. The Epyc 3255 which is similar to my CPU allows cTDP down to 35 Watts.

AMD has come far since my time using the Opteron 170 back in 05, but still some minor platform issues here and there noting huge though.

I am sorry, It is not reinforcing the point, as I fail to see the issue.. Maybe elaborate more? As I mention I am using the 5900X 12C/24T CPU (currently not OC) on a Gigabyte Aorus X570 Xtreme v1.1 Motherboard and when I was in the BIOs playing with features, I could tune the heavy features down or off, and Tune the Processor to a certain power envelope. Heat and power went down nicely and might of gone down further if I did not have a board that was designed solely for performance and stability. Part of the power and heat issues are that of the processor used followed by the Board used. Even if you use a SuperMicro board, which is tuned for a more specific role, it still can introduce issues of its own. Also the big thing you cannot change is the chipset, as AMD currently uses the B550/X570 Chipsets for both Workstation/Performance and Gaming/Performance/Enthusiast cases. The biggest changes will be in layout, features utilized, and mosfets/chokes/components on the board. Gigabyte currently rains at the top due to the mosfets/chokes they use provide the most accurate information and Realtime performance. MSI/ASUS follow behind, but I have seen ASUS release some new revisions of boards as well, adding slightly higher components or features, without much of a price change.

SuperMicro fits more the specific case scenarios then a full fledgling of a product stack. For instance you may get more ethernet ports, but at the sacrifice of other hardware features, however one ethernet port may also carry IMPI support, which you do not typically see on a normal non-workstation class motherboard. So there are trade offs. IMO and from some of what I looked up, the chokes/mosfets and components used to support the processor are ok and help keep stability for what SuperMicro minimum specifications of the board were for, but the components on the bigger brands are typically of higher-end quality especially once you get to the 150-200.00 mark, and at around 260-290 they start using the best of the best. Warranty may also become a factor based on the board you you chose, but for me I do not always look at that as I tend to do a lot of my homework first on what I want and stick to brands I know are 9 out of 10 times reliable and of quality. The smaller items, I am more willing to maybe take a risk, but not on the bigger and/or more expensive items/parts.

The cTDP I believe is only in the Pro parts and possibly the X parts. Non X parts are already under a slightly lower TDP. I also believe any parts from the 3000-5000 series can support the cTDP capabilities. As for the Epyc processors, I can not speak to them as I do not even have a board and processor to test with, but from what I can read and see, I would believe they have that as they were heavily designed as Workstation/Server class system all around. The only thing I cannot recommend at all is the Intel cpus as they are more power hungry and harder to control their power delivery. As for the S3 that I am not aware of a full scale issue, and that may be specific to the hardware but as you mentioned, it is not really needed for a 24/7 Router appliance device. Good chat though. I was getting board today lol

 

[avtella]

Point I made was SMT off increased temps due to clocks going up that was clearly shown by Anandtech. I also gave my assumption as to why clocks went up as well... which was increased power headroom due to idle units within the CPU, however power draw also went up. My 3251 runs a cool 32-38C in regular use with two relatively low RPM 40mm fans so no temps were never critical or anything... temps climbed by roughly 4-5C with SMT off. Only time I’d ever hit 80C-85C is a continuous long running 16 Thread load with all cores at 3.1 GHz like when I did SSL benchmarking at which point fans would hit 4-5K RPM. I never stated one should necessarily disable SMT but pointed out that it’s a quirk that exists in my testing, this is something that doesn’t happen with turning Intel’s HT off.

As for the S3 issue it’s specific to Zen chips it, AMD has said it’s how it’s designed in regards to the S3/SMT quirk. Not a big deal for most of us.

Like wise I enjoy our conversations

 

[Maverick009]

Point I made was SMT off increased temps due to clocks going up that was clearly shown by Anandtech. I also gave my assumption as to why clocks went up as well... which was increased power headroom due to idle units within the CPU, however power draw also went up. My 3251 runs a cool 32-38C in regular use with two relatively low RPM 40mm fans so no temps were never critical or anything... temps climbed by roughly 4-5C with SMT off. Only time I’d ever hit 80C-85C is a continuous long running 16 Thread load with all cores at 3.1 GHz like when I did SSL benchmarking at which point fans would hit 4-5K RPM. I never stated one should necessarily disable SMT but pointed out that it’s a quirk that exists in my testing, this is something that doesn’t happen with turning Intel’s HT off.

As for the S3 issue it’s specific to Zen chips it, AMD has said it’s how it’s designed in regards to the S3/SMT quirk. Not a big deal for most of us.

Like wise I enjoy our conversations

You know what I think we both were on the same page, but where I meant SMT off has no affect on heat or power draw and it is similar with SMT on, is when they are locked at pretty much the same clock speed. I did and did not fully equate PBO into the equation as that can boost and draw more power to reach higher clock speeds and performance based on cooling solution and other factors.

In the long term it is better with SMT on as you can squeeze slightly more performance from the chip, plus there is more threads to handles heavy loads, or multiple device traffic in the case of a router/server. As for S3 and SMT, you may want to look into the Zen3 based 5000 series, as AMD did a lot of improvements, with L3 unified cache being a big one. There was also an update to improve the cache performance and enhance the 5000 series chips further. Still not a big deal specific use case scenarios.

 

[avtella]

Yeah Zen 3 has made decent improvements, biggest being no more 2x 4 Core CCX within each CCD unlike Zen 1/2 , just 8 core CCD now with no CCX. Much better for inter core latency and even inter CCD latency also got a nice improvement.


In regards to networking, one case where I now recall having seen HT and SMT recommended off a while back was for low latency networking. AMD’s own recommendation for low latency servers is similar. Again not something me and most people would need to worry about lol.

But yeah overall we are on the same page.

 

[sfx2000]

In regards to networking, one case where I now recall having seen HT and SMT recommended off a while back was for low latency networking. AMD’s own recommendation for low latency servers is similar. Again not something me and most people would need to worry about lol.

This mirrors my experience with telco class app servers - we disabled HT/SMT on the big Xeons...

(SMSC, AAA/HLR, HSS, etc)