Recommendations for discrete Gw/Fw etc. Router

[System Error Message]

If your needs are 2 WAN ports that would work. You also have to look at the CPU. The xeon-D is new so prices are around what is mentioned ($400 for CPU) for the quad core so if the brix has quad core CPU it will be a bit faster for much less.

My suggestion however is to get the xeon L56xx series as they're really cheap now and have loads of cache even for the quad cores. Using them with consumer LGA1366 (2nd hand boards) they typically came with a few PCIe x16 slots so you have a lot of expansion capability unlike with the brix. The L56xx series is the 32nm of the 1st gen iseries and have hardware AES so will be cheaper than the brix and can actually be faster than the brix for less if you overclock them. The xeons overclock very well so you can get double frequency using very little voltage bump.

For comparison in quad cores:
xeon-D 6M
xeon L56xx 12M
embedded i7 6M (8M if extreme CPU). I would check the brix's CPU to confirm.

So looking at clock speeds the lga 1366 xeon can be overclocked to 4Ghz via the bus to give a huge performance increase not only just with CPU but with other components because of the bus. The extra PCIe x16 lanes will allow you to go with many ethernet slots of even 10Gb/s and wifi AC.

The gigabyte brix would do what you want but if its the i3 variant it will be slower than the xeon L56xx even at those clocks. If this is for pfsense it depends on what you're running and it also depends on what NICs the gigabyte brix uses. Realtek NICs use more CPU and have less compatibility than intel NICs.

In terms of raw stock performance
Xeon- D, Xeon L56xx, i3 but the L56xx's larger cache and overclocking and low price gives it a big advantage.

 

[jalyst]

Don't care about Wifi or 10Gbe for this build, I'll have sep. dedicated switch with 10Gbe capability, & have other plans for Wifi. No need for 10Gb+ between this gw device & the main Switch, but from the main Switch to the rest of the (W)LAN I may do.

Do you have more intimate knowledge of these 3 platforms (Rangeley/Xeon-D/Skylake-U), & how exactly they compare "across the board", in terms of their various strengths/weaknesses for the primary focus of this build?

Yikes, 230am GN/thanks-again!

 

[System Error Message]

i cant remember about Rangeley but you can tell from the series number. for example i3/5/7 xyyy. X denotes the generation, if it is missing (i.e. i7 920) it means 1st gen. 2nd gen is sandybridge. So skylake is 6th gen. The performance across all 1series is very similar. newer generation has like a 5% increase in IPC (instructions per clock) or a process shrink so thats how you skew the clock per clock comparison. Each newer generation may include an additional instruction set (avx, avx2,...) that only has to do with math.

The xeons usually have an equivalent to their iseries desktop counterpart for some, while others are usually made differently.

However the older LGA1366 has the advantage of triple memory channels (unlike 2 for xeon 2 and mainstream), more PCIe lanes but dont have PCIe 3.0 or usb3. Routers dont need much math performance so newer stuff like avx is useless as the benefit is very little. Faster memory (more channels, more clocks) benefit more and this is why i suggested the lga1366 in the first place as overclocking it gives a big increase. Sure even the newer generations do have variations with 3 or 4 memory channels too but it depends on how much you're willing to pay as the CPU and boards are expensive.

In all the CPU's we've discussed, as long as they're not atoms or celerons or pentiums, The CPUs compare linearly with up to 20% difference between 1st gen and latest in IPCs (this means at same clock new gen is 15-20% faster) but the lga1366 (1st gen with 45nm and 32nm) is the only variant with huge bus overclocking room. All other gens require unlocked CPU.
Starting with the 32nm 1st gen iseries till now, most of their CPUs include hardware AES and this is important for vpn (and perhaps ssh?) performance.

What differentiates a xeon from a desktop CPU is 3 things
1) they come from the better part of the wafer and overclock better (can only overclock on lga 1366 desktop boards)
2) they support ECC ram (though some i3s are the exception)
3) they support extended virtualisation (not just CPU but also hardware too) and some features not present in desktop CPUs (the bit in intel ark right below)
4) some of them are specially optimised like more cache, lower voltages, etc
Some desktop CPUs support virtualisation and some of those features right below but they arent consistent across models.

When i overclocked my 6 core xeon on the lga 1366 to almost double, i managed to get the L3 cache to be very very close to the speed of the L2 cache. I found that if i overclocked further, if the speed of L3 cache exceeds L2 cache it becomes unstable. This is determined by the one of the busses (cant remember which) which needs to be 50% (if 32nm CPU) or double (if 45nm) to speed of ram. So the benefit of overclocking lga 1366 via the bus is huge and unlike the newer Iseries you can lock speeds of some things like the PCIe clock for instance, set all the memory, QPI and other busses multipliers/settings.

 

[sfx2000]

What differentiates a xeon from a desktop CPU is 3 things
1) they come from the better part of the wafer and overclock better (can only overclock on lga 1366 desktop boards)
2) they support ECC ram (though some i3s are the exception)
3) they support extended virtualisation (not just CPU but also hardware too) and some features not present in desktop CPUs (the bit in intel ark right below)
4) some of them are specially optimised like more cache, lower voltages, etc
Some desktop CPUs support virtualisation and some of those features right below but they arent consistent across models.

Quad Core Xeon's are pretty much the same as Quad Core i3/i5/i7 - enable the ECC flag, as that's about the only difference - just be advised that some Xeon's are not hyperthread enabled - check Intel's ARK site to see what features are supported for a given model. Should also note that most Xeon's do not have a GPU, so they are different in some ways from the Desktop Core chips.

That and many Xeon's support 2 and 4 socket, whereas desktops are one socket only...

Where Xeon's get interesting - once one goes above 4 discrete cores, the uncore and memory controllers are unique to the Xeon family, and there, the $$$ goes up

Ivy and Sandy bridge based Xeon's are cheap in the market these days if one is looking for Xeon - there's a lot of data center gear - and it's fairly cheap relatively speaking...

 

[System Error Message]

Quad Core Xeon's are pretty much the same as Quad Core i3/i5/i7 - enable the ECC flag, as that's about the only difference - just be advised that some Xeon's are not hyperthread enabled - check Intel's ARK site to see what features are supported for a given model. Should also note that most Xeon's do not have a GPU, so they are different in some ways from the Desktop Core chips.

That and many Xeon's support 2 and 4 socket, whereas desktops are one socket only...

Where Xeon's get interesting - once one goes above 4 discrete cores, the uncore and memory controllers are unique to the Xeon family, and there, the $$$ goes up

Ivy and Sandy bridge based Xeon's are cheap in the market these days if one is looking for Xeon - there's a lot of data center gear - and it's fairly cheap relatively speaking...

but other than the 1st gen iseries xeons, all the other iseries gen xeons are locked so you cant really overclock them. Uncore and memory controllers arent unique to the xeon family as all the lga1366 CPUs have it.

Where xeons differ is mainly because they come from the better part of the silicon. You can get quad cores with more cache than what i7 quad cores have. Trying to overclock an i7 to twice the frequency requires water cooling and a lot of voltage increase. Overclocking a xeon to twice the frequency requires some voltage bump and air cooling. Other than the ECC flag there are physical differences in the silicon itself.

unlike sandybridge and newer the LGA1366 had the PCIe controller on the chipset on the board so some boards had lots of PCIe lanes while some not many. The CPU had a big bus to communicate with chipset. Nowadays the PCIe lanes have been shifted away from chipset and onto CPU which is good until you need more lanes where no intel chipset had more than 4 lanes to spare.

The xeons and desktop cores themselves are the same when manufactured. The xeons get taken from the good parts of the silicon wafer and placed onto a chip with more cache and other things while the desktop ones get less stuff but they get an IGP on the chip. However some xeons may use the same slot but are very different (such as more cores) which desktop variants, not even the enthusiast line have. So core per core xeons are same as desktop variants but the chips themselves are different and the part of the silicon wafer to get taken from is different too.

 

[sfx2000]

but other than the 1st gen iseries xeons, all the other iseries gen xeons are locked so you cant really overclock them. Uncore and memory controllers arent unique to the xeon family as all the lga1366 CPUs have it.

Where xeons differ is mainly because they come from the better part of the silicon. You can get quad cores with more cache than what i7 quad cores have. Trying to overclock an i7 to twice the frequency requires water cooling and a lot of voltage increase. Overclocking a xeon to twice the frequency requires some voltage bump and air cooling. Other than the ECC flag there are physical differences in the silicon itself.

unlike sandybridge and newer the LGA1366 had the PCIe controller on the chipset on the board so some boards had lots of PCIe lanes while some not many. The CPU had a big bus to communicate with chipset. Nowadays the PCIe lanes have been shifted away from chipset and onto CPU which is good until you need more lanes where no intel chipset had more than 4 lanes to spare.

The xeons and desktop cores themselves are the same when manufactured. The xeons get taken from the good parts of the silicon wafer and placed onto a chip with more cache and other things while the desktop ones get less stuff but they get an IGP on the chip. However some xeons may use the same slot but are very different (such as more cores) which desktop variants, not even the enthusiast line have. So core per core xeons are same as desktop variants but the chips themselves are different and the part of the silicon wafer to get taken from is different too.

No... not completely accurate - not in the slightest...

Single Socket Xeon's are similar to the "E/EP" core chips, but outside of that - they're their own... 2 socket and 4 socket Xeon's are their own product line, and are not bin sorted Core i3/i5/i7... and they have their own support logic around them...

Intel has dropped a couple of the Xeon single socket chips into the enthusiast community, which perhaps is the basis for your assertions - but don't apply that logic that across the entire product lineup...

 

[jalyst]

Sigh I don't know, what you're suggesting just seems like overkill, sure it may be much cheaper, but the route I was suggesting is already much cheaper than the more specialised bare-bone units I was originally looking at. If the primary areas of use for this build, are going to be routinely heavily hitting all that compute power, then the much higher power draw (on idle or when used heavily) would I guess be justifiable, & the savings from the total build cost would be 'icing on the cake'.

I just want something that's more than powerful enough, for all the potential applications of such a narrowly focused build*, whilst still being an impressive sipper of power. I want it in a 1U form-factor, & I don't want it to be ridiculously priced, $900 I maybe could stomach, but a cent above that is just getting silly. It must have 2 quality Intel Gbe NICs (ideally 4), it doesn't need them OOTB, but there should be a way to add them neatly. Ideally it also has a PS well tuned for the range of power-draw, so there's impressively minimal wastage etc. And the chassis ideally is quality, good stock cooling/ventilation, & well built & laid out.

Is that asking for too much? :-/

*EDIT*
Quality 1U Chassis & more optimal PS I could probably forget in the short-term, just use what the Brix comes with.
Then buy some good 2x 8GB RAM (don't think ECC's possible?), & use a SSD I have, think it's 2.5" S6G (must check).
One of the Intel NICs can be WAN, the 2nd can be LAN, for 2 more WAN interfaces & possibly 1 more LAN;
I guess I could use the USB3 ports? Two are USB3.1 gen 2 (1 Type-A & 1 Type-C), & one is USB3.0.

Ugh, god their bare-bones page is ridiculous, so useless at breaking down all the models & their differences.
At least the first page of that AT article I linked explains how to break them down, but it's still far from ideal.
Reading up on the various platforms I'm considering here, though TBH I'm not much closer to being 100% decided.

*must check, but quite sure the best speed I can get (before any sort of aggregation/load-balance) is ~105/5MBPS, & I'll sometimes want VPN on such a link, so that must be factored-in too.

 

[System Error Message]

You could always go with the xeon l56xx, cheap with plenty of headroom, 2U case where all the cards fit well with good cooling. its easier to cool a rack case because the fans cover the entire chassis and the cooling is streamlined. My overclocked 6 core xeon doesnt even reach 60C on load. I estimate that it would cost you around $500 at most to go with that solution. both quad port NICs i have are half width so they fit perfectly in 2U.

 

[sfx2000]

If one's need go beyond what Rangely can offer, the Xeon-D chips are very interesting from a performance and power perspective - uATX format, drop this in a short 1U chassis, and pretty much done...

https://www.supermicro.com/products/motherboard/Xeon/D/X10SDV-TLN4F.cfm

Gigabyte and ASRock Rack also have similar boards if SuperMicro is not available in a particular market...

 

[jalyst]

You could always go with the xeon l56xx, cheap with plenty of headroom, 2U case where all the cards fit well with good cooling.

Right, but did you see what I wrote. Don't agree? Thoughts on that?

I'm still trawling through ARK for the 3 main platforms I'm most curious about, trying to make sense of it all.
Doesn't help much as aside from all those raw specs, you need a raft of RW applications sitting alongside.

Is this the range you've been consistently advocating?
http://ark.intel.com/products/series/47915/Intel-Xeon-Processor-5600-Series#@All

Earlier night tonight (just), thanks again, GN/BR!

 

[System Error Message]

well sfx2000 has laid out the xeon-D stuff, so the choice is really up to you to either go with the gigabyte brix with dual core CPU (dont know what NICs they use), what sfx2000 suggested or what i suggested.
If you're planning to rackmount than the board size does not matter. When browsing for a case it will indicate what boards are supported and what PSUs (in the case of 2U). 1U cases actually have noiser fans and more of them. 4U cases have much quieter fans while 2U is in between in terms of noise. The best cooling for a rack case is one with fans that cover the entire area inside (from one side to the other)

Either way both suggestions involving the xeons will fit in 1U but my suggestion on using the lga1366 board will require a PCIe riser to fit the server NIC card in 1U but in 2U will have no issue.

Yes i am suggesting the xeon 5600 series (specifically the L56xx) for its low power and bigger cache while still having hardware AES. But you can go with the X series xeon instead for more cores.

My suggestion for the case is to go with a 2U short chassis as it gives you the flexibility to fit cards (without risers/extenders), PSU choice and choice of drivers (3.5 inch or 2.5 inch or even dual 5 inch bay you can add stuff to). 2U also allows for fitting some good cool.

For your requirements
- low power : all 3 solutions
- performance: xeons
- memory performance: xeons (DDR4 dual channel or DDR3 triple channel)
- low cost: gigabyte brix or lga 1366 xeon (2nd hand)
- flexibility: xeons (PCIe, memory, sata, etc). LGA 1366 xeon has huge headroom to overclock

Some things for you to weigh. If you go with the lga1366 you will have to find the board and CPU. The xeon-D already has 4 NICs embedded but i dont know if the 10Gb/s is compatible with gigabit ethernet. If its SFP+ than you can just get the module very cheap. You also need to consider the RAM price as well (i would avoid the value line and get standard or better).

If your internet is through fiber optics you could get SFP card and module and ditch modem. Linux supports SFP cards well while those some cards dont work on windows so avoid using windows/windows server on the router, use pfsense or a linux/unix distro.

The LGA1366 will yield the lowest price, the xeon-D is fastest out of the box but if you do overclock the LGA1366 like i said than their performance will be comparable/similar. The huge L3 cache running almost at the speed of the L2 cache gives a big performance boost even for the ram being overclocked as well. In my case my asus board automatically selects SPD for ram closest to the clock it is at when overclocking.

 

[sfx2000]

The upside of the Xeon-D's is that as a SoC, pretty much everything that would be needed is present - 8 cores, 16 threads, good onboard SATA and Gigabit*4 Intel NIC's, along with a out of band service processor... plus one still has a PCI-e 3.0 slot to support a 10GB NIC if needed...

Looking at Anandtech's review of the Broadwell based Xeon-D, it's a pretty compelling solution for a small VM lab, or a high end NAS - and as a Router running whatever flavor of SW, it's got more than enough horsepower to run a small enterprise...

http://www.anandtech.com/show/9185/intel-xeon-d-review-performance-per-watt-server-soc-champion

http://www.anandtech.com/show/10592...servermark-vdi-on-the-supermicro-sys5028dtn4t

Resisting the urge to order one for a build - it really does seem like the right chip for the job if Rangley isn't enough - the 8 core Rangely chips are also pretty well suited for routing work, but suffer in single thread performance compared to the Xeon-D if one has compute needs somewhere in the future...

 

[sfx2000]

One could do a Xeon E3-1200 v3, but Xeon-D does more with less...

Supermicro offers three different chassis for their boards - two 1U shorts (different storage options), and the Minitower in Anandtech's review, which could be a good basis for a NAS...

But running Debian or Centos with KVM, doing the routing on the Host, and Storage on a hosted VM - one still has the options for LXC/LXD and Docker for other items...

Don't get me too close to the (BUY) button, I might hit it by accident

 

[jalyst]

Can't sleep, had one last search for SYS-5018A-FTN4 & surprisingly found some much better deals this time:
http://www.ebay.com.au/itm/SUPERMIC...N4-1U-RM-BB-BLACK-C2758-1600MHZ-/351829673604
Some of them seemed good but had stupid postage, there may be some more slightly better deals, will have a closer look tomm...err today.

Now I know this isn't in the same league as the Xeon-D, Brix i-U, & L56xx in terms of GP compute abilities.
But it's surely more than enough for the intended focus*, & to boot should be waaay more power-efficient???
Remember, I have plans under-way for a separate build (NAS/jack-of-all) that will be much beefier.

*remember my link I think is no more than 105/5 (before any LB/aggregation with xDSL & 4G/LTE), it may even be much less, as I may have quality issues with Cable in my area. We're currently only on 30/1, but I'll upgrade to the fastest residential Cable plan soon to test. FTTN is still a few years away last I checked, our suburb lucked out when it comes FTTH despite being inner-city, FML!

*UPDATE*
The newer iterations from Netgate & pfSense (pfSense used to use SMicro's units), AFAICT they still use C7258, right?
So what are the notable fixes/improvements/refinements now that they use ADIE/Other, surely there's been some?

 

[sfx2000]

For Routing, it's a pretty good box...

 

[sfx2000]

The ADI boxes tend to be more focused on the gateway functionality rather than general purpose...

Depends on the particular config - the 4 port/6 port boxes are pretty much same, but how they populate them, and which C-series is used..

The 2440 I'm using is a 2-core Rangley, C2358, and 4GB RAM (plus an 8GB eMMC, but I'm running mSATA SSD instead... 4 ports (intel based), and passively cooled, so no fan - doesn't have IPMI/Service Processor for out of band management, but it does has a console port which is handy...

The 4860 is just more of the same - more cores, more RAM, more ports, bigger eMMC...

The build quality on the ADI/Netgates is very good - not very stylish, but it does the job and that's that...

I went with the 2440 as it scales very well to the needs here at the house - heck, it's overkill - let's just that

 

[sfx2000]

And FWIW - the ADI/Netgate boards do tend to be a bit spendy compared to some of the alternate choices (low volume perhaps) - case in point is that the bare board for the 2440 is $254USD, which for a 2-core Silvermont is a bit on the high side..

 

[mtganzer]

One could do a Xeon E3-1200 v3, but Xeon-D does more with less...

Don't get me too close to the (BUY) button, I might hit it by accident

I have a E3-1230v3 Supermicro-based ESXi 6 server, and that is enough to run my whole house! Before I got the ER-L, it also ran my firewall router (OpenWRT x86 in a VM), but I household politics made me move to a separate router.

If I was doing it now, the Xeon-D 1520 is not much more expensive that separate Haswell CPU & MB, and the Supermicro board can take up to 128 GB RAM, has dual 10 GB NIC's, and draws less power. You do give up a bit on CPU clock at the top end though.

 

[System Error Message]

The intel C7258 is an intel atom based core. This is very low power but the xeon-D and xeon 56xx series all beat it in performance clock per clock, core per core. They also have more cache too, more memory bandwidth which translates to faster routing. If multi gigabit WAN is something you need, go with xeon rather than intel atom (c7258)

 

[sfx2000]

The intel C7258 is an intel atom based core. This is very low power but the xeon-D and xeon 56xx series all beat it in performance clock per clock, core per core. They also have more cache too, more memory bandwidth which translates to faster routing. If multi gigabit WAN is something you need, go with xeon rather than intel atom (c7258)

Clock for clock - the silvermont cores in the Rangley are about 1/2 the speed of the large Broadwell cores in the Xeons, and the big core clock higher to boot...

For routing purposes, Gigabit WAN, one is better off with the big cores, same with 10GB on the WAN side...

Under 500 MBit WAN, the Rangley chips are probably sufficient, and no issues with the LAN at 1Gb...

One of the upsides on both Rangely and Xeon-D - they both natively support QuickAssist, which can help out in certain situations - however, not many packages out there support this at present (it's on pfSense's roadmap) - QuickAssist is to SSL as QuickSync Video is to H.264, the cores do not have to get involved as it's all handled in HW...