Today's delightful new problem (10gbit/s connection incoming)

[sfx2000]

For example the TILE has low internal latency from the NIC all the way to the CPU. Same with PCIe ports connected to the CPU as well (one good thing about some of the SoCs in routers too). In epyc, the PCIe ports are directly connected to the CPU but intel's xeon/high performance equivalent have some PCIe ports on CPU, some on chipset and its very confusing as well.

Plenty of differences, and latency actually doesnt matter internally because of the way routing works, pushing those packets only requires internal bandwidth more than latency, but latency from any recent x86 far outclasses the smaller SoCs like ARM and MIPS. Clocks arent everything too as i was saying. LTT has been using servers with manycore many CPU xeons at 2+ghz pushing multiple 10Gbe for purposes of storage, video editing and so on.

In a router/switch, there's more than just the CPU... in an ideal solution, the "CPU" provides supervisory purpose only from a kernel perspective - the flow tables for both stateless (packet forwarding) and stateful (NAT, QoS, etc) can and should be done outside of the kernel in user space...

DPDK provides just this - used to be intel centric, but it's now available for other platforms.

BTW - one of the quirks about Epyc is where one has to move data around - it can be worked thru, but if one is not careful, one can have code running on one of the CCX's, accessing memory that is not local to that CCX. Again, that can be fixed, just something to be aware of - applies to more than just networking...

 

[System Error Message]

In a router/switch, there's more than just the CPU... in an ideal solution, the "CPU" provides supervisory purpose only from a kernel perspective - the flow tables for both stateless (packet forwarding) and stateful (NAT, QoS, etc) can and should be done outside of the kernel in user space...

DPDK provides just this - used to be intel centric, but it's now available for other platforms.

BTW - one of the quirks about Epyc is where one has to move data around - it can be worked thru, but if one is not careful, one can have code running on one of the CCX's, accessing memory that is not local to that CCX. Again, that can be fixed, just something to be aware of - applies to more than just networking...

every x86 has this weakness in being a router, that is why to achieve 10Gb/s you need 4-8x the ram and bus bandwidths, whereas with the CCR, just 2 channels of DDR3 is sufficient. Epyc has 8 DDR4 channels so i dont see it having this issue, so many of their boards already come with 2 10Gbe ports.

 

[sfx2000]

Yeah both the MIPS and PPC cost a lot more than ARM for the performance too, not to mention the development and support as well.

MIPS and PPC are still pretty cool - most of the big stuff these days is focused on ARMv8 and X64 - even big MIPS shops like Cavium have moved over to ARM these days.

Interesting to note that Tilera, while defunct, the concept is still alive - Mellonox own's the IP these days, and they've redesigned it around the A72 ARMv8 cores...

BlueField is pretty cool...

 

[System Error Message]

MIPS and PPC are still pretty cool - most of the big stuff these days is focused on ARMv8 and X64 - even big MIPS shops like Cavium have moved over to ARM these days.

Interesting to note that Tilera, while defunct, the concept is still alive - Mellonox own's the IP these days, and they've redesigned it around the A72 ARMv8 cores...

BlueField is pretty cool...

you mentioned it before, still Tilera's concept is the manycore mesh based method where you have many cores that can communicate with any other core that are connected via a mesh, can be applied to any architecture, infact facebook's and google's Tilera cards dont use MIPS, have 100 cores and run apache, linux, firewalls and 4xSFP+.

 

[sfx2000]

infact facebook's and google's Tilera cards dont use MIPS, have 100 cores and run apache, linux, firewalls and 4xSFP+.

Source of that info?

 

[System Error Message]

Source of that info?

only a guess, because theres a difference between the regular TILE and the TILEGX used by mikrotik. TileGX is network focused which is why it uses MIPS (good CPU for it), whereas the 100 core TILE used by facebook is application focused, and to fit 100 cores on a PCIe card when mikrotik's best 72 core requires 100Watts just to operate (the PCIe cards dont have a power connector, so they draw from the board). The 100 core one i think is clocked higher.

They may have started with a different architecture but used a MIPS based one for the tilegx and mikrotik managed to make one of the best price to performance routers. So technically i guess MIPS isnt so expensive just the industry moving to ARM pressuring many to move to it.

 

[sfx2000]

only a guess

oh, ok - so kind of imaginary then?

 

[System Error Message]

oh, ok - so kind of imaginary then?

Tileras first manycore, using the results i made a guess based on it. im usually good at this sort of stuff, same with the economics of pricing behind non DIY NAS, many things you can profile based on facts and business performance.

The popular MIPS 24k in the past had its limits, would not scale well in a 100 core mesh. I have one of the weirder MIPS (SMP based single core one) which is a bit bigger sitting useless as i cant get openWRT installed on it (damn zyxel). There are plenty more archs in the mix that one can pick and scale, such as freescales plenty archs to pick from, or even a few other brands, so if one was making a new architecture, it is likely it would be picked from this as part of the scaling including FPGAs are both in parallel and clocks. While FPGAs dont clock high they are massively parallel too so its likely the first TILE is more like one of the architectures used in architecture development, as an ASIC version could clock much higher, with each core being small and scaling well to being massively parallel, would run code well, only not the best match for networking, something MIPS is well known to be good at at the time.

 

[Magebarf]

While sketching out my plan (and waiting for the ISPs customer support to root out some issues with their internal routing) my ISP has presented their suggestion for getting some speed over their fiber.

They published their news item today (english version of article is just below the swedish text on the same page):
https://www.bahnhof.se/press/press-...router-for-hemanvandare-snabbast-pa-marknaden

Expected delivery of devices are within 8-10 weeks.

In essence, it seems they've customized the Huawei HN8255WS (https://support.huawei.com/enterprise/en/access-network/echolife-hn8255ws-pid-22280247) which seems to be originally built for GPON usage, by putting a SFP+ port in place.

A few users/customers/potential customers have requested some additional details about what components are in this device, so we'll have to wait and see what it is capable of.

SFP+ and 1 10GBASE-T LAN-port does bode well, but no idea yet about throughput. I admit I'd rather see a NBASE-T enabled port, but for my use case it will be going straight into a 10GbE switch, so not much of an actual problem.

 

[Samir]

Thank you for the update. Looking forward to seeing the service in action. I wonder what site you could even speedtest it at.

 

[Rainmaker]

Thank you for the update. Looking forward to seeing the service in action. I wonder what site you could even speedtest it at.

There are several 10Gbps iperf3 servers available for public speed tests in Europe, at least. One command, reliable results.

iperf3 -c bouygues.iperf.fr -p 5002 -P 8 -4 -R

Easy. You can grab iperf3 on Linux(apt install iperf3 / pacman -S iperf3, xbps-install -S iperf3, etc), BSD (pkg-add install iperf3), macOS (brew install iperf3) and Windows (choco install iperf3).

ETA: Ah, I've been lurking here for years. I didn't realise I'd never actually posted before haha. Hi everyone.

 

[Magebarf]

Yep, will be doing some iperf3 troubleshooting with the ISPs technical support a bit later, as they seem to have some odd internal routing going on from my town over to the larger connection points, so at the moment I'm happy to reach 200Mbit/s.
As far as I can tell they do have some internal iperf3 setups available whenever the support deems it a good idea to use them for testing performance (I assume to keep the load on them down, to ensure whoever needs the bandwidth to test has it available).

As for the gear they presented today, they provided some measurements going through the device showing 9.2-9.3Gbit/s both directions.
I guess the question is if that is with NAT enabled and what type of firewall rules was in place when measuring.

Seeing as how specialized it is as a device I can only assume that NAT is properly offloaded and hopefully the performance well adjusted for the rest of the device's capacity.
I have seen worse misses with regards to system architecture however, so I won't hold my breath.

 

[Samir]

Thank you for the iperf3 server! I've been looking for one for a while now.

I have a cygwin ported version of iperf3 and it worked perfect with that server. Can't wait to see test results from a 10Gb line.

 

[Samir]

It will be interesting to see what that router can and cannot do. A 10Gb router is still deep in the realm of enterprise equipment so the ISP will cut every corner they can to bring their cost down.

 

[Magebarf]

Thank you for the iperf3 server! I've been looking for one for a while now.

I have a cygwin ported version of iperf3 and it worked perfect with that server. Can't wait to see test results from a 10Gb line.

Another tip if you want to test "realistic" applications is that quite a few of the Ookla test servers (speedtest.net) offers alternative/additional http(s) frontends where you can download files up towards 1000MB in size. It can be a bit tricky to find out the right URLs for these standalone web versions though, I can't really recall exactly how I figured out the URL for my two nearest test locations.

One of them is http://speedtest.tele2.net/ at least, and that ISP has servers hosted in quite a few locations around the world.

 

[Rainmaker]

Thank you for the iperf3 server! I've been looking for one for a while now.

I have a cygwin ported version of iperf3 and it worked perfect with that server. Can't wait to see test results from a 10Gb line.

Serverius (NL) has an iperf3 speedtest available to the public also, and their web front end has some binary files for downloads as well. It gets a little more busy than the French one I linked earlier, as they only offer one port (5002). As you probably know, iperf3 can only use one connection/port at a time. By only advertising one public port the Serverius server is therefore also limited to a single test at a time. The French (bouygues.iperf.fr) server has multiple ports available, so can run more tests.

Another tip if you want to test "realistic" applications is that quite a few of the Ookla test servers (speedtest.net) offers alternative/additional http(s) frontends where you can download files up towards 1000MB in size. It can be a bit tricky to find out the right URLs for these standalone web versions though, I can't really recall exactly how I figured out the URL for my two nearest test locations.

One of them is http://speedtest.tele2.net/ at least, and that ISP has servers hosted in quite a few locations around the world.

Very true. I also can't remember how I found the URLs, but some do indeed host their own front ends. One such useful website is linked above your quote in this reply. Good luck with your connection, Magebarf. I'm very envious and would likely be shopping for hardware for a nice VyOS or similar setup in your position.

 

[Magebarf]

Alright, slow and steady progress.

In essence, the "consumer" router the ISP opened up pre-orders for still haven't been shipped, and latest ETA is at the end of month.

So, while waiting for that, I've started rolling my own solution, after coming across a thread on reddit by a user with the same ISP and connection as me; https://www.reddit.com/r/PFSENSE/comments/9ne9mb/pfsense_on_a_10gbit_home_internet_connection/

So, current phase is hardware acquisition.
Just as the user in the thread I linked to I actually managed to get a hold of a cheap Dell Optiplex 9020, with an Haswell i7-4770, and 8GB of RAM (likely to be expanded at some point whenever I come across a good deal on DDR3).
Much in line with the recommendations from @sfx2000 it has very good performance per core and runs at a high base frequency with a even higher turbo frequency (3.4GHz base and up to 3.9GHz turbo on a single core, while 4 cores loaded seems to stay at ~3.5Ghz).
Memory bandwidth as I've tested it is around 75Gbit/s.

I still haven't decided on exactly what NIC I'm going to use, as I see two paths (as I have a CPE consisting of a SFP+ to NBASE-T media converter with a SFP+ fiber module):
Either I go with a 2 x SFP+ NIC and move the SFP+ module there, and then use a DAC to connect over to the media converter.
Option 2 is go for a 2 x RJ45 NIC (10GBASE-T or NBASE-T, not that important as it's only going into a switch just next to it), and leave the media converter to do its thing.

For now, I just purchased a Solarflare SFN5122 NIC from ebay to start testing out the hardware.
I may possibly replace this with a Intel X520/540/550 at some point, if I find reason and/or good deals enough, or possibly a SFN7122 which can be found quite cheap on ebay (and is DPDK compatible).

So, first thing when receiving the NIC is testing out the performance of the NIC itself, both in bare metal mode, and also through ESXi.
If not too bad performance penalty with ESXi I may experiment a bit with "next generation" routing (DPDK and user land network stacks) while keeping to pfSense/OPNSense as a "stable" setup.

I anticipate to reach at least a few Gbit/s of throughput when dealing with larger packets, but even at imix I'm doubtful if the setup will perform near 10Gbit/s.
Will have to see around what number of pps it starts choking.

And on the topic of pps choking, that's my secondary use for this machine once the Huawei HN8255Ws arrives; I'm gonna torture the cr-p out of that box to see what speed it manages to maintain in and out with/withouth NAT and different Firewall settings. Who knows, if the Huawei is capable enough, the Optiplex may switch roles to a dedicated iperf node or some other funny task.

 

[sfx2000]

For now, I just purchased a Solarflare SFN5122 NIC from ebay to start testing out the hardware.
I may possibly replace this with a Intel X520/540/550 at some point, if I find reason and/or good deals enough, or possibly a SFN7122 which can be found quite cheap on ebay (and is DPDK compatible).

So, first thing when receiving the NIC is testing out the performance of the NIC itself, both in bare metal mode, and also through ESXi.
If not too bad performance penalty with ESXi I may experiment a bit with "next generation" routing (DPDK and user land network stacks) while keeping to pfSense/OPNSense as a "stable" setup.

Chelsio T520-SO-CR and Intel Intel X520-DA1/2 or similar... and then choose the cabling options from there...

Basically anything that's supported by FreeBSD 11...

Haswell i7-4770

Good choice here - and hint, turn off hyperthreading in the BIOS on that box - 4 cores/4 threads is very good, hyperthreading actually can impact performance on this specific task...

 

[Magebarf]

Chelsio T520-SO-CR and Intel Intel X520-DA1/2 or similar... and then choose the cabling options from there...

Basically anything that's supported by FreeBSD 11...

Yep. Both Chelsio and Mellanox are on my list of possible candidates as well.
My gripes with them is the pricing of Chelsio gear in Sweden, and at least with their newer generations the air flow requirements that Mellanox seems to be alone about (slightly higher power draw leads to that I guess).

I guess my first step will be to see how well the 2 x SFP+ approach works in practice, if it's better to keep the media converter and then leave 2 x RJ45 in the router, or if it works alright.

Second thing I really am curious about is all of these offload features, and how much of a performance difference the make. For the software routers, and general network stacks, I assume they need to fall back to the lowest common denominator, which means they avoid using vendor specific features like this unless they're highly integrated in the drivers. Is BSD better on this part, abstracting away offloading from the networking stack and drivers, compared to Linux, where it seems it's more of a wild west what offload features you can trust to be enabled?

Also, a big problem for me trying to snag NICs at "junkyard sales", or what you want to call ebay deals, is what models there are clones and copies of, and also for which of these the clones behave somewhat the same.
My understanding is that the Intel X540 is so widely cloned by now you can't really be sure when you find a deal.
Is the Chelsio boards as much a target for copies, or can you be sure to get legit NICs for now?

Good choice here - and hint, turn off hyperthreading in the BIOS on that box - 4 cores/4 threads is very good, hyperthreading actually can impact performance on this specific task...

Yeah, good note! I'll most likely turn Hyperthreading off permanently, but thought I'd test performance a bit before doing that call.
I also guess the bare-metal vs ESXi may come into play in this question as well.

Thank you very much for your feedback as always!

 

[sfx2000]

I also guess the bare-metal vs ESXi may come into play in this question as well.

Metal in your case is recommended - also consider HyperV on Win10Pro - it's interesting if one is deep into Windows

I don't have many kind words on ESX/ESXi - other than they are a good revenue source for the shareholders...

Anyways - single site - ESX/ESXi - not much benefit there compared to what one gets for free on bare metal....