Teaming on Quad NIC + win server 2012 + unmanaged server

[Denywinarto]

Hi, we have a winserver 2012 with intel quad nic which is used as a ccboot server (pxeboot) to 30 clients, due to the switch rarity in my country, we prefer to use nonmanaged switch.. My question is, if we team a quad nic, then do we need 4 nonmanaged switches to get the benefit for clients? Also which teaming mode is better for pxeboot? Thanks
Edit : sorry i mean unmanaged switches, cant edit title

 

[System Error Message]

Not a great idea. You need managed switches in order to benefit from LACP as bonding needs to be done at layer 2 as smoothly as possible. Get a managed switch that supports teaming used by servers/routers not just with other switches.

With unmanaged switch the only possibility is failover.

If you dont do teaming you can use 4 unmanaged switches. Simply configure each NIC as a seperate network instead and you will need to configure the server for PXE boot on all 4 networks. All you would have to do is equally divide clients amongst the switches. Each NIC will connect to a seperate switch so this makes internet access complicated unless you have another NIC on your server with your server acting as gateway.

 

[Denywinarto]

Not a great idea. You need managed switches in order to benefit from LACP as bonding needs to be done at layer 2 as smoothly as possible. Get a managed switch that supports teaming used by servers/routers not just with other switches.

With unmanaged switch the only possibility is failover.

If you dont do teaming you can use 4 unmanaged switches. Simply configure each NIC as a seperate network instead and you will need to configure the server for PXE boot on all 4 networks. All you would have to do is equally divide clients amongst the switches. Each NIC will connect to a seperate switch so this makes internet access complicated unless you have another NIC on your server with your server acting as gateway.

Then i guess this would be better ?
Server quad nic 》8p managed switch 》 16x2 unmanaged switches 》 30 clients

Would the clients still get the benefit from teaming?

52p managed switch is rare in my country and most likely i have to import it... and thats not a good idea in the long term for warranty reason.. but 8p 16p still exists i think..

 

[System Error Message]

a 48port semi managed switch is the ideal solution as you could plug in 30 clients and 4 ports of your server.

Otherwise if you cant because of cost or physical locations, you will have to use 4 unmanaged switches, each connected directly to server for different network.

If you require everyone to be on the same layer 2 network than what you suggest can be used but your LACP efficiency is reduced.
I'll try put this into perspective with a simple drawing (hope you're using widescreen)
Server====|Managed|---unmanaged switch
NICs ====| Switch | --- unmanaged switch
This is your suggested configuration. 2 of the server NICs are just useless because there is only 2x1Gb/s links after managed switch so to make full use you need 4 unmanaged switches rather than 2. Dont try to connect 2 cables to the managed switch from unmanaged as it will create loop for unmanaged or the managed switch may just use failover if STP is used.

However if you use managed switches everywhere, you can use a cheaper managed switch like this-
Server====|Managed|=== cheap semi managed switch
NICs ====| Switch | === cheap semi managed switch

As long as the cheap switches support LACP you can make more effective use of your server's LACP as switch to switch LACP is much simpler for the switch than server to switch LACP because server can use protocols the switch cannot. If you look at mikrotik wiki about bonding it will list the types of LACP and even mikrotik only allows for varying protocols on bonding on CPU interfaces rather than switch based LACP and this limitation is apparent on all switches.

There are many brands you can consider. For example you can use cisco semi managed switch (SG series, pretty sure many can suggest what models) and for your cheaper ones you can go with netgear, tp-link, etc. When it comes to reliability i wouldnt count on tplink or dlink (talking about semi managed) but netgear switches do well in reliability but they tend to have buggy firmwares so with netgear its more of a question of what feature is buggy. The cisco semi managed line will definitely support teaming with your server while when using the netgear or other cheap semi managed switch there will be less compatibility issues when doing switch to switch LACP. The other reason for using cisco is more STP variants which is important in a multi switch config. If using RSTP set it to root switch and also make sure to use RSTP on other switches. You wont get this on unmanaged.

 

[sfx2000]

Hi, we have a winserver 2012 with intel quad nic which is used as a ccboot server (pxeboot) to 30 clients, due to the switch rarity in my country, we prefer to use nonmanaged switch.. My question is, if we team a quad nic, then do we need 4 nonmanaged switches to get the benefit for clients? Also which teaming mode is better for pxeboot? Thanks
Edit : sorry i mean unmanaged switches, cant edit title

Kind of - Win2012 Server can build VLAN's, and most modern switches will honor those VLAN tags - just be aware that you do lose some flexibility with unmanaged switches, so you'll need to be pretty precise with how you build the VLAN's with regards to policies... but once there, you can stand up AD, and do some policy management/enforcement based on user account profiles.

For pxeboot on the server - keep it on a private LAN port reserved for pxeboot and management of the WAN/LAN interfaces (e.g. that port is to manage the server).

 

[Denywinarto]

Sorry for the late reply.. been scratching my head looking for most viable solution...
Thanks for the answers System Error Message and sfx2000
Guess i have to go through managed switch option... It's too risky using dumb switches..

D-Link Gigabit 28p + 4 SFP port is still available in my country,
Planning to use it with 10G NIC on server.. the diagram is like this :

Diskless Server + single 10G NIC ===> SFP port to 28p Gigabit switch +SFP module ==> Gigabit Clients

if it works, single 10G is still better than the 4G i have now
Question is, is this effective or is it going to cause bottleneck?
I still dont understand the data flow in switch that mixes SFP port with Gigabit port

thanks

 

[System Error Message]

They're usually called uplinks or server port. SFP can do a lot such as supporting various different fibre optics so when building a backbone it helps. for 10G you need SFP+.

If you are uploading from server to all 20+ clients simultaneously you'd still get around 500Mb/s per client so thats not bad. If you were to combine 2 SFP+ ports that will eliminate the network bottleneck. Your server will need some fast drives and some configs to properly make use of it. Just plugging in 10G and using a ramdisk doesnt mean you can instantly use 10G, many have made videos covering it and results depending on many things.

 

[Denywinarto]

They're usually called uplinks or server port. SFP can do a lot such as supporting various different fibre optics so when building a backbone it helps. for 10G you need SFP+.

If you are uploading from server to all 20+ clients simultaneously you'd still get around 500Mb/s per client so thats not bad. If you were to combine 2 SFP+ ports that will eliminate the network bottleneck. Your server will need some fast drives and some configs to properly make use of it. Just plugging in 10G and using a ramdisk doesnt mean you can instantly use 10G, many have made videos covering it and results depending on many things.

looks like you're right SFP doesn't always mean 10G...

https://www.amazon.com/dp/B00AI9H628/?tag=snbforums-20

the SFP module that D-link provide only has data rate of 1 GB..
https://www.amazon.com/D-Link-DEM-311GT-1000BASE-SX-Mini-GBIC-Ethernet/dp/B000087LFF/ref=pd_sim_147_1?ie=UTF8&dpID=41MkF22dfeL&dpSrc=sims&preST=_AC_UL160_SR160,160_&psc=1&refRID=YN9JY9SH9TER6SBE1H69

Well i guess its not worth it.. it's still less than 4Gb that i get from teaming Quad NIC now..
have to search for another switch with SFP+..

 

[System Error Message]

looks like you're right SFP doesn't always mean 10G...

https://www.amazon.com/dp/B00AI9H628/?tag=snbforums-20

the SFP module that D-link provide only has data rate of 1 GB..
https://www.amazon.com/D-Link-DEM-311GT-1000BASE-SX-Mini-GBIC-Ethernet/dp/B000087LFF/ref=pd_sim_147_1?ie=UTF8&dpID=41MkF22dfeL&dpSrc=sims&preST=_AC_UL160_SR160,160_&psc=1&refRID=YN9JY9SH9TER6SBE1H69

Well i guess its not worth it.. it's still less than 4Gb that i get from teaming Quad NIC now..
have to search for another switch with SFP+..

I use a mikrotik CRS226 as that has 2 SFP+ and at the same time probably one of the cheapest with SFP+ and managed. The only problem with the CRS is that its an incomplete switch, still under development so the main 2 issues with the switch itself is lack of STP (loops can form so changing the network it is in frequently can also cause hang) and it sometimes can have issues with SFP+ interface (this does show up in log as link up and link down a few times). Price is good and mikrotik's own SFP+ to SFP+ direct cables are very cheap and so are used SFP+ cards. Ubiquiti does have similar switch line too. It is also fully managed and uses different terminologies.

 

[Denywinarto]

I use a mikrotik CRS226 as that has 2 SFP+ and at the same time probably one of the cheapest with SFP+ and managed. The only problem with the CRS is that its an incomplete switch, still under development so the main 2 issues with the switch itself is lack of STP (loops can form so changing the network it is in frequently can also cause hang) and it sometimes can have issues with SFP+ interface (this does show up in log as link up and link down a few times). Price is good and mikrotik's own SFP+ to SFP+ direct cables are very cheap and so are used SFP+ cards. Ubiquiti does have similar switch line too. It is also fully managed and uses different terminologies.

Yeah thats very tempting, and there's authorized dealer in my city.. but i googled and found some bad reviews about it... link down and hang are pretty fatal for my business, probably could work around link down with active backup teaming.. but hang means all clients offline.. does it happen often?

I also found Quanta LB4M but apparently its discontinued and only available second hand in ebay.. ever consider it?

 

[System Error Message]

I once did have issues with the link downs but i updated the firmware which helped. The other thing is that if you are using LACP than a link down wont matter.

Link downs only occur after a while but that is mainly due to the loops that can accumulate in the switch itself (lack of STP for example). If you end up with link downs than try not using LACP. After months of owning and using this i've only had 2 link down issues with my CCR and the switch chip hanging (loop issue again). However STP will be introduce in ROS 7 for switching.

Switches has never been mikrotik's strong point but if you need 10Gb/s at a low price it is something you can consider. There are other variants with 10Gb/s too.

There are many 2nd hand server cards on ebay, i even bought one which refused to start on me. My problem is with OS compatibility but if you use server OS that shouldnt be an issue. Just beware of hidden whitelists.

 

[sfx2000]

There are many 2nd hand server cards on ebay, i even bought one which refused to start on me. My problem is with OS compatibility but if you use server OS that shouldnt be an issue. Just beware of hidden whitelists.

Some of the intel server cards are dual-mode - ethernet and infiniband - so check the card there for a mode switch...