Looking for replacement of power hungry SAN

[ddaenen1]

This probably going to be my last post on this. I finally found time to pull the server out of the rack and swap out the X8SIL-F motherboard and X3430 CPU with the newer X9SCM-F and E3-1230. It booted instantly but i did experience network issues as the NIC's on the X9 are different from the X8 and since i was running the existing install, that didn't go down well. Doing a new install initially fixed this, but when uploading my config into TrueNAS, it was messed up again. Took me some time to figure out how i could get it to work through the shell. Now all is up and running really well. I am quite surprised with the noticeable performance improvement from the E3-1230 compared to the X3430. Both Plex and Nextcloud seem to running smoother and especially when playing 4K UHD movies through plex, the difference is huge. Where previously, a lot of buffering was needed and the film sometimes stopped to buffer again, all this is completely gone now.

I think i will leave this to run now for some time, unless ofcourse i bump into a Supermicro X11 motherboard and matching CPU for cheap

 

[ddaenen1]

Well, i am at it again. Since i had an EV charging point installed at my home as i went electric several months, that also came with a real-time power meter. I can now see my real-time power consumption and as we were away during x-mas, i noticed that my average daily power consumption when everything is on idle is around 10kWh per day. Makes sense when i had 5 kWh in the night after the mods i made last time but i also know now that 50% of that is my network consumption so about 5 kWh a day.

Now things are different: i own a limited company (LC) and any investments i need to make in the network, can be done through the company whereas the power consumption is private with a small portion charged back from the LC.

Which is making me wondering:

- What power consumption reduction would i gain from exchanging the four 3.5" SAS drives in the NAS by 2.5" SSD's?
- Would i really benefit from exchanging the current X9SCM-F and Xeon E3-1220v, X550-T@ combo in the pfSense box by something like a Supermicro X11SDV-8C-TLN2F board?
- Maybe time to swap out the Cisco WAP571's (although i am very happy with those) by Cisco 150AX AP's of which read they consume a lot less power?

Open to any suggestions and comments/remarks.

 

[Tech9]

Calculate what's the difference in electricity bill and how long it is going to take to recover your new hardware cost.

 

[ddaenen1]

Calculate what's the difference in electricity bill and how long it is going to take to recover your new hardware cost.

Well, that is in this case less relevant because for the LC the cost of a couple of thousand is really insignificant but any savings from reduced consumption go directly into my own pocket.

 

[Tech9]

You own the LC and the cost of new equipment still comes out of your pocket, no? If there is some tax play - do it.

 

[Tech9]

3.5" SAS drives in the NAS by 2.5" SSD's

All my data servers run on HDDs, upgraded in 2022.

Supermicro X11SDV-8C-TLN2F

Much faster, but not much power savings. Not needed for home setup.

Cisco WAP571's (although i am very happy with those) by Cisco 150AX AP's

May be worth it for Wi-Fi performance, but not much for power savings.

 

[coxhaus]

I don't run any high wattage CPUs anymore. I might have 1 if I was to game again. I also don't spin any hard drives as I buy SSDs. It all adds up. My rack is turned off. I tried a bottom of the line NAS but I had power problems after a while, so I am running a small Dell low wattage PC for my NAS now and it draws a little more power but no issues so far and it is using 2 SSDs, one for boot and another for storage. When it gets a little older, I may mirror the NAS SSD part.
I don't think swapping Cisco WAP571's for Cisco 150ax will save you much power. It will give you Wi-Fi 6.
My older Cisco WAP581 worked well and had better roaming. My Cisco Wi-Fi 6 150ax are faster but they drop a couple of words on roaming. They do not drop WiFi calls. It could be the Wi-Fi 6 spec as it is new. I don't know.

What I remember SAS drives pull some power so maybe there could be some savings there. They are designed for fast access for big networks at 10,000 rpm.

When I had my rack, I changed motherboards and CPUs all the time. I never bought servers, just motherboards and I used the same cases and power supplies. I only had one power supply die over many years, but I was not loading down the power supplies except for my hot swap 8 drive server. Nowadays I would not run that anymore as I am now more power conscious now like you.

I had a 20-amp dedicated circuit run for my rack and 220v circuit run for my Window AC to cool my rack. The room was too hot without a separate AC even though I had central AC for the house. I don't think like this anymore.

 

[ddaenen1]

You own the LC and the cost of new equipment still comes out of your pocket, no? If there is some tax play - do it.

Indeed there is. I work as a consultant through the LC so all the revenue goes into the LC and i pay myself a salary. As salaries are heavily taxed over here, it is more beneficial to to leave as much as possible money in the LC and pay as much as possible costs directly through the LC so it isn't taxed and you can recover VAT.

 

[Tech9]

I do it myself for number of things for the same reasons. Happy New Year!

 

[ddaenen1]

- Would i really benefit from exchanging the current X9SCM-F and Xeon E3-1220v, X550-T2 combo in the pfSense box by something like a Supermicro X11SDV-8C-TLN2F board?

Pulling this thread from under the dust again as i am unsure this deserves a new thread. I have always been happy with the above pfSense setup and it is running flawless but i have been wanting to revert back to pfSense CE given the change in direction from Netgate and when i got the opportunity to acquire a Supermicro X10SDV-4C-TLN2F (Xeon D-1521 with 2 x 10Gbe ports and IPMI) last week i couldn't resist as i was able to buy it for a measly 50 Euro.

The plan is to put that in a SC505-203B enclosure with front I/O access which is also something i wanted for some time already and squeeze in 2 x 16Gb DDR4 RDIMMs. I haven't really made up my mind if i am going install pfSense on an NVMe drive instead of the current ZFS mirror setup or keep the 2 SSD's i have in the current setup. With NVMe being as reliable as SSD these days i don't even know if there is any use in using SSD's anymore.

My thinking is that it should definitely consume less power given that the idle power should be somewhere around 25W instead of Xeon E3-1220v2 with TDP of 69W together with the X550-T2. It will also make my pfSense box a bit more future proof and should be plenty fast to keep up if i ever decide to spend money on a 10-Gig switch.

Will keep you posted on progress...

 

[coxhaus]

You are going to notice a difference between a 69-watt CPU and a 25-watt CPU. Heat wise, costs wise not so much. Your CPU fan should be quiet at 25-watt.

My 10gig Cisco switch runs fans in the summer here in Texas during the day. This is Cisco small business switch so it is much quieter than a Cisco enterprise switch which is designed for a wiring closet but you can still hear it.
My granddaughter has grown up and does not have enough time for us so it might be time to put it into play now since it is in her old room. To use it I must switch to Pfsense CE also. I think I will switch when Pfsense 2.8 CE comes out.

I may need to buy another NIC as my 10gig Broadcom NIC had issues running last time I used it. It took like 100 ms to decode stuff in Pfsense last time I used it.

A while back I figured my sever rack costs me about $10 a month to run and if you add my 220v room AC to cool it in the summer it added an additional $10 to $15 a month. Without room AC even though I had central AC I still sweated on my keyboard. So, room AC was required in the summer.

 

[ddaenen1]

The board came in but my stash is limited nowadays so except for a 256Gb M2 SATA, all needed to be ordered which i did:

Supermicro CS-505-203B chassis
2 x 16Gb 2x4R DDR 2133 RDIMM

Both on their way. I might need a couple of fans but waiting until i get more visibility on the passive cooling. Next up is putting it all together and testing the setup. If that works out fine, i'll install pfSense CE 2.7.2 on it and upload the config. from my active box. Do some more testing and then the most critical step...switch over current with new where timing will be key considering that school vacation is starting. This may well be in the 2nd half of july.

 

[coxhaus]

The board came in but my stash is limited nowadays so except for a 256Gb M2 SATA, all needed to be ordered which i did:

Supermicro CS-505-203B chassis
2 x 16Gb 2x4R DDR 2133 RDIMM

Both on their way. I might need a couple of fans but waiting until i get more visibility on the passive cooling. Next up is putting it all together and testing the setup. If that works out fine, i'll install pfSense CE 2.7.2 on it and upload the config. from my active box. Do some more testing and then the most critical step...switch over current with new where timing will be key considering that school vacation is starting. This may well be in the 2nd half of July.

Too bad you are not close as would have sold you my server rack cheap. I have 3 SuperMicro servers in it which you can replace the motherboard in. It is a four-foot rack on wheels with ball bearing pull outs. It also has a large rack mount APC in the bottom.

 

[ddaenen1]

Seems my setup is very similar to your old one. 4 foot rack, APC in the bottom, 2 Supermicro servers (1 pfSense and 1 Truenas) and also a Synology as file server. I could have opted to pull out the current pfSense box and reuse the chassis but it would cause too much down time as the kids are currently stuying for the exams and also i like to test new hardware thoroughly with pfSense before adding it to my network, especially the HAproxy/ACME cert packages as they provide external access to my cloud which i use for business purposes.

 

[ddaenen1]

Small update: parts came in yesterday and initial assembly done. Couple of setbacks. The I/O shield that comes stock with the chassis is not compatible with the board. Need another one. The CPU runs quite hot with the passive heatsink. Not sure if this is because of the initial boot with open case and no fans. For the time the testing, i used a 40mm case fan on top the sink which did the trick to keep it cool. An active fan seems available for this board but hard to come by, at first glance. The system boots fine and the memory and M2 SATA stick working fine too. IPMI also works as should but i encountered a problem with the 10Gbe NICs. They are visible in the BIOS and IPMI but a quick test install of pfSense reveals a problem. When booting, the system returns an NVM checksum mismatch on both NICs and then halts. Contacted Supermicro support and apparently this is a known issue so they sent me a link to a file to update the NVM. I need some time for this so probably a weekend job.

 

[ddaenen1]

Not good news on the NVM issue. The NVMupdate didn't succeed. I shared all findings with the Supermicro support rep and he eventually shared with me that this board can be salvaged but would require repair by Supermicro but since it is has passed the age of 5 years after out of warranty, this is not possible anymore. The software to do this is not available for the end-user. I have tried another attempt with the NVMupdate tool and some additional flash parameters but no luck either. I am not giving up yet but i could end up with a dead board. Just for completeness, the seller has a no-return policy as it is a refurbished board. From what i read, this issue can occur when updating the BIOS. As both BIOS and BMC are on the latest level, this is where it went wrong. Alternative would be to plug in a PCIe NIC like the X550-T2 i currently have in the other box. Thinking about maybe using an SPF+ card like an X520 instead as they are cheaper, less power hungry and i could connect it to the switch with a DAC cable (if that works with 1G SFP on one side and 10G SFP+ on the other side). The only issue is see is that WAN-side i currently have RJ45 and i understand that SPF+ RJ45 modules are expensive and quite sensitive.

Thoughts?

 

[coxhaus]

Sorry for the late reply as I was up fishing in Alaska last week.

I would try turning off the on-board NICs and see if you can run them off after market Intel NICs. Check to make sure your PICe slot is big enough.

 

[ddaenen1]

Sorry for the late reply as I was up fishing in Alaska last week.

I would try turning off the on-board NICs and see if you can run them off after market Intel NICs. Check to make sure your PICe slot is big enough.

This is probably where this is going. I have a Supermicro PCIe riser and an i350-t4 lying around that i am going to use for a first trial. Should work fine as i did something similar with the X9SCM-F board and an X550-T2. If that works, i will probably get another X550-T2 or even an X710-T2L to make it 2.5/5/10Gb

 

[sfx2000]

Sorry for the late reply as I was up fishing in Alaska last week.

And... this is a good thing, IMHO...

As we go into our end-days, the career accomplishments are something looking back - yes we can take lessons learned and share...

But things like Golf and Fishing and a bit of travel - they're all good.

If you have a bit of time, consider the Bourbon Trail over in Kentucky - great stories, and yes, a fair amount of tasting, so finding a designated driver, lol...

 

[coxhaus]

And... this is a good thing, IMHO...

As we go into our end-days, the career accomplishments are something looking back - yes we can take lessons learned and share...

But things like Golf and Fishing and a bit of travel - they're all good.

If you have a bit of time, consider the Bourbon Trail over in Kentucky - great stories, and yes, a fair amount of tasting, so finding a designated driver, lol...

Yes, I have spent a week in Louisville, Kentucky drinking some fine Burbons at different distilleries.