Spec For Software RAID 20 Discs

[nullack]

Hi,

So for my fileserver I am going with software raid over hardware raid. I want to have the benefit of filesystems like ZFS and BTRFS that have advanced checksumming at the file level that is impossible to do under a hardware raid controller. The other key advantage is that by using BSD or Linux software raid, I’m not forced to use enterprise grade drives that feature TLER, as in software the HDD is allowed to go into deep recovery and the software wont falsely drop it from the array like a HBA card will.

Let’s say I’m exploring the Norinco 20 bay hot swap sas/sata rack mount enclosure for a 20 disk fileserver. What exactly are my options for hooking them all up?

Looking at the detail I’m a bit confused generally about multilane expanders, single lane expanders, SATA port extenders and all the gizmos in this storage space

I don’t want to buy a hardware raid card with say 24 ports, because thats expensive and I don’t need it

Seeing that modern sas and sata drives are on 6 Gb/s bus links, I don’t believe that I need a full lane for each of the 20 disks because the sustained throughput of 7200 rpm drives is well under the 6Gb/s limit. Ok in caching it’s a different story but I’m willing to sacrifice some maximum burst for reduced costs. Most 7200 rpm SATA drives seem to peak around a sustained 120 MB/s or so which is well off from the 600 MB/s that the SATA III bus provdes.

So I was thinking, what if I had a single SATA III 6 Gb/s pcie card with four multilane miniSAS connectors. Then, I could use a backplane extender for each of those four ports, for single lane 5 sata III ports to the one minisas port. So that would give me a total of 20 ports, where from the backplane extender with 5 drives across one SATA III bus, thats a shared 120MB/s of bandwidth allocated to each individual drive.

Is this possible? Can anyone please help with actual part URLs and some guidance on specifics please.

Thanks

 

[GregN]

Oh, I am so bummed! Peeved even. I wanted to be the first guy to try this.

Ok, take a look at the backblaze forum posts, and the backblaze blog (2.0 design) uses the expander backplanes from somebody called Chyang Fun Industry. Most of the site in in chinese, so you'll probably have to e-mail them.

The BackBlaze design isn't built for speed, so you'll probably want a different SATA card. You going PCIe I assume, not PCI-X, right?

PCIe2 x4 SATA III is tough, SATA II is much easier, SuperMicro appears to make a well regarded II card , the other is the Syba which does SATA III. Make sure to stick with FIS based switching. Two of the Sybas, you might look for a four port card

The HighPoint (not a fav ) cheap raid card appears to be on sale at newegg.

Beware the Norco 4220 case is noisy, but price is right.

One of the advantages of Raid cards is they support staggered spin-up, which the sata cards generally don't. You'll need a larger PSU.

What Chipset are you looking at? The new Sandy Bridge boards look attractive.

Hope that helps

 

[nullack]

Thanks mate

Looking at a UP system with:

* Xeon E3-1275 4 core, 8 with Hting 3.4Ghz 8MB L3 cache
* CPU Includes an on die HD Graphics P3000 GPU
* 16 GB (4GB sticks) of ECC UDIMMS because the chipset doesnt support RDIMMs, I'd prefer 32GB but no one makes reasonably priced 8GB ECC UDIMMS
* Some sort of UP Intel / Tyan / Supermicro LGA1155 mobo with quad Intel NICs
* These sorts of boards all seem to use the Intel C200 series chipsets

Yeah I looked at the backblaze setup and they are limited by going for SATA II not SATA III, plus I think they are a bunch of cowboys for selling 135TB high density cloud storage without using ECC RAM

So possibly two of the Sybas for the needed 4 SATA III lanes

Or the rocket raid which is a nice price but I dont know how to set it up so its not RAID but just presenting the disks as seperate devices for software raid. This isnt JBOD either as you'd know. Im not sure if the card could even do it?????

Where does the noise come from in the 4220? The mobos I'm thinking of are all passive except the CPU cooler. I'll use an aftermarket CPU cooler and being 4U high I've got room for a nice one. I'll turf the Norco chassis fans and run some decent Noctua ones or similar. Have my eye on the gold Corsair PSU which is very quiet.

I'll look intyo FIS based switching I dont know what that is.

I'm stuck on the expanders. I see on the Norco 4220 site they say:

"Five internal SFF-8087 Mini SAS connectors support up to twenty 3.5" or 2.5" SATA (I or II) or SAS hard drives;"

So is this case limited to only SATA II?

And do I need to go for mini SAS out on the pcie card, hook into one of the five mini SAS backplanes and then finally plug in a SATA III HDD into the hotswap caddy? In which case I now need 5 mini SAS ports to provide all the lanes I ideally want?

EDIT: Some updates. I don't think this 4220 case supports 6Gb/s SAS backplanes, I think it is SATA II backplanes only. Seems the price of 6 Gb/s mini sas backplane server rack chassis rises fairly quickly. The price/performance sweet spot could be better at a 16 drive chassis with a 4 port SAS HBA, and I use larger capacity disks. Which is better from a reliability point of view because each added drive increases the risk of failure. Better to run higher capacity drives in less numbers I think.

 

[nullack]

Ok so I've learnt a few things. I'm an IT pro but not a storage expert.

The pcie card I want is a host bus adapter. The difference between a HBA and a raid card is that the HBA will present the disks as is, without hardware raid, which is exactly what I want.

Only SAS supports multilane and expanders, not SATA.

SATA drives are physically compatible with SAS, but not the reverse.

So I want a 6 Gb/s SAS HBA. The Norco has SAS backplanes, 5 of them. Each one of the 5 backplanes goes to four hot swap drive caddys.

What I'm confused about now is if I want atleast 5 mini sas ports on my 6 Gb/s SAS HBA card for a lane one each of the 5 backplanes

OR

Is a less port HBA OK and I use SAS expanders?

I'd like to understand the performance implications of this.

If any of my posts are wrong, please advise me

 

[GregN]

* Xeon E3-1275 4 core, 8 with Hting 3.4Ghz 8MB L3 cache

Nice. Going new not used then. Probably more muscle than you need. Dual core i3 is probably enuf for a NAS.

* CPU Includes an on die HD Graphics P3000 GPU

Shouldn't need it, Ol' Shuck runs headless.

* 16 GB (4GB sticks) of ECC UDIMMS because the chipset doesnt support RDIMMs, I'd prefer 32GB but no one makes reasonably priced 8GB ECC UDIMMS

Even the 4Gig sticks are not cheap, going used can save you some cash. Solid state and all that.

* Some sort of UP Intel / Tyan / Supermicro LGA1155 mobo with quad Intel NICs

I'm a huge fan of SuperMicro, they phone supported my used antiquated, no longer extant 2005 MB. And I think their products are top notch. Intel NICs is a must have, but dual is enuf unless you plan on running alot of VMs.

Yeah I looked at the backblaze setup and they are limited by going for SATA II not SATA III, plus I think they are a bunch of cowboys for selling 135TB high density cloud storage without using ECC RAM

I think their Ideal is lowest dollars per Terabyte, performance is not a priority since it is across the net.

So possibly two of the Sybas for the needed 4 SATA III lanes

That appears to be the choice of the day if you want SATA III. SuperMicro has a III board but I think it is pricey.

Or the rocket raid which is a nice price but I dont know how to set it up so its not RAID but just presenting the disks as seperate devices for software raid. This isnt JBOD either as you'd know. Im not sure if the card could even do it?????

I'm not sure what you mean, I use my 3ware card to presents single disks and the OS handles it, a drive within a volume, no problem.

Where does the noise come from in the 4220? The mobos I'm thinking of are all passive except the CPU cooler. I'll use an aftermarket CPU cooler and being 4U high I've got room for a nice one. I'll turf the Norco chassis fans and run some decent Noctua ones or similar. Have my eye on the gold Corsair PSU which is very quiet.

The Norco-4220 is a fan brace between the MB and the drives, it has 4 80mm fans, plus two in the back. The specs say Sata III, are you looking at their site? ( more pictures from AVS, where the Norco is a fetish object )

Norco has a 3 120mm replacement fan brace, if you search for it you'll find it. You provide your own 120mm fans....

The Corsair PSUs have a nice warranty, and the modular ones are easy to work with. You may need an extra connector for the Server MB, floppy power yes, extra MB no...

I'll look into FIS based switching I don't know what that is.

FIS is for SATA multiplier command handling. Frame something or other, it allows multiple drives to be addressed at the same time, instead of blocking on a single command. Some SATA boards do port multiplication but not really, and don't have that as a feature.

I'm stuck on the expanders. I see on the Norco 4220 site they say:

"Five internal SFF-8087 Mini SAS connectors support up to twenty 3.5" or 2.5" SATA (I or II) or SAS hard drives;"

What site, the site I referenced says SATA III in the specs, which I'm running in my case without an issue. Is there an Aussie version or something?

And do I need to go for mini SAS out on the pcie card, hook into one of the five mini SAS backplanes and then finally plug in a SATA III HDD into the hotswap caddy? In which case I now need 5 mini SAS ports to provide all the lanes I ideally want?

Look at the pictures you have twenty sata connectors, the SAS connectors are HBA board specific, none in the RPC-4220. Not sure how you'd handle it, card to backplane to Norco connector.

 

[nullack]

Greg thanks very much, Ill share a beer with ya if you ever come to Aus

I think the Norco has a SATA II backplane, and while SATA III will work it is doing so in backwards compatability mode. I've emailed Norco in relation to this. AFAIK there is only one model internationally.

What I'd like is a mini SAS backplane enabled chassis. I've got my eye on this Chenbro:

http://www.chenbro.com/corporatesite/products_detail.php?sku=144

I could run a 4 port mini SAS HBA, and have a full 600 MB/s going to each of the four mini SAS backplanes. Thats 150 MB/s per drive, being four into 600 MB/s for each lane.

No port extenders, no SAS expanders, no more than a single pcie SAS HBA, just four 80887 internal mini SAS cables nice and neat and F A S T yeah yeah

EDIT: trying to find prices, found one for $USD619 so far at http://www.spectrumservers.com/ssproducts/pc/viewPrd.asp?idproduct=238

 

[nullack]

Just an update in case anyone is following my misadventures with this

Assuming I find a mini sas enabled backplane chassis for the right price I have this HBA in mind for the shopping list:

http://www.lsi.com/products/storagecomponents/Pages/LSISAS9201-16i.aspx

It's a 16 port internal SAS / SATA 6 Gb/s HBA

With it's four mini sas 8087 ports, it can support 2400 MB/s continuously through it's pcie x8 interface. Obviously this will be cache burst only from my HDDs but it's certainly allot. Hmmm how I wish 10GB/e switches were cheap LOL

 

[GregN]

No port extenders, no SAS expanders, no more than a single pcie SAS HBA, just four 80887 internal mini SAS cables nice and neat and F A S T yeah yeah

EDIT: trying to find prices, found one for $USD619 so far at http://www.spectrumservers.com/ssproducts/pc/viewPrd.asp?idproduct=238

If ya got the budget, the Chenbro is nice, I can also recommend the Supermicro cases (with cooling separators). The Chenbro is like two to three times more expensive than the Norco. My Selection of the Norco was because it was cheap (SAN under $1K).

If you go mini-SAS (I like it, looks good) take a look at the non-raid LSI or even better Areca HBAs (smallest is eight, but is very very sweet )

Folks have been talking about buying used IBM M1015 boards and flashing them with the most recent LSI firmware giving you both 3TB and 6Gbps support.

 

[nullack]

So I hear back from the 4220 people,

"Thank you for contacting us about the backplanes. The backplanes does
support sata III. The description is outdated and was not updated. If you
have any other question please don't hesitate to ask.

Best Regards,
Raymond Hsu"

So the backplane is a SATA III backplane in that chassis