Looking for tips on building a large capacity file server

[Jorsher]

Hi!

I've been Googling around a bit for information but don't quite know what the options are.

I would like to build a ~24+ drive file server at the lowest cost possible without resorting to low quality parts. It will be used for streaming my media around the house. I'm a huge movie and music junkie, and prefer to rip high quality copies of my media to a network drive so that I can access it from any room (or anywhere with internet) without having to search for a disc. Being able to quickly view things like imdb information from the media player is also a great benefit to digitizing my collection. I'm currently at about 10TB of media spread across multiple drives, which is a hassle to work with.

Hardware or software?
I planned to use software RAID instead of hardware RAID to save money. This is another topic where I've read conflicting things. Some say it's not reliable, but I've also heard that it's just as safe to use as hardware RAID. I planned to install the OS on a drive separate from the RAID. If that hard-drive failed, would I lose the ability to read the RAID when I replaced the drive, or would keeping a backup of the OS be wise?

Is the CPU the determining factor of the performance (other than drive speeds), or is something else the bottleneck? I planned to use a low-power quad-core desktop processor, but I don't know if RAID software supports multi-core. If it doesn't, I'd thinks a higher clock single or dual core proc would be the better route.

ZFS RAID-Z3
I would like to use a triple-redundancy RAID. From what I can tell, this is only available on OpenSolaris, but I'm interested in knowing about other OSes that might support this.

Deduplication
Does anyone have any idea of how beneficial this would be on a media server? I understand that media doesn't compress well, but I think across a huge amount of data, there may be enough duplicate blocks to make it worthwhile. I assume a smaller block size would increase the probability of duplicate blocks, but I wonder how bad it will effect performance using small blocks for huge files. Any experiences?

How to hook it all up?
This is probably the source of my most confusion. What would be the lowest-cost solution for handling this number of drives? I am looking at this card. From what I gather about that card, port #8 is always used to connect to the RAID/HBA, optionally (and preferably) you could connect port #9 to the RAID/HBA to double the bandwidth, however all ports except #8 can be used for up to four hard-drives each, giving it the ability to handle up to 32 drives. My question is, what parts am I looking for to actually connect the drives, and what am I looking for to connect to ports #8 and #9? Is there a better, cheaper way of connecting 24-32 drives to use in a single RAID volume? I'm not fishing for exact parts (although they'd be welcome!), and I'd be satisfied with some keywords that'll lead me to what I need.

Hard-drives
Now, for the most important part. I would like to use 2TB drives. They are at a great price/capacity point. I wanted to use WD20EARS drives, however I've heard conflicting reports on whether these Green drives should be used in a RAID or not. I know enterprise grade drives are recommended, however they are much more expensive, I've never had a drive fail (knocks on wood), and no important data will be stored on here. In short, if I can use 24 "economy" drives (of any brand) in a RAIDZ3 with a decent chance that I'll never lose four drives at once, I'm willing to risk this (relatively) unimportant data. I like the Green drives simply because they're from a brand I trust, cheap, quiet, and low power. This will be in my home so I do not want a loud heat-pump

I'm not trying to cheap out, I would just like the best bang-for-buck for my particular situation. Although reliability is a concern, it's not data that I can't replace.

Apologies for this wall of text and the numerous questions. When I complete this project, I will be sharing the information and the build process. Credit will be given where credit is due, and I appreciate any tips and info in advance!

 

[Jesse B]

I would go the ZFS route using 2TB Samsung F4 drives personally. I know that any Solaris or BSD derivative supports ZFS. This leaves you with many options to choose from. ZFS doesn't require expensive RAID cards, and yields very high performance. I think this would be a excellent choice for your applications. I've seen a few 40TB/100TB etc. builds running off ZFS, and they seem to be working quite well.

One thing to note is that ZFS will eat A LOT of RAM up. This is just part of its nature. You can use an SSD as a cache as well, which will help out speeds.

What kind of network do you have set up as well? Is it all Gigabit?

 

[PeterO]

Hi Josher,

Perhaps you've already come across these links but here are a few that I found very insightful in concept and practice when building my very, very humble file server.

ZFS -- Constantine Gonzalez:
http://constantin.glez.de/blog/2010/02/seven-useful-opensolaris-zfs-home-server-tips


ZFS -- ECC and Registered RAM (James Hamilton):
Note the "Categories" section on the right for more insight
http://perspectives.mvdirona.com/2009/09/03/SuccessfullyChallengingTheServerTax.aspx


Very large file server, build details:
http://www.servethehome.com/category/the-big-whs-30-drive-whs/


Cheers


~~~~~~~~~~~~~~~~~~~
EDIT (25 Jan 2011): To Tim and the crew, maybe because it was so obvious to me that SmallNetBuilder and the people in the SNB forum are far and away my greatest educative destination; that it has given me an enormous sense of confidence as I grow my network and those of my family; and, that being such a greenhorn I feel I've taken in more than I've given back to the forum, it didn't occur to me that posting out links may be frowned upon. When I saw Jorsher had designs on ZFS, I jumped and posted links to the personal blogs of two ZFS researchers and to one giving a parts recipe to a unique storage array and the design thinking behind it. If I did cross a line, or it seemed discourteous, please accept my apologies. In no way was it my intent. SNB, the authors, and the forum formed the cornerstone to my network (now 12 nodes, 2 sub-nets, 3 vlans, remote server backup, and plans for more) and remains my ongoing schooling. From your perspective I may appear as an anonymous IP address but from mine, SNB is a locomotive of growth. From me and the packets, I send you a big and ongoing Thank You.

 

[Jorsher]

Thanks to you both I really appreciate it. It's hard finding some good information since it's not a common project like building a computer, and the fact that I don't know exactly what to look for.

The only thing I'm worried about with OpenSolaris is hardware support, but the website seemed to have a decent sized hardware compatibility list.

I just recently upgraded my network to gigabit. I'm going to look for a motherboard that supports dual ethernet connections like in my PC, both for redundancy and increased bandwidth.

Fortunately ram is cheap, but thanks for mentioning that it will need a lot. I was honestly going to buy a small, cheap amount, but it makes sense that that file system would need a lot to perform well.

Thanks again for the links. Now I'm going to read over them

 

[Jesse B]

Thanks to you both I really appreciate it. It's hard finding some good information since it's not a common project like building a computer, and the fact that I don't know exactly what to look for.

The only thing I'm worried about with OpenSolaris is hardware support, but the website seemed to have a decent sized hardware compatibility list.

OpenSolaris should be just fine with most hardware. FreeBSD is known to be quite compatible, and also supports ZFS, so that might be worth looking into as well.

Note that there are dozens of OpenSolaris/FreeBSD based OS's made and readily available. FreeNAS, OpenIndiana, Nextenta, OpenFiler... the list goes on Definitely worth checking out some alternatives as well just to find the OS best suited for you.

I just recently upgraded my network to gigabit. I'm going to look for a motherboard that supports dual ethernet connections like in my PC, both for redundancy and increased bandwidth.

Depending on which LAN chipset your motherboard uses, you might want to consider getting a/a couple of Intel PCI-e NIC's. They will help you squeeze every bit of performance out of your network. You can definitely test the onboard LAN to see how it performs before hand as well though, just to make sure it's a worthwhile investment. Be sure to get PCI-e and not PCI, as PCI lacks the bandwidth to run dual Gbit connections.

Fortunately ram is cheap, but thanks for mentioning that it will need a lot. I was honestly going to buy a small, cheap amount, but it makes sense that that file system would need a lot to perform well.

Due to your pool size, I would start with no less than 8GB of RAM. More would be beneficial for sure. Again, you can add more if you feel the need to, but this is all up to the individual. Due to all the advanced features ZFS has to offer, you have to pay for it with more memory. It's still cheaper than buying dedicated hardware-RAID cards though!

Good luck with your project, looking forward to see how it turns out.

 

[Jorsher]

Thanks again Jesse.

The primary reason I'm leaning towards OpenSolaris is for the RAIDZ3. I'm willing to give up another 2TB for the additional peace of mind. As far as I could tell, Z3 is only available on OpenSolaris.

I was only going to use 2-4GB of ram in the NAS, but I didn't even consider how it would effect the RAID. Since you suggest at least 8GB, I think I will go with 12GB to be sure there's plenty!

I will definitely post back when I have it set up, although I'll most likely only start with about 10TB of usable capacity just to migrate my media over to it. I wouldn't have considered Samsung so I'm glad you mentioned them! It appears that many people are very satisfied with the speed, noise, heat, and power consumption on these drives, and I saw many people reporting to use them in large RAIDs without trouble.

Currently I'm designing a custom desk for it, and I think people will enjoy my idea It will be a couple months at the minimum before I have it all ready, but I will be sure to report back.

 

[Jesse B]

Thanks again Jesse.

The primary reason I'm leaning towards OpenSolaris is for the RAIDZ3. I'm willing to give up another 2TB for the additional peace of mind. As far as I could tell, Z3 is only available on OpenSolaris.

I overlooked this detail. As far as I know, you are correct that only OpenSolaris supports this. I've heard no mention of it in FreeBSD, only RAIDZ and RAIDZ2. OpenSolaris should work just fine however

I was only going to use 2-4GB of ram in the NAS, but I didn't even consider how it would effect the RAID. Since you suggest at least 8GB, I think I will go with 12GB to be sure there's plenty!

It will work with 4GB or 8GB of RAM, but the more you add, the better performance you'll have. From what I've heard, 12GB is the most that's really worth putting in, but I have no way to confirm this, as I've never made a similar system myself.

I will definitely post back when I have it set up, although I'll most likely only start with about 10TB of usable capacity just to migrate my media over to it. I wouldn't have considered Samsung so I'm glad you mentioned them! It appears that many people are very satisfied with the speed, noise, heat, and power consumption on these drives, and I saw many people reporting to use them in large RAIDs without trouble.

Yep, the new 2TB F4's are pretty amazing drives. They're only 5400RPM, so they're low on power, but they perform amazingly. They perform on par with the average 7200RPM drive from a few years back. You'll most likely want to upgrade the firmware however, apparently there were some issues that have been fixed.

Currently I'm designing a custom desk for it, and I think people will enjoy my idea It will be a couple months at the minimum before I have it all ready, but I will be sure to report back.

Sounds very cool, looking forward to see what you have in store.

 

[Jesse B]

Hopefully there's no rules against linking to other forums, but here's a list of Solaris-based OS's that support ZFS, might find something interesting.

 

[Jorsher]

Sorry to bump up this old thread, and thanks everyone for the replies.

I'm still "lost" on the hardware aspect. What would be a cheap method of getting 12-16 drives that can be software-raided? The cards I've found seem to have SAS ports instead of SATA, and I assume I'd need some sort of backplane or expander (not sure of the proper term) to connect the SATA drives?

I had this project on hold, but with Seagate buying Samsung and WD buying Hitachi, I'm afraid the "cheap" drives that are good for RAID may be coming to an end.

 

[chiadog]

I had this project on hold, but with Seagate buying Samsung and WD buying Hitachi, I'm afraid the "cheap" drives that are good for RAID may be coming to an end.

Excuse the bad pun but I'm not buy that. Even with the consolidation, the 2 giants will still rub horns instead of collude. IMO media prices is nothing to worry about, as they still have to face the ever increasing densities of solid state media. The only advantage magnetic media has is density per dollar. They would not shoot themselves in the foot.

As for the SAS, you can use SAS splitter cables to connect your SATA drives.