Proposed New Build (suggestions welcome!!)

[re0]

Need to build an almost infinitely expandable nas for a client. Budget for parts is $3000 and storage to start needs to be 12T
(already have 4 2tb drives)
Debates

which os Freenas or Nexenta or WHS or open filer or napp-it( http://www.napp-it.org ) or another linux based os
primary use is media storage/sharing

biggest thing is being able to search the the nas (not through a windows box but rather through the os's gui?)

which case

super micro
http://www.newegg.ca/Product/Product.aspx?Item=N82E16811152124

or
NORCO RPC-4224 4U
http://www.newegg.ca/Product/Product.aspx?Item=N82E16811219038

(in which case i'll need all the cables and a psu)

also what's the best Raid configuration


my criteria is
1. Speed
2. data rebuild time
3. cost

so do i do software raid of 6 or 50 or do i do clusters of raid 6 hardware with a software raid overtop to make it one big disk

this is the intended mobo cpu combo

SUPERMICRO MBD-X9SCL+-F
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813182262

cpu
Intel Xeon E3-1220 Sandy Bridge 3.1GHz 4 x 256KB L2 Cache 8MB L3 Cache
http://www.newegg.ca/Product/Product.aspx?Item=N82E16819115084


so any help/suggestions would be great......

the three major ones are
1. os
2. raid/hba card(s)
3.raid/hba/filesystem config

side notes

green technology (shut down unused drives etc) suggestions


Thanks in Advance

 

[GregN]

Sounds like an interesting build.

Here is what I would do:

Decide if hardware or software RAID. As stated in other threads, Hardware RAID will offer better performance, and better rebuild time, at the cost of flexibility and price. Software Raid reverses that. Hardware RAID will be more expensive, so the added performance (factor dependent on which controller) is at a significant cost.

If Hardware RAID, what controller(s)? Areca, LSI/3Ware, Adaptec are the industry leaders.

Software RAID, and ZFS appears to be the best choice these days, which distro? FreeNAS is the most popular, but other choices, as you note, are out there. Someone just posted that there is a new release of EON.

On the case, your choices are like choosing between a Ford Fiesta and a Volvo, and not the base model, but the really comfortable one. People don't buy the Ford Fiesta because they love nice rides, they do it because they want basic transportation at the lowest cost, that is the Norco - Hot Swap but basic. The SuperMicro includes nice redundant PSUs, and probably could survive being thrown off the roof of a data center. Given the choice, without considering budget, folks will pick the SuperMicro. Every time. Chenbro would probably be the next step up from the Norco. But I wish I could afford the SuperMicro (at at a third of your budget, can you? )

On the SuperMicro's PSU, it is dual redundant ( need that? ), and includes the PFC green feature. Nice. But I run 12 drives on a green 450W PSU (4yr old Dual Xeons), 900W is probably more than you need, a silver/gold 650W would probably do it, and is probably greener.

The Motherboard and Processor combo is very good, depending on your choice of RAID, if you go hardware, you could find a cheaper and greener CPU, you won't need as much processor, and you can probably drop the imbedded graphics, the 1220L?

Hope that helps...

 

[re0]

I've decided on the hardware!!!

Ok here are the specs for the new build

Mobo

SUPERMICRO MBD-X8DTH-6F-O Dual LGA 1366 Intel 5520 Extended ATX Dual Intel Xeon Server Motherboard
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813182174

Cpu

Intel Xeon E5504 Nehalem 2.0GHz 4 x 256KB L2 Cache 4MB L3 Cache LGA 1366 80W Quad-Core Server Processor BX80602E5504
http://www.newegg.ca/Product/Product.aspx?Item=N82E16819117187CVF

CPU cooler
Intel BXSTS100A Active heat sink with fixed fan
http://www.newegg.ca/Product/Product.aspx?Item=N82E16835203002CVF


Ram

Kingston 24GB (3 x 8GB) 240-Pin DDR3 SDRAM ECC Registered DDR3 1333 (PC3 10600) Server Memory Model KVR1333D3D4R9SK3/24G
http://www.newegg.ca/Product/Product.aspx?Item=N82E16820139273

Case

NORCO RPC-4224 4U Rackmount Server Case with 24 Hot-Swappable SATA/SAS Drive Bays
http://www.newegg.ca/Product/Product.aspx?Item=N82E16811219038

PSU

SuperMicro PWS-865-PQ 865W Single Server Power Supply - OEM
http://www.newegg.ca/Product/Product.aspx?Item=N82E16817377004


Hard Drives
8 x HITACHI Deskstar 5K3000 HDS5C3020ALA632 (0F12117) 2TB 32MB Cache SATA 6.0Gb/s

http://www.newegg.ca/Product/Product.aspx?Item=N82E16822145475


Cache Drive

Crucial M4 CT064M4SSD2 2.5" 64GB SATA III MLC Internal Solid State Drive (SSD)
http://www.newegg.ca/Product/Product.aspx?Item=N82E16820148441

USB Boot device
Mushkin Enhanced Mulholland 4GB USB 2.0 Flash Drive
http://www.newegg.ca/Product/Product.aspx?Item=N82E16820226077


Raid Type

Software 3 x 8 drives in raidz2 (software raid 6) all in same zpool

File System

ZFS


OK so things that remain

1.Choose OS! So confused!!!!

2. assemble.....gonna be fun!!!

3. Testing -----any suggestions for a test proceedure?

4. anyone see anything i'm missing etc?

 

[Hydaral]

Something else you may want to consider is if you want the system/boot drives to be in a RAID, I didn't use FreeNAS because it cannot have redundant ZFS system drives, whereas NexentaStor can, not sure about Eon.

 

[GregN]

Cool. Looks good.

Are you going to be running Dual Xeons? From your Mem spec, I would guess not, why a dual cpu board, for expansion?

Folks have recommended having two USB drives that are mirror, so when one fails the other can be popped in.

How comfortable are you at the command line? Eon, Nexenta (even with Napp-it) are much more CLI intensive. FreeBSD is geared toward booting off of thumb drives, NexentaStor ( you are below the 18TB limit ) I think has the sweetest GUI.

Are you going to be running DeDup?

As for Testing I'd use the standard benchmark suite used here NASPT from Intel.

Keep us abreast.

 

[re0]

All the parts are here!!!!

Ok so all the parts have arrived i plan to start the build tonight......(hope the boss doesn't expect me to be too productive tomorrow! )

a few things have been decided in regards to the os

1. ZFS Raidz2 is the filesystem
2. support for add ins or programs (so a full os not a stripped down one)
3. so chocies are FreeBSD, OpenIndiana, or OpenSolaris
4. a gui for any one of the above would be nice but not required.....


Software is proving to be the most complicated part of the build........help suggestions would be great....


Hydaral:

Thanks for your input. i don't need to run the boot drives in a RAID as i plan to boot from a usb and keep a spare ready preloaded if the first one dies/corupts.

GregN

Your input here and on many other posts has proven very valuable during this proccess
"Are you going to be running Dual Xeons? From your Mem spec, I would guess not, why a dual cpu board, for expansion?"

possibly but no not now (yes the othe cpu would have trouble without ram lol)

"How comfortable are you at the command line? Eon, Nexenta (even with Napp-it) are much more CLI intensive. FreeBSD is geared toward booting off of thumb drives, NexentaStor ( you are below the 18TB limit ) I think has the sweetest GUI. "

only experience with cli is dos based...........but i can read so if it means a better more stable system with more features then i'll learn. NexentaStor is out because this box will grow to 34 TB in about 2years and probably be around 100 TB in under 5yrs. (external cases with sas links) Did i mention this is for a client who is crazy about media!!

Where is a good place to look for FreeBSD programs. Specifically a gui torrent downloader. also a comprehensive search engine is required. any thoughts.

"Are you going to be running DeDup"


forgive the ignorance but what is that?

also was looking at http://zfsguru.com/ ?

 

[Terry Kennedy]

SUPERMICRO MBD-X8DTH-6F-O Dual LGA 1366 Intel 5520 Extended ATX Dual Intel Xeon Server Motherboard
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813182174

That's a wonderful motherboard for a variety of reasons. One of the best ones is that all expansions slots are the same - PCIe x8 in an x16 form factor, so you can shuffle cards around for the best possible airflow. It also has an incredible amount of hardware monitoring on board - in addition to the usual voltages and fan (all 8!) speeds, you can read the temperature from each memory DIMM (if supported on the DIMM) and also communicate with the power supplies over PMbus. Here is a snapshot of the type of data I can capture from the system.

This is the same motherboard I use (non-SAS version X8DTH-iF) in the second-generation RAIDzilla:

Picture

This has dual E5520 CPUs, 48GB RAM, 256GB PCIe SSD, 3Ware 16-port SATA, and dual external SAS (for attaching tape backup).

You've selected the motherboard version with dual SAS ports as well as 6 on-board SATA ports. Looks like you have 8 data drives - will you be using SAS/SATA fanouts? What will you do if you want to expand beyond 8 drives?

Will you be installing a CD/DVD drive to install from?

File System

ZFS

OK so things that remain

1.Choose OS! So confused!!!!

I'm running FreeBSD 8-STABLE (the maintenance branch, which has ZFS v28 in it).

3. Testing -----any suggestions for a test proceedure?

For starters, install the iozone benchmark. Fine-tune your ZFS performance for local operations (I get > 500MB/sec sustained for days at a time, 5-second burst over 4GB/sec), then work on tuning SAMBA to make the best use of what you already have. You can use iozone on the client system, and/or the Intel NAS PT suite. Here is a sample iozone write graph from my system.

Per another user's comments about ZFS dedup, I'd recommend against it unless you have loads and loads of redundant data. A scrub of 32TB of data normally takes about 16 hours. With dedup on (and consider that I have 48GB of RAM installed), the same scrub takes nearly a week.

 

[GregN]

Re0,

DeDup is a feature of ZFS, it is sort of like uber-compression, if two blocks are identical, it stores only one of the blocks (using reference pointers everywhere else). A very cool feature, but as you can imagine, performance and memory intensive process.

I was not recommending running DeDup, just wondering what your plans are. You do have the processor muscle, though you would probably need more memory.

FreeNAS, given your choice of boot media, appears to be your best option.

I run FreeBSD just on my router, which is turn key so I don't know the best place for software other than the regular repositories, bot Deluge and RUtorrent/Rtorrent appear to available through pkg-add. Deluge has an interesting thin client architecture. RUtorrent is the web front end to RTorrent.

Quite a few folks also recommend using Ports to install additional packages.


Terry,

Looks like an excellent build, what are you using as a case, it look like the Norco, but with those fans....


I am also a big fan of SuperMicro's products.

 

[Terry Kennedy]

Looks like an excellent build, what are you using as a case, it look like the Norco, but with those fans....

Customized version of the CI Design NSR-316.

 

[re0]

Terry:

What will you do if you want to expand beyond 8 drives?

http://www.newegg.ca/Product/Product.aspx?Item=N82E16816118112 with 8 more drives and then a raidz2 and add that dev to the zpool? (i think i can do this....lol)

the norco case has 6 sas backplanes (each fan out to four drives)

Will you be installing a CD/DVD drive to install from?

No just use one temporarily to install software. this will be a headless system.

It also has an incredible amount of hardware monitoring on board

what packages/ports are you using to capture and alert you of this info. like do you get an email if your raid becomes degraded?

I'm running FreeBSD 8-STABLE

I want to run this as well with a full scrub every 30 days
1. is there a torrent software port/package
2. is there a pre built live cd
3. can i use your freebsd config?

(remember i know relatively nothing about UNIX period nevermind FreeBSD)


Greg:

FreeNAS, given your choice of boot media, appears to be your best option.

can you not load FreeBSD on a usb stick?

 

[Terry Kennedy]

http://www.newegg.ca/Product/Product.aspx?Item=N82E16816118112 with 8 more drives and then a raidz2 and add that dev to the zpool? (i think i can do this....lol)

Make sure that it is supported under the FreeBSD release you'll be using. Most of the modern LSI cards use the mpt or mps drivers, which are somewhat opaque - not the greatest error recovery and their logging consists of uninformative "MPT event X" type messages.

what packages/ports are you using to capture and alert you of this info. like do you get an email if your raid becomes degraded?

A large number of things - I get emailed alerts from the 3Ware controller if there's a drive problem, the IPMI on the motherboard emails if any parameters are out of limits, and so on. One thing not well understood is that ZFS "autoreplace" isn't "auto" on FreeBSD - an autoreplace drive is a hot spare. In FreeBSD (at least at the present) you'd need to write a devd script to receive the notification of a drive failure and then tell ZFS to use the spare drive. Since you don't have a hot spare planned, this shouldn't be an issue.

I want to run this as well with a full scrub every 30 days

That's configurable on the stock system (in /etc/periodic.conf).

1. is there a torrent software port/package

I don't know - I don't use it. But you could check http://www.freebsd.org/ports.

2. is there a pre built live cd

For each -RELEASE version, yes. It is part of the stock release. It won't have things added since a release, though. As an example, ZFS v28 was added after 8.2 was released (8.2 shipped with ZFS v15), so the 8.2 live CD won't be able to mount a newer zpool. mfsBSD is a live CD that is updated whenever major new features (of interest to that person, at least) are added. One of the ones on that site is built with ZFS v28, for example.

3. can i use your freebsd config?

If you have a particular question, I'll try to answer it or you can ask over at the FreeBSD forums. Asking a question like this is sort of like saying "Can I have a copy of your Windows registry?" - it won't do anything by itself, has lots of stuff you don't need, and is missing stuff you do need. It isn't that hard to get up to speed on FreeBSD and soon you'll be installing and configuring stuff on your own.

can you not load FreeBSD on a usb stick?

You can - in fact, one of the pre-built distribution kits is one to copy to a USB stick. There are a couple reasons why you might not want to use a USB stick for this, though. The main one is that FreeBSD will do a lot of writing (log files, swap area, etc. not to mention the amount of writing when you recompile the whole system). This tends to be rather hard on flash-based media, particularly USB sticks. SSD's at least try to perform some evening out of the writes across the media.

In the picture linked from my original post, you can see a pair of 2.5" drives (320GB Scorpio Black) on top of the drive cage. These hold the OS for my RAIDzilla, so that I can completely nuke them if I want without needing to worry about my ZFS pool. They're a mirror set, so I can lose either one and still be working fine.

[Please excuse any typos or incomplete sentences in this post - my cat keeps trying to climb over the keyboard onto my lap.]

 

[GregN]

can you not load FreeBSD on a usb stick?

Re,

FreeNAS is built on top of FreeBSD. The Advantage of FreeNAS is the Web UI for disk (and pool) admin.

As I've said before, I'm not the biggest fan of thumb boot drives, but given your choice, you should be able to do both FreeBSD or FreeNAS. The advantage of FreeNAS is that you get the GUI, with FreeBSD you'll be facing significantly larger command line involvement.

 

[Terry Kennedy]

FreeNAS is built on top of FreeBSD. The Advantage of FreeNAS is the Web UI for disk (and pool) admin.

As I've said before, I'm not the biggest fan of thumb boot drives, but given your choice, you should be able to do both FreeBSD or FreeNAS. The advantage of FreeNAS is that you get the GUI, with FreeBSD you'll be facing significantly larger command line involvement.

Right now there's a transition going on (FreeBSD 9.0 will be officially announced Any Day Now) which means that a number of wrappers like FreeNAS are likely to change version in the near future as well.

If the original poster's enclosure has the space for it, they could consider using some old SATA drive for the system disk. That avoids the issue of having to deal with excessive writes on a USB memory stick. Just be sure, if you do this, that you get drives that can have their "intelligent parking" 'feature' turned off - otherwise you'll have your system drive(s) spinning up and down every 8 seconds. For the WDC drives, the utility is called wdidle3.exe. Not explicitly stated, but true - you need to completely power off the drive after disabling the idle timer, or it won't change anything.

My advice would be "Plan to throw one (or more) away" - try some of the nice GUI frontends and see if they can handle everything you need. As all of the stuff underneath is configured in command-line mode, you may find things that you can't accomplish in the GUI, and worse - if the GUI doesn't understand a config option you manually edited into a text file, strange things can happen.

 

[re0]

Pictures

https://skydrive.live.com/redir.asp...3636A87A8C9461!144&parid=B73636A87A8C9461!141

 

[re0]

A Few Things Remain

Things Decided

1. FreeBSD is the os specifically FreeNAS
2. ZFS in Zraid2 (8 X 2 TB drives) with separate ssd cache drvie
3. I HAVE A LOT MORE TO LEARN!
4. Virtual Machines are great for testing!!!!!
5. IPMI with KVM on supermicro mobo rules......the server is in the cold basement and i'm on the couch!!!!! i can even turn it off and on!!!!

Things left to do

1. mount the ssd somewhere in the case. if anyone has done this any help/links/pictures/ would be helpful as the cas comes with lots of hardware and no manual and i'm sure i can rig something but i don't want the hack up the case

2. configure weekly scrub. is there an editor to edit the /etc/periodic.conf file?

3. Testing via Iozone

4. Find and install torrent software. i was thinking vuze (as client already uses this ) and then installing the remote client on his desktop. i think this will take all torrent processes and put them on the server. (i have no idea where to start on this one lol)

again any helpwould be great!

 

[rototiller]

There are some "el cheapo" stand alone NASes like dlink dns 323 that have a torrent client/app built into it that can be downloading torrents w/o a computer running. You can get them on ebay for around $50 probably. Your client can use that nas to d/l his torrents and then move them to the big one that you are building for him and erase it off the dlink nas after transferring. May be a little redundant, but an option. But if he likes torrenting and already uses Vuze, then just tell the client to use his computer running Vuze, to d/l his torrents on to his computer then transfer them to the NAS that you built for him and then delete them from his computer's HD.