Building High Performance NAS / HTPC

[Tyrix]

Hi,

I want to build a high performance but also relative less power consuming NAS.
It should backup important files, videos, pictures, music, etc...

I'm thinking of these components:

CPU: AMD X2 4850e AM2
MB: Gigabyte MA78G-S2H or Asus M3A78-VM both are AMD 780G Chipsets
PSU: Enermax MOD82+ PRO 385W or S12II-330 both with 83% eff.
RAM: cheapest DDR2-800er 2GB

CASE: Silverstone SG03 or Antec Mini P180 (I have the standard ATX P180 here, so it would be his little brother )

OS: Windows Home Server, Unix, I have also access to Windows Server 2008 (MSDNAA) but that would be overkill?, ... maybe you can recommend me one?


Thanks in advance, and sry for my english

 

[claykin]

Why AMD over Intel? My 2 cents, stick with Intel and an Intel chipset (Intel or Asus motherboard) and save yourself aggravation. Trust me, I've done this for far too long!

Why not buy an appliance?

 

[00Roush]

Why AMD over Intel? My 2 cents, stick with Intel and an Intel chipset (Intel or Asus motherboard) and save yourself aggravation. Trust me, I've done this for far too long!

Why not buy an appliance?

Why Intel over AMD? For file serving tasks either has more than enough computing horsepower. Also the components chosen have a price advantage over comparable Intel components and will offer similar power usage.

My opinion is that the hardware selected would work just fine and most likely offer much better performance than a off the shelf NAS for a similar price.

So what kind of performance are you wanting? When I hear high performance I think in the 100+ MB/sec range. Also what OS is on the clients that will be connecting to the NAS? Knowing this can help narrow down what OS might work best.

00Roush

 

[valentin]

I use the same motherboard MA78G-S2H, RAM and CPU. This is very good combination, and it would serve anything you want.

I use it connected to a TV for gaming (I added also HD3850), for movies, and for torrent downloads.

It has an eSATA port so you could easily add external storage and use it for NAS and backups.
________
GLASS WEED PIPE

 

[Tyrix]

At first, thanks for all your replies, claykin, 00Roush and valentin!

@claykin: I think AMD fits better in this segment... the chipset and the "cheap" CPU's are ideal to build a low power consuming and fast system. I can also buy a Phenom II CPU later that does fit (many AM2+ MB's are ready for AM3 ). Intel is not userfriendly as AMD is atm (I have in my CAD-Gaming-Sys Intel inside (C2D) )

@00Roush: Yes, I want writing / read speeds at nearly or above ~100MB/s. The client systems are on Vista Ultimate 64bit and XP Pro 32bit.

@valentin: Yep, I think I'm going to purchase the MA78G over the ASUS. I heard, that the MA78G is out as rev 2.0 but I didn't find it anywhere.

Which CPU Cooler are you using? I assume that the boxed is too loud for me... I want it to be silent as possible.



The PSU will be the Seasonic S12II 330W.


Thanks again for your replies.

 

[Dennis Wood]

Valentin, would be great if you could post a pic or two of your setup. Barring that, a hardware list would be great. I've got a spare MB sitting here and would like to incorportate the silent HTPC option for our Ubuntu based NAS project.

 

[valentin]

@Tyrix,
yes, I do use the boxed cooler of the CPU. I find it ok, silent enough for a living room.

Mine MA78G-S2H is rev.1. The northbridge was quite warm, I glued a 4cm fan on it. Had to use an adapter for the back fan, as rev.1 motherboards have only 1 fan header besides the cpu fan header, but rev.1.1. has an additional header so this would be no problem for you. The back fan works on low rpm through the adapter, so it is silent.

@Dennis,
I am sure you would want something looking much better than what I have, but anyway I shot few pictures. I do not know how to attach images here, so I put them on the small gallery on my linkstation:
http://vnaumov.mooo.com/
Look at the bottom four images. One of them shows the arrangement in the living room, the next is the motherboard, below the list of the hardware (I keep accounts in GnuCash). The last one is in our bedroom, the old TV, VCR, on top of it my latest buy - captiva hdd media player (exactly the same as Plextor PX-MPE640U) and also the linkstation.

The images are w/o the video card - I packed it today. I do not need it anymore inside, as FarCry 2 got too boring. I will put it in again when the next good game comes

You may see there also pictures of my wooden computer - the second link from the top. I built custom case from wood and acrylic few years ago, and this year I made a new wood-only case for the PC for my daughter.
________
HERBALAIRE

 

[00Roush]

@00Roush: Yes, I want writing / read speeds at nearly or above ~100MB/s. The client systems are on Vista Ultimate 64bit and XP Pro 32bit.

First off I want to say that your Win XP Pro client will probably not be capable of 100MB/s transfer speeds over the network. From my testing of Win XP Pro as the client the best I have seen is around 65 MB/sec for reads and 75 MB/sec for writes.

Lately I have been testing with Vista Ultimate 64bit SP1 as a client. Using Win XP Pro SP2 on the server I see read and write speeds of ~90-100 MB/sec for large files. Using Ubuntu Server on the server I see read speeds around ~85 MB/sec and write speeds at ~100 MB/sec. These results are what Vista is reporting for copy speeds and are consistent with recorded network throughput.

To get the best performance out of Win XP Pro windows needs to be set to use a large system cache when used as a server. Here are the steps from the Microsoft website.

1. Click Start, click Run, type sysdm.cpl in the Open box, and then press ENTER.
2. Click the Advanced tab, and then under Performance click Settings.
3. Click the Advanced tab, and then under Memory usage use one of the following methods:
• Click Programs if you use your computer primarily as a workstation instead of as a server. This option allocates more memory to your programs.
• Click System cache if your computer is used primarily as a server or if you use programs that use a large system cache.
4. Click OK to save preferences and close the dialog box.


Based on my testing I would say Win XP Pro or maybe Windows Home Server would give the best performance for your mixed OS environment. Ubuntu would also work but I have found read/write speeds to be bit slower for Win XP clients. Win Server 2008 is probably overkill like you mentioned but could possibly offer better performance than any other OS. You might consider testing out a few different OSes to find what really works best for you.

Don't forget to make sure your hard drive setup on the server and client is fast enough to support the speeds you want. This means both the server and client need drives capable of 100+ MB/sec read and write.

00Roush

 

[Tyrix]

Thanks for this detailed answer!

So, XP Pro, WHS, Ubuntu would possibly fit. I did install Debian Etch on a VM yesterday and I did some experiments setting it up etc... How about Debian? I know that Ubuntu is based on Debian but is it "better" than Debian for file servers (maybe only out of the box)?

What do you think of a WD Caviar Green (WD10EADS 333GB Platter). I think its very good in terms of power consumption but is the speed enough for a file server?

Thanks in advance.

 

[00Roush]

I think out of the box Ubuntu Server might be better but as you mentioned it is based on Debian so the differences are just in the configuration. Also Ubuntu Server does a minimal install without a graphical interface. For my setup I use webmin to manage the server remotely. It works great.

The WD Green drives would work fine but just remember they are not tuned for performance. According to the HD performance charts at Toms Hardware the max read performance is 97 MB/sec for the drive you mentioned. Close to the 100 MB/sec mark but only at the beginning of the drive. By the end of the drive you would be around 45 MB/sec. You would probably need 2 of the drives in a RAID 0 array or 3 drives in a RAID 5 array to get to 100 MB/sec or better. You might consider the WD Black drive (WD1001FALS) as it is about 10 MB/sec faster across the whole drive. Pretty much the same drive just tuned differently and it uses a bit more power. Just for reference I currently am using two WD Caviar SE16 WD3200AAKS 320GB (single platter) drives for my testing. One in the server and one in the client.

If you like WD drives you might also check out the 640GB drives. Both the standard version and Black version are reasonly priced and fast.

00Roush

 

[thiggins]

In the Fast NAS series, I have yet to see drives make a difference. Once you get past caching effects (file size larger than RAM size on either client and/or NAS), I see no better than 70MB/s in RAID 0, even with Velociraptor drives. RAID 5 has even more overhead so would be slower.

 

[00Roush]

In the Fast NAS series, I have yet to see drives make a difference. Once you get past caching effects (file size larger than RAM size on either client and/or NAS), I see no better than 70MB/s in RAID 0, even with Velociraptor drives. RAID 5 has even more overhead so would be slower.

My latest testing using a 2 drive RAID 0 array on the server and the client came in at 112 MB/sec writing and 110 MB/sec reading a 20.1 GB file from the server. Vista Ultimate x64 SP1 was on the client and Win XP Pro was on the server. These numbers were from the Vista copy window statistics just before the file was done copying. This was also consistent with recorded network throughput. So far my Iozone results using the same command line used in the NAS charts are not reflecting these results for Vista. I believe this has to do with Vista using much larger "record sizes" for reads than Win XP. I have tested using record sizes up to 2 MB and the results look better but I still need to find out why they don't match.

00Roush

 

[thiggins]

I don't know how Windows calculates its throughput. Vista contains numerous file copy optimizations. Mark Russinovich's blog post is probably the best reference for details.

You can look at the iozone data for different record sizes and see if that supports your theory. Our NAS Charts report only the 64 KB file size, which is what Windows used to use.

If you manually calculate the transfer rate by doing a timed drag-and-drop, does it correlate with the speed that Windows reports?

I'll check with Don Capps and see what he has to say about Windows' reported file copy Stats.

 

[Dennis Wood]

Tim, I'll be very interested in your findings after discussing with Don. Vista does have several technologies, including dynamic TCP tuning that may factor in here. My measured results more or less agree with Vista's reports. I haven't been using IOzone a whole lot lately...just our measured/timed results as the IOZONE tests with load balancing connections under-reports speeds significantly.

What I've seen from Vista SP1 is that a very large file transfer (read) from the NAS slowly builds in speed until it caps out about 90MB/s. It takes nearly 10GB transferred before the file transfer speed reaches its max.

What's also weird with the latest beta QNAP code is that writes under 4GB to the NAS (with 4GB in the NAS) are no longer spiking over 112MB/s (as reported by Vista) and then gradually slowing to 45MB/s or so. Now the write speed is staying up there in the 70MB/s range. Apparently the new code uses updated SAMBA, so I'd be curious about Don's comments there too. I'm a bit cautious about these numbers as the RAID 5 writes are nearly twice as fast with just a code update. More testing required here for sure.

What I'm seeing is the NAS with the 5x WD "green" 5400rpm drives on a NetGear GS108T switch is consistently slower than NAS 2 with the 5 x Seagate 7200rpm enterprise drives, and a 3COM switch. I'd guess that the drives are the major factor, but I'm beginning to wonder about the switch too.

 

[justin2net]

i'm sure you will need a dedicated raid controller for RAID5 ~100 MB/s, esp. with large disks. 2 gigabit trunk will probably be necessary. (depends on what gigabit interface, pci or pcie)

 

[00Roush]

I don't know how Windows calculates its throughput. Vista contains numerous file copy optimizations. Mark Russinovich's blog post is probably the best reference for details.

You can look at the iozone data for different record sizes and see if that supports your theory. Our NAS Charts report only the 64 KB file size, which is what Windows used to use.

If you manually calculate the transfer rate by doing a timed drag-and-drop, does it correlate with the speed that Windows reports?

I'll check with Don Capps and see what he has to say about Windows' reported file copy Stats.

I tested iozone using sizes from 16k to 16MB and looked at the trace data provided by the Process Manager program that was in the article you linked to. Along with this I did a normal file copy and used Process Manger to trace what happened.

According to the trace data my drag and drop file copies are using a 32k request size and are doing multiple request to the network sequentially to my Win XP server. Which is keeping the network pipe full. Once the data arrives from the network it is written directly from memory to the disk sequentially in 32k blocks.

The trace data for reads shows iozone is requesting the size I set, but directly from memory. The system looks to be reading ahead of these requests but not at a quick enough rate. So what happens is the system requests the data from the network then iozone requests the data but the data has not actually arrived from the network. This means iozone has to wait until the data arrives before it will issue another read request. So multiple requests are not being made at a time and in turn the network pipe is not being kept full. My results testing with 8MB request sizes show much faster reads but much slower writes. I believe the faster reads are because the network pipe can stay full. I will explain the slow writes in a bit.

It looks like when iozone writes it writes directly to memory using the request size set. From what I can tell the whole file is written to memory if possible, then the file is flushed to disk or in this case the network. When this happens the system sequentially writes the data to the network in 1 MB blocks as fast as possible. I think the write results shown by iozone are only measuring how long it takes to write the file to memory that is why they are so high. I believe this can be fixed by using the -e switch on the command line. When using request sizes of larger than 256k with iozone for writes, Vista breaks them down into 256k blocks and writes them directly to the network. I believe this extra processing is what was causing my slow write speeds using larger than 256k record sizes.

I did test with 16MB request sizes and found that reads were much slower than with 8MB. This larger size should help keep the network pipe full but when I looked at the Process Monitor trace I found that the requests were being broken down into 64k request sizes to the network. Whereas the 8 MB requests were being broken down into two 4MB requests. On a side note I did try some of the larger sizes with iozone running on a Win XP Pro machine and did not see any increase in read speed.

This is of course just my conclusions based upon my own testing and understanding of how this stuff works. I gladly welcome any feedback.

00Roush

 

[00Roush]

Almost forgot...

I did get a chance to do a timed drag and drop copy and here's the results:
Read was 3 minutes 25 seconds for a 20.1 GB file. So about 100 MB/sec
Write was 3 minutes 5 seconds for the same file. So about 112 MB/sec
My 110 MB/sec read I mentioned in my previous post must have been a fluke because I ran this a few times and each time was within a second or so.

00Roush

 

[thiggins]

Thanks for the detective work, 00Roush.

How much memory on both systems and what are the disk performance / caching settings?

I'll post Don Capps' response when I hear back.

 

[00Roush]

Thanks for the detective work, 00Roush.

How much memory on both systems and what are the disk performance / caching settings?

I'll post Don Capps' response when I hear back.

Vista client has 4 GB RAM and Win XP Pro server has 1 GB RAM. Both systems are running a two drive RAID 0 array using Western Digital Carviar SE16 320GB drives. According to the benchmarks I have run these arrays are capable of a max of about 210 MB/sec for sequential reads/writes. Minimum looks to be around 115 MB/sec.

The Win XP Pro server has been set to use a large system cache. Is that what you meant about "caching settings"?

00Roush

 

[thiggins]

The Win XP Pro server has been set to use a large system cache. Is that what you meant about "caching settings"?

I was thinking more about the "enable enhanced performance" switch in Vista's hard drive properties.

Thanks for the other info. I'll have a post up shortly with Don's response.