Benchmarking your NAS

[travist]

I started a thread on benchmark tools a couple of weeks back. Since this is a slightly different topic, I think it is deserving of a new thread.

I've recently built my DIY NAS, and so far I'm very happy with it, although there are many things I haven't decided on yet due to the lack of performance results. I'm trying to benchmark the box in several different configurations so I can compare to see what works best for me.

I have tried NASPT, but because my client box is an AMD machine (as is the NAS) as well as a windows 7 x64 box, it's not the most stable machine to run the software from. I have gotten it to successfully run the batch test, but it takes a LONG time...

So I decided to give the ATTO bench32 a try. Worked great for one iteration of testing. Network drive showed up in the dropdown, I ran through several tests using each of the different total length options and saving each of the results. I powered off the box and unlocked the second core on the NAS to see if any speeds were affected. No matter what I try, the mapped network drive does not show up in the bench32 drop down. Restarts, re-mapping, etc - nothing seems to work.

Any suggestions on how to collect usable data to evaluate the different settings before putting my NAS into production?

 

[00Roush]

I haven't tested it too much but Iometer has given me some decent results. But you just need to know how to set it up.

Generally I have just setup a large test file (4 GB to 20 GB) and done some timed file copies to test different configurations. The reason I do this is the copy engine in Windows Vista and 7 is designed to keep multiple I/O's in flight at a time and allows for high speed file transfers over a network.

The problem I see is some of the benchmark programs out there don't do multiple I/O's so they don't accurately measure actual performance. Here a couple that I have found give some usable results... SiSoftware Sandra's file system benchmark and Nodesoft Disk Bench. There are others but this is all I could think of at the moment.

00Roush

 

[travist]

Thanks for the recommendations on those two. I'll definitely give those two a try. I have downloaded/installed IOZone, but after installing I realized it was a command line tool, and haven't taken the time to figure out how to use it. I will likely explore that more as well.

I'll also add the two benchmark tools to the thread started about benchmark software, and maybe add a quick review of how helpful I found them so others will have it a little easier than I did...

Thanks!

 

[travist]

I stumbled across IOMeter yesterday, and so far this is working like a champ. Nice interface, nice features, and I can export to excel and chart my results - comparing the different configurations should be easy.

Now the only problem is I'm not very smart on what all of the settings do, and not sure how to demand the most from my network/NAS to compare results. It seems that most of the included (default) tests don't come close to fully saturating the LAN, while I've made some adjustments and got over 800Mbs transfer rate.

I've had this new NAS built and sitting under my desk for almost two weeks now - I'd sure like to start using it! Are there any good built in tests to run and use as the "standard" or can anyone suggest settings to use to get a good idea of true performance?

 

[00Roush]

So for IOmeter I first select the drive I want to test and change the "# of Outstanding I/Os" to 4 as this is similar to what the copy engines in Vista/Win 7 do. For reading... the standard 32k 100% read seems to work okay on my hardware. You could try bumping it up to 64k. For writing I have found it best to make my own access pattern. So 100% write, 100% sequential, and change the request size to at least 64k. Actually from what I have seen Windows 7 uses 1 MB request sizes so you could try that. But this is between a Windows 7 machine and a Server 2008 box.

If you have a lot of ram in the server or client I recommend running the benchmark for like 5 to 10 minutes and see what the performance is.

00Roush

 

[00Roush]

I got a chance to play some more with IOmeter and saved a configuration file which should make it easier to setup IOmeter. http://www.mediafire.com/file/ymujbgzv3y2/iometer.icf This should set some of the defaults like maximum disk size and # of outstanding I/Os. Also it should just have 3 access specifications. 1MB read and 1MB write are what I have been testing with. Just make sure the disk you are testing does not have the iobw.tst file on it. This ensures IOmeter will "prepare" the disk (create a new iobw.tst file) and give accurate resutls. If it already does have the file erase that file and restart IOmeter. When you start your test you should see "Preparing Drives" at the bottom of the IOmeter window. This means IOmeter is in the process of creating the 4GB test file (8192000 sectors @ 512 byte sector size) on the disk you are testing.

Let me know how it goes. I still need to test some more to see if the 1MB sizes are actually used by Windows 7 file copy engine when accessing a linux box or a win XP box.

00Roush

 

[travist]

Thanks for posting up that info. I've finally gotten to a point where I think I can begin running some ZFS / SRAID benchmarks. Looks like that's all I'll be able to do unless I can figure out the onboard RAID problems I'm having.

I'll be sure to post up some info once I gather all of the test results.

So far, I've run 8 workers with the 1MB read and 1MB write access specs on SRAID 5 with 4 1TB disks, I'm getting about 80 MBps Write and 70 MBps Read. Is this about normal, or should I be shooting for higher numbers? My CPU is not pegged on the NAS, although it's running above 75% for most of the write cycle. It will be interesting to compare SRAID and ZFS.

 

[00Roush]

Are you running the 8 workers with read AND write at the same time? That is a hell of a workload.

Generally I have just been selecting just one worker and setting a drive for that worker. Then adding only the 1 MB read access spec OR the 1 MB write access spec. My test drive is a single drive empty drive (1 TB Hitachi 7K1000.C) in my server. With this setup I have been seeing about 111 MB/sec with the read test and about 108 MB/sec with the write test. But I would consider this a best case as I am running Windows Server 2008 R2, a quad core, and the test drive is capable of 135 MB/sec read/write. My guess is with your hardware you should be able to at least see 90-100 MB/sec.

I can't remember if I have tested FreeNAS on my latest server build so I don't know if performance is much lower than with Windows Server. I will say that in the past when just using a single drive to test against I was able to get 90-100 MB/sec file copy speeds using FreeNAS as the server. In my limited RAID tests and ZFS tests with a couple of drives performance was LOWER. But it has been a while since I tested FreeNAS with RAID or ZFS.

Just for kicks I would just try testing FreeNAS with a single test drive using the native UFS file system and see what kind of performance you get. This way you can eliminate the possibility that the RAID or ZFS setup is limiting performance. I also recommend you try testing with your SAMBA buffers at 0 instead of the default. In general I have noticed higher performance on FreeNAS when doing this.

You know I never asked... what is your goal with the benchmarking? Are you wanting to know what your maximum performance is? Multiple user performance?

00Roush

 

[00Roush]

Well I went ahead and tested a little with FreeNAS, IOmeter, and a standard drag n drop file copy. It looks like the 1 MB read/write sizes are only used by Windows 7 when accessing another Windows machine that is running Vista or newer. ie SMB 2.0 is only used on those OSes. So when testing with IOmeter you should use 32 K as the read/write size. 32 K is the sized used by Windows 7/Vista when copying files to/from a FreeNAS box.

Am I confusing you yet??

00Roush

 

[thiggins]

The large block sizes are Windows-specific file transfer optimizations. I don't think this is related to SMB 2.0.

And you need to use Vista SP1 and higher to get the larger block sizes.

 

[travist]

00Roush,

Although I'm not very smart on the disk specifics, I do understand what you're recommending from a high level.

I am currently testing with 8 workers, each assigned a read and write. Although I tried adding them in different orders to each worker (with the intent of 4 reads and 4 writes at a time), it didn't seem to matter and ran the writes then the reads. In a 4 drive sRAID 5 configuration, I saw about 80 write and 70 read. CPU utilization was above 75% for the entire write, and it was high for the reads although not quite as high.

I dropped one disk to bring the array into a degraded state, and ran the same test over again. No change in performance.

Reconfigured FreeNAS to a 4 drive RAID 0 array, and performed the same test over again. Same results.

Reconfigured to ZFS / RAID 5, again using 4 disks. Write performance jumped to about 90 while reads were 65. Degraded the array and retested - writes dropped to 85 and reads stayed essentially the same. CPU maxed out during tests.

What does seem a little odd to me is that I'm consistantly writing faster than reading, and I thought that it was typically the other way around.

I'll try some of the other settings you recommended and post the results.

My main goal with these benchmarks is to a) gain knowledge about the different capabilities of NAS's b) determine the fastest performing configuration c) use gathered info to make an informed decision on the configuration in which to store my data (Music, Video, Pictures, Documents).

My environment is a windows domain with W7 clients. I would like to use the iSCSI target for a VMWare server that I will do testing / low volume production use.

Thanks again for all of your help!