777 MB/s from a TS-470 PRO NAS!

[Dennis Wood]

I haven't posted a great deal of late, but here are some performance numbers that might surprise you.

The arrival of "reasonably" priced 10G hardware has precipitated another round of testing in my test lab to see if the team at Cinevate can take advantage of shared storage at 10Gbe speeds for our media projects using the Adobe CC suite. We deal with a lot of HD (and now 4K) footage as well as raw product photos etc. As luck would have it, my old friends at Qnap and I were discussing my ambitions for 10G and they kindly sent out a TS-470 PRO with dual port 10G interface installed. Terry Kennedy from the forum here gave us a great deal on a used Netgear 10G switch..so the testing began.

After about 2 weeks of testing, some frustration and a lot of learning, you can see the results below of two windows workstations hitting up the TS470-PRO NAS with simultaneous "real world" requests for 10GB of media data.

I'll be doing a series of blog posts that will be helpful for those looking to do the same. If anyone has any requests, please indicate the in this thread.


1. Pcie 4x slot vs 8x slot Intel x540 10G NIC considerations. Basics of PCIe bandwidth and motherboard PCIe lane limitations.

2. 10G performance over cat6 and cat5e cable runs. (Not tested yet)

3. Use of NTttcp and ramdrives in 10G troubleshooting. Use of simple benchmarking tools that accurately reflect 10G file performance.

4. Qnap 4.1 and samba 3 asynchronous I/O considerations. I had to reinstall 4.1 code to get samba3 windows options back after disk volume deletion and replacement on the NAS.

5. Windows SMB2 vs SMB3 considerations. I had posted about SMB1 vs SMB2 performance several years back that changed the platform used for NAS testing at smallnetbuilder.com. It is obvious in my tests that SMB3 performance in Windows 8.1 will precipitate another change in testing. Samba SMB3 support in qnap 4.1 firmware was well timed indeed

6. Antivirus, impact of file I/O and CPU saturation at 10G transfer speeds.

7. Disk I/O, SSD/RAID vs ramdrive and testing.

8. Windows 10G driver configuration and jumbo frame considerations.

9. Performance results of Adobe CC suite products using QNAP 10G shared storage.

 

[Butzi]

Amazing NAS Progress

Dennis,

I'm looking forward to your testing. The integration of 10GbE has changed the NAS landscape quite a bit, along with all the other enabling pieces you mentioned.

I'm going to be doing some preliminary testing on the new SSD cache feature, using a TS-870 platform. I'll post a link when it's up.

Bruce.

 

[Dennis Wood]

Here's a screenshot of the Windows 8.1 file transfer window. It's a 9GB file being copied from the TS-470 PRO NAS, to a 12GB ramdrive on the workstation. I have to admit that this type of performance on a 10G network (and particularly at this cost!) is something I wasn't expecting to see for a few years yet. The switch and two 10G Intel NIC cards was ~ $1000. Certainly will change a lot of workflows in the video/photo world! I'll be sharing the step by step tests, configurations and tweaks (about 2 weeks worth) to get to these speeds. The good news is that you won't need 2 weeks to replicate..just an hour or so.

Large file transfer from TS-470 PRO to Windows 8.1 workstation (ramdrive).

Below is a copy of several large media files between two Windows 8.1 workstations. I've shared ramdrives on both machines as my disks are not capable of keeping up with these speeds

 

[Dennis Wood]

Very impressed with the Hitachi 4TB Deskstar NAS drives @ 7200 rpm. Here's what four of them in RAID zero will do over 10G. The real work write/read speeds in Windows 8.1 look to be ~ 550 MB/s. I'd ignore the 1000MB/s results as cache effect.

and Intel NASPT results. The Windows 8.1 workstation was restricted to 2GB of RAM using the msconfig tool in Windows. The workstation has 2 x Intel 530 SSD's in a Raid zero configuration, so should not be a limitation in this test. I should point out that the NAS is updated to QTS 4.1 so therefore is using SMB3 with asynchronous file I/O as a default.

 

[sinshiva]

just now reading this thread; those are some nice results. i used hitachi for my 4x raid0 a long time ago, they were the 7k1000.c single platter 500gb drives iirc. mine were hitting 500+ on the shortstroked volume of ~400gb/2tb, but the only way i got to see the numbers back then was linear read/writes to a ramdisk.

i mostly used my setup for ripping movies, etc. not too far off from what your doing

what i would do is leave the source material on the hdd and send the output to ramdisk. then copy it back. this was because i was limited by ram quantity, though. i suspect a better approach might be to copy the source to the ramdisk and output to the disk. obviously, in a perfect world, you'd just be swimming in ram and could input/output everything to ramdisks, then move to storage when you're finished.

the only problem with the ramdisk software i was using is that i noticed it added some instability to the OS. i wasn't using ECC, but i never found any glaring issues using the ramdisks themselves. the best way to explain it is that when i would try and use a decent memory consuming program like a game, they would (ofte enough) crash spontaneously, despite there being more than sufficient ram, swap, etc.

[edit/]

this is back from christmas 2009 lol, how time flies.

http://img.photobucket.com/albums/v295/utspyder/sleepy_raid.png

 

[Dennis Wood]

I would only use the ram disk software for testing ( at least for now) although a server with a large ram disk would be likely quite effective as a shared cache drive for adobe CC. I'm ramping up for the final build based on all the test results. The server will be a dual core based on a supermicro board incorporating 10g and the LSi 2208 raid chipset on board.

Interestingly enough, this week I finished our R&D, marketing and management transition to windows 8.1 pro, as well migration to SSd drives in the few remaining workstations that don't have them. On two workstations (adobe cc/graphics and Solidworks) SSD drives were already installed so the only change was windows 7 64bit upgraded in place to 8.1 pro 64bit. Both users reported "snappier" performance, much shorter boot times and zero issues with any of our existing software. Once the windows disk clean utility was run with the options to remove old system files, upgrade log files etc., disk space with windows 8.1 has actually decreased..maybe the first time ever for new windows versions?? I have not tested SMB3 performance as it relates to our older NAS units, however my sense after imaging and recovering to SSD over the network for several days.. was that there is a noticeable performance increase. Over the course of 10 workstation upgrades there have been zero problems that were not easily resolved.

For anyone on the fence with regard to windows 8.1, I am now on the "do it" side of things. The smb3 features (particularly multichannel) and performance improvements in server 2012 or windows 8.1 are substantial. In fact, until samba implements multichannel in their smb3 feature set, storage server 2012 will have an obvious advantage in a windows 8.1 workstation environment.

 

[scaigs]

QNAP-TS x70 & 10 GbE

Good morning Mr. Wood,

I have a similar setup:

• QNAP TS-870 (non-pro) + QNAP 10GbE RJ45 dual-port card + SMB3 activated.
• 7x WD Re 4TB HDD
• 1x GForce GS SSD 256Gb (for cache)
• RAID 5
• Intel X540T2 for my ESXi 5.5 servers running Win 7 Pro 64 bits and WIn 8.1 Pro 64 bits
• NetGear XS708E GbE Switch
• VMware vxnet3 network drivers.

Well first I was trying to achieve the speeds you mentioned in your post and was very far from them.

In my NASPT tests the maximum transfert rate that I achieved was below 300 MB/sec, an also other tests like Parkdale and still have to try sqlio.


Then I noticed something, I was using firmware 4.0.5 and decided to upgrade to 4.1 Beta, also, your are using RAID 0 and I RAID 5, as far as I know nobody uses RAID 0 in a real NAS or SAN environment, so yes your are achieving pure performance in a non realistic day to day setup.

Firmware 4.1 is better now closer to the 400 MB/sec but still far away from the 777 MB/sec posted with RAID 0 even when using Windows 8.1 (I see the performance in task manager capping at 2.x Gbps on 10 Gbps available in some tests).

Cache disk just kills write performance and when I turn it off I gain about +/- 10% write speed.

So personally, I'm a little disappointed in the small gain with the 10GbE hardware after investing like $3K in it.


Cheers from brisky Montréal.

 

[scaigs]

QNAP-TS x70 & 10 GbE

I kept on running my tests on the QNAP TS-870 last evening from my ESX servers.

Performance mesured against a file share on the TS-870 not an iSCSI LUN.

a) Performance is better under a Win 7 Pro 64 bits VM than a WIn 8.1 64 Bits one.
b) Again under 400 MB/sec using tools like NASPT and Parkdale.
c) The I used Microsoft's SQLIO benchmark tool (http://www.microsoft.com/en-ca/download/details.aspx?id=20163) and this time it hit the 700+ MB/s throughput, CPU went through the roof on the TS-870 (peaking at over 90%) and SSD cache was disabled of course. From the moment I activated the SSD Cache the performance went down a few percentage points.

So now I'm more confident that the gain is there but it is which tools and under the which conditions/environment that the tests are performed that the results will vary.

One last note, with Firmware 4.1.0, my ESXi 5.5 servers can't see the iSCSI LUNs on the TS-870 and the QNAP VMware add-on refuses to connect to the NAS because of firmware 4.1.0. So now if I want to keep on working and decommission my old Synology NAS I have to revert to firmware 4.0.5. and transfer the data.

Hope that QNAP provides the "go back" procedure to firmware v4.0.5 because I tried using the GUI and it gave me an error 3 times.

Cheers everyone.

 

[Tonica@QNAP]

Hi! Thanks so much for that feedback and information. I am going to share that last note with my team and would like to provide some insight as to whether or not our new upgrade has resolved that issue or if there is something we can do to make that experience better for you in the future.

 

[Dennis Wood]

Sorry I missed your posts..

Keep in mind that this unit tested was a TS-470, not an 870...and I did use raid0 really just to demonstrate max IO. I also tested with SSD drives x 4 to explore the io limits of a single connection over smb3. There were clear advantages over smb2 using smb3 in windows 8.1. Test first with a "real" client and then with virtual as you are introducing too many variables with a VM.

You may want to look over my blog series on how I tested : http://www.smallnetbuilder.com/labels/Cinevate

Parkdale and Naspt are usefull, but not as accurate as ATTO. I found setting up a Ramdrive on windows client and testing large file transfers were efficient ways to tune. Virtualization settings in the intel x540 NIC advanced properties are enabled by default (server 2012) and really hurt performance. Antivirus like bit defender also dramatically drop performance. I'd suggest ram drive testing, and watch CPU loading on both NAS and client.

I did review SQLIO during windows storage spaces parity SSD journaling testing...it looks like an excellent load testing tool...your 700MB numbers are in line with what I measured. Keep in mind that single threaded connections (at least in my tests) will max out at 750 MB/s using Samba SMB3. To get above 1000mb/s required Microsoft SMB3 on both client/server with RSS queues tuned, jumbo frames and four core processors.

Look for my TS-870 pro tests posted here. With 8 x 4tb hitachi in raid 5 I had no issues hitting 550 MB/s writes, and reads in the 740 MB area. Client was windows 8.1 with an i7 4770K processor and intel x540 nic, with tuned NIc settings. You may want to test jumbo frames on both NAS and client