TS509, Link Aggregation, 3COM 2916 = good

[Dennis Wood]

We just got a 16 port switch in here specifically because it was the only layer 2 managed switch I could find for under $300 that had full support for 803.2ad...or automatic link aggregation. Trunking and link aggregation was supported on the previous switch, the DLINK 1216T, but it does not fully support this specification...only static trunking.

Enter the TS509 with dual ports and the magic claim that it supported load balancing. With a static trunk set up and 2 LAN cables used, the TS509 did not load balance on the trunk which was not a surprise. QNAP specifies the switch must support automatic link aggregation, 803.2ad. Well, there are some issues there under multiple loads (and I'm working with Qnap on this) but what a surprise in testing today. The test workstation uses an ASUS (P5W-DH) board with Marvel dual LAN PCIe connected gigabit and Vista SP1 with ICH7 based RAID 0 drives. The Marvel driver supports 3 modes, basic, static, dynamic which are used depending on the switch capabilities. Dynamic is what you want, and can only be used with switches fully supporting 803.2ad....like the 3COm 2916. This switch has an amazing feature set for under $300.

My previous tests with link aggregation on the DLINK 1216T switch (using the aforementioned workstation) showed that link aggregation worked, but only in "static" mode on the Marvel driver with a trunk configured. Now with the 3COM switch, I can use "dynamic" mode, and the TS509 does work in load balancing mode.

Performance from the same workstation to the TS509 (neither teamed) was previously 45MB/s write, and about 60MB/s read. With the same workstation now configured with two gigabit LAN connections, the TS509 connected in the same manner, and the LACL trunks set up, I got a surprise. Write speeds jumped to 50MB/s and read speads are much higher at 83 MB/s using our 5.3GB test set of files. This is with the TS509 using RAID 5.

In our large file copy (200GB) tests from the NAS to the desktop, Vista is now jugging along at sustained 97.5 MB/s read. Link aggregation is not supposed to increase speed of one "pipe", but clearly something else is going on here. Any LACL experts out there? It would seem that "dynamic" LACL is capable of doubling the pipe, and increasing a single workstation's (IP address) performance.

Note that with the latest firmware 2.0.2, performance with 2 workstations loading the unit simultaneously is completely unacceptable at less then 3MB/s ... so some more code updates will be required.

 

[thiggins]

So "load balance" on the TS509 Pro is the same as teaming? Or only with the dynamic mode on the 3Com switch?

 

[Dennis Wood]

For those who are not aware, the QNAP TS509 has two gigabit LAN ports that support dual LAN, failover, or fault tolerant operation. The TS509 network interface allows you to choose "Load Balancing" as an option, but the literature indicates you must use an 803.2ad compliant switch for this to work. So you may choose this, with both LAN ports connected, but with a normal switch...there's no load balancing. All of the traffic goes to one port.

With a switch like the DLink 1216T that supports manual (static) trunking, teaming, or static link aggregation (used to describe the same thing by various manufacturers)...no joy. The TS509 still only uses one port, even with a static trunk setup on the switch. Now if you have a motherboard with the Marvel Yukon family dual gigabit ports...that will work as a server and load balance just fine, as long as the driver is set to "static" mode. So the Marvel driver will support basic "teaming" even on a cheap switch but the links cannot dynamically configure themselves, and two identical teamed workstation will never be any faster with respect to LAN traffic between them. These types of motherboards would be a perfect base for a cheap but high performing NAS under multiple loads as each (assuming 2) workstation would end up with it's own pipe. The Nvidia 680i motherboard chipset supports teaming as well, but it seems to be very basic. It does not work if a trunk is configured on either of the switches, static or LACL.

With the 3COM switch you can set up either static trunks, or LACL trunks. An LACL trunk is evidently a fully compliant 803.2ad construct. From what I understand of the 803.2ad spec, this spec allows dynamically reconfiguring of the the ports used in a trunk. A trunk can contain up to 8 ports on the switch. With a four port PCIe NIC, you could therefore create a 4 port trunk that would likely reconfigure as required for higher bandwidth. I believe that what I'm seeing now is the absolute limit of the TS509 unit drive performance. 50MB/s write and 83MB/s read is by far the best RAID 5 performance I"ve seen over the LAN in our testing. To compare, the best we could do with an onboard ICH7 RAID 5 was 35MB/s local write!

Next step...our own NAS with dual LAN ports, and the RocketRaid 4 port controller?

 

[thiggins]

The Broadcom BRCM 5787 Ethernet chips in the TS509 PRO are PCIe. So there must be some other limitation that is preventing a single PCIe Ethernet port from reaching the > 100 MB/s that my IxChariot testing showed that it was capable of.

 

[Dennis Wood]

Yes, it may be at the NAS end, or the workstation end...hard to tell. Iozone, for the first time, is not accurately measuring large file read and write rates between a workstation and NAS with LACL used on both. What we're actually measuring is about 20% higher.

In fact, none of the tools work in this configuration. I suspect the 3COM switch is routing upstream and downstream traffic dynamically over the 2 LAN connections and causing some measurement confusion. It's the only way to explain the 25% improvement in read rates from the NAS.

 

[iwod]

Yes, it may be at the NAS end, or the workstation end...hard to tell. Iozone, for the first time, is not accurately measuring large file read and write rates between a workstation and NAS with LACL used on both. What we're actually measuring is about 20% higher.

In fact, none of the tools work in this configuration. I suspect the 3COM switch is routing upstream and downstream traffic dynamically over the 2 LAN connections and causing some measurement confusion. It's the only way to explain the 25% improvement in read rates from the NAS.

So what you are saying is the results inaccurate?
Becasuse it sounds too good to be true

 

[Dennis Wood]

What I'm saying, after many, many tests is that if you hook up the TS509 in fault tolerance mode (2 ports connected to the switch), to a an LACL capable switch, with a trunk configured, here's what you will see in actual measured file transfer times. This assumes you're using Vista SP1 on a machine with at least 2 drives in RAID 0 (to keep up with the NAS). It also assumes that this workstation has PCIe connected gigabit and you're connected to the switch with one port.

1. Read speeds will exceed 85 MB/s on a mixed set of files. If you are copying a single 200GB file off the NAS, Vista will report read rates at about 97 MB/s. This I believe is as good you'll ever get over gigabit ethernet. If you try copying two different 200GB files off the NAS to two different Vista SP1 workstations (both with RAID 0), each will report 63MB/s for a total rate of about 120MB/s. This is outstanding.

2. Write speeds will be at 50MB/s.

If you load the NAS from two workstations (at least in our tests) the write speed will drop dramatically. I'm very confident this will be addressed with a code update. Confident enough to order a second TS509 to see how it performs with rsync (synchronizing two units for offsite backup over the WAN)

 

[Dennis Wood]

A few more tests with the TS509 using RAID5 and in load balancing mode using our 5.3 GB mixed file set.

Single drive XP SP3 workstation with PCI gigabit:

Read: 38 MB/s
Write: 36 MB/s

Single drive Vista SP1 laptop with PCIe gigabit:

Read: 30 MB/s
Write: 44MB/s

Using Storagecraft backup, we were able to back the laptop up to the NAS (56GB) at a rate of approx. 45 MB/s which is excellent.

Simultaneous two workstation loading (Vista SP1, RAID0)

Read: 61MB/s (total aggregate read rate at 122MB/s)
Write: 3 MB/s (appears to be a code issue with the NAS)

Single workstation, Nvidia 680i SLI chipset, Vista SP1, RAID 0 onboard)

Read 83MB/s
Write: 52MB/s

 

[SirBC]

Confident enough to order a second TS509 to see how it performs with rsync (synchronizing two units for offsite backup over the WAN)

Dennis, did you get a chance to test the remote replication feature yet? I would be interested to hear your thoughts as I am considering purchasing two of these for the same purpose.

Cheers,
Dave

 

[Dennis Wood]

Dave, I've got 5 WD green drives 1TB sitting here and a TS509 on the way. I'll definitely post my thoughts on rsync performance when they're here. It will be interested to test the NAS with the WD green drives and compare to the unit using the 5 Seagate Enterprise 1TB drives which case $100 more each. With the money saved, one could purchase six of the green drives, and still have $400 in your pocket. We use several of the WD green drives in external drive enclosures as theoretically they should run cooler.

On another note, I've just tested some beta firmware and the dual workstation loading test shows much improved results. The TS509 is still in load balancing network mode to an LACL capable switch. Reads are still (to a RAID 0 Vista SP1 workstation) an impressive 97MB/s on large files, and writes at around 47MB/s.

On two workstations loading the TS509 simultaneously, write rates were dropping to 3MB/s and are now improved to approximately 10MB/s. in the beta code for an aggregate total of approx. 20MB/s. These are XP workstations, and I have not tested two Vista workstations yet. I believe the QNAP folks can potentially tune this to where I believe it should be, about 40 to 45MB/s aggregate.

 

[Dennis Wood]

Well, the second NAS arrived today, earlier than expected so we loaded it up with drives, updated code and started testing.

So far, rsync is working like a charm which I honestly didn't expect to see at all (call me a cynic). The user manual is fairly limited in it's helpfulness here, but there really wasn't much to mess up. What I really like is that once you schedule an rsync job, the NAS attempts the connection to ensure no mistakes were made in your paths, or port assignments and let's you know if there's a problem. Toggling SSH encryption is as easy as a mouse click on a check box, ensuring the ports you've set on the NAS match with what you've entered in the rsync job. The rsync and SSH ports can be set to whatever you like. With both TS509s in load balancing network mode, and two LACL trunks configured on the switch, the local sync tests reflected some very impressive speeds. Once the first local sync is done (8 staged jobs), we'll take unit 2 remote and see how it does over the WAN. Replicating several Terabytes of data over a DSL connection here would take a very long time, so a local sync to start makes a lot of sense. Rsync does a delta backup (assuming you've checked off the "incremental" check box) of just binary changes to a file so the subsequent replication is quite efficient. There are plenty of options for monthly, weekly or daily scheduling in the interface, and you can select an entire share, or a sub-folder in a share to replicate. This means that if you want to backup 8 different shares, you'll need to schedule 8 different rsync jobs. Another check box toggles file deletion so if you delete a file on NAS1, it gets deleted on NAS2 during the sync.

The goal of all of this is simply unattended, secure off-site backup with no user intervention required, no data limits (well 5TB I guess), and no licensing/or bandwith costs to worry about.

 

[SirBC]

Dennis, thanks for the info. I’m glad to hear it worked as well as it did. I assume you are using the built-in Remote Replication feature and not running Rsync on its own? Did you notice a difference in transfer speed when SSH was enabled? Any problems with the WD Green drives (I am looking at using those as well)?

Cheers,
Dave

 

[Dennis Wood]

Yes, I'm using the built in replication. It isn't clear (nor does the manual explain) what the differences are between toggling "Enable backup from a remote server to the local host" or "Allow remote Rsync server to back up data to the NAS". I'd guess that option two (and both can be enabled at the same time) may allow an rsync service running on a third party Linux box to back up the NAS. From what I can tell, both use rysnc, and both use the same port you define in the setup dialog.

The WD green drives are just fine. The NAS likes them, and so do our eSATA external drive enclosures. They're also listed on QNAP's drive list as OK. They aren't particularly speedy by themselves, but (tests coming) I would guess zero impact with 5 of them in RAID5. Gigabit LAN speeds would come into play long before aggregate drive bandwidth would. We do have an Adaptec hardware RAID card (PCIe) coming to do some more tests with and drive speed likely will come into play there.

Just ball parking the sync times (based on share sizes vs event log times) with and without encryption (SSH)..and assuming both NAS units are locally connected:

With SSH: ~14MB/s
Without SSH: 43MB/s

Given that SSH would only be toggled typically for WAN traffic, the 14MB/s is obviously far faster than a typical DSL or cable connections's bandwidth...and therefore not a concern IMHO. Once NAS2 is remote, we'll have a look at how efficient rsync is with regard to delta binary backups.

 

[SirBC]

Good stuff, thanks! I went ahead and ordered two, can't wait to try them out.

- Dave

 

[Dennis Wood]

I'm quite impressed with how efficient the remote replication is working. It's almost certainly rsync based given the fact that a modified database file (just deleted a SQL query) on the source NAS was synchronized over the WAN in about 1 second..and it's a 4.66MB file. Finding that a bit hard to believe, I checked the corresponding file on the backup NAS, and sure enough, it was altered with the deleted query. Very cool.

 

[Dennis Wood]

It would appear that my results with the beta firmware should have been better...and are under-reported a few posts back. The problem is that one of the two test workstations was using an LACL trunk, as was the QNAP TS509. Once both workstations were set to a single port, the results are actually better. This could be due to the workstation configuration, weird switch interactions, or the TS509 driver. This time I used a single 2.5GB file at the request of the nice folks at Q central

Simultaneous 2 workstation performance XP SP3, PCI NIC, single HD:

Reads: 34MB/s
Writes: 20 MB/s

Simultaneous 2 workstation performance Vista SP1, RAID 0, PCIe NIC:

Reads: 27MB/s
Writes: 19 MB/s

Note that the SMB2 performance is not as good as SMB. This is a surprise as the single workstation performance (Vista) reads at 77MB/s and writes at 51MB/s on the same file. Normally I'd be a bit concerned about this, however there are more tweaks coming down the line. As much as I don't like testing in this way, the QNAP engineers have shown a genuine interest in refining performance and it's refreshing to see their attitude.

For those interested in the remote replication performance...it's been working flawlessly. We've got 9 "jobs" scheduled and to my surprise, they work concurrently. In other words job 1 can run concurrently to job 2 etc. This is good news as some jobs make take a day or two to run if for example you've created a new share, and haven't done a "local" replication first. Another plus is that the rsync jobs don't seem to bog the NAS down for normal network use. CPU usage hovers at 1 to 2% with a remote replication job running (SSH enabled).

 

[SirBC]

I'm glad to hear the remote replication is working as well as it is. That will be the main function of my two Qnap's. Since I haven't received mine yet I am not familiar with the options for the replication feature, but I am guessing you can't use regular expressions. I need to backup a folder but only files with certain file names should be backed up (XX-io37.spi, xx-io38.spi, etc). I believe I can do this with a manual Rsync job, but I believe that standalone Rsync isn't supported by Qnap. So it may take some creativity to get the jobs working like I will need.

 

[Dennis Wood]

No filtering option exists in the QNAP gui...and as far as I can tell, you can't enable outbound rsync. That said, I know folks are doing this via the console.

Exclusions would be a very nice addition, however we've just placed those files on a share that is not replicated. You can configure those shares to backup to an eSATA drive via the standard backup schedule, and take those offsite.

 

[Dennis Wood]

If you are using the TS509, you'll want to grab the latest firmware (1016) released yesterday.

Two workstations are now able to load the NAS simultaneously with large file writes at about 24MB/s each. It is able to do this while the NAS is writing large files to a 1TB eSATA drive at a rate varying from 5MB/s to 33MB/s This is good It looks like a few vagaries with Wake On Lan (WOL) in Vista have been addressed as well. Certainly the unit is ready finally to be called a "business use" device.

I'm almost certain the NIC drivers (load balancing to an LACL switch) in the NAS have been updated as the switch port stats on single workstation writes have changed...it appears to be load balancing differently now. Two workstations hitting it simultaneously will each get a port...whereas earlier, it appeared as if one NAS gige port was doing RX and the other TX.

 

[SirBC]

Good news. I just finished formatting the drives and error checking on both devices. Time for some firmware goodness