Improving Network Performance

[Jesse B]

I recently purchased a new switch (Dell 2716), and have been playing around with my server a bit. Unfortunately, my results this far aren't overly satisfying. I wasn't expecting full Gbit speeds with the current setup I have, but I was expecting a lot more than I'm getting.

First things first, my NICs aren't the best. This is just a fact of life right now, as I'm in the process of hopefully upgrading some things. My server has a Realtek 8111 (boo!) and my Laptop has a JMicron something-or-other (not 100% sure what it is). Both these computers have 4GB of RAM. Both computers are connected via Cat6 patch cables to the Dell switch. No configuration has been done to any of these 3 devices.

I've done a bit of testing, so I do have some numbers and results to throw out there. First things first, here's iperf:

That all pans out (I did the test running in both directions as well, forget what it's called, and got the same results. Just didn't get a screenshot).

Next is the benchmark on my mirrored array (2x Samsung F4 2TB's) on my server:

Probably not the best method of testing, but it should still be able to nearly saturate a Gbit network, so that's obviously not the issue.

I ran IOzone on my Laptop, don't really know how to decipher the results, so here they are:

If you guys want me to run something else just lemme know.

Finally, transferring a large file to/from my Server/PC. Not a real accurate test, but this is just real world results.

As you can see, those results are quite pitiful.

I don't really know where to go from here next. Changing NICs isn't an option at this point. I'm running Cat6 and a high-end Gbit switch, so that shouldn't be an issue. At this point I'm thinking it has to be primarily configuration, and maybe my laptop drive is slow as well. As I mentioned, my server/Laptop have default network configurations, and the Dell 2716 is at system defaults currently.

Any suggestions on how I can improve my speeds?

Thanks,


- Jesse

 

[GregN]

I recently purchased a new switch (Dell 2716), and have been playing around with my server a bit. Unfortunately, my results this far aren't overly satisfying. I wasn't expecting full Gbit speeds with the current setup I have, but I was expecting a lot more than I'm getting.\
- Jesse

As it has it, I'm doing tuning for part 3 of the SAN series, here are a few tips:

Change the mirrored disks to Write Back instead of Write Through: Go to properties for the disk in the device manager, policies tab, check both boxes. ( downside, if you lose power during a write, you'll lose data )

Optimize your TCP/IP connection using Speedguide's optimizer: http://www.speedguide.net/downloads.php

More important than anything else, try to up the amount of core you have in each machine, largest amount of the fastest memory you can handle.

Isolate choke points, starting from one machine ( the one least likely to be a problem) run j/i perf as you slowly add components to your network. Do this sans internet initially, and add pieces bit by bit. A bad cable, bad port, bad switch can spoil an entire network ( retrans and fragmentation can slow the entire net down ).

Turn off all of the various Green options while testing.

Consider using subnets, and isolating say the laptop there. Some switches , and wifi in-particular will run at the speed of the slowest element.

You are correct to be concerned about the controllers, Realtek and alike often slice performance off the CPU esp on Checksum handling. Lamprey controllers and weak cpu with limited memory is a sure recipe for killing a network.

NASPT from intel is a good "real world" test of network performance.

Solving problems like yours, there are very few quick fixes, it often takes methodical multiple iterative measure and change cycles to get to a satisfactory feeling that you've exhausted your options.

There is also a MS doc you might want to look at:
http://download.microsoft.com/downl...7-4CF8-8534-9C2AAA0117F4/Perf-tun-srv-R2.docx

This is geared towards MS Server 2008, but has some significant tips for Win7 ( treat it as voodoo, which it might be since you aren't running MS Server, try doing some invocations, then measure )

Hope this helps

 

[Jesse B]

Thank you very much for all your suggestions! I'll have to go through all that tonight after work.

Thanks again,


- Jesse

 

[Jesse B]

Change the mirrored disks to Write Back instead of Write Through: Go to properties for the disk in the device manager, policies tab, check both boxes. ( downside, if you lose power during a write, you'll lose data )

I'll have to figure out how to do this on my current OS (OpenIndiana). I don't have any overly important data, so it wouldn't be the end of the world if that situation occurred. I can't see that happening anyways.

More important than anything else, try to up the amount of core you have in each machine, largest amount of the fastest memory you can handle.

This will have to wait a little while. I'm in the planning stages of completely revamping my server (hopefully within the next month or so). The new rig will have a much more powerful CPU, much more RAM, and proper Intel-based NICs (multiples!). That should help

Isolate choke points, starting from one machine ( the one least likely to be a problem) run j/i perf as you slowly add components to your network. Do this sans internet initially, and add pieces bit by bit. A bad cable, bad port, bad switch can spoil an entire network ( retrans and fragmentation can slow the entire net down ).

Basically here's the situation. I'm living somewhere temporarily (just for the extent of the summer), so I can't do too much to their network. I basically just have a Cat6 cable run from the router (which all other devices that aren't mine are connected to as well) to my switch, then my devices are connected to my switch. There's not a whole lot I can do until I move back into my own place unfortunately.

Thank you very much for taking the time to wrote up that post. It was extremely informative. I've started working on a few things, but will obviously have to do a bit more research and work in general.

Thanks again,


- Jesse

 

[GregN]

Continued

You are running OpenIndiana, how are you sharing the disks, iSCSI or as a NAS? What OS are you sharing with? Raid controller or software raid? If you are doing iSCSI than you configure the disks under windows. Otherwise take alook at Samba tuning.

What kind of switch you running? Sounds like you are stuck with whatever speed is coming off the switch ( and using the switch as a router.... ). I'd do some testing without the switch, and see what kind of effect that is having. Try using a router instead.

Hope that helps

 

[00Roush]

Based on your Iperf results your network looks to be working just fine. The sustained read/write speed of you ZFS pool in the server looks just fine too. Have you also checked the disk speed in your client (laptop)? If that is good to go then your sub systems look to all be up to speed and the next thing to try and find would be a software bottleneck.

I noticed you were testing using FTP and from what I can tell from the screen shots maybe running Windows Vista/7. Since you are running a Windows setup I would recommend you also test using CIFS/SMB sharing. Generally I have found FTP (FileZilla on Windows) offers good performance but on occasion I recall seeing poor performance.

If you have another Windows machine to test with you can try sharing files between two Windows machines to see what a best case might be.

As GregN mentioned it might not be a quick fix. Part of what makes it a bit hard in your case to diagnose is the OS you are using on the server. I am not sure if out of the box the OS needs any tuning to provide good file copy performance. This is in comparison to knowing that out of the box a Windows 7 setup can copy files over a network connection at disk speed (up to about 113 MB/sec) provided the computer being connected to is capable. From what I recall NexentaStor didn't need any special tuning when I tested it so I would guess OpenIndiana shouldn't need much tweaking either.

In general I like to start simple with a client setup I know can do high performance file copies. In my case I know my main desktop is capable of sustaining copy speeds of 113 MB/sec (Gigabit is pretty much maxed out) when copying files 20 GB in size and larger. Then I try to start simple with my server setup by testing with only a single hard drive at first to see where performance is. This eliminates any possible configuration problems with RAID. When sharing over the network I try for the simiplest setup first which usually is a samba share with no security.

It has been while and I can't remember for sure but I think I tried messing around with OpenIndiana a while back and couldn't get very good performance... Things worked good using NexentaStor Community Edition so I didn't really feel like trying to get OpenIndiana to work. Maybe I'll give it a go again and see how it turns out.

00Roush

 

[Jesse B]

Thanks for your input. I was actually considering changing my server's OS. OpenIndiana is nice and all, it just doesn't really feel complete to me.

The reason I was using FTP was in part because when I initially tried setting up Samba on OI, I just couldn't get the damn thing to work. I'm exploring other options currently.

I've got a lot of testing and tweaking to do, so I'll be sure to post back as soon as I get the time to do all this.

Thanks,


- Jesse

 

[Jesse B]

So I've done a bit more testing here. I have my Desktop again, so I mounted my NFS share. Ran the command "time cp "file-to-destination-etc", which gave me the time it took to transfer, and I knew the size of the file before hand. Here are the results:

4.3GB *.iso
Server -> Client
61m22.101s
~1.2MB/s

4.3GB *.iso
Client -> Server
1m29.817s
~49.0MB/s

This CLEARLY shows that my desktop is in dire need of a new hard drive. Yikes, that was appalling. 49MB/s isn't too awful, I'd still like more however, and I'm wondering what the likelihood of my read-speed limiting the transfer rate is. For the time being, that's acceptable however. I'll have to do some more tweaking and configuration still.

Just thought I'd share this,


- Jesse

Edit: Ran the same commands on my laptop on a wired connection, results:

4.3GB *.iso
Server -> Client
1m34.027s
~46.8MB/s

4.3GB *.iso
Client -> Server
2m14.224s
~32.8MB/s