Home build nas: How improve transfer speed

[ugo1]

Hi

I finally started to build my home nas based on ubuntu hardy 64 bit.

I will have 5 hard disk so I choose the Asus P5E VM HDMI which has 6 sata ports. The Ubuntu installation hadn't problems abd all gone well. Before installing all the things I started some speed test via ftp from my computer and the server. I'm able to transfer 32 GB in 8 minutes from XP SP2 (not bad at all) using the integrated gigabit lan. My files can reach 12 GB each. Actually I have 2GB of ram installed. The data will be saved on two disks in raid1 (the test above is made on system disk)

Questions

1- I notice that certain point for 3 / 4 second the lan trasfer was at 0, what can be success?

2- there is some action that i can take to improve a bit the transfer speed?

Regards

 

[scotty]

Look at SNB's 'How to Build you Own NAS' series. Tim goes through the steps and documents some of the things that make a big difference. In short:

- Use PCIe GigE NIC's, normal onboard/PCI nics wont cut it.
- Try to use Jumbo frames if you can
- The hard drives make a difference. Normal 7200rpm sata drives dont go much beyond 70MB/s by themselves.

Lots of good info in the articles.

 

[00Roush]

I am a little confused on your configuration. Were your test results from the server you are trying to increase transfer speeds or was that previous results? Could you give a few specs on the server and client(s) you are using. Things like OS, CPU, and network card.

I recommend you test your setup to see where you stand first. This means first testing raw network speed using a program like Iperf. Then testing hard drive transfer speeds with a program like HDTach.

If raw network speeds are above 800 Mbps move on to testing hard drive speeds. The hard drives on both the client and server need to be benchmarked as the slowest drive will determine the maximum transfer speeds you will see. Especially for very large files.

Testing on Windows and Ubuntu should be fairly easy. For Windows you can download HDTach and Iperf. For Ubuntu you should be able to use hdparm and then you should be able to install Iperf by typing "sudo apt-get install iperf". Here is the command lines I usually use for Iperf and hdparm:
Server: Iperf -s
Client: Iperf -w 64k -c 192.168.0.2 -r (r tests the connection in both directions)

hdparm -t /dev/sda

If your network transfer speeds are within 5-10 MB/sec of benchmarked hard drive speeds you probably won't be able to improve transfer speeds much.

For more information about command line options with Iperf and hdparm type "man iperf" or "man hdparm" into google.

On a side note...if you are just looking at FTP performance you might try using Filezilla Client on your Win XP SP2 client. I have found that it has better performance than the windows implementation.

00Roush

 

[ugo1]

hi,

first my configurations

Client
Win XP SP2
2GB of ram
QX6700
and on board Marvel Yukon ethernet card

server
Ubuntu 64 bit
E2140
2GB of ram
and on board Atheros L1 PCI-E ethernet card

second

I made the test suggested

on the server

iperf -s

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------

iperf -w 64k -c 192.168.0.10 -r

KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.10, TCP port 5001
TCP window size: 128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[ 5] local 0.0.0.0 port 60228 connected with 192.168.0.10 port 5001
[ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec

hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 344 MB in 3.00 seconds = 114.58 MB/sec

On client side

HDtach test

read test from 80MB/s 40MB/s average read 67,5
burst test 187,5

Regards

Ugo

 

[ugo1]

Look at SNB's 'How to Build you Own NAS' series. Tim goes through the steps and documents some of the things that make a big difference. In short:

- Use PCIe GigE NIC's, normal onboard/PCI nics wont cut it.
- Try to use Jumbo frames if you can
- The hard drives make a difference. Normal 7200rpm sata drives dont go much beyond 70MB/s by themselves.

Lots of good info in the articles.

You say that a pcie card is better than the onboard one?

other question

I notice that with filezilla (I configured to trasfer a single file at time from the client to the server)

file of 360 MB or less are trasferred at 30MB/s max
1 st 11GB file is trasferred between 64MB/s and 60MB/s
2 nd 11GB file is traferred between 53MB/s and 57MB/s
3 rd 11GB file is trsferred between 50MB/s and 55MB/s

regards

Ugo

 

[00Roush]

Well you can have a PCIe network card that is onboard. Like in my case I have a Marvell Yukon 88E8056 PCIe based onboard network card for my client. In many cases though onboard network cards are on the PCI bus not the PCIe bus.

Not sure why you didn't get iperf results...
Just for reference the IP address you type in is the one for the server you are trying to connect to. So in my case the server IP is 192.168.0.2 and the client has an IP of 192.168.0.4. Should look something like this...

Server:

C:\IPERF>iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------

Client:

C:\iperf>iperf -w 64k -c 192.168.0.2 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.2, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[156] local 192.168.0.4 port 56069 connected with 192.168.0.2 port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec
[164] local 192.168.0.4 port 5001 connected with 192.168.0.2 port 1054
[ ID] Interval Transfer Bandwidth
[164] 0.0-10.0 sec 1.10 GBytes 945 Mbits/sec

For reference this is between a Vista SP1 client and Win SP2 server. I can share more specs if you would like.


Your server disk speed looks good.

Your HDTach results along with your filezilla results make me wonder if your client hard drive might be slowing you down a little.

It looks like you are saying your client hard drive has a max read of 80 MB/sec and min read of 40 MB/sec. This would mean when your hard drive is reading and writing data at the beginning of the drive it will at around 80 MB/sec. Then when you are reading and writing data toward the end of the drive it will be at around 40 MB/sec. I am just guessing based upon the size of your test files that your client hard drive was not empty for your tests. So your read/write speeds will be somewhere between 40 and 80 MB/sec. The fuller the drive the closer to the minimum you will get.

You mentioned 3 file copies you did with filezilla... Were these 3 different files you copied one after the other? If it was then you did a total file copy of about 33 GB right? My thought is that as the drive being copied to filled up, write speeds slowed down a bit that is why you saw progressively slower file copies. That is just a guess though. Not sure why you saw slower speeds on the smaller files though.

One thing I did want to add was if you want to see peak transfer speeds make sure that the drives you use are separate from the OS drive. This helps eliminate any unnecessary (unwanted) disk accesses from the OS during file transfers.

Not sure if you mentioned it yet... are you running Ubuntu Desktop or Ubuntu Server.

00Roush

 

[ugo1]

Hi

I didn't understand that I should run iperf at the same time on server and client

client 192.168.0.10
iperf -w 64k -c 192.168.0.6 -r

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.6, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[148] local 192.168.0.10 port 49420 connected with 192.168.0.6 port 5001
[ ID] Interval Transfer Bandwidth
[148] 0.0-10.0 sec 339 MBytes 284 Mbits/sec
[172] local 192.168.0.10 port 5001 connected with 192.168.0.6 port 45311
[ ID] Interval Transfer Bandwidth
[172] 0.0-10.0 sec 109 MBytes 91.7 Mbits/sec

Server (Ubuntu server Hardy 64 bit) 192.168.0.6

iperf -s

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.10, TCP port 5001
TCP window size: 128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[ 5] local 192.168.0.6 port 39499 connected with 192.168.0.10 port 5001
[ 5] 0.0-10.0 sec 109 MBytes 91.8 Mbits/sec
[ 4] local 192.168.0.6 port 5001 connected with 192.168.0.10 port 49394
[ 4] 0.0-10.0 sec 306 MBytes 256 Mbits/sec

How you have obtained the yours

the onboard lans (client and server) are on the PCIe bus

Answers
yes the client disks is almost full

You mentioned 3 file copies you did with filezilla... Were these 3 different files you copied one after the other? If it was then you did a total file copy of about 33 GB right? My thought is that as the drive being copied to filled up, write speeds slowed down a bit that is why you saw progressively slower file copies. That is just a guess though. Not sure why you saw slower speeds on the smaller files though.

Yes I move (form client to server) a file at time and they are saved in the OS hard disk (I must still install two separate disk in the server) and yes total file are about 33GB. to be sharp I have 2 files of 11GB (file 1 and file 2) 1 file of (9GB file 3) and some files of 300MB / 360MB.

Can the amount of ram installed on the server help?

Regards

Ugo

 

[00Roush]

It has been a while since I had Ubuntu on my server... I recall having to use different window sizes with iperf to get accurate results. Instead of using -w 64k try -w 32k or -w 128k. This changes the window size which is basically how much data can be in flight at one time. Your iperf results seemed a little off considering you are transfering data at about 60 MB/sec (or about 480 Mbps) using FTP. Iperf should show at least that high as it is measuring raw tcp/ip throughput and you already know from your FTP results you can do at least 480 Mpbs. Try the different window sizes and see what you get.

As I mentioned in my previous post I think your bottleneck is most likely your client hard drive. If you are looking to see better performance out of your current hardware you could try testing with hard drives that don't have a lot of data on them. While this is not how they likely will be used, I feel it is good to see what the best case scenario might be. Another option would be to use faster hard drives and see if transfer speeds increase.

You asked if the amount of RAM in the server will help with performance. I say yes and no. If you have less than 512 MB of RAM I think file transfer speed could suffer. Having more than 1 GB of RAM can help but overall will not increase transfer speeds when files are read/written directly from the disk. More memory basically will allow for more data to be cached. This means there would be more chances for the server to read directly from memory instead of reading from the hard drive. But this will only be for files that are smaller than the amount of RAM and read more than once.

In my opinion 1 GB of RAM will work great but with the price of RAM being so cheap 2 GB is the way to go.

00Roush

 

[ugo1]

Hi

here some other test

iperf -w 32k -c 192.168.0.6 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 32.0 KByte
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.6, TCP port 5001
TCP window size: 32.0 KByte
------------------------------------------------------------
[160] local 192.168.0.10 port 49799 connected with 192.168.0.6 port 5001
[ ID] Interval Transfer Bandwidth
[160] 0.0-10.0 sec 329 MBytes 276 Mbits/sec
[168] local 192.168.0.10 port 5001 connected with 192.168.0.6 port 40685
[ ID] Interval Transfer Bandwidth
[168] 0.0-10.0 sec 109 MBytes 91.5 Mbits/sec


iperf -w 128k -c 192.168.0.6 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.6, TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[156] local 192.168.0.10 port 49800 connected with 192.168.0.6 port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0-10.0 sec 341 MBytes 286 Mbits/sec
[120] local 192.168.0.10 port 5001 connected with 192.168.0.6 port 40686
[ ID] Interval Transfer Bandwidth
[120] 0.0-10.0 sec 110 MBytes 92.0 Mbits/sec

nothing change

On the server and on the client I already have 2GB of ram.

Regards

Ugo

 

[00Roush]

WTF?!

Maybe I will setup a Ubuntu computer here so I can test this. I remember something about getting weird iperf results with Ubuntu but I can't remember for the life of me. Been meaning to setup a Ubuntu computer again anyway. One difference I did notice is that you are running the 64 bit version of Ubuntu Server. Not sure if that might make a difference. I have only used the 32 bit version.

00Roush

 

[ugo1]

New tests nice results

Hi

I worked around Win XP (read re-installed it) and get some nice results
from the client
iperf -w 64k -c 192.168.0.6 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.6, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[1844] local 192.168.0.10 port 1041 connected with 192.168.0.6 port 5001
[ ID] Interval Transfer Bandwidth
[1844] 0.0-10.0 sec 1.00 GBytes 860 Mbits/sec
[1944] local 192.168.0.10 port 5001 connected with 192.168.0.6 port 33544
[ ID] Interval Transfer Bandwidth
[1944] 0.0-10.0 sec 752 MBytes 631 Mbits/sec

I tryed filezilla and all the 3 bigger was uploaded to the server with a tranfer rate over the 70MB/s mostly over 75 MB/s with the first big file always over the the 80MB/s with peak to 92 MB/s

the only problem that I still have are releated to the "smaller" one where still get an average speed of 30 MB/s

Regards

Ugo

 

[00Roush]

Hi

I worked around Win XP (read re-installed it) and get some nice results
from the client
iperf -w 64k -c 192.168.0.6 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.6, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[1844] local 192.168.0.10 port 1041 connected with 192.168.0.6 port 5001
[ ID] Interval Transfer Bandwidth
[1844] 0.0-10.0 sec 1.00 GBytes 860 Mbits/sec
[1944] local 192.168.0.10 port 5001 connected with 192.168.0.6 port 33544
[ ID] Interval Transfer Bandwidth
[1944] 0.0-10.0 sec 752 MBytes 631 Mbits/sec

I tryed filezilla and all the 3 bigger was uploaded to the server with a tranfer rate over the 70MB/s mostly over 75 MB/s with the first big file always over the the 80MB/s with peak to 92 MB/s

the only problem that I still have are releated to the "smaller" one where still get an average speed of 30 MB/s

Regards

Ugo

Now that is what I am talking about!! Glad to see the better results.

Have you tried just using ftp in windows to copy your smaller file and see if you get the same performance? Maybe it could be just filezilla that is causing the slowdown on smaller files. Do you notice with your big files that the speed ramps up? I mean does it start out at say 20 MB/sec then ramps up to the 70 MB/sec? I only ask as it could be the smaller files might not be large enough to really ramp up the speed. Maybe you could test some files in the 500-1000 MB range.

Almost forgot... did you try the different window sizes again to see if they made a difference with iperf?

00Roush