What's the speed secret to "Build Your Own Atom-based NAS - Part 2"?

[milo]

Hi All,

This is my first post but I have done a lot of searches, plus testing of various software but am now just stumped, so I must ask..... What is the secret to getting the speeds shown in Build Your Own Atom-based NAS - Part 2?

I was very inspired by the article and went out to purchase some gear. I ended up with the following:
- Intel BOXD945GCLF2 Atom 330 ITX Motherboard/CPU Combo
- Samsung Spinpoint EcoGreen F2 HD103SI 1TB Hard Drive (x2 for RAID1)
- Samsung 1 GB DDR2 PC6400 800MHz Memory - M378T2863QZS-CF7
- Slim DVD/CD-ROM IDE drive
- IN-WIN CASE BM648.AD120TBL MINI ITX MINI TOWER Case

I have used both a IDE to CF Adaptor with a 128MB CF card for OS when using FreeNas and have also used an old 10GB Travelstar 2.5" IDE drive with both FreeNas and Ubuntu.

I think this set-up is pretty close to that in the article listed above but my throughput is MUCH slower. In the start I was using FreeNas i386 0.69.2 with the drivers added for the on-board GBe. At that time I was seeing 5-8 MBytes/s when transferring from my XP Pro system to the Nas via wire GBe. I then jumped to Ubuntu (tried both 8.04 LTS & 9.04 64-bit versions) but had a problem getting the Webmin to work correctly. After setting up the RAID1 via the command line, I got a small increase with Ubuntu to 6-10 MBytes/s. Using the same computer running Vista 64-bit I was at 12-18 MB/s.

Depressed and frustrated with the Webmin problems, I when back to FreeNas and tried "Tuning". Tuning helped a little but didn't do much so I tried jumping to the 0.7.4706 nightly build of FreeNas which got me up to 15-25 MBytes/s. On the 0.7 build tuning doesn't seem to make much of a difference and I also am not seeing too much of a difference between XP pro and Vista 64.

My throughput has gotten a lot better than the start of this project but still not near the 35-45 MB/s write and 55-60 MB/s read speeds seen in the article. Also, my read and write speeds seem to be about the same. Any thoughts on what the problems might be? Things to turn off in bios or to turn on? etc.

Thanks,

Milo

P.S. One last thought, my SATA controller shows up in FreeNas as SATA300 but both my drives are listed as SATA150 even though they are SATA300 drives. Is this normal?

 

[00Roush]

My first thought would be the fact that you are using the onboard network card. In the article that was disabled and a Intel PCI network card was used which is most likely much better supported under FreeNAS and Ubuntu.

I would recommend doing some tests with Iperf to see if your network performance is good. Then move on to file transfer speeds.

On FreeNAS make sure you change the send and receive buffers to 65536 under the CIFS settings.

My last install of Webmin on Ubuntu Server 8.04 ran into some problems as well. Here is how I got it to work... Add deb http://download.webmin.com/download/repository sarge contrib to your sources.list file as recommended in the Atom article. Then execute the following commands:
cd /root
sudo wget http://www.webmin.com/jcameron-key.asc
sudo apt-key add jcameron-key.asc
sudo apt-get update
sudo apt-get install webmin

Hopefully that should get it installed for you. Also I would generally recommend trying the 32 bit Ubuntu Server versions first to limit the chance of problems.

I usually disable serial ports, parallel ports and onboard sound in the BIOS on any computer I plan to use as a server.

If you are looking for the best SMB/SAMBA speeds make sure you are testing with Vista SP1 on the client.

00Roush

 

[milo]

Thanks for the tips. I believe the network card in the article was only a 10/100 on the single-core Atom board. That was one of the main reasons I went with the dual-core Atom instead. I figured a dual core couldn't hurt and I'd rather have GBe built-in. Although there still might be some performance differences between the Intel PCI card vs my Realtek built-in Ethernet. I'm not familiar with Iperf so it looks like I have more reading and testing to do. Thanks for pointing me in the right direction.

As far as buffers, I played with the buffers under the tuning option in System|Advanced|sysctl.conf but never tried (or noticed) those setting under the CIFS setting before. I am a newbie to this kind of stuff so it looks like I have a few more things to try with FreeNas before I give up. I am also a bit surprised at the jump I saw with 0.7 vs 0.69 so maybe there is hope.

Thanks for the tips on Webmin also. I put the 10G drive back into the system today to go back to Ubuntu but haven't reconnected it yet and still have FreeNas running off the CF card for now. I'll see what I can do with FreeNas with the new info and then compare Ubuntu again.

Thanks

 

[roamer]

Hello,

I build a similar configuration after read this nice article about Ubuntu on Atom-platform.

My config:

Intel D945GCLF2 board (with Realtek 8111C onboard)
2x 1TB WD EADS hdd as mirror
2 GB RAM
4GB CompactFlash for Ubuntu
and Ubuntu 8.10 server

The write performance from a PC with Vista (core2duo,4GB RAM) was between 50-60 MB/s with a bigfile (>2,5GB).



- Have you tested the speed with ftp instead of samba ?
( you have to install a ftp-server on the NAs e.g pureFTPd )

- login on your nas and start the 'top' tool during the file transfer
is there any process that is consumpting a lot of CPU or RAM resources?

-Are you disks in the nas are ok - can you locally on the nas copy such big files from one directory to another?
have simply tested speed of a single disk with ubuntu
e.g hdparm -tT /dev/sdc

You should also check on your NAS the logfiles in the directory /var/log for errors. during the filetransfer Especially in the messages files.

 

[thiggins]

Thanks for the tips. I believe the network card in the article was only a 10/100 on the single-core Atom board.

The onboard NIC was disabled and an Intel PRO/1000 PCI used.

 

[milo]

roamer, thanks for the info. It looks like your system really flies.

My test computer is a Asus G50VT-A2 laptop and I have been using files in the 1-1.25 GB and 4-4.5 GB range during testing. It is also a core2duo (2.53Ghz) with 4GB ram (well only 3GB when using XP Pro).

I did not try FTP using Ubuntu but have tried it in FreeNAS. In FreeNAS 0.69 the FTP speeds where a bit faster but now with version 0.7 the Samba and FTP speeds are about the same.

Last night I played with the buffers (and other settings) under CIFS as 00Roush mentioned and was able to get FreeNAS running between 28-35 MB/s. (Thanks 00Roush!) I have checked the logs and nothing seems obviously wrong but then again it just might not be obvious to me.

I haven't tried NAS to NAS file transfers or the other things you mentioned so I'll look into those too.

Is there a reason you chose 8.10 server over 8.04 LTS? Also, are you running 32-bit or 64-bit? I'm about to jump back to Ubuntu tonight.

EDIT: I saw in another post of your that you are running 32-bit, but I still wonder why 8.10 over 8.04 LTS or 9.04?

 

[milo]

The onboard NIC was disabled and an Intel PRO/1000 PCI used.

Hi thiggins,

Sorry if my post came across wrong. I knew the on-board was disabled and a GbE PCI card was used. I was simply trying to point out the differences between the two on-board NICs and the possible / probable reason a PCI card was used instead of correcting the driver for the non-working on-board NIC.

Thanks for the GREAT articles on the Atom based NAS. I had purchased a Linksys WRT610n router in hopes of using it but I never check this site before my purchase and only realized how snail slow it was, afterwards. Now I'm simply using it for GbE and N wireless.

 

[roamer]

Is there a reason you chose 8.10 server over 8.04 LTS? Also, are you running 32-bit or 64-bit? I'm about to jump back to Ubuntu tonight.

EDIT: I saw in another post of your that you are running 32-bit, but I still wonder why 8.10 over 8.04 LTS or 9.04?

I prefer 32bit
- because less then 4GB RAM is used
- there were some trouble in past with Realtek NIC and 64bit e.g here, I want to bypass such problems

Why not 8.04 -> It's older then 8.10
Why not 9.04 -> It was not released
Why not still upgraded -> don't toch a running system

 

[milo]

Thanks for the info, once again.

I still have FreeNAS running for now and just tried the NAS to NAS transfer. I have to say that I'm really confused with the results. I started by making two directories on the NAS drive via windows. I then transferred a 4.35GB file from my computer into one of the folder in 172 seconds (about 25 MB/s). I then tried using windows to copy the file from one folder to another on the NAS and it took forever! Five minutes into the transfer I was only half way through so I canceled the copy. I then sshed into the NAS and tried again with a cp command. This time it took 235 second or about 18MB/s. Why would the NAS to NAS take longer than the computer to NAS transfer. Is it due to my Raid 1 set-up? Basically reading and writing at the same time. Even if this is it, I figure the drive would work faster than the network(?). I'm confused and done for the night.

I think I'm going to give Ubuntu 9.04 32-bit server a try tomorrow. Thanks again.

 

[roamer]

Thanks for the info, once again.

I still have FreeNAS running for now and just tried the NAS to NAS transfer. I have to say that I'm really confused with the results. I started by making two directories on the NAS drive via windows. I then transferred a 4.35GB file from my computer into one of the folder in 172 seconds (about 25 MB/s). I then tried using windows to copy the file from one folder to another on the NAS and it took forever! Five minutes into the transfer I was only half way through so I canceled the copy. I then sshed into the NAS and tried again with a cp command. This time it took 235 second or about 18MB/s. Why would the NAS to NAS take longer than the computer to NAS transfer. Is it due to my Raid 1 set-up? Basically reading and writing at the same time. Even if this is it, I figure the drive would work faster than the network(?). I'm confused and done for the night.

I think I'm going to give Ubuntu 9.04 32-bit server a try tomorrow. Thanks again.

Give Ubuntu a try, if you have read my previous posts in another thread then you know that I have also not reached more the 25MB/s with FreeNAS.

--------------
To copy a file via windows client from one directory on the NAS to another directory on the NAS is not a good idea.
That means your are transfering the file from NAS -> PC -> NAS and for shure the LAN connection is busy in both directions.


My suggestion was
- login into the NAS via ssh with e.g. putty and use the 'cp' command to copy a file locally from one disk to another. In this situation the LAN is not involved and it can't be the bottleneck. This steps should be executed without raid configuration from oone single disk to another, if you have still low throughput, you should check yor HW e.g. SATA cable.

If everything works fine locally
- install the ftp-server on the NAS
Transfer files from your PC to the NAS via ftp. with e.g. winscp
With this step the LAN is involved, but not SAMBA.
The troughput should be even higher then with smb/cifs shares, because the ftp protocoll has not so much overhead.
If you have still low throughput, you should check yor HW e.g. LAN-cable.
And it's not wrong to login into the NAS via ssh and start the 'top' tool on the NAS. There you can see which processes consumes CPU-load.

 

[00Roush]

Thanks for the tips. I believe the network card in the article was only a 10/100 on the single-core Atom board. That was one of the main reasons I went with the dual-core Atom instead. I figured a dual core couldn't hurt and I'd rather have GBe built-in. Although there still might be some performance differences between the Intel PCI card vs my Realtek built-in Ethernet. I'm not familiar with Iperf so it looks like I have more reading and testing to do. Thanks for pointing me in the right direction.

I wanted to get back to you about Iperf... http://en.wikipedia.org/wiki/Iperf It basically is used to measure the best case TCP/IP throughput your network is capable of. This can help you to find out if your network or network card is a bottleneck. This thread might help you figure out how to use it... http://forums.smallnetbuilder.com/showthread.php?t=1118 Now a program like HDTach (windows) lets you know what your best case disk throughput is. For Ubuntu you could use hdparm -t /dev/sda to give you an idea of disk throughput as well. Testing with these programs is usually where I would start when trying to track down where a bottleneck might be.

00Roush

 

[milo]

00Roush, Thanks for the detailed information. After your first post I googled Iperf and found the Wikipedia page but wasn't ready to try that yet. Last night I reinstalled Ubuntu and used 9.04 32-bit server this time. Using your info, I was able to get webmin installed (with the cert) and it seems to work better this time. At this point I only have Samba and Ssh running and have the two hard drives partitioned independently (not in Raid). My transfer rates from my computer (XP pro) to NAS are still in the 28-30 MB/s range. I then tested a drive to drive copy on the NAS using ssh and got 60 MB/s. To me this still seems very low for a drive to drive copy with these drives.

I will look into using the hdparm tool and see if I can figure out what's happening. I have ran the "top" command as suggested but did not find anything hogging resources. So I think it might have something to do with the drive setup. I still wonder why the SATA300 controller is located but the drives are setup/showing SATA150 when they are really SATA300 drives. Or could it simply be the fact that I only have 1GB of ram rather than 2GB.(?) I was even thinking out trying an install of windows xp or windows 7 RC just to test the transfer speeds with that setup.

Thank you all for the help! As I dig further and learn more, I'll make sure to give updates.

 

[milo]

UPDATE: OK so I have done some more testing on Ubuntu and am now confused. Using hdparm I ran the following commands and got the following results:

> hdparm /dev/sdb
/dev/sdb:
IO_support = 0 (default)
readonly = 0 (off)
readahead = 256 (on)
geometry = 56065/255/63, sectors = 1953525168, start = 0

> hdparm -Tt /dev/sdb
/dev/sdb:
Timing cached reads: 1208 MB in 2.00 seconds = 604.10 MB/sec
Timing buffered disk reads: 322 MB in 3.02 seconds = 106.76 MB/sec

> hdparm /dev/sdc
/dev/sdc:
IO_support = 0 (default)
readonly = 0 (off)
readahead = 256 (on)
geometry = 56065/255/63, sectors = 1953525168, start = 0

> hdparm -Tt /dev/sdc
/dev/sdc:
Timing cached reads: 1188 MB in 2.00 seconds = 594.18 MB/sec
Timing buffered disk reads: 322 MB in 3.02 seconds = 106.77 MB/sec

If I read this correctly, I have getting 106 MB/s reads on both drives. If this is correct I don't understand why my drive to drive copies are only at 60 MB/s. I also tried to change the IO-support to 32-bit mode but got an error.

Is there something I'm missing?

Thanks

Milo

 

[roamer]

UPDATE: OK so I have done some more testing on Ubuntu and am now confused. Using hdparm I ran the following commands and got the following results:

> hdparm /dev/sdb
/dev/sdb:
IO_support = 0 (default)
readonly = 0 (off)
readahead = 256 (on)
geometry = 56065/255/63, sectors = 1953525168, start = 0

> hdparm -Tt /dev/sdb
/dev/sdb:
Timing cached reads: 1208 MB in 2.00 seconds = 604.10 MB/sec
Timing buffered disk reads: 322 MB in 3.02 seconds = 106.76 MB/sec

> hdparm /dev/sdc
/dev/sdc:
IO_support = 0 (default)
readonly = 0 (off)
readahead = 256 (on)
geometry = 56065/255/63, sectors = 1953525168, start = 0

> hdparm -Tt /dev/sdc
/dev/sdc:
Timing cached reads: 1188 MB in 2.00 seconds = 594.18 MB/sec
Timing buffered disk reads: 322 MB in 3.02 seconds = 106.77 MB/sec

If I read this correctly, I have getting 106 MB/s reads on both drives. If this is correct I don't understand why my drive to drive copies are only at 60 MB/s. I also tried to change the IO-support to 32-bit mode but got an error.

Is there something I'm missing?

Thanks

Milo

read is always faster then write...

your values quite normal and 60MB/s write is not wrong for a single disk.

 

[milo]

read is always faster then write...

your values quite normal and 60MB/s write is not wrong for a single disk.

Thanks for the info and confirming these values look OK. My network read and writes are at the same speed and about half the disk to disk speed. Therefore I assume the problem must be something to do with the network.

Time to jump into Iperf and see what I can find.

Thanks again for everyone's help. I thought I knew a bit about this stuff BUT I sure feel like a newbie trying to get this system going. I guess my bit of networking and Linux knowledge is just enough to get me in trouble.

 

[roamer]

just install 'pure-ftpd' on your nas, before you starting with creation of raid...
Check the throughput via ftp from PC to your nas with a single disk.
This step takes only 2minutes and you know what's on.

 

[00Roush]

read is always faster then write...

your values quite normal and 60MB/s write is not wrong for a single disk.

I tend to disagree with that. Write speeds should be very close to read speeds if those disks are empty. Now with journaling file systems like ext3 write speeds tend to be a little lower but not 40 MB/sec slower.

Milo, what program did you use to do your copy between disks? Also what file system are you using?

00Roush

 

[milo]

I tend to disagree with that. Write speeds should be very close to read speeds if those disks are empty. Now with journaling file systems like ext3 write speeds tend to be a little lower but not 40 MB/sec slower.

Milo, what program did you use to do your copy between disks? Also what file system are you using?

00Roush

I am currently running Ubuntu 9.04 32-bit server with the ext3 file system on all the drives (OS Disk 10GB, Disk1 1TB, Disk2 1TB). Both the storage (1TB) drives are basically empty other than one or two ~4GB files.

I used Putty from my XP laptop to ssh into the server and then ran the following command for the drive to drive copy; "cp /mnt/Disk1/test.mpg /mnt/Disk2/test.mpg". This file was 4.35GB in size and resulted in about 60MB/s from the time I hit enter to the time the cursor reappeared on the screen (~72sec).

Tonight a ran several tests with Iperf and Jperf. I set up Iperf on the server running as a server and used Jperf on my laptop. I ran all my tests for 10 seconds, reporting every second with MByte output using port 5001. Using the default TCP setting (nothing checked) I got the following:

bin/iperf.exe -c 192.168.1.104 -P 1 -i 1 -p 5001 -f M -t 10
------------------------------------------------------------
Client connecting to 192.168.1.104, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[1912] local 192.168.1.103 port 1061 connected with 192.168.1.104 port 5001
[ ID] Interval Transfer Bandwidth
[1912] 0.0- 1.0 sec 44.3 MBytes 44.3 MBytes/sec
[1912] 1.0- 2.0 sec 37.3 MBytes 37.3 MBytes/sec
[1912] 2.0- 3.0 sec 45.8 MBytes 45.8 MBytes/sec
[1912] 3.0- 4.0 sec 46.6 MBytes 46.6 MBytes/sec
[1912] 4.0- 5.0 sec 47.1 MBytes 47.1 MBytes/sec
[1912] 5.0- 6.0 sec 47.2 MBytes 47.2 MBytes/sec
[1912] 6.0- 7.0 sec 47.4 MBytes 47.4 MBytes/sec
[1912] 7.0- 8.0 sec 45.1 MBytes 45.1 MBytes/sec
[1912] 8.0- 9.0 sec 46.6 MBytes 46.6 MBytes/sec
[1912] 9.0-10.0 sec 46.9 MBytes 46.9 MBytes/sec
[1912] 0.0-10.0 sec 454 MBytes 45.3 MBytes/sec
Done.

I was able to tweak my "Buffer Length" and "TCP Window Size" setting to greatly improve my throughput. The best combination (after many rounds of testing) was using 64 KBytes for Buffer Length and 32 KBytes for TCP Window Size, which gave me the following results:

bin/iperf.exe -c 192.168.1.104 -P 1 -i 1 -p 5001 -w 32.0K -l 64.0K -f M -t 10
------------------------------------------------------------
Client connecting to 192.168.1.104, TCP port 5001
TCP window size: 0.03 MByte
------------------------------------------------------------
[1912] local 192.168.1.103 port 1064 connected with 192.168.1.104 port 5001
[ ID] Interval Transfer Bandwidth
[1912] 0.0- 1.0 sec 108 MBytes 108 MBytes/sec
[1912] 1.0- 2.0 sec 107 MBytes 107 MBytes/sec
[1912] 2.0- 3.0 sec 108 MBytes 108 MBytes/sec
[1912] 3.0- 4.0 sec 109 MBytes 109 MBytes/sec
[1912] 4.0- 5.0 sec 105 MBytes 105 MBytes/sec
[1912] 5.0- 6.0 sec 109 MBytes 109 MBytes/sec
[1912] 6.0- 7.0 sec 109 MBytes 109 MBytes/sec
[1912] 7.0- 8.0 sec 109 MBytes 109 MBytes/sec
[1912] 8.0- 9.0 sec 108 MBytes 108 MBytes/sec
[1912] 9.0-10.0 sec 108 MBytes 108 MBytes/sec
[1912] 0.0-10.0 sec 1080 MBytes 108 MBytes/sec
Done.

My problem is that I have tried to use regedit to make these values my defaults but can't seem to get them to stick. I tried setting TcpWindowSize, DefaultReceiveWindow, DefaultSendWindow in the registry but they didn't seem to change anything.

I also tried to install FTP and test but haven't been able to get all the permissions sorted out to allow me to write to my 1TB drives.

Thanks for the help,

Milo

 

[00Roush]

I am currently running Ubuntu 9.04 32-bit server with the ext3 file system on all the drives (OS Disk 10GB, Disk1 1TB, Disk2 1TB). Both the storage (1TB) drives are basically empty other than one or two ~4GB files....

You shouldn't have to make any changes to the registry... The window size is auto tuned in Windows and to my knowledge also in Ubuntu. You should only have to set buffer size or window size in your tests. I generally use this command line iperf -w 64k -c 192.168.0.2 -r as the results in Mbps make it easier for me to see the network speed. Also this will test both directions.

00Roush

 

[milo]

I was going by this article when trying to adjust the setting and edit the registry, as the article talks about editing the registry for the TCP Window Size. TCP Windows Size made a bit of difference in my reading (from 45 to 65 or so) but the real impact was adjusting the Buffer Length. I then checked this MS regedit guide but wasn't able to figure out exactly which parameters corresponded to Buffer Length in Jperf.

I will run a few more tests tonight with the Mbps output format and I'm also going to try and get FTP working.

My hope really was lying with the tuning of either the hard drives or networking in Ubuntu. When I had FreeNAS tuned, I was getting the same transfer rates between FTP & Samba. Those transfer rates were also very close to what I am currently getting with Ubuntu.