Ubuntu 9.10 with ext4 filesystem on Linux RAID5, superb performance!

[sammynl]

Hi,

Just wanted to share with the Ubuntu DIY folks, i did a clean install of the new Ubuntu Server 64-bit release, version 9.10 and formatted the Linux RAID5 array (6*1TB and 256K chunksize) with the new, now native ext4 linux filesystem (mkfs -t ext4 /dev/md0). I heard ext4 would be faster and it is!

Reading 10 GB file from NAS: speed at least 108 MB/sec or more.

Writng 4 GB file to NAS: speed at least 82MB/sec or more.

Writing 10 GB file to NAS: speed at least 70MB/sec or more.

Compared to previous tests with Ubuntu 9.04 - ext3, performance is abt. 10-15% better with Ubuntu 9.10 - ext4 and there is no significant speeddrop when working with very large files like 4 -10GB HD media stuff. I think Ubuntu Server 9.10 with Samba and Linux RAID makes the perfect NAS.


Root and RAID array with ext4:

NAS hardware:

• Intel Celeron Dual Core 2 Ghz.
• Intel DG965OT microATX motherboard. (6 o/b SATA ports)
• 6 * Samsung Spinpoint F1 HD103UJ - 1TB.
• 4Gb DDR2 667 Mhz RAM.
• Promise TX 2300 controller for 2.5" 160GB OS harddrive.
• Antec NSK6580 casing (with 6 * 3.5" drive slots).

 

[erezsolan]

Very nice. BTW, how did you test speed? I can see that was a regular SMB copy, but I guess you didn't copy the file from a single drive machine into the NAS. Am I correct?

 

[sammynl]

Very nice. BTW, how did you test speed? I can see that was a regular SMB copy, but I guess you didn't copy the file from a single drive machine into the NAS. Am I correct?

Hi, that's correct the client is a Vista X64 SP2 machine with network mappings to samba shares running on a 4-disk RAID0 array (Samsung HD501LJ disks) hence client performance is always better than the RAID5 NAS performance. Test results are the maximum the NAS can do if the client is not being the bottleneck so i think these results are fairly correct just to see what the NAS can do as Samba server.

 

[zhamrock]

Hi Sammy - I built similar setup using ubuntu 9.10 64 bit, however, im getting only max 19mb/s from a vista client

Network - Tplink unmanaged gigabit switch with supports for jumbo frames

Client Vista 32 SP1
-Interface has 4kb enabled jumbo frames
-1 gbps full duplex

Here's my Mini ITX server spec:
- AMD 3250e
- Minix 785g
- 2GB so-dimm ram
- 3 * 1.TB raid 5 - ext4 FS.
- Sandisk CF for OS

Raid5 was formatted with EXT4 as well(mkfs -t ext4 /dev/md0)
-interface MTU=9000

Though the raid is still in rebuilding state, i was wondering if you could share your client and server's config to achieve max performance.

Regards'
zham

 

[msl]

I would wait for the array rebuild to complete and then check performance.

 

[kroken]

awsome!

cant wait to get my 4 f2 disk á 1 tb and get started with raid!!!

 

[bobcov]

How did you do this?

Hi, that's correct the client is a Vista X64 SP2 machine with network mappings to samba shares running on a 4-disk RAID0 array (Samsung HD501LJ disks) hence client performance is always better than the RAID5 NAS performance. Test results are the maximum the NAS can do if the client is not being the bottleneck so i think these results are fairly correct just to see what the NAS can do as Samba server.

What network card are you using? Do you think I would need to manually tweak a QNAP 639-PRO to get anywhere close to your results in a 32 bit environment?

 

[zhamrock]

Raid 5 rebuild is complete, however, im still getting 20mb/s max samba copy. sata to sata copy is slow as well, 35 mb/s using rsync from one disk to raid5.

I have spent hours in google to improve the performance, including tweaking bios to use AHCI from native IDE, still no joy

Any help would be appreciated.

 

[zhamrock]

I managed to increased the SMB copy but still far beyond from 70-90mb/s by:
- tweaking samba and mount options
- Upgrade kernel to latest version 2.6.31-16-server

fstab:
/dev/md0 /mnt/data ext4 defaults,noatime,norelatime,errors=remount-ro 0 1

00Roush program
----------------
Source disk c:\
Destination disk z:\
System disk C:\
Source drive is DRIVE_FIXED
Destination drive is DRIVE_REMOTE
File copied at 45.044 MB/sec

Fastcopy
-------------
TotalRead = 4848.0 MB
TotalWrite = 4840.0 MB
TotalFiles = 0 (1)
TotalTime= 105.97 sec
TransRate= 45.67 MB/s
FileRate = 0.00 files/s

I will try to change the client's cable this weekend to see if this increase the speed.

 

[00Roush]

zhamrock-

What is your setup on the client? Wondering which CPU, HD, and network card you have as they are the main components that will determine performance.

To minimize the possible causes of your slow speed I would start with a basic setup. So no RAID, no SAMBA tweaks, and no jumbo frames. Just setup and format a single drive to use as a Samba share. Then do some testing and see how the performance is.

Other options would be to test your network throughput using Iperf or test your RAID 5 disk array locally with DD, Bonnie, or Iozone.

Have you gotten a chance to test read speeds from the server?

00Roush

 

[zhamrock]

Thanks for the reply.

zhamrock-

What is your setup on the client? Wondering which CPU, HD, and network card you have as they are the main components that will determine performance.

-Windows Vista 32bit SP2
- AMD Athlon 64 4800+
- Maxtor 80GB HD IDE
- Onboard NIC - 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)

To minimize the possible causes of your slow speed I would start with a basic setup. So no RAID, no SAMBA tweaks, and no jumbo frames. Just setup and format a single drive to use as a Samba share. Then do some testing and see how the performance is.

Other options would be to test your network throughput using Iperf or test your RAID 5 disk array locally with DD, Bonnie, or Iozone.

Have you gotten a chance to test read speeds from the server?

00Roush

I will do the above this weekend and get back to you.

 

[00Roush]

Well right off the bat I can tell you your hard drive in the client is probably limiting your performance. If you get a chance run HD Tach on your client drive and see what your numbers are. My guess will be somewhere in the 50-60 MB/sec range.

Now sammynl ensured his client drive(s) would not be a bottleneck by using a 4 drive RAID 0 array in the client. The thing you have to remember is, when transferring files the slowest component will determine performance. For example if you only had a 100 Mbit network your maximum transfer speed would be limited by the speed of the network. Same goes for hard drives. If your drive in the client or server can only read/write at 50 MB/sec the maximum possible speed you can transfer files is 50 MB/sec. This is assuming no caching it taking place.

Let us know how it goes.

00Roush

 

[zhamrock]

It seems that the bottleneck is on the client side - Thanks a lot 00Rush!

Windows Vista Client - HD Tune
-------------
I only get around 50-55MB/s

Here's the Iperf, Bonnie++ and DD test outputs, i cant tell if these are normal result for my setup.

Ubuntu NAS server
-----------------------
$ iperf -c 192.168.1.9
------------------------------------------------------------
Client connecting to 192.168.1.9, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.11 port 57048 connected with 192.168.1.9 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 140 MBytes 117 Mbits/sec

Using 15gb file
--------------------
$ iperf -c 192.168.1.9 -F Ubuntu\ 64-bit\ \(experimental\)-flat.vmdk
------------------------------------------------------------
Client connecting to 192.168.1.9, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.11 port 57049 connected with 192.168.1.9 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 126 MBytes 105 Mbits/sec

Windows Client
----------------
c:\>iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
OpenSCManager failed - Access is denied. (0x5)
[140] local 192.168.1.9 port 5001 connected with 192.168.1.11 port 57048
[ ID] Interval Transfer Bandwidth
[140] 0.0-10.1 sec 140 MBytes 116 Mbits/sec
[144] local 192.168.1.9 port 5001 connected with 192.168.1.11 port 57049
[ ID] Interval Transfer Bandwidth
[144] 0.0-10.0 sec 126 MBytes 105 Mbits/sec

On NAS Server:

Bonnie++ - looks like i got the same result for read/write on my RAID 5
--------------
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
bubuyog 3504M 38805 97 98267 39 49765 18 40039 95 160321 23 444.1 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 17541 74 +++++ +++ 29118 99 18957 100 +++++ +++ 24881 98
bubuyog,3504M,38805,97,98267,39,49765,18,40039,95,160321,23,444.1,1,16,17541,74, +++++,+++,29118,99,18957,100,+++++,+++,24881,98

DD
------------
$ dd if=/dev/zero of=10gbfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 127.242 s, 84.4 MB/s

Regards'

-zham

 

[00Roush]

Looks like you found your bottleneck.

Man those Bonnie results are hard to make sense of. The DD results are probably about right for your setup. 84 MB/sec for writes is pretty good with RAID 5.

As for the iperf results... Since you are testing maximum network throughput you would want to see results close to 1000 Mbps. Usually I see around 930 Mbps on my network at home. Also to test for maximum performance the window size needs to be bigger than what Iperf uses as the default. I usually use a command line like this... iperf -w 64k -c 192.168.0.2 -r.

Glad I could help.

00Roush

 

[nbat58]

Hi Guys,

Just some advice please, I have just set up my NAS drive and haven't done any tweeking at all but wondered, could I improve my speeds somehow, I am getting around 85mbps transfering files from a 500gb samsung f3 drive on Windows 7 64 bit pc to the NAS samsung f2 1.5tb.

I have noticed that my ram speeds never get more than 28% and my cpu speed no higher than 14%, is this somehow related to speed and can I do any tweeking to FreeNAS settings to make it any quicker.

I have also noticed although on a gigabyte switch (HP Procurve) the RJ45 nic connection on the NAS drive shows as orange solid light opposed to the usual green for connection and flashing amber for data, is there anything I can do to improve this?

Oh forgot to say that FreeNAS server is set up as ZFS RaidZ.

Many thanks for all your help.

 

[00Roush]

Hi Guys,

Just some advice please, I have just set up my NAS drive and haven't done any tweeking at all but wondered, could I improve my speeds somehow, I am getting around 85mbps transfering files from a 500gb samsung f3 drive on Windows 7 64 bit pc to the NAS samsung f2 1.5tb.

I have noticed that my ram speeds never get more than 28% and my cpu speed no higher than 14%, is this somehow related to speed and can I do any tweeking to FreeNAS settings to make it any quicker.

I have also noticed although on a gigabyte switch (HP Procurve) the RJ45 nic connection on the NAS drive shows as orange solid light opposed to the usual green for connection and flashing amber for data, is there anything I can do to improve this?

Oh forgot to say that FreeNAS server is set up as ZFS RaidZ.

Many thanks for all your help.

First thing I recommend is to take a look here. Also an orange light probably means for some reason that connection is at 100 Mbps instead of 1000 Mbps. I would be happy to try and help more... But could you start another thread? I don't want to get too far off topic with this thread as it has more to do with Ubuntu, ext4, and RAID5.

00Roush

 

[nbat58]

Many thanks for your help, sorry to divert from topic , I will start another thread.

 

[ugo1]

Hi,

Just two quick questions.
1- Have you tried to simulate an hard drive fail and rebuild it? How much time it takes?
2- And what about file corruption on large files bug?

Regards

Ugo

 

[msl]

2- And what about file corruption on large files bug?

I am using Ubuntu 9.10 64-bit server with ext4 on a RAID10 array (not RAID5). I tried to replicate the large file corruption bug by copying large files to/from the ext4 partition and comparing checksums before entrusting data to it. All the checksums were fine and I couldn't reproduce any problem on my system.

 

[kroken]

what results will hdparm -Tt on your raid array give you?