Software RAID 5 speed issues

[merlinn31]

Maybe you guys can help me out.

I initially started with Ubuntu 10.04 (Spiderman) on a hacked ss4200 (Celeron 420 w/ 512 RAM) with 2x1TB WD Green EARS and 2x1TB WD GREEN EADS. I set up a software RAID 1 across the boot partition on 2 drives, and a software RAID 5 (64k chunk) across all drives. All filesystems are ext4.

I've upgraded to a DIY solution with Ubuntu 10.10 (Beast) and an Atom D510 1.66ghz w/ 4GB RAM, 5x2TB Samsung F4 drives. I put an old laptop drive in and the OS is installed to it. I've configured a software RAID 5 (64k chunk) across all 5 drives. It's finished syncing, and all file systems are ext4.

What I'm noticing is that Beast seems to be performing significantly worse on writes than Spiderman:

root@Spiderman:~# dd if=/dev/zero of=/media/checkit.file
6206255+0 records in
6206254+0 records out
3177602048 bytes (3.2 GB) copied, 40.4401 s, 78.6 MB/s

root@Beast:/storage# dd if=/dev/zero of=/storage/checkit.file
3044046+0 records in
3044046+0 records out
1558551552 bytes (1.6 GB) copied, 39.8188 s, 39.1 MB/s

I'm ruling out the processor as I think the Atom d510 far outperforms the Celeron 420, and more RAM is better than less RAM. The only other thing I can think of is the drives, but I've read reports that the F4s perform almost as well as 7200rpm drives of the same size.

Initial SMART disk checks are okay, and I'm running extended self diagnostics on them now.

What do you guys think?

 

[thiggins]

Why is Beast's checkit.file half the size of Spiderman's?

Don't assume Atom is better than Celeron. Have you checked CPU usage during the copy?

Memory probably won't make much difference if you are moving large sequential files.

 

[merlinn31]

It's half the size because it's writing at half the speed

Check the length of time I let dd run -- approximately 40 seconds on both machines.

This Celeron is much older and single core. The Atom has a faster clock speed, more cores, and same cache. Why would the Atom not be faster than the Celeron?

 

[thiggins]

Atom is designed for low cost and low power consumption. Cores don't tell the whole story. But I could be wrong. Check the CPU usage during your test.

 

[merlinn31]

My logs are also full of messages like this, every once and a while. Maybe a correlation. I can consistently make it happen by making and remaking an ext4 file system. It'll hang for a few seconds, then these appear in /var/messages:

http://article.gmane.org/gmane.linux.utilities.smartmontools/7428

I've patched all the F4 drives, so I don't know what's going on here. Anyone else have the Samsung F4 drives and messages like the ones in the link?

 

[merlinn31]

So I aligned the 4k boundary by starting each raid partition on sector 56. This fixed the logs in messages. So I did some further tests. For whatever reason, dd without any options still writes about 1/2 as fast on Beast than Spiderman. As soon as I add a bs with some counts, the speeds behave like normal. Here's what I found:

root@Spiderman:~# dd if=/dev/zero of=/media/bigfile bs=1024K count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 63.8444 s, 82.1 MB/s

root@Beast:/storage# dd if=/dev/zero of=/storage/bigfile bs=1024K count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 34.796 s, 151 MB/s




root@Spiderman:~# dd if=/dev/zero of=/media/bigfile bs=1024 count=5000
5000+0 records in
5000+0 records out
5120000 bytes (5.1 MB) copied, 0.583233 s, 8.8 MB/s

root@Beast:/storage# dd if=/dev/zero of=/storage/bigfile bs=1024 count=5000
5000+0 records in
5000+0 records out
5120000 bytes (5.1 MB) copied, 0.0997241 s, 51.3 MB/s




root@Spiderman:~# dd if=/dev/zero of=/media/bigfile
4247913+0 records in
4247913+0 records out
2174931456 bytes (2.2 GB) copied, 27.07 s, 80.3 MB/s

root@Beast:/storage# dd if=/dev/zero of=/storage/bigfile
2186236+0 records in
2186235+0 records out
1119352320 bytes (1.1 GB) copied, 28.4251 s, 39.4 MB/s


I'm going to try some iozone and bonnie++ tests and see what I come up with.

 

[thiggins]

Why don't you just try some good ol fashioned file copy tests?

 

[merlinn31]

Thank you Tim for the constant helpful input while I struggle through this. I did some Samba read/write tests, rsync read/write tests, and some iozone tests.

Samba read writes with a 5 gig file, gigabit ethernet, hooked straight to the switch, with options "socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65535 SO_SNDBUF=65535":

Beast: Read=~53MB/s, Write=~51MB/s
Spiderman: Read=~45MB/s, Write=~62MB/s

I changed the chunksize on the RAID 5 on Beast to 128 at the last minute, so I wonder if this is why reads are faster but writes are slower when compared to Spiderman (chunk size 64).

Rsync between the machines both ways result in approximately 19-22MB/s, which isn't good news when I decide it's time to migrate all my data. Not sure why this is so much slower than Samba.

iozone results were pretty remarkable, and much more consistent with what I expected to see. The test always finished much faster on Beast than on Spiderman.

iozone -R -l 5 -u 5 -r 4k -s 100m -F /storage/f1 /storage/f2 /storage/f3 /storage/f4 /storage/f5 | tee -a /tmp/iozone_results.txt &

Spiderman:
" Initial write " 108292.28
" Rewrite " 109015.82
" Read " 146325.08
" Re-read " 2107213.84
" Reverse Read " 1712078.26
" Stride read " 1844392.61
" Random read " 1838471.96
" Mixed workload " 979890.22
" Random write " 4660.33
" Pwrite " 144616.44
" Pread " 252777.69

Beast:
" Initial write " 502539.34
" Rewrite " 832838.78
" Read " 2770575.31
" Re-read " 2784939.28
" Reverse Read " 2387685.95
" Stride read " 2295398.25
" Random read " 2440343.97
" Mixed workload " 1688672.44
" Random write " 881467.16
" Pwrite " 528180.89
" Pread " 2123829.34

I'll keep tweaking settings, but either way, I think these tests indicate that my new DIY isn't that much slower (or slower at all) than the hacked ss4200. Before I formatted my laptop and switched to the new drive, I could have sworn I was getting ~75-80MB/s reads from Spiderman, so this loss in speed may be due to the differences in laptop hard drive.

 

[thiggins]

I'll keep tweaking settings, but either way, I think these tests indicate that my new DIY isn't that much slower (or slower at all) than the hacked ss4200. Before I formatted my laptop and switched to the new drive, I could have sworn I was getting ~75-80MB/s reads from Spiderman, so this loss in speed may be due to the differences in laptop hard drive.

I'd put most of my faith in the Samba tests, since they most reflect real-world.

With iozone, you need to test with a file size > the RAM in the target system or you'll get mostly cached performance on write. Using the -c switch will provide a more realistic view of performance.

D510-based commercial NASes in the charts for comparison are:
Synology DS1010+, DS411+
QNAP TS-459 Pro, TS-559 Pro
Thecus N4200
Cisco NSS322, 324, 326

I'd also not use jumbo frames if you are using them.