Nas OS Comparison (unRaid, eon, Freenas,...)

[sdesbure]

Hi Guys,
Having to upgrade the size of my NAS, I've decided to quickly review the different OS solutions.
My NAS hardware is the following:

• Intel D945GCLF2 (Atom 330 1.6Ghz Bi Processor Hyperhtreading, i.e. 4 vCPUs -- 2 physical, 2 "virtuals")
• 2G RAM
• OS on USB Disk
• Realtek Gbe NIC
• one Sil3124 card (4 SATAII ports on 1 PCI slot)
• 3 1,5To Samsung disk (two on the board sata ports, one on the Sil3124)

My client is the following:

• Intel Core 2 Quad Q9550
• 4G of RAM
• Asus P5K Motherboard
• Intel 82574L Gbe Nic
• Ubuntu 10.04 OS on disk

I want to test unRaid, EON, Freenas and Ubuntu, all at the latest version (even if beta/alpha)
The test will be only on CIFS.
I've created a 31G file using dd (dd if=/dev/zero of=./zerofile.tst bs=1k count=30000000)
it has taken 344,2s to do it (681 Mb/s rate).
edit: I've created a 10G randomized file using dd too (dd if=/dev/urandom of=./randomfile.tst bs=1k count=10000000)
edit2: I've also created a 10G file filed with zero in order to speed up the tests (dd if=/dev/zero of=./zerofile.tst bs=1k count=10000000)
all the following test will be done using the same behavior.
First mount of the share on the nas with default options (mount -t cifs //NAS_IP@/share /mnt/cifs -o user=sylvain,pass=xxx)
First copy of the file on the nas (time cp zerofile.tst /mnt/cifs)
The copy of file from the nas (time cp /mnt/cifs/zerofile.tst .)
edit: Then do the same thing with the random file

Specific things done on the nas will be described on the different part

unRaid Part
Tests with unRaid 5.0 beta2
parity disk is the one on the sil card
No specific things except that (used the Webpage to administer the nas).

• Zero File
  • Write: 271Mb/s
  • Read: 211Mb/s
• Random File
  • Write: 241Mb/s
  • Read: 213Mb/s

Surprinsingly, reading takes longer time than writing...

E.O.N Part
Tests with eon 0.60

ZFS with Gzip-1 Compression and deduplication
command passed to create the filesystem (viewed on the eon site of course )
zpool create tank raidz c1t0d0 c2d0 c3d0
zfs create -o casesensitivity=mixed -o sharesmb=on tank/media
zfs set sharesmb=name=media tank/media (create a ZFS filesystem = media on zpool = tank, casesensitivity has to be set at creation)
cd /tank ; chmod 755 media (set permission to full control for owner and read, execute for group and others)
groupadd -g 600 media
useradd -u 501 -g 600 -s /bin/bash sylvain
cd /tank ; chown sylvain:media media (set owner = sylvain and group = media on the tank/media dataset)
chmod A+owner@:full_set:fd:allow,everyone@:read_set/execute:fd:allow media (for proper unix permission inheritance on windows created files)
zfs set compress=gzip-1 tank/media (optional: enable gzip-1 compression, a good balance of compression ratio and performance)
zfs set dedup=on tank/media (optional: enable deduplication on dataset tank/media. dedup=on is the same as dedup=sha256,verify)
passwd sylvain (set the passwd for sylvain also so that it will be recognized for CIFS login)
smbadm join -w workgroup

eon:62:/tank/media#zpool status
pool: tank
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c2d0 ONLINE 0 0 0
c3d0 ONLINE 0 0 0

errors: No known data errors

eon:63:/tank/media#zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
tank 4.08T 123K 4.08T 0% 1.00x ONLINE -

• Zero File
  • Write: 307Mb/s
  • Read: 90Mb/s
• Random File
  • Write: 80Mb/s
  • Read: 69Mb/s

Surprinsingly again, reading takes longer time than writing...

The good thing is the compression that make the 30G file virtually empty (of course in real life it won't happen).
ZFS without Compression and deduplication
The command are the same except the two on media which activates compression and deduplication

• Zero File
  • Write: 147Mb/s
  • Read: 92Mb/s
• Random File
  • Write: 150Mb/s
  • Read: 92Mb/s

Freenas Part
I wanted to use Freenas 0.8alpha but never managed to make it boot (issue with mounted file system).
So the tests are done with Freenas 0.7.2.5266 (RC version)
Everything is done via the Web page

UFS

• Zero File
  • Write: 235Mb/s
  • Read: 150Mb/s
• Random File
  • Write: 228Mb/s
  • Read: 147Mb/s

ZFS without compression

• Zero File
  • Write: 176Mb/s
  • Read: 187Mb/s
• Random File
  • Write: 178Mb/s
  • Read: 172Mb/s

ZFS with lzjb compression

• Zero File
  • Write: 9481Mb/s
  • Read: 143Mb/s
• Random File
  • Write: 153Mb/s
  • Read: 183Mb/s

ZFS with gzip-1 compression

• Zero File
  • Write: 10417Mb/s
  • Read: 143Mb/s
• Random File
  • Write: 85Mb/s
  • Read: 180Mb/s

As we can see, compression can significantly increase performance if the file to be compressed can be easily compressed.

ubuntu Part
I've used ubuntu server 10.10 beta installed on an usb key (using ext4 for the system).
the share is install on a software raid5 made with md during installation. On top of this raid5, I've used ext4 or btrfs

ext4

• Zero File
  • Write: 9413Mb/s
  • Read: 216Mb/s
• Random File
  • Write: 295Mb/s
  • Read: 198Mb/s

Btrfs no compression

• Zero File
  • Write: 365Mb/s
  • Read: 198Mb/s
• Random File
  • Write: 303Mb/s
  • Read: 189Mb/s

Btrfs Compression

• Zero File
  • Write: 10557Mb/s
  • Read: 213Mb/s
• Random File
  • Write: 288Mb/s
  • Read: 194Mb/s

I was really surprised with write speed using ext4 (i did several times the test to be sure...).

Conclusion
Heres the numbers:

As you can see, EON is far behind (perhaps Sun's SMB implementation is not good) and the three others are quite close.
My opinion is then the choice of paid os (unRaid) or free os (freenas) knowing that the unique feature of unRaid is very very interesting!

Personaly, I think I'll go with unRaid.

 

[00Roush]

Are your results in MB/sec or Mbps?

Thanks for sharing your results.

00Roush

 

[Hydaral]

I am presuming it's in Mb/s as the OP indicated, although I can't even comprehend how you can get 10557 Mb/s (1302 MB/s) over gigabit ethernet, that's is more than 10 times the THEORETICAL limit. I'm presuming it's over ethernet because you are using a CIFS share, maybe you are using 10Gb.

Although if this testing is all on the local machine you should really add a sync command to the end of the writes so the write is not just cached, but actually written to disk.

 

[smitbret]

Are the files being shared over a wireless network connection? They all seem pretty slow if we are talking Mb/s and not realistic if we are talking MB/s. An unRAID system should run 480-640Mb/s with a modern HDD, it wouldn't really be choked by the Atom CPU. I would think that should be the slowest of the bunch.

Your NAS isn't really built for ZFS (FreeNAS), either. The CPU will probably bottleneck it and you should have about 1GB of ECC memory for each TB of storage. Even then, a properly striped FreeNAS with 3 modern drives should put you above 200MB/s (1600Mb/s) but that would then be bottlenecked by a Gb network connection.

I'm not familiar with E.O.N. and I have never used Ubuntu server.

 

[sfx2000]

The Intel D945GCLF2 was a nice board, but I never got the Realtek GBe nic to be stable... so replaced it with an Intel GBe single port card, problem perhaps solved.. but limited then to onboard SATA/PATA...

The 330 is getting along in years, it's been, what 5 now?

J1800/J1900 is well beyond the Atom 330 in overall performance, no matter what OS is use...

 

[Brutalizer]

Your NAS isn't really built for ZFS (FreeNAS), either. The CPU will probably bottleneck it and you should have about 1GB of ECC memory for each TB of storage.

This is not really correct. ZFS has a really efficient disk cache, so ideally you would have several GB of RAM to utilize the L2ARC disk cache. Ideally you would have at least 4-8GB RAM which gives you a decent sized disk cache. If not, ZFS performance will degrade to disk speed, which is acceptable.

The requirement of 1GB RAM per 1TB disk is when you use ZFS deduplication - which is broken and should be avoided. Do not use ZFS deduplication. But always use compression, as it increases performance. It is faster to read 1000 blocks and pack them up to 2000 blocks in RAM - than to read 2000 blocks from disk.

People have used ZFS servers with 4GB RAM to great effect. I have run a Solaris server with 1GB RAM for a year without problems.

The main reason to use ZFS is not performance, it is to protect your data against data corruption such as bit rot, etc. All common filesystems, NTFS, ext3, etc are unsafe and not designed to protect your data against data corruption. But ZFS is. According to researchers to ZFS is the safest filesystem out there.

 

[Cake]

I suppose someone could just install some Linux distro and install webmin. From there you could install the Samba stuff and setup your shares from the webmin GUI. I have never tried freenas or others, but why have a putty knife when you could have a swiss multi knife.

 

[sfx2000]

This is not really correct. ZFS has a really efficient disk cache, so ideally you would have several GB of RAM to utilize the L2ARC disk cache. Ideally you would have at least 4-8GB RAM which gives you a decent sized disk cache. If not, ZFS performance will degrade to disk speed, which is acceptable.

On Solaris - ZFS is pretty nice - memory hungry, but most folks running solaris in a production system know this and plan accordingly...

I'm not sure I would do ZFS on something other than solaris... as there was a license change/cut-off by Oracle that essentially forked development - so there's Oracle's version, and the open-source version.

ext4 on Linux - still pretty good - btrfs is improving, but I'm not convinced yet that it's production ready, but that hasn't stopped vendors from deploying it.

From a linux perspective - recovering from a crash using mdadm/lvm/ext4 is a known thing, and most sys admins can do this with little effort - and there's many tools that can help with the recovery for the lay person that is technically inclined..

Remember - with any kind of RAID - the failure opportunities are much higher based on the number of disks in the array compared to a single volume on a single disk. The more disks, the more the MTBF works against you...

RAID is not a backup strategy in the longer view - still need to have a backup plan there for data on the array, no matter what filesystem or OS is in use.

Similar things are cloud oriented micro-services - more servers, better the chance that any single server could fail...

 

[SickOfResetting]

Solid info here, although the OP is from 2010.

 

[ddaenen1]

Solid info here, although the OP is from 2010.

Would anybody have an update on this?