ZFS Testing

[corndog]

I keep reading everywhere that ZFS is the up-and-coming panacea of storage that will totally "change the game" and that we'll all switch to it and wonder how we ever lived without it.

Sounds nice, right?

So I'm thinking about doing some torture testing on ZFS to see how well it stands up to the kind of abuse I put on my storage.

The two most demanding things on my storage systems are my gentoo portage tree and my scheduled backups. Interestingly, they are about as exactly opposite as you could possibly get.

My backups are basically 7 or 8 jobs that collect large chunks of data files of all sizes (anywhere from 40G up to 300G total job size) and compress them into large archive files, storing them on a staging area on my QNAP. From there I bulk-copy the large archive files onto a portable NAS to take off-site.

My gentoo portage tree is totally different. This is the database of installable software available for a gentoo system. It is periodically synchronized from gentoo linux servers to most active running gentoo systems. In my mind it is a bloated nightmare that would be better housed in a database, but that's just me. It is basically about 200MB total size and that 200MB contains over 110,000 files that are so small they can take up to 1.2G of space on disk depending on how inefficient the filesystem is with small files. Every time I install a piece of software on a Gentoo system this portage tree gets scanned for settings and dependencies, which puts a big hit on the cache, and the CPU of the file server. Because the files are so small and so many of them are read, performance is way lower than you'd expect for the total amount of bytes of data read. When this tree of files is synchronized from the Gentoo project's distribution points, the usage is even higher, as rsync works through every folder and file.

I'm looking to start putting my backup staging and also a copy of my gentoo portage tree on ZFS on a NAS and then put them through regular use, to see how the system behaves. I'll post my experience with this.

 

[corndog]

Just as a backgrounder, I set up Gentoo Linux (because zfs-fuse has a nice easy install on gentoo) on a Norco DS-520 D.I.Y. NAS system. This is basically a 1GHz Celeron M Server with 1Gig of RAM, 5 hot-swap SATA bays, and a CF slot for your Custom NAS OS. It is summarized here, but don't let that engadget summary mislead you - this little box is completely lame compared with a TS-509Pro and a ReadyNAS Pro, and it has barely enough power to get ZFS to run at all. However, just to set some expectations, here are some numbers. I put two 140G Raptors in the Norco and set them up with Linux RAID-0. Then I used the dd command to put a 30Gig file full of zeroes on the array. After this I rebooted the NAS, and then used dd to read the 30gig file off the array. I repeated this for ext3, reiserfs, xfs, zfs-fuse, and ntfs-3g. I included ntfs-3g because it also uses the fuse architecture, and I wanted a comparative benchmark to see if I should blame ZFS or FUSE for performance challenges. Here are my numbers, in MB/sec:

Filesys Read Write
------- ---- -----
ext3 88.2 95.7
reiserfs 42.8 75.7
xfs 92.5 91.8
zfs 46.7 12.7 (zfs on Linux software RAID)
zfs 50.6 12.4 (zfs on zpool with stripe)
ntfs-3g 42.4 8.1

So, you can see there is a sizeable write overhead with zfs-fuse. However, this could be attributed to the checksum operation that zfs does on all writes. The 1GHz Celeron M is probably just a wee bit light for this kind of work. (ZFS sites all recommend powerful 64bit CPUs and minimum 2G of RAM, which the Norco does not have) In addition, there is a "big-writes" patch for ZFS that increases the buffer size for writes so there are fewer context switches between usermode and kernel mode, which is supposed to bring a big boost in performance. This patch is not in the stock zfs install from gentoo, so I will have to figure out how to incorporate that as well - but that will have to wait for a more powerful system.

Noticing also that the ntfs-3g numbers were quite pathetic - and that was with ensuring I used 4kB write blocks to give ntfs-3g the best possible chance at speed. So fuse could be a major issue here.

Anyway, more experiences coming as I work through this.

Of note: this implementation uses fuse version 27 and zfs-fuse 0.5.0. Also, the SATA bays in the Norco are driven by a Marvel SATA chip, whose driver is marked "extremely experimental" in the Linux kernel sources. Nice...

 

[thiggins]

You sure are a curious guy, corndog. Thanks for writing this up.

I think ZFS is designed more for Enterprise use. I love this quote from the Wikipedia article:
"The limitations of ZFS are designed to be so large that they would never be encountered, given the known limits of physics."

Sorta sounds like overkill for a little bitty consumer NASes.

 

[corndog]

Hi Tim,

Yep - it does seem a little grandiose, and Sun's storage systems that run ZFS are definitely enterprise class - although they hover at the bottom end of enterprise, starting at around $22,000. However, looking closely we are seeing ZFS start to trickle down into our Small Net world. Two examples come to mind:

1. FreeNAS, which is based on FreeBSD, has an experimental cut of ZFS included in it's development branch - FreeNAS 7.0 Not sure how long it will take before that one becomes production - they talk about ZFS still being a bit too raw in FreeBSD to trust it in your BSD-based NAS.

2. Thecus, whose products regularly grace your site, is starting to support ZFS in their higher end systems. Of their Pro line, the N8800 and N7700 both support ZFS. Admittely, these are larger systems with 7 and 8 drives. It would be great if Thecus would put ZFS into the 4100Pro and 3200Pro as well. It would really expose it to a lot more curious minds.

In addition to these, there is eon (which I discovered today in your forum) and also Nexentastor and good ol' zfs-fuse on Linux, all of which are attempting to bring this enterprise idea into our realm. Also, with free OpenSolaris which boots from ZFS, and MacOS almost having it now too, it's coming...

The little bits of testing I've done with zfs have shown me just how tempting it can be - best to be ahead of the curve and be an "expert" by the time it becomes prevalent, right?

 

[thiggins]

What are you seeing as the advantages of ZFS? Not performance, apparently. I'll have to ask Thecus what their motivation is for ZFS support.

 

[corndog]

That's a huge question, and it requires some thought to answer it properly. I guess it is a combination of many things. I've been running a serious home storage since my wife and I got a digital camera. Then a digital movie camera. Then digital music collection. Then we "back up" all the DVD movies we buy, and play them over the LAN. Then we need a converted version for the PSP, and for the iPod, and for the Touch. Besides, we both work in IT, so there's Ghost images, ImageX images, Acronis images, Windows backups, ISO files, VMX backups, installation distributions, all of our documents, etc etc etc, oh and not to ignore the DVR that keeps recording every sci fi on tv, and before you know it I have nearly 6 Terabytes of data I'm responsible for. Of that, nearly 1.6 Terabytes are irreplaceable. i.e. they need to be backed up and protected.

Now, even with the best NAS you can buy, it takes a good while to recover from problems. Even with a good backup, it takes a week and a half to recover from a catastrophe - believe me, I've had to do it.

And then, because we're both in the business, we have opportunities to upgrade and migrate, which takes time also. So I keep having these issues that are outstanding:

1. Backups take forever, and are highly network intensive. Even worse, quite often the machine being backed up is dog slow during the backup.
2. Restores are even worse
3. I have way too much data to backup in the cloud. It would take forever and my ISP would have my hide.
4. Why can't I migrate to a new NAS or DAS without downtime?
5. Why does a multi-terabyte RAID array take most of a day to initialize? There's no data on it! Does it need to make copies of nothingness for hours on end?
6. Why can't I take drives out of a dead NAS or DAS or server and have them "just work" in another one to recover the data? I don't mean "another one of the same kind". What if the NAS or DAS is out of production and the company is bankrupt?
7. If I don't access about 90% of my data very much, and then all of a sudden I really need it one day, how do I know it hasn't quietly corrupted on my drives?

These are a few issues. There are more. I would like to have my data on about 5 or 6 main shares with proper security reflecting the needs of my family and friends. All of those 5 or 6 main shares should be on one big volume, so I don't have to worry about running out of space on "photos" while I have lots of extra unused room on "Videos". I should be able to run this storage on a server, and when that server gets too old, I can migrate it all to a bigger, faster one, and everyone keeps accessing it without even knowing it's happening. If I run out of room, I should be able to bring a new array online and migrate my stuff over to the new array without anyone knowing that's happening either. Same for faster disks, faster enclosures, faster network access, you name it.

There are a few projects that do some of this.
- journalling filesystems for data robustness
- dynamic-growing filesystems with redundancy, such as X-RAID
- copy-only-the-changes backup strategies, such as rsync
- live array migration - on most NASes
- providing redundant storage only for your actual data - not for the whole empty disk - such as on the Drobo
- live volume relocation - such as evms
- live snapshotting - such as lvm2
- virtualization of entire servers and even multiple ones, with live migration during regular use - such as Microsoft DFS

All of these solutions are great pieces of the puzzle, but none of them provide the whole thing. And many of these solutions don't work with each other. And some of them that looked so great are orphaned, such as EVMS and reiserfs.

ZFS provides the whole thing from start to finish, with the possible exception of that last one (storage virtualization like Microsoft DFS). You just start with plain old partitioned hard drives, and ZFS provides a dynamically virtualized array, using the equivalent of any RAID method you like. You can migrate this array from one physical storage method to another dynamically, while the system is running. You can take snapshots at any time, and they are intended to be recoverable and useable (not like ripping a RAID-1 mirror out of a NAS and expecting it to work as a restore later). And your data is dynamically checked for accuracy during write, and also randomly during system inactivity at any time. On top of this array is a proper journalling and error catching file system for your important data. And if your system dies, you can recover your data in any server or NAS whether it is a Thecus, a Mac running Snow Leopard, or a D.I.Y. clone running FreeNAS, FreeBSD, OpenSolaris, or Linux. One piece of software provides the whole thing - stem to stern. That's why it is so compelling.

And while my zfs-fuse test on Linux hasn't produced any nice performance so far, I have to remember that the Linux implementation is alpha at best, and I really need to try a "real" setup like OpenSolaris or Nexenta to give it a proper test. Who knows? It might actually be fast too!

 

[thiggins]

Very nice explanation.

I wish you luck in your exploration. Keep us posted. I suspect, however, that all of that ZFS wonderfulness is going to require some serious compute power...more than you'll find on even "prosumer" NASes.

 

[szaka]

Noticing also that the ntfs-3g numbers were quite pathetic - and that was with ensuring I used 4kB write blocks to give ntfs-3g the best possible chance at speed. So fuse could be a major issue here.

Please note FUSE is not the reason for the lower results. The Open Source NTFS-3G driver performs 10-20 times worse than the Commercial NTFS-3G driver which is also FUSE based: http://ntfs-3g.org/performance.html

The best ever measured, sustained, non-cached NTFS-3G write performance is 1.8 GB/s.

 

[eon]

Zfs

ZFS is too resource and control hungry to expect the visible gains/performance under fuse.

To add to the great info from corndog:
you can also be proactive with zfs and add dynamic hot spare(s).
You can have unlimited snapshots, which you can recover a file delta from that instance.
You can have on the fly lzjb(zlib) - gzip 9 compression.
You can have separate ZIL (logging devices) and L2 Arc(ssd or ramdisk) to boost performance
you can build a never ending zpool (limited by ports, power on the machine) that you can grow to fit your needs
zpool create raidz pool dev1 dev2 dev3
later you outgrow this simply
zpool add raidz1 pool dev4 dev5 dev6
repeat

 

[corndog]

eon,

That's interesting additional info. Thanks!

It seems that the biggest two challenges I have with ZFS right now are:

1. It is big and bloated, power-hungry, and demanding. I'm assuming this is because it is still "young". Sun needs to optimize it better.

2. It is tied by Sun's CDDL license to Solaris, and hence can't really go properly into Linux. This is a major complaint I have about Sun. OpenSolaris' hardware support is really terrible - I can't even test it on any of my test machines. And because Sun is only semi-commited to the open source community (evidenced by the fact that they couldn't just use the GPL - they had to roll their own license) we now have to either use the FUSE kludge with an OS that properly supports our hardware (Linux), or stick to an OS that usually doesn't work on our systems (OpenSolaris).

Sun is repeating exactly the same mistake again which doomed Java to NOT rule the world. They can't seem to just get their act together and do it right, and on time when the technology is hot. They finally saw the light and put Java under the GPL when a good chunk of the world had already passed Java by in frustration. They'll probably release ZFS under the GPL in another 5 years or so, but it will be too little too late again, because there will be competing projects (tux and btrfs filesystems) which have been created specifically because zfs is so license-hobbled.

Sad...

In any case, I continue testing it and learning it because I believe in spite of the above crap, it still has enough technical merit to be worth the pain. I really hope Sun can see more than 6 inches in front of their own faces and do this right.

 

[eon]

Eon

EON tries to remove some of the bloat for you.

give it a look as a storage device you only need it to recognize your nic, sata devices. There is a Samba and suns in kernel CIFS module version. It's also 32/64 bit capable.

http://eonstorage.blogspot.com/

 

[corndog]

I will try both, and compare performance...

 

[corndog]

So after reading all the warnings online that ZFS only really begins to work with over 2G of RAM, I decided to bite the bullet and do what I've always wanted to do anyways - build a super-fast NAS.

So I ordered the Chenbro 4-bay D.I.Y. case that seems so popular around here, and picked up an Intel DG45FC motherboard for it. I also got a Core2Duo E8400 and 4G of DDR2 800 RAM to put in this thing. So far it has a pair of 134G WD Raptors in it for testing. I have just finished hand-building a nice 64bit Gentoo Linux environment for it, which boots over NFS from my Netgear ReadyNAS Pro, leaving the Raptors to be used only as a test array.

I'll baseline this system with some nfs, rsync, and samba file copies, along with a run of Intel NASPT from a test workstation, using ext2, ext3, reiserfs, xfs, and ntfs-3g, and then try ZFS again. I want to give ZFS a good workout in at least 5 different configs:

- ZFS-FUSE on Gentoo Linux x64
- ZFS on FreeBSD or FreeNAS if I can manage it.
- ZFS on eonstorage
- ZFS on Nexentastor
- ZFS on full-blown latest OpenSolaris

The plan is to test this thing with nfs, rsync, samba, and NASPT as above, with all five of these ZFS configurations, to give them a good comparison. I'll post these numbers as they arrive.

Then, because Sun is so bone-headed with their half-way commitment to Open source, which is causing people to look for and create alternatives to ZFS, I'll do the same type of testing with btrfs - a filesystem started by Oracle and released to the community with full GPL licensing.

 

[corndog]

BaseLine

Ok, here are the first baseline numbers:

The test involved is writing a 30GB file of zeroes with the following command:

dd if=/dev/zero of=testfile bs=1024000 count=30000

This totally overwhelms all caches and pushes the system hard. After this write is finished, I reboot the NAS and then read the file back with:

dd if=testfile of=/dev/null

Once again, this is too big for caching to have any effect. I did the same thing over NFS with another Linux box, to compare. Remember - this is a software RAID0 on a pair of Raptor drives. Here are the results:

ext3:
local system write - 145MB/sec
local system read - 138MB/sec
nfs write - 60.2MB/sec
nfs read - 73.1MB/sec

zfs-fuse:
local system write - 65.4MB/sec
local system read - 85.0MB/sec
nfs write - 68.2MB/sec
nfs read - 38.7MB/sec

The NAS system was running Gentoo 64bit, and the remote NFS test system was running Gentoo 32bit. I also tried reiserfs and xfs but their figures weren't as steady as ext3 so I used ext3 as a good baseline.

 

[corndog]

What about 32bit?

As I was testing this D.I.Y. NAS I began to see a lot of strange things - especially situations where reads seemed to be always a lot slower than writes - this even when overwhelming cache, and using local file systems. As an example, I noticed that if I tried reiserfs, I was getting 134MB/sec writes but only 50.9MB/sec reads. This was using dd and file sizes of 30G as explained in previous posts.

So just for fun (and a certain level of masochism I guess) I re-implemented the NAS exactly the same, but using 32bit Gentoo instead. WOW! What a difference!

Here are my numbers - same style of test as before - using dd to write 30Gigs of zeroes, and then reading the same file to /dev/null after a reboot.


ext3:
dd write all zeroes - 136MB/sec
dd read all zeroes - 169MB/sec

reiserfs:
dd write all zeroes - 140MB/sec
dd read all zeroes - 161MB/sec

xfs:
dd write all zeroes - 165MB/sec
dd read all zeroes - 174MB/sec

zfs-fuse:
dd write all zeroes - 63.5MB/sec
dd read all zeroes - 71.8MB/sec

So, at very least, reiserfs is no longer the great file system it once was - at least on my system. If I am braving the new world of 64bits, I'll have to leave reiser behind.

So, continuing from here in 32bit mode to maximize my hardware's capability.

Maybe that's why most of the fast NASes we see here still run in 32bit mode?

 

[eon]

ZFS score on your HW

Corndog,

It would be interesting to see a raw ZFS score on this same hw.

Is reiser still developed? The creator was convicted for killing his wife if IRC.

 

[corndog]

eon,

I'm trying, but the hardware support in eon is so sparse I can't get it going. I had it half-booting on an old Sony USB DVD drive, but the drive died, so I replaced it with an LG Super Multi Blue USB drive, and now it won't boot at all. OpenSolaris won't start either. So I replicated the whole experiment on another system - an ASUS Striker Extreme with a PentiumD 3.2GHz CPU, SiL3132 SATA, and I put a 3com 905 card in it because the Nvidia network interfaces were a challenge.

Once again, even though eon starts, it won't look at the SATA drives.

I can't seem to find any real life system that solaris will even work on.

I figured instead I would try FreeNAS because the alpha version 7 is supposed to have ZFS, but FreeNAS produces less than half the throughput of Linux, and that's on UFS - it's native file system. So I wasn't even going to bother trying ZFS on FreeNAS because it's so slow...

Still trying to get eon or OpenSolaris running, but my patience is running out.

 

[eon]

The drives don't show under eon because you mentioned the sil3132, those drivers not included. I will give a look see if they included at all on opensolaris dvd and post them for you.

I posted the sil3124 for someone else here dunno if they would work (might).
http://github.com/eonstorage/eon/downloads

cpio -idumbv sil3124.cpio

cd /kernel/drv/amd64 or cd /kernel/drv/
add_drv -v sil3124

So opensolaris won't install on any of your hardware you mentioned at all ?

 

[corndog]

Well, there's still the main D.I.Y. NAS that I started all this business on - the system with the SiL3132 is a diversion - if I really want to pursue this it should be on the main one. This system runs an Intel DG45FC Mini-ITX board - if I recall, it has an Intel e1000 network adapter (for some reason there seem to be a lot of different varieties of this adapter. This one in particular is the 82567LF-2) and an Intel ICH10 AHCI SATA adapter. That is the preferred platform. Is it possible to make eon support this?

I also boot this system from a externally attached USB BluRay drive, and neither eon nor OpenSolaris will boot this way, with this drive. I have to take the NAS apart and put in a slim-line IDE CD drive just to get it to boot eon.

 

[eon]

e1000g support

Corndog,

There is a e1000g driver in EON. It works fine with my badaxe2 w ICH7 82573E/L lan controller. I know there was some hit and miss support with this but I think it should work for your 82567.

If you boot eon on this board, do you see the nic? do a ifconfig -a plumb?
You should see a e1000g0 interface if all went well.

It may be the bluray driver why eon won't boot from that. However once you get eon installed to a USB key you won't need the CD drive anymore not even for upgrades.

As per the drivers if for the sata if they exist for solaris it easy to pull them from the install dvd and add them. EON is built on the concept of adding only what you need, which is why they are not included.

Which is why I recommended booting the opensolaris live CD ( osol-0906-109-x86.iso http://www.genunix.org/ no install needed ) and see if it sees the drive. Then I'd could tell which drivers it uses if found.