ESXi - FreeNAS - Upgrade to ZFS

[Hannibal80]

Hello all,
I'm new to this forum so, first of all, I'm Lorenzo and I'm glad to receive any help from you.

That's the story.
About one year ago I had the mad idea to build a NAS on my own.
I was planning to refresh my home PC and I thought to recycle the old one.
I needed a box that could hold my storage and, hopefully, act as a small home server.
So I began to study ESXi and FreeNAS.
I found that my old MB was perfect for ESXi, I didn't need to change NIC or SATA controller. Then I built my box using this configuration:

ASUS p4g8x deluxe
p4@2800 (533mhz FSB)
4x512MB DDR RAM
2x1TB Hitachi SATA HD
1x80GB IDE disk
1x20GB IDE disk

The smallest drive contains the Hypervisor, the FreeNAS VM guest system OS and the Win XP guest system OS.
The 80GB disk is used by XP and the 2 terabyte disks are used by FreeNAS in RAID 1 configuration.

Now... it's time to maintain the system, because I'm running out of space and out of patience when I have to move big files around the net.

I was thinking to add a new disk and build a 3 disks RAID 5 array with ZFS file system over, but I'm really not sure that my box can handle the job.

I bought a PCI 4x SATA board (no RAID) to extend the capacity of the system because the MB has only 2 SATA interfaces.

Last but not least, I would like to improve the poor performance of the box, may be adding a new NIC, but I'm not sure this can really help.

Actually, using Samba and/or FTP, I can achieve something between 80Mb/s and 160Mb/s around the network. Note that between my workstation and the box there is a DLink gigabit router.

Any help would be precious,
thanks a lot

Hannibal.

 

[00Roush]

I think you should be able to see better performance than that. My guess you could get 30-80 MB/sec out of that setup. Depends on what bus the network card is on and settings used in FreeNAS. This is assuming no VM. I recommend you at least test without ESXi and see what kind of performance you get. With FreeNAS I have found the best performance by changing the Samba buffer sizes to 0 or to 65536. From my understanding 0 allows the buffers to be set dynamically.

As for the ZFS or Raid 5... the box might be a bit underpowered. I would say go ahead and test it if you can. First without the VM then with just to get an idea of lost performance. My guess is the older hardware takes quite a hit in performance doing virtual OSes. I never did ask... what NIC is on the board? If anything a cheap Intel PRO/1000 MT or GT nic (PCI) might help. Heck I have at least one I could send ya if you want to pay shipping.

If I get a chance tomorrow I might try doing some testing... I have some similar hardware here that I could use to get an idea of a best case performance.

00Roush

 

[Hannibal80]

Hello Roush and thanks for the reply.

Yesterday evening I rebuilt the box, adding the new SATA controller and a caviar green 1T drive. The volume was formatted in ZFS with the devices in RaidZ.
In the SMB page I setted a buffersize of 64240, both sending and receiving, the large read/write checkbox is enabled as the tuning of kernel variables.

Everything is still inside a VM in ESXi and yes, I think you are right, the poor performances could be imputed for the most to the virtualizated environment. Common SMB file transfer speed is something around 10-15MB/s and a good clue that the problem rely on the VM is the graph below. As you can see, peaks are over 240mbit/s but are completely unstable.

Anyway I need the other VM and installing ESXi wasn't so easy so I'm really scared to install FreeNAS deleting the hypervisor...

The MB has a Broadcom BCM5702 Gigabit Ethernet controller but some time ago I read that it was placed in some bus so that would not be able to deliver max performances, anyway I cannot remember nothing more.

The SATA embedded controller is a Sil 3112 with the 2 sata ports used. I added a PCI Sil 3114 to connect the third drive.

I would be glad to know something about the use of a second NIC...

 

[00Roush]

The nice thing about FreeNAS is you don't have to install it! You could just put it on a CD and load from the CD. That is one reason why I like FreeNAS. Within 5 minutes (well maybe 10) I can go from no OS, to up and running with several file shares. This is what I meant about testing FreeNAS without ESXi. I would just disconnect the drives that have any important data and test with just your new drive.

Your graph kind of reminds me of some problems I encountered testing ZFS with a single drive. With two drives or more I had much better luck with ZFS. But I have not really tested much with ZFS on FreeNAS, just EON. To eliminate possible bottlenecks I would recommend running FreeNAS from a CD and then just formatting your drive with the default format of UFS. Testing with UFS will help eliminate any slowdowns that might be caused by ZFS.

It looks like your network card should be able to provide decent performance. It is on a PCI bus so it is a bit limited but no more than the Intel cards I mentioned. I spent some time testing on a similar machine here at home and my Intel PRO/1000 desktop card actually does pretty well when testing with iperf. Actual file transfer speeds are fairly solid at about 60 MB/sec read and write using UFS. With ZFS on a single disk transfer speeds drop to 30-40 MB/sec and the network throughput graph starts to see-saw like yours. ZFS with two disks was a little better but not much.

Just for reference here are the specs of the machine I tested on:
Gateway compuer with a Intel Motherboard (210882)
P4 @ 2.4 Ghz with hyperthreading (800 Mhz FSB)
2 x 256 MB PC2700 DDR RAM
2 WD SE16 320 GB hard drive (SATA)
Intel PRO/1000 MT desktop (PCI)

I would consider my results as a best case but based on your hardware I would say you could see similar performance. Your ZFS speeds might be better as you have more RAM.

00Roush