Intel SS4200 running FreeBSD 7.1 with 4 x 1.5 TB seagate drives

[simonlok]

I just bought an Intel SS4200-E. I found the built in EMC software to be lacking (fails in odd ways, needs to be rebooted twice some times, upgrades sometimes don't take, factory reset brings it back to the original version it came with and then it cannot be upgraded again without work, console access is a pain). I tried FreeNAS and OpenFiler and I was unhappy with those as well. OpenFiler was way too much of a pain to get even a single share up. FreeNAS was better, but being based on FreeBSD 6.4 it's ACPI support and whatnot is basically nonexistent.

What I finally ended up doing is installing FreeBSD 7.1 on a small hard disk that is plugged into the ATA port (I removed the DOM). FreeBSD 7.1 has full support for serial console and works quite well. It is very resilient on the box. Replacing the root hard disk with another one and the software RAID volume is found and mounted instantly. It gives me full root shell access and I can do whatever my heart desires with it. I am very happy with the SS4200 now. I should have bought the SS4200-EHW and saved the $75 on the EMC software.

In case anyone is interested, here are the things that I did to get the machine up and running. I used the standard FreeBSD images available off the FreeBSD website and did a "minimal" installation.


created /boot/boot.config

-Dh -S115200


added to /boot/loader.conf

boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
console="comconsole,vidconsole"
geom_vinum_load="YES"


edit /etc/ttys

change ttyd0 to on and vt100 from off and dialup


create /etc/rc.local

dhclient em0
/usr/local/sbin/smbd


create /etc/mainarray.gvinum

drive r0 device /dev/ad8a
drive r1 device /dev/ad10a
drive r2 device /dev/ad12a
drive r3 device /dev/ad14a

volume mainarray
plex org raid5 512k
sd drive r0
sd drive r1
sd drive r2
sd drive r3

- or -

volume mainarray
plex org striped 512k
sd drive r0
sd drive r1
plex org striped 512k
sd drive r2
sd drive r3



edit /etc/fstab

/dev/gvinum/mainarray /space ufs rw 2 2


run the following commands as root:

fdisk -BI ad8 && bsdlabel -wB ad8
fdisk -BI ad10 && bsdlabel -wB ad10
fdisk -BI ad12 && bsdlabel -wB ad12
fdisk -BI ad14 && bsdlabel -wB ad14

bsdlabel -e ad8
bsdlabel -e ad10
bsdlabel -e ad12
bsdlabel -e ad14

gvinum create /etc/mainarray.gvinum

gvinum print_config

# to start the build out of the mirror
gvinum start mainarray.p1

# verify that it's rebuilding
gvinum print_config

newfs /dev/gvinum/main_array

mkdir /space

mount /space

pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7-stable/net/samba-3.2.7.tbz

setenv PKG_PATH ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7-stable/All/

adduser admin

/usr/local/bin/smbpasswd -a admin

create /usr/local/etc/smb.conf

[global]
workgroup = WORKGROUP
server string = SS4200_1
security = share
log file = /var/log/samba/log.%m
max log size = 50

[space]
path = /space
public = no
guest ok = yes
force user = admin
force group = admin
writable = yes
printable = no

[public]
path = /space
public = yes
only guest = yes
writable = no
printable = no

 

[simonlok]

Today I pulled the system apart again to try Linux to see if I would get any better performance. Sadly Ubuntu Server 8.10 and Fedora 10 both do not boot on the SS4200 if you wish to use a PATA device as the root. You can install with the OS on one of the four SATA drives or you can disable all those and only have the PATA drive, but no combination of kernel options and BIOS twiddling that I tried today allowed me to get all 5 drives up and running on Ubuntu or Fedora. Back to FreeBSD for me.

 

[dlong]

simonlok,

* The EMC s/w has a way to enable SSH via the web interface (via support.thml).

* I have seen reports that Ubuntu have been successfully installed on the SS4200 using a larger DOM, not sure which version of Ubuntu.

* How did you mount the (laptop?) hard drive ?

Thanks,

-d

 

[simonlok]

simonlok,

* The EMC s/w has a way to enable SSH via the web interface (via support.thml).

I do not understand how anybody uses the EMC stuff. It is unreliable (sometimes when you click on things and say apply, it doesn't take, it needs to be rebooted twice every so often, you have to hit upgrade six times to get it to upgrade to the latest version, etc.), slow (takes a full day or two to get it's act together setting up a RAID array, read and write on the RAID 5 was pathetic, etc.) and the level of instrumentation that you have is basically nonexistent (which I suppose would be solved if they let you enable SSH that I didn't find). If you are using it without issue then clearly I am missing something.

* I have seen reports that Ubuntu have been successfully installed on the SS4200 using a larger DOM, not sure which version of Ubuntu.

I looked around and I can't seem to find a link to how that was done. As far as I can tell, the kernel in Ubuntu 8.10 and Fedora 10 both have the same issue which is that it does not even detect the PATA drive. You can even go so far as to install onto a PATA disk using a different machine, then plug the PATA disk into the SS4200. Set the BIOS to boot off of the PATA drive and the boot loader will come up and then load the kernel. The kernel will come up and will not be able to find the root filesystem. Looking through the kernel messages, the PATA disk is never found by the kernel. Exact same behavior on Ubuntu and Fedora. I tried messing with the BIOS to load the PATA disk before the SATA, after the SATA, in compatibility mode, in IDE mode, etc., to no avail.

I am going to try ZFS on FreeBSD soon. That should be quite interesting.

* How did you mount the (laptop?) hard drive ?

Thanks,

-d

It's a desktop drive. Remove the DOM, unplug it from the 4 pin molex power. Remove the ATX power connector to the motherboard. Connect the 4 pin molex to the desktop drive along with ribbon cable for 40 pin ATA. Slide the drive with the front part (opposite connectors) facing towards outside of the chassis. The molex 4 pin will for you to put the hard drive in a particular spot due to the limitation of slack and the other things in the chassis. Plug the ATX power supply connector on top of the hard drive. This cable has very little slack as well and basically locks the drive down into place. I did some shake testing, this works very well.

 

[simonlok]

ZFS on FreeBSD 7.1

I redid everything on the SS4200 again. This time using FreeBSD 7.1 and ZFS. I have messed around with volume managers before but ZFS really brings it to a new level.

Apparently ZFS under FreeBSD 7 requires some tuning to make work right. Here is what I did, and everything has been working great on the SS4200 with very high loads. Performance is awesome compared to the geom software raid 5.

in /etc/loader.conf

zfs_load="YES"

vm.kmem_size="512M"
vm.kmem_size_max="512M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.arc_min="16M"
vfs.zfs.arc_max="128M"
kern.maxvnodes="400000"


run the command:

zpool create space raidz ad8 ad10 ad12 ad14


No partitioning, labelling or newfs is necessary. The raidz specifies the ZFS form of software RAID5. You can also put a raidz2 if you want to do double parity but with only 4 disks you are better off doing stripes of mirrors. The best part about it is that the ZFS RAIDZ is up and ready to go in minutes as opposed to hours or 1.5 days with the other software RAID solutions.

After some more reading, I found that you want to really make:

vm.kmem_size="1024M"
vm.kmem_size_max="1024M"

But to do anything above 512M requires a custom kernel. That was really easy.

cd /usr/src/sys/i386/conf
cp GENERIC SS4200
vi SS4200

add the following line:

# for ZFS
options KVA_PAGES=512


then you:

cd /usr/src
make buildkernel KERNCONF=SS4200
make installkernel KERNCONF=SS4200
shutdown -r now


Combining this methodology with the serial console methodology from my first post and it works wonders on the SS4200. Installs easily and quickly, set is simple and works the first time, very stable and local read/write performance is excellent.

admin@ss4200 /mainarray/tmp]$ dd if=/dev/zero of=/mainarray/tmp/test bs=1m count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 44.416255 secs (94431735 bytes/sec)

[admin@ss4200 /mainarray/tmp]$ dd if=/mainarray/tmp/test of=/dev/null bs=1m count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 24.277846 secs (172762608 bytes/sec)
[admin@ /mainarray/tmp]$ ls -la

Remote read/write performance over Samba is very good but not amazing. Using a Vista client, copying a multigigabyte file to and from the SS4200 (accessed in explorer via \\ss4200\space, measured using perfmon), I get a solid 40+ MB / sec write and a solid 75+ MB / sec read. In addition, I've been getting approximately just over 40 MB/sec read and just under 40 MB / sec write via a Mac Pro (mounted smb://ss4200/space via Finder and copying multigigabyte files to and from ss4200 with measurements made using Activity Monitor). That's good enough for me but I'm sure there are some out there that would say there is a lot of performance being left on the table.

I'm sure I can get more if I started tweaking the smb.conf and kernel but for now, I think that this is an excellent solution. It is very easy to setup and appears to be highly reliable and fast.

 

[eon]

If you're interested in ZFS

give this image a try

http://eonstorage.blogspot.com/
http://eonstorage.blogspot.com/2009/01/eon-0589-based-on-snv104-is-released.html

 

[Gidi]

Ended @ZFS also

Simon,

I'm running the box also with ZFS but did install OpenSolaris instead of *BSD/*X
and its native CIFS service which puts any Raid and SAMBA config procedure
and scripts to a shame.

One of the first things I did was upgrading the 512MB Memory with a 2GB DIMM
which gave the box a nice boost because ZFS loves memory.

I had similar experiences with OpenFiler and FreeNas and couldnt make
Ubuntu happy on the IDE port.

The EON package did arrive to late for me because I've gone trough all the pain
with installing any other OS on that box before and I'm running now the box
with a OS 2008.11 with a 64bit Kernel. So if there would be a 64bit kernel
with EON I might look at it again.

The only thing I miss with OpenSolaris is a usable DLNA server but other than that
I'm more than happy with the box now (Great FS, Snapshots, Scrubbing, Performance, Cifs, iSCSI etc).

 

[eon]

Eon 64

Gidi,

The 64-bit versions (82 MB CIFS, 95MB Samba) have been uploaded on genunix.org and will be ready as soon as Al provides the green light on the links.

Couple of things to try and convince you to still give EON a shot.
- It's one drive greener than a HD install without a performance hit.
- Upgrade path is safe and reversible if not pleased. With a new release you can safely test the upgrade (snv_xxx) without sitting through a install/upgrade. If you don't like it (currently at snv_104) you put back the previous image and reboot.
- Because it runs from RAM, a OS drive failure will not halt the show.

 

[dlong]

I do not understand how anybody uses the EMC stuff. It is unreliable (sometimes when you click on things and say apply, it doesn't take, it needs to be rebooted twice every so often, you have to hit upgrade six times to get it to upgrade to the latest version, etc.), slow (takes a full day or two to get it's act together setting up a RAID array, read and write on the RAID 5 was pathetic, etc.) and the level of instrumentation that you have is basically nonexistent (which I suppose would be solved if they let you enable SSH that I didn't find). If you are using it without issue then clearly I am missing something.

You can enable SSH in http://device/support.html (undocumented) along with a few other items -- not sure how useful they are. I do agree that the EMC LifeLine, as designed are very much lacking in features. And setting up a RAID does take a very long time. My original 2 1.5 TB drive RAID-1 took 11 hours. I added 2 more drives yesterday and RAID-5 took 6.5 hours to setup. However, via the EMC gui, I made a change to the NAS box name and it rebooted the system at 5 percent and when it finished it didn't like the finished product -- said RAID was fine but nothing was available -- strange (not sure if mke2fs would continue properly after the reboot and why EMC would allow a function which would reboot the device in the middle of building a RAID system is beyond me?).

However, instead of changing the to RAID 0+1 in an attempt to get a working RAID -- I should of tried "Attempt to recover RAID" in the support.html page first. Anyway, it is at 20% now after 2.5 hours.

I looked around and I can't seem to find a link to how that was done. As far as I can tell, the kernel in Ubuntu 8.10 and Fedora 10 both have the same issue which is that it does not even detect the PATA drive. You can even go so far as to install onto a PATA disk using a different machine, then plug the PATA disk into the SS4200. Set the BIOS to boot off of the PATA drive and the boot loader will come up and then load the kernel. The kernel will come up and will not be able to find the root filesystem. Looking through the kernel messages, the PATA disk is never found by the kernel. Exact same behavior on Ubuntu and Fedora. I tried messing with the BIOS to load the PATA disk before the SATA, after the SATA, in compatibility mode, in IDE mode, etc., to no avail.

You are right. Ubuntu 8.1 does not have the PATA driver loaded -- in the other post, the user had to patch the kernel to add the PATA driver. I miss-read the post then when he went to upgrade he installed a new kernel without the PATA driver.

Here's a post regarding "EMC SOHO environment":
http://forum.wegotserved.com/index.php?showtopic=6210&mode=linear

It's a desktop drive. Remove the DOM, unplug it from the 4 pin molex power. Remove the ATX power connector to the motherboard. Connect the 4 pin molex to the desktop drive along with ribbon cable for 40 pin ATA. Slide the drive with the front part (opposite connectors) facing towards outside of the chassis. The molex 4 pin will for you to put the hard drive in a particular spot due to the limitation of slack and the other things in the chassis. Plug the ATX power supply connector on top of the hard drive. This cable has very little slack as well and basically locks the drive down into place. I did some shake testing, this works very well.

I am having a hard time visualizing how the HD is mounted/held in place -- there seems to to be very little space for a hard drive. Unless you are just temporary leaving it there until proper mounting?

I have the cable to connect to the serial pinout on the SS4200 MB but not the F-to-M (or vice versa) adapter so that I can connect it to my computer. So far, I have only looked around on the base EMC image. I don't have any experience with configuring RAID in Linux.

I was hoping there was an easy way to add Apache, php, mySQL, perl to EMC's SOHO environment but I tried adding some binary software but it wouldn't run for some reason (even made it 777). I am hoping some studious soul will post an image (of *IX with the other packages installed) and I could just restore (via DD) it onto the DOM.

-d

 

[eon]

64 Bit version released

Gidi,

The 64 bit version is now posted

http://eonstorage.blogspot.com

 

[Gidi]

Dlong,

had a strange experience too when I rebooted the box during a raid resync
which was caused by a movement of a installed drive. I've also rebooted the
box during that operation and the resync was not restarted/continued after
boot. This didnt improve my confidence into this raid implementation.

I didnt play around with ubuntu for long but also saw that the driver was missing
and the same happens with OpenSolaris. So I run OS from a USB stick
(sometimes with 2 of them mirrored ).

EON,

you made me curious seen it avail on http://eonstorage.blogspot.com/
I'll give it shot on a second box I have for testing....can I wish to have
comstar (native iscsi) in one of the next updates included

Guido

 

[Master One]

Hi guys,

I am running FreeNAS on my SS4200-EHW, and even with the latest versions, I have the problem, that shutdown is not working, it just halts and the chassis fans go to maximum, instead of the unit powering off. It's a well known (ACPI) issue, and I thought it's a FreeBSD problem, not particularly a FreeNAS one.

I just tried with the following versions:

FreeNAS 0.69 Kwisatz Haderach (revision 4276) amd64-embedded
based on FreeBSD 6.4-RELEASE-p3 (revision 199506)

FreeNAS 0.7 Sardaukar (revision 4347) i386-embedded
based on FreeBSD 7.1-STABLE (revision 199506)

Does powering off work with plain FreeBSD on the SS4200?

BTW The amd64-version of FreeNAS 0.69 does not recognize a connected PATA device, which is why I had to kick the DOM and go for an installation on an USB stick (which works better and is more comfortable anyway). The FreeNAS devs claim this issue to be an upstream one as well, so FreeBSD 6.4 amd64 does not contain the driver for the used IDE controller, but FreeBSD 6.4 i386 does?

FreeBSD or OpenSolaris / EON with ZFS looks tempting, but ZFS still does not include full drive encryption, does it? With FreeNAS I have my SoftRaid5 encrypted, and I really don't want to miss that functionality.

 

[Gidi]

Master One,

ZFS encryption is planned to come with the snv_111 release (we are currently
@ snv_107, OS 2008.11 was snv_101). So keep a eye on it

 

[Master One]

Well, ZFS is not on top of my priority list, but the still existing power off problem with FreeBSD/FreeNAS is.

Can anybody here running a pure FreeBSD check, if the a shutdown really powers down the unit, or if it is indeed a still existing FreeBSD ACPI problem?

 

[eon]

Acpi

Gidi,

What's the difference between the comstart iscsi target and the oem iscsitgtd?

thanks

MasterOne,

It's probably quicker/easier to burn a live iso BSD and test yourself.

 

[Master One]

It's probably quicker/easier to burn a live iso BSD and test yourself.

Therefor I'd need a live iso which supports the use of a serial-console right out of the box, and is based on FreeBSD 7.1 (since all previous versions lack the updated ACPI support and therefor cause the mentioned issue for sure).

Any hints?

 

[dlong]

Therefor I'd need a live iso which supports the use of a serial-console right out of the box, and is based on FreeBSD 7.1 (since all previous versions lack the updated ACPI support and therefor cause the mentioned issue for sure).

Any hints?

Maybe:

Mount the iso as read-write (mount -o loop,rw -t iso9660 /path/to/file.iso /media); modify the necessary grub/lilo menu item to support console, unmount and then burn it?

I never actually tried the above but it seems reasonable.

-d

 

[brianwtl]

I have a question on DMA on IDE port. I have tried Ubuntu 8.04 and Debian 5.0 and both fail to enable the IDE DMA.

I tested all the BIOS IDE setting from Enhanced-IDE/RAID/AHCI to legacy mode. Only legacy mode give DMA but only two SATA drives are populated.

I recompiled the Kernel 2.6.26.1 to disable CONFIG_IDE (ref: http://www.thinkwiki.org/wiki/How_to_configure_and_use_libata_SATA_/_PATA_drivers) but still it cannot discover the IDE HDD.

I would like to know does FreeBSD or OpenSolaris enable the IDE HDD DMA with 4 SATA drives. Thanks!

 

[drbiggly]

I have a question on DMA on IDE port. I have tried Ubuntu 8.04 and Debian 5.0 and both fail to enable the IDE DMA.

I tested all the BIOS IDE setting from Enhanced-IDE/RAID/AHCI to legacy mode. Only legacy mode give DMA but only two SATA drives are populated.

I recompiled the Kernel 2.6.26.1 to disable CONFIG_IDE (ref: http://www.thinkwiki.org/wiki/How_to_configure_and_use_libata_SATA_/_PATA_drivers) but still it cannot discover the IDE HDD.

I would like to know does FreeBSD or OpenSolaris enable the IDE HDD DMA with 4 SATA drives. Thanks!

You know this sounds oddly like an issue I had...

Simple Ubuntu box with an IDE boot drive and a single SATA HDD. I had Ubuntu 7....uh, I cannot recall if it was 7.04 or 7.10. Anyway, I did a full distro upgrade and suddenly it could no longer see the SATA drive.

Apparently with some of the kernel modifications in 8.04 and beyond, it is impossible for the system to play nice with an IDE boot drive and then SATA drives afterwards. It assumes some sort of SCSI driver to make the IDE drive 'faster', but then this completely does not work with SATA drives as well since they are not also on the same bus, though Linux seems to think that they are. Huge flaw IMO.

Can you guys try an older than 8.04/8.10 version of Linux on this? 7.04/7.10 perhaps? This would likely allow you to boot from the IDE drive without problem.

After all of the trouble, I eventually gave up on Linux for what I was doing after it completely crashed my volume when a backup job (push to the SMB share from a Windows box) exceeded the space on the drive. It literally took me powering off the system and back on twice (fully rebooting Linux) before it would see it again. Note that I was using JFS...supposedly a robust file system.

Now that I have gone off on a tangental rant...have you guys tried an earlier Ubuntu? I think it might work.

-Biggly

 

[00Roush]

I used Ubuntu Server 8.04 in my server for a while with IDE and SATA drives without any problems. 80GB IDE boot drive with a 320GB SATA data drive.

00Roush