FreeNAS 8 reboots continuously when all 10 SATA ports filled

[Mr. Bungle]

I just put together my first system to run FreeNAS:

• Asus M4A89GTD motherboard (http://www.newegg.com/Product/Product.aspx?Item=N82E16813131631)
• Kingston KVR1333D3E9SK2/8G DDR3-1333 8GB (2x4GB) ECC memory (http://www.newegg.com/Product/Product.aspx?Item=N82E16820139262)
• AMD Athlon II X2 260 Regor 3.2GHz CPU (http://www.newegg.com/Product/Product.aspx?Item=N82E16819103873)
• (10x) HITACHI Deskstar 0S03230 3TB SATA 6.0Gb/s hard drive (http://www.newegg.com/Product/Product.aspx?Item=N82E16822145493)
• (2x) HighPoint Rocket 620 PCI-Express 2.0 SATA 6.0Gb/s controller card (http://www.newegg.com/Product/Product.aspx?Item=N82E16816115072)
• IDE to Compact Flash CF Adapter (http://www.monoprice.com/products/p...=10407&cs_id=1040705&p_id=2105&seq=1&format=2)
• Transcend 4GB Compact Flash TS4GCF133 (http://www.newegg.com/Product/Product.aspx?Item=N82E16820208339)
• Intel EXPI9301CTBLK PCI-E network adapter (http://www.newegg.com/Product/Product.aspx?Item=N82E16833106033)
• Antec EarthWatts Green EA-380D 380W PSU (http://www.newegg.com/Product/Product.aspx?Item=N82E16817371033)
• Antec Three Hundred Illusion ATX case (http://www.newegg.com/Product/Product.aspx?Item=N82E16811129066)
• Cooler Master STB-3T4-E3-GP 4-in-3 HDD cage (http://www.newegg.com/Product/Product.aspx?Item=N82E16817993002)

It's all assembled now, and I loaded FreeNAS 8.0.1-RC2 onto the CF card and was able to boot up successfully and log in via the web interface. However, I noticed while booting that only 9 of my 10 HDDs was recognized in the BIOS, which was also reflected in the FreeNAS UI. After some troubleshooting, I narrowed it down to a bad SATA cable, so I swapped that out, and now all 10 drives are recognized in the BIOS.

However, now FreeNAS won't boot. I get through the POST and BIOS steps just fine, but right when I'd normally start to see FreeNAS booting up, the machine just resets and goes through the process again, and does this continuously.

Not sure what had happened, I tried everything I could think of to narrow this down:

• changing the boot device settings in the BIOS
• removing the Intel NIC (I saw the Intel Boot screen, which I hadn't noticed before, and thought maybe the NIC was trying to initiate a network boot)
• installing FreeNAS on a USB stick and booting from that
• reinstalling FreeNAS on the CF card and trying again

None of those steps fixed the problem, but then I remembered that everything had worked fine when only 9 HDDs were connected at first. I removed one of the drives, and it worked! FreeNAS booted again.

So, I've narrowed the problem down to FreeNAS perpetually rebooting only when I have all 10 SATA ports filled. This happens whether I have the IDE port filled with the IDE/CF adapter and am booting off of a CF card, or whether I remove that adapter and boot from USB.

Does anyone have any ideas about what may be happening, and possible steps to fix it? Thanks in advance!

 

[GregN]

Dude, you're back.

I'll take a look, but when I've seen this behavior before, it has been a BIOS/BIOS setting problem ( boot drive or etc ) or memory issues.


Are you hearing any BIOS Beep Codes?

( or this )

Given the 9vs10 Drive issue, I'd lean towards a BIOS boot drive issue.

Is the tenth drive off of the Highpoint card or the MB? If MB, try swapping the tenth drive off of the Highpoint, or see if some combo makes a difference. Highpoint card could be failing to post, the full MB could be forcing a Sata boot.

Do you have an IDE drive laying around? I try that initially, replacing the CF card with the IDE Drive.

Are you seeing any difference on BIOS post, 9 Drives vs. 10 Drives ( IDE banner, Sata bad drive count, High Point cards posting? )

 

[Mr. Bungle]

Thanks, GregN! (BTW, I left a quick reply in my old thread as well.)

Everything seems to function properly with the system during POST:

• I don't hear any beeps (assuming you're referring to audible beeps?) - the motherboard didn't come with a speaker, although maybe there's one already on it.
• The system itself POSTs just fine every time; during the process, it shows me first all of the SATA drives that are connected to the motherboard (up to 6); then I see each of the two HBAs POST, each of which shows up to two connected drives; then the Intel Boot screen (I guess that's my NIC), and as soon as that's done, I see the Asus BIOS screen pop up for a fraction of a second and then the screen goes black. Then the computer restarts and does that all over again.
• Each time this process happens, every connected SATA drive is properly recognized, and I don't see any complaints or error messages from the system.
• When I get it running with just 9 drives, all of the above steps happen in the same way, but then the Asus BIOS screen pops up again (for the second time, BTW; I see it both at the very beginning and after POST) for a few seconds, and then I see FreeNAS start up and everything seems to run fine.

I've paid attention to my boot drive settings, and I'm pretty sure they're correct. When I boot from my CF card, I select that as the first boot device and disable all other boot devices. Same for USB. I see the brand name of those cards/drives in the BIOS so I'm confident that I'm picking the right setting.

I just experimented a little more and found that's I get the problem when all of the motherboard's SATA ports are full; the HBA's ports don't seem to matter. Disconnected one or both of those from each cart still causes the problem, but when I disconnect from one or more of the motherboard's native ports, FreeNAS boots successfully.

I might have an IDE drive around; good suggestion - I'll give that a shot if so.

 

[Mr. Bungle]

Some more clues - I removed both of the Highpoint HBAs, and the system booted fine with just the 6 onboard SATA ports. What do you think that means?

The slots I have available on my motherboard (as listed in the manual) are:

• PCIe 2.0 x16_1 slot (gray, at x 8 link)
• PCIe 2.0 x1_1 slot
• PCIe 2.0 x4_1 slot
• PCIe 2.0 x16_2 slot (blue, single at x 16 or dual at x 8 link)

I have the HBAs plugged in to the two x16 slots. Should I try moving them? If so, where to?

Also, this reminded me that I'd seen some talk of somehow updating the HBAs (maybe via firmware?) to be effectively transparent to the system; in other words, they'd function more or less like native motherboard SATA ports. Do you know anything about that? Could the fact that I'm seeing both of them POST individually during the boot process mean that they might not be set up ideally since I just need them for extra SATA ports and nothing else?

 

[GregN]

You should be able to run them in the x1 and x4 slots.

This appears to be a BIOS Boot Select problem, which was my guess. Have you enabled AHCI?

What is the jumper setting for IDE on the MB, Cable select or Master/Slave? Setting on the CF adapter?

I would continue to try combinations of cards and drives, but from what you are saying, you can try Asus support....

Can you Bail on the HighPoints?

Take a look at the conclusion here:

While the Rocket 620 is a great option for those who don’t have a SATA 6Gbps port, or who want to maximise their SATA 6Gbps performance, it does have limitations that are solved by the RocketRAID 640. Adding a second drive when using the 620 resulted in heavy bottlenecking when both drives were used simultaneously, whereas we saw great two-disk JBOD performance when using the 640.

However, the RocketRAID 640 brings its own problems, and while a two-disk JBOD setup works great thanks to the dual Marvell 9128 controllers, adding a third or fourth drive in JBOD configuration will result in very poor performance.

 

[Mr. Bungle]

I tried the HighPoint HBAs in the x1 and x4 slots, but no joy (same continuous reboooting issue). I also removed the Intel NIC when I did that just to take that out of the equation.

I have enabled AHCI for all drives in the BIOS.

I don't think the motherboard has a jumper setting - at least I don't see one, and the motherboard's manual makes no mention of it (it just refers to "drive jumper setting"). The jumper on the IDE/CF adapter is set to "Single Device." This also happens when I boot from a USB thumb drive, so that seems to rule out the IDE port being the issue.

I'd rather not bail on the 620s just yet, although I will if I have to. Moving up to 640s would be pretty costly - that would actually be more expensive than just using 4 ports from an 8-port controller like the 2720. Also, removing just one of my 620s doesn't help, so that doesn't give me much confidence that a pair of 640s (or even just one 620 with 4 HDDs connected) would solve the problem.

 

[Mr. Bungle]

Can you Bail on the HighPoints?

Take a look at the conclusion here:

While the Rocket 620 is a great option for those who don’t have a SATA 6Gbps port, or who want to maximise their SATA 6Gbps performance, it does have limitations that are solved by the RocketRAID 640. Adding a second drive when using the 620 resulted in heavy bottlenecking when both drives were used simultaneously, whereas we saw great two-disk JBOD performance when using the 640.

However, the RocketRAID 640 brings its own problems, and while a two-disk JBOD setup works great thanks to the dual Marvell 9128 controllers, adding a third or fourth drive in JBOD configuration will result in very poor performance.

I just read through that, and they're using SSDs for the test drives. It makes sense that they'd be bandwidth-capped on the 620, since the 500MB/s bandwidth of PCIe 2.0 x1 has to be split between both drives, leaving 250MB/s for each (which heavily limits the SSDs' 350MB/s+ throughput).

Since I'm just using plain ol' SATA 3.0 HDDs, though (maximum throughput more like 100-150MB/s), I *should* be fine with the 620s unless, for some reason, they happen to impose their own speed cap that's well below the PCIe 2.0 x1 standard.

 

[GregN]

OK, let me see if I got this:

One Rocket 620 in, all SATA ports filled on MB - It doesn't find the IDE drive to boot.

One Rocket 620 in, 5 of the 6 SATA ports filled on MB - It DOES find the IDE drive to boot from. Once booted it sees all 7 drives.

No Rocket Card, 6 of 6 SATA ports filled on MB - it does find the IDE drive to boot from.

AHCI makes no difference.

Disconnecting the IDE, and booting using USB, same results.

IDE Banner appears under all scenarios.

You see the Banner(s) for the Rocket Cards under all scenarios ( where a card is in )

-<>-

Do I have that right?

Do you see any indication that the boot process is dying after talking to IDE or USB ( the USB light flashing, CF card adapter lights indicate anything? ). Any indicators of bootstrapping?

Do you have a Windows/Linux install that you can try on the CF/IDE (eliminates the possibility that it is FreeBSD).

You could try installing temporarily FreeNAS on the first MB Sata drive, then add the sixth drive...

IDE drive, just to see.


You have me stumped, need more symptoms

 

[Mr. Bungle]

This may be useful - after playing around with hardware configuration and BIOS settings some more, I discovered that the AHCI/IDE setting does make a critical difference. I hadn't anticipated that AHCI would cause any issues, so I'd chosen that mode in the BIOS from the beginning; however, changing all ports back to IDE now allows the system to boot with all 10 drives attached.

To summarize:

• If I set all 6 motherboard SATA ports to IDE mode, the system boots successfully. The BIOS allows you to specify AHCI/RAID/IDE separately for ports 1-4 (as a group) and ports 5-6 (as a group). You can set all 6 as AHCI, all 6 as IDE, or 1-4 as AHCI and 5-6 as IDE (the BIOS says something about using the latter to allow the system to recognize optical drives for installing the OS).
• If I set ports 1-4 to AHCI and ports 5-6 to IDE, the system also boots successfully.
• If I set all 6 motherboard SATA ports to AHCI mode, the system does not boot when all 10 drives are connected. To get it to boot, I can either (A) remove one or more SATA cables from the motherboard's ports, or (B) remove one of the Rocket 620 HBAs from its PCIe 2.0 slot (or just both SATA cables from it, still leaving the card in the slot). I was wrong about the latter - I thought I'd tried it with no success, but apparently that was not the case!
• One approach I kind of expected to work that does NOT work is to remove one SATA cable from each Rocket 620 HBA (8 drives total). I have to remove both SATA cables from either Rocket 620 HBA.

FWIW, I noticed that the BIOS for the 2 Rocket 620 HBAs list "Mode: CPU AHCI" so I assume that the Rockets can only run (or at least run by default) in AHCI mode.

So, with all that in mind, some new questions:

• Does this new information reveal anything more about the cause?
• Should I just give up and leave the motherboard SATA ports in IDE mode so I can boot this thing already? Or is AHCI worth more troubleshooting?
• Does this suggest any particular component(s) as the culprit, or safely rule anything out? I ask because I've already submitted a trouble ticket to HighPoint, but I'd like to also submit one to Asus and/or FreeNAS if that might help.

 

[Mr. Bungle]

One Rocket 620 in, all SATA ports filled on MB - It doesn't find the IDE drive to boot.

One Rocket 620 in, 5 of the 6 SATA ports filled on MB - It DOES find the IDE drive to boot from. Once booted it sees all 7 drives.

No Rocket Card, 6 of 6 SATA ports filled on MB - it does find the IDE drive to boot from.

I actually gave up on the IDE/CompactFlash adapter for now. I just wanted to remove it from the equation. I've been booting exclusively from the USB thumb drive for some time now. If I can get to the bottom of this problem, I'll probably try the CF card again, but if it doesn't just work, I'll just stick with USB.

AHCI makes no difference.

Looks like it makes all the difference in the world, actually - thanks for suggesting this, as it prompted me to play with those settings and make some more progress on the issue! (See my last post.)

IDE Banner appears under all scenarios.

Not sure; but I do always see all of the connected drives either on the initial BIOS screen (if the drives are in IDE mode) or on the next screen (if the drives are in AHCI mode). I haven't seen any cases where I drive that I have plugged in to the motherboard has failed to be recognized.

You see the Banner(s) for the Rocket Cards under all scenarios ( where a card is in )

Correct.

Do you see any indication that the boot process is dying after talking to IDE or USB ( the USB light flashing, CF card adapter lights indicate anything? ). Any indicators of bootstrapping?

I haven't dug into this yet; I'll poke around here next step.

Do you have a Windows/Linux install that you can try on the CF/IDE (eliminates the possibility that it is FreeBSD).

I don't, but I can probably set one up. I might try using my spare IDE DVD-ROM drive to try to boot from the Windows 7 DVD instead of FreeNAS. I don't have an extra IDE HDD after all.

Thanks for your help! Interested to hear if this gives you or anyone some more clues.

 

[Mr. Bungle]

Here's the FreeNAS ticket: http://support.freenas.org/ticket/838

 

[GregN]

Sounds like you have found a sol'n, run ports 5 & 6 as IDE.


Trying to see if FreeBSD is starting to boot, then failing, try any linux live cd or even FreeNAS on CD, to see if it hits the drive with a single IDE drive in.

My theory here, given your latest findings, is that the BIOS is unable to find the boot drive with all of the sata ports running ahci.

Just a guess, A possible explanation is that your IDE drive was running in slave mode, without either an open Sata port or another IDE port as a master the BIOS could not boot from the slave. It wants a pair of IDE ports.
This shouldn't be the case, but fits the symptoms.

Have you attempted to contact Asus for a possible explanation?

 

[GregN]

Found This:

Boot Sequence Issue, BIOS Setting for Advanced PnP

What is the setting for Adv. PnP in your BIOS?

This looks like it might be worth investigating.

 

[GregN]

Here's the FreeNAS ticket: http://support.freenas.org/ticket/838

Dude,

You're going to someone else for help, double timing us? I feel so used!