Entware broken after reboot on 380.64_2

[swetoast]

go to the ab-solutions thread and ask

 

[Jay Culley]

Well Entware died again, twice. In test 1 I did nothing. Entware was working, went to sleep, then 6 hours later it had failed. There were no reboots or anything happening other than very light traffic. The USB thumb drive had unmounted. I found this in the syslog:
EXT4-fs error (device sda1): __ext4_get_inode_loc:4908: inode #7077889: block 14155778: comm hotplug: unable to read itable block

In test 2 after reinstalling Entware the drive unmounted after 4 reboots, killing Entware again. I had installed AB-Solution and added the post-mount and services-start scripts mentioned by @NeoDragon I had also flashed the FW back to 380.64_1 that never had a problem. The first time I saw this was with FW 380.64_2

I am preparing for test 3. I have never had a swap partition. I am currently formatting the drive with a 1 GB linux swap logical partition and with the remaining space, about 30 GB, a primary EXT4 partition. The AC5300 has 500MB of RAM so a 1 GB swap file should be fine. I am not sure that a $10.00 Lexar USB drive is suitable for a swap partition, so if the lack of a swap file is the problem then I am thinking I might need something like a WD My passport for $50.00.

 

[tomsk]

Are you sure its not the method you are using to partition the stick?...

This works good on Windows: MiniTool Partition Wizard Free

Gparted works pretty good too

http://gparted.sourceforge.net/download.php

 

[Jay Culley]

I used MiniTool Partition Wizard free. I first created the logical 1GB linux swap partition, then a primary ext4 partition with the remaining space. It added the 2 jobs on the left side and it added in a job to mount or do something prior to creating the linux swap partition. Then I clicked apply and all 3 jobs ran fine. I stuck it in the 5300 and it's all there, just like I expected. I then installed Entware, no problem.

 

[tomsk]

I used MiniTool Partition Wizard free. I first created the logical 1GB linux swap partition, then a primary ext4 partition with the remaining space. It added the 2 jobs on the left side and it added in a job to mount or do something prior to creating the linux swap partition. Then I clicked apply and all 3 jobs ran fine. I stuck it in the 5300 and it's all there, just like I expected. I then installed Entware, no problem.

Sounds like you did everything right... but from your logs it definitely looks like a filesystem problem. Maybe you can try with a simple single EXT2 ( no journaling like EXT3/4) partition and see if it behaves.With 500Mb of RAM onboard you shouldn't really have to worry about swap unless you plan on installing some of the really memory intensive Entware packages.

 

[Jay Culley]

I have tried entware on ext2,3, and now 4. I just rebooted and I now have 1 gig swap space running, have not seen it used yet but it's there. OF course the reboot killed entware, but the drive was still mounted. Just reinstalled entware, the router will boot in 6 hours, we shall see if it lives then,

I wonder if I install ReiserFS if that will work! I doubt if he gets anything for that in prison?!

 

[tomsk]

You have re-flashed router firmware versions back and forth, tried different USB sticks with different flavours of linux filesystems. If you are creating the filesystems correctly then you have to start wondering about the router hardware itself. Do you have any adverse external conditions that might cause this? ( I'm thinking unstable power causing brownouts, electrical interference, etc) I notice in another post you mention a USB hard drive connected on the USB3 port. Does the USB2 stick behave if this isn't connected?

 

[thelonelycoder]

I did not check if you did @Jay Culley but did you format your jffs partition lately?
In Administration / System tick 'Format JFFS partition at next boot'.
This will erase content and reformat the partition.
You will have to start over new with your Entware and AB stuff.

 

[latenitetech]

You have re-flashed router firmware versions back and forth, tried different USB sticks with different flavours of linux filesystems. If you are creating the filesystems correctly then you have to start wondering about the router hardware itself. Do you have any adverse external conditions that might cause this? ( I'm thinking unstable power causing brownouts, electrical interference, etc) I notice in another post you mention a USB hard drive connected on the USB3 port. Does the USB2 stick behave if this isn't connected?

I have to agree with @tomsk on this. I think you have a fundamental hardware incompatibility or failure issue going on. Before going to the trouble of reinstalling entware and applications again and again, I'd beat the heck out of the USB stick with diskcheck (e.g. e2fsck -cfv), and if that passes, just do some massive read/write to the drive and see if it fails. A properly performing drive will read/write continuously for hours/days/years without a single failure, and you're seeing it in a matter of a few hours with light activity.

If you can reliably get it to fail with a simple e2fsck or read/write activity, then you can start to narrow down the source by removing other devices, swapping to the other USB port, etc. Through trial and error you should be able to identify the culprit. For the most part, this stuff just works (for most of us), so there is something unique to your device or setup that is causing all this grief.

 

[Jay Culley]

Entware did fail again after the scheduled reboot, but the drive is still mounted. @tomsk and @latenitetech you read my mind. It is a fairly new router from Asus, new AC5300 technology, I have always wondered why the guy was selling it on eBay, which is where I purchased it from. I would think if it did have a hardware problem it would still be under warranty. He said it was just overkill for his apartment, who knows. The router is on a shelf I installed near the center of the house about 7 feet off the floor. Power is connected to an APC 450va NEW UPS. No sunlight hits it, no cats, no other sources of RF in the room except cell phones. @thelonelycoder I have only formatted jffs once, when it was created December 3rd. The other drive on the USB port is formatted NTFS and has never had a problem except for a couple errors in syslog about 4 or 5 times a week, bot tagged smbd[2052]

error: smbd/sesssetup.c:1355(reply_sesssetup_and_X)
error: reply_sesssetup_and_X: Rejecting attempt at SPNEGO session setup when it was not negotiated.

These are all great suggestions so here are the next tests:

1. REmove usb3 drive, reboot, install entware reboot
2: leav usb3 drive disconnected and format /jffs
3. Leave usb3 drive disconnected and run e2fsck -cfv /dev/sda. I just ran this and it failed:/dev/sda is in use. e2fsck: Cannot continue, aborting so I will probably need to reboot to run that.

 

[latenitetech]

...
3. Leave usb3 drive disconnected and run e2fsck -cfv /dev/sda. I just ran this and it failed:/dev/sda is in use. e2fsck: Cannot continue, aborting so I will probably need to reboot to run that.

The partition must be unmounted (umount /dev/sda) before you can run e2fsck. And 'umount' will fail if there is anything active on the drive, such as entware applications started from rc.unslung. So kill anything running from entware on that drive, then umount it, then run e2fsck.

Edit: you may also need to kill samba, as it may prevent umount from working. (service stop_samba)

 

[Jay Culley]

well so far I have disconnected the 1TB drive on the USB3 port, upgraded the fw to the brand new 380.65. Every single time I have upgraded the FW it has killed entware, and normally reboots kill it. Since disconnecting the drive on the USB3 port entware has survived 2 fw upgrades (i did the beta3 first not knowing the release came out last night) and 2 reboots. This has never happened.

Health check on the usb2 drive takes about 30 seconds and returns no results. Throws out a couple syslog entries from the monitor tag for manual scan and idle but no other errors. I am going to run the e2fsck checks on it but I think the other drive is causing the issue.

 

[tomsk]

well so far I have disconnected the 1TB drive on the USB3 port, upgraded the fw to the brand new 380.65. Every single time I have upgraded the FW it has killed entware, and normally reboots kill it. Since disconnecting the drive on the USB3 port entware has survived 2 fw upgrades (i did the beta3 first not knowing the release came out last night) and 2 reboots. This has never happened.

Health check on the usb2 drive takes about 30 seconds and returns no results. Throws out a couple syslog entries from the monitor tag for manual scan and idle but no other errors. I am going to run the e2fsck checks on it but I think the other drive is causing the issue.

Do a search for other threads on NTFS issues that others have had in the past..... i believe Merlin uses newer drivers for NTFS than stock ASUSwrt. Its just media on the hard drive?

 

[Jay Culley]

I was writing the IP traffic history to the ntfs drive so I changed that to the thumb drive on usb2. Other than that the ntfs drive is mostly media, I had it shared via FTP and Samba. Leaving it disconnected is no biggie, it is a backup of the original media files, i have a copy job that backs stuff up from the main NAS to that drive, I will just suspend the backups until I figure out what I am going to do with it. EXT4 will probably play better with Samba, may just format it that then share it back out with samba.

I had seen the smbd errors in the syslog ever since I connected the ntfs drive and I could never resolve them or determine any negative effects (scroll up for errors) but I may have just found the negative effects!

I have installed AB-Solution and it went flawless, after a reboot even!

 

[Jay Culley]

It's been 6 hours or more and all is well. My family wants to kill me but entware and all software including ab-solution is working. No errors. I think I can safely say the root cause was the 1TB NTFS drive I had connected to the USB3 port.

the family is griping that dns lookups are slower, I said for now deal with it, I still think the PROs out weigh the CONS with ab-solution.

Now, what to do with it. Jettison NTFS and format it EXT4? put all the backup media back on it, share it out with samba again?