Entware broken after reboot on 380.64_2

[Jay Culley]

I could not find another thread for this so here goes: Entware and most things I install do not work after rebooting running FW 380.64_2. This did not happen on earlier releases of the FW. I have an ARM proc, ASUStek AD5300 with USB 2 thumb drive formatted EXT3. Not doing anything fancy, I just installed Entware with entware-setup.sh then install tcpdump, nano, mc, iftop, htop. Then after I reboot nano works but nothing else does, opkg fails, tcpdump, mc.

Anyone else seen this?

 

[swetoast]

Nano is included in Merlin firmware so my guess is that your usb partition isnt mounted properly

 

[Jay Culley]

Nano is included in Merlin firmware so my guess is that your usb partition isnt mounted properly

Well it's EXT2 mounted as sda5. Looking at /proc/mounts it shows
/dev/sda5 /tmp/mnt/Lexar ext2 rw,nodev,relatime,user_xattr 0 0

It is shared via samba so I can see it in windows.
/etc/fstab is empty, busybox does not have fsck so i ran the health check in the gui, it ran but did not display any information, so I looked at syslog and it contains:
USB partition unmounted from /tmp/mnt/Lexar fail. (return -1, Bad file descriptor)

is it OK if fstab is empty?

Thanks -N00b

 

[latenitetech]

Well it's EXT2 mounted as sda5. Looking at /proc/mounts it shows
/dev/sda5 /tmp/mnt/Lexar ext2 rw,nodev,relatime,user_xattr 0 0

It is shared via samba so I can see it in windows.

If opkg isn't found, yet Samba is sharing the volume, then it's likely the problem is the entware directory isn't being linked to '/tmp/opt' properly in '/jffs/scripts/post-mount.' A quick test of this is run 'ls /opt'. Do you see the entware directory contents listed out?

If no, then I'm wondering if you're referencing your "entware partition" by it's device name (/dev/sda5) rather than its volume label (/tmp/mnt/Lexar) in '/jffs/scripts/post-mount'. The latter is by far the safest way to go as the "dev" name can change between boots, while the label will always be the same.

So open your '/jffs/scripts/post-mount' file, and look at the block of code that should look something like:

if [ "$1" = "/tmp/mnt/Lexar" ] ; then
    ln -nsf /tmp/mnt/Lexar/entware /tmp/opt
fi

If you're seeing it reference "/dev/sda#" (probably not sda5), then change it to reference the label path instead and manually execute that script ("./post-mount /tmp/mnt/Lexar" from the /jffs/scripts directory) to see if it fixes things.

/etc/fstab is empty, busybox does not have fsck so i ran the health check in the gui, it ran but did not display any information, so I looked at syslog and it contains:
USB partition unmounted from /tmp/mnt/Lexar fail. (return -1, Bad file descriptor)

is it OK if fstab is empty?

fstab is not used by default in this environment, so being empty is normal.

The firmware does include a full set of fsck commands, but they're named differently based on the file system to be checked. For Linux filesystems (EXT#) the command is 'e2fsck' (they're all located in /usr/sbin).

Hopefully that helps. Let us know what you find.

 

[Jay Culley]

Thanks for the quick reply! I think the link in post-mount script looks ok, it is referencing the volume label:

if [ "$1" = "/tmp/mnt/Lexar" ] ; then
ln -nsf $1/entware /tmp/opt
fi

I think I will run e2fsck later tonight when I can boot and mess with things. If that does not help I am thinking about replacing the thumb drive with a new one. I do get other errors in the syslog like for SPNEGO: Rejecting attempt at SPNEGO session setup when it was not negotiated. Probably not related but I will try to eliminate it.

 

[latenitetech]

Thanks for the quick reply! I think the link in post-mount script looks ok, it is referencing the volume label:

if [ "$1" = "/tmp/mnt/Lexar" ] ; then
ln -nsf $1/entware /tmp/opt
fi

What was the result of 'ls /opt'? Did it show the contents of the entware directory? If yes, then I would certainly expect 'opkg' to be in the path. So when you earlier said opkg didn't work, was it not found, or was it found but didn't behave as expected? Just trying to sort out if the problem is the entware directory not linked properly, or it's linked, but potentially corrupt.

 

[Jay Culley]

Sorry @latenitetech I reinstalled entware before starting this thread. So, as of now the link for /opt looks right

ls /opt
bin etc lib sbin share tmp usr var

 

[latenitetech]

Sorry @latenitetech I reinstalled entware before starting this thread. So, as of now the link for /opt looks right

ls /opt
bin etc lib sbin share tmp usr var

Hmmm, well based on the symptoms you shared, I took my best shot. The other thing I'll suggest is perhaps the default timeout in '/jffs/scripts/services-start' is too short. If that's the case, you'll see this error in the log file on boot: "Could not start Entware." It's a trivial fix if you're hitting that, just edit the line "i=30" to something like "i=60." The actual value isn't all that important so long as it's long enough to not time out during your normal boot cycle. But, that error wouldn't cause opkg to not work. It would only result in entware-installed services from starting upon boot.

So if things are still not working, you'll have to share more specifics on what's failing. A command not found (like opkg), or opkg is found and executes but errors out (what specifically is the error)? I highly doubt there is anything specific to the 380.64_2 firmware that's causing your problem. I (and many others) are running that firmware version with entware with no issues.

 

[NeoDragon]

Had this problem a while ago. Adding a sleep time in service-start and post-mount inside the jffs partition did the trick.

at the beginning of services-start and post-mount, add sleep 60
Save & Reboot.

 

[Jay Culley]

i will reboot the router and that will cause the failure then start digging in the logs. Before that I am going to put a new USB thumb drive on it, install the beta of the FW, 380.65_0 I think it is so we can start fresh. One thing that may be causing the problem is that I used partition magic to format the USB drive last time. I will use Busybox this time and use EXT3.

 

[Jay Culley]

SOLVED. It was filesystem corruption on the USB thumb drive. Like I said earlier, I ran a health check from the GUI and it did not return any info. There were also some SPNEGO errors from samba in the syslog. So i bought a new 32 GB drive, inserted it in the USB 2.0 connector and made an EXT3 filesystem with BusyBox right there on the router. Then I flashed the firmware to 380.65_beta3. After the reboot it was mounted as SDA1. I installed entware, tcpdump, and some other stuff, booted and now entware is retained and works after booting.

Before I had formatted the USB EXT2 with Partition Manager on a Windows machine. I think Partition Manager may be the root cause.

 

[thelonelycoder]

SOLVED. It was filesystem corruption on the USB thumb drive. Like I said earlier, I ran a health check from the GUI and it did not return any info. There were also some SPNEGO errors from samba in the syslog. So i bought a new 32 GB drive, inserted it in the USB 2.0 connector and made an EXT3 filesystem with BusyBox right there on the router. Then I flashed the firmware to 380.65_beta3. After the reboot it was mounted as SDA1. I installed entware, tcpdump, and some other stuff, booted and now entware is retained and works after booting.

Before I had formatted the USB EXT2 with Partition Manager on a Windows machine. I think Partition Manager may be the root cause.

This works good on Windows: MiniTool Partition Wizard Free

 

[Jay Culley]

I spoke too soon. After 3 or 4 boots its started doing it again. Entware and anything I installed from the repository fails. I loaded the latest beta, 380.65_beta3 fails on that as well as 380.64_2. I guess ext3 changed between fw releases, I know busyBox did I can see the mount point /dev/sda1 but after flashing the FW to beta3 I cannot write to that drive, chmod 777 /dev/sda1 (which I did as soon as I created the filesystem) now returns no such file or directory (chmod by itslef returns usage instructions). fdisk will not let me delete the partition, says it failed with no other info. I removed the USB stick an reinserted it, now no matter what I do it will not mount, not even after a reboot.

I am lucky to have internet access. I could throw the whole router in the f***-it bucket and load the stock FW. I am not real sure why I bought the AC5300 anyway, MU-MIMO, yeah right.

 

[swetoast]

So here is an idea reformat the usb drive to ext4 (most practical to use these days) and reset to factory settings and redo the installation of the firmware.

https://github.com/RMerl/asuswrt-merlin/wiki/Entware

thats what i would do atleast before just trying to fix a broken installation

 

[john9527]

Hate to ask the obvious, but have you tried a different USB stick? They do sometimes fail in strange ways.

 

[Dragon]

Hate to ask the obvious, but have you tried a different USB stick? They do sometimes fail in strange ways.

They did, yes, I quote "So i bought a new 32 GB drive, inserted it in the USB 2.0 connector and made an EXT3 filesystem"

 

[john9527]

They did, yes, I quote "So i bought a new 32 GB drive, inserted it in the USB 2.0 connector and made an EXT3 filesystem"

Ahhh....thanks...I missed that one. Now, my next question, was it another Lexar drive? I had a Lexar 32GB USB3 stick that would disconnect randomly when connected to the router. Worked fine under windows.

 

[Jay Culley]

Sadly enough, it is another learn. I cannot mess with the router until later tonight but I have a new 32GB SanDisk I will try next. I am going back to FW 380.64_2 that Entware worked on, I will format it with that OS

 

[NeoDragon]

Have you tried what I've posted earlier? Add a sleep timer.

services-start

#!/bin/sh
sleep 60
RC='/opt/etc/init.d/rc.unslung'
i=30
until [ -x "$RC" ] ; do
  i=$(($i-1))
  if [ "$i" -lt 1 ] ; then
    logger "Could not start Entware"
    exit
  fi
  sleep 1
done
$RC start

Add a sleep timer to post-mount as well if you are using ab-solution

post-mount

#!/bin/sh
sleep 30
# generated by AB-Solution 1.07
if [ -d "/tmp/mnt/Entware" ];then
   sleep 2
   service restart_dnsmasq
   logger "AB-Solution added entries via post-mount"
fi

 

[Jay Culley]

Man i love Asustek! I wish I'd bought this when it first came out! One reason is that I do believe that Asus routers are brick-proof. I did the FW rescue mode for the first time tonight and it did not take any longer than a normal firmware upgrade. After that I installed FW 380.4_1. Entware did not give me any problems until 380.64_2. Imported the config, setup jffs and restored my customizations. Then I formatted a new 32GB Sandisk Cruzer USB thumb drive EXT4. Installed Entware no problem. Installed tcpdump, htop, iftop, mc and booted. Let it run a bit and booted a few more times and Entware survived. Then I installed ab-solution with the #2 medium host file. For the first time I was able to follow the log! WooHoo!

Thank you all very much. @NeoDragon, if it starts twitching, your sleep timer code is the first thing I am going to try, Thank you. @thelonelycoder thanks for all the input. I enabled (wc) to get weekly emails, you refer to one stat ads, is this the number of ads blocked? @swetoast @john9527 @latenitetech you guys all obviously know your stuff, I just hope I am not posting in this thread again in a few day because entware stops working! If it does NOT fail then I am going to flash to the 380.65_beta3 again to see if that fails, I suspect it will but who knows.

The only other optional thing I have not done with ab-solution is pixelserv (ps). That is just for ssl/tls blocking right?