Segmentation Faults/Tainted Binaries/Inconclusive Disk Scans and other woes... ;)

[Viktor Jaep]

I have a feeling my AC86U is on its last legs, or my flashdrive is on its last legs... either way, looking for input on what I can do with this. I first started noticing this when I ran nmap, and I would get this response: "Segmentation Fault".

This would follow in the logs:

Dec 11 10:50:05 kernel: pgd = ffffffc0093d7000
Dec 11 10:50:05 kernel: [8057364a7e] *pgd=0000000000000000, *pud=0000000000000000
Dec 11 10:50:05 kernel: CPU: 0 PID: 12253 Comm: 7za Tainted: P           O    4.1.27 #2
Dec 11 10:50:05 kernel: Hardware name: Broadcom-v8A (DT)
Dec 11 10:50:05 kernel: task: ffffffc009287500 ti: ffffffc009bec000 task.ti: ffffffc009bec000
Dec 11 10:50:05 kernel: PC is at 0x7fadb0623c
Dec 11 10:50:05 kernel: LR is at 0x7fadb07cc0
Dec 11 10:50:05 kernel: pc : [<0000007fadb0623c>] lr : [<0000007fadb07cc0>] pstate: 20000000
Dec 11 10:50:05 kernel: sp : 0000007fcdc801d0
Dec 11 10:50:05 kernel: x29: 0000007fcdc801f0 x28: 0000007fadaee6b0
Dec 11 10:50:05 kernel: x27: 0000000000000000 x26: 0000007fadb28000
Dec 11 10:50:05 kernel: x25: 0000000000000001 x24: 0000007fadb29cd8
Dec 11 10:50:05 kernel: x23: 0000007fcdc803c8 x22: 0000008057364a7e
Dec 11 10:50:05 kernel: x21: 0000007fadb28000 x20: 0000000000000000
Dec 11 10:50:05 kernel: x19: 000004f600000101 x18: 0000000000000001
Dec 11 10:50:05 kernel: x17: 0000000000000064 x16: 000000000000000c
Dec 11 10:50:05 kernel: x15: 0000000000554e47 x14: 0000000000000000
Dec 11 10:50:05 kernel: x13: 0000007fadb2d008 x12: 0000000000000041
Dec 11 10:50:05 kernel: x11: 0000000000000028 x10: 0101010101010101
Dec 11 10:50:05 kernel: x9 : 0000000000000000 x8 : 7f7f7f7f7f7f7f7f
Dec 11 10:50:05 kernel: x7 : 0000000000000000 x6 : 0000000000000001
Dec 11 10:50:05 kernel: x5 : 0000000000000000 x4 : 0000007fadb29cd8
Dec 11 10:50:05 kernel: x3 : 0000007fadb28358 x2 : 0000007fcdc803c8
Dec 11 10:50:05 kernel: x1 : 0000007fadb28000 x0 : 0000008057364a7e

But now I'm seeing these exact same errors regularly throughout the day... I'm not seeing any degradation or loss in functionality (yet), but it's definitely worrisome that something is about to imminently fail. 7za seems like a 7zip binary of some sort. Recently I had some other issues with the DIG command no longer working due to some corrupt files, and had to uninstall/reinstall various packages to get that working again. Again, most likely due to my flashdrive about to bite the dust.

Also... when I click on the flashdrive health scanner, click "rescan", it runs... but then doesn't display any results in the results window. Another sign?? Any other way to try to get to these results, or seeing if there's a way to fix the drive without dealing with downtime or having to reload?

Good thing I have a GT-AX6000 sitting here ready and waiting to be installed, with a nice Samsung SSD to function as my external jffs storage... but I want to keep this 86U running as long as possible in the interim until I have some time to make the switch...

Thanks in advance for your advice...

 

[ColinTaylor]

Have you installed the "Disk check script" in amtm? If so, just reboot the router and check the disk check log file for errors.

You could just run 7za from the command line and see if it crashes: 7za -?

 

[Viktor Jaep]

Have you installed the "Disk check script" in amtm? If so, just reboot the router and check the disk check log file for errors.

Thanks @ColinTaylor ... I do... and it comes up clean. I had rebooted earlier this morning:

Sun Dec 11 10:20:44 EST 2022 Probing 'ext4' on device /dev/sda1
 Running disk check v3.1, with command 'e2fsck -p' on /dev/sda1
 ASUS-USB: clean, 1530/476720 files, 1096464/1903229 blocks
 Sun Dec 11 10:20:45 EST 2022 Disk check done on /dev/sda1
 ---------------------------------------------------
 END FILE

Just tried doing this:

opkg install --force-reinstall nmap

And the error changes to nmap now being tainted:

Dec 11 11:08:55 kernel: CPU: 1 PID: 22705 Comm: nmap Tainted: P           O    4.1.27 #2
Dec 11 11:08:55 kernel: Hardware name: Broadcom-v8A (DT)
Dec 11 11:08:55 kernel: task: ffffffc00806f500 ti: ffffffc0091fc000 task.ti: ffffffc0091fc000
Dec 11 11:08:55 kernel: PC is at 0x7f9b70323c
Dec 11 11:08:55 kernel: LR is at 0x7f9b704cc0
Dec 11 11:08:55 kernel: pc : [<0000007f9b70323c>] lr : [<0000007f9b704cc0>] pstate: 20000000
Dec 11 11:08:55 kernel: sp : 0000007fcdbe3620
Dec 11 11:08:55 kernel: x29: 0000007fcdbe3640 x28: 0000007f9b6096b0
Dec 11 11:08:55 kernel: x27: 0000000000000000 x26: 0000007f9b726380
Dec 11 11:08:55 kernel: x25: 0000000000000001 x24: 0000007f9b724278
Dec 11 11:08:55 kernel: x23: 0000007fcdbe3818 x22: 0000008044e7fa7e
Dec 11 11:08:55 kernel: x21: 0000007f9b726380 x20: 0000000000000000
Dec 11 11:08:55 kernel: x19: 000004f600000101 x18: 0000000000000001
Dec 11 11:08:55 kernel: x17: 0000000000000064 x16: 000000000000000c
Dec 11 11:08:55 kernel: x15: 0000000000554e47 x14: 0000000000000000
Dec 11 11:08:56 kernel: x13: 0000007f9b72a008 x12: 0000000000000041
Dec 11 11:08:56 kernel: x11: 0000000000000028 x10: 0101010101010101
Dec 11 11:08:56 kernel: x9 : 0000000000000000 x8 : 7f7f7f7f7f7f7f7f
Dec 11 11:08:56 kernel: x7 : 0000000000000000 x6 : 0000000000000001
Dec 11 11:08:56 kernel: x5 : 0000000000000000 x4 : 0000007f9b724278
Dec 11 11:08:56 kernel: x3 : 0000007f9b7266d8 x2 : 0000007fcdbe3818
Dec 11 11:08:56 kernel: x1 : 0000007f9b726380 x0 : 0000008044e7fa7e

 

[ColinTaylor]

Maybe mixed levels. Try opkg update; opkg upgrade

 

[Viktor Jaep]

You could just run 7za from the command line and see if it crashes: 7za -?

Same kind of crash:

ViktorJp@RT-AC86U-BE10:/tmp/home/root# 7za -h
Segmentation fault
ViktorJp@RT-AC86U-BE10:/tmp/home/root#

 

[Tech9]

Also... when I click on the flashdrive health scanner, click "rescan", it runs... but then doesn't display any results in the results window.

I believe this GUI Health Scanner is only checking FAT/NTFS. You have your USB stick on ext4.

 

[Viktor Jaep]

Maybe mixed levels. Try opkg update; opkg upgrade

Unfortunately no difference. I just tried reinstalling nmap again right after this command, and still getting a "segmentation fault".

 

[Viktor Jaep]

I'm sure it has to do with some base packages that need to be uninstalled/reinstalled to get past some corruption in one of them... <sigh>

So now both 7za and nmap are complaining about tainted binaries in the syslog. I can live with it.

I thought I may have had success by running, "opkg install --force-reinstall 7-zip" but, alas no dice. All 3 of these binaries (7z, 7za and 7zr) all come up with segmentation faults when I run them.

 

[badtoast]

Hi @Viktor Jaep, I am running into a similar issue you have described above, except mine has to do with sqlite3 and ntpmerlin:

Jan 17 21:19:08 kernel: potentially unexpected fatal signal 11.
Jan 17 21:19:08 kernel: CPU: 1 PID: 25392 Comm: sqlite3 Tainted: P           O    4.1.27 #2
Jan 17 21:19:08 kernel: Hardware name: Broadcom-v8A (DT)
Jan 17 21:19:08 kernel: task: ffffffc00acee0c0 ti: ffffffc016b38000 task.ti: ffffffc016b38000
Jan 17 21:19:08 kernel: PC is at 0x7f7fa59d34
Jan 17 21:19:08 kernel: LR is at 0x7f7fa52be8
Jan 17 21:19:08 kernel: pc : [<0000007f7fa59d34>] lr : [<0000007f7fa52be8>] pstate: 20000000
Jan 17 21:19:08 kernel: sp : 0000007fc936d0a0
Jan 17 21:19:08 kernel: x29: 0000007fc936d0a0 x28: 0000007f7f593f08
Jan 17 21:19:08 kernel: x27: 0000007f7f580130 x26: 0000007f7fa7fa20
Jan 17 21:19:08 kernel: x25: 000000000000bca0 x24: 0000007f7fa7e6f8
Jan 17 21:19:08 kernel: x23: 0000000000000001 x22: 0000007f7f57d000
Jan 17 21:19:08 kernel: x21: 0000007f7fa78a60 x20: 0000007f7f590f68
Jan 17 21:19:08 kernel: x19: 0000007f7f58bc40 x18: 0000000000000001
Jan 17 21:19:08 kernel: x17: 0000000000000064 x16: 000000000000000c
Jan 17 21:19:08 kernel: x15: 0000000000554e47 x14: 0000000000000000
Jan 17 21:19:08 kernel: x13: 0000007f7fa7f008 x12: 0000000000000040
Jan 17 21:19:08 kernel: x11: 0000000000000030 x10: 0101010101010101
Jan 17 21:19:08 kernel: x9 : 0000000000000000 x8 : 00000000000000e2
Jan 17 21:19:08 kernel: x7 : 45ff2f2d325e4242 x6 : 0000000000008000
Jan 17 21:19:08 kernel: x5 : 0000007f7f593f20 x4 : 0000007f7f593f08
Jan 17 21:19:08 kernel: x3 : 0000007f7f5feed0 x2 : 0000007f7fa7fa20
Jan 17 21:19:08 kernel: x1 : 0000100000080220 x0 : 0000007f9f5dae58

Based on this suggestion from @Jack Yaz, I ran:

opkg update
opkg remove --autoremove sqlite3-cli
opkg install sqlite3-cli

Unsurprisingly, when removing sqlite3-cli, the stream of endless errors in my System Log ceases. But when I install sqlite3 again or if I try to run ntpmerlin, it attempts to create database table indexes, but does nothing beyond saying "Creating database table indexes...":

badtoast@router:/tmp/mnt/PNY_30GB/entware/share/ntpmerlin.d# ntpmerlin
Installing required version of sqlite3 from Entware

Downloading https://bin.entware.net/aarch64-k3.10/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Installing sqlite3-cli (3400000-1) to root...
Downloading https://bin.entware.net/aarch64-k3.10/sqlite3-cli_3400000-1_aarch64-3.10.ipk
Configuring sqlite3-cli.
Creating database table indexes...

When I attempt to view the /opt/share/ntpmerlin.d/ntpstats.db, I too get a "Segmentation fault" message:

badtoast@router:/tmp/mnt/PNY_30GB/entware/share/ntpmerlin.d# sqlite3 /opt/share/ntpmerlin.d/ntpdstats.db
Segmentation fault

Any luck with your situation or any suggestions?

 

[dosborne]

You could always try running without any scripts or USB stick and see how stable it is

Never had a need for them myself. I offload as much as I can to other devices such as a raspberry pi so that my router can focus all resources on the basics, routing.

 

[badtoast]

True. I've been running the router without sqlite3-cli for approx 1/2 hour now and the logs are not showing that "tainted" message anymore. So far so good, but now I'm wondering, does the native ntp server perform similarly?

 

[jj22038]

"Tainted" has nothing to do with the program that crashed (7za, nmap, etc.). That is a reminder that the kernel is not pure open-source, something that the GPL originators feel very strongly about. Every stack trace will say "Tainted". Because of the binary blobs that Merlin has to work with, the Asus kernel will always be tainted.

 

[badtoast]

I think I may have fixed my issue. After having done the steps I mentioned above, I then uninstalled ntpmerlin, then unmounted/unplugged the USB flash drive. Then re-installed the latest firmware 386.9 again. Then restarted the router. Then restarted again with the USB flash drive plugged back in. Made sure that the router functioned normally with my other scripts running (Skynet, scMerlin). Finally, I reinstalled ntpmerlin via amtm. All seems back to normal - no more System Logs filled up with "fatal signal" and "tainted" message. Hope it stays this way!

 

[Viktor Jaep]

Any luck with your situation or any suggestions?

Frankly, after going to the AX6000 with a new SSD, I also formatted a newer (more namebrand) flashdrive when I reflashed everything from scratch on my old AC86U. I think it all just needed a fresh start. My old flashdrive was just hobbling along as more and more stuff started getting corrupt over time.

"Tainted" has nothing to do with the program that crashed (7za, nmap, etc.). That is a reminder that the kernel is not pure open-source, something that the GPL originators feel very strongly about. Every stack trace will say "Tainted". Because of the binary blobs that Merlin has to work with, the Asus kernel will always be tainted.

For some reason, when something came up as "tainted" in my logs, when executing these, it would also generate "segmentation faults". Thus I equated "tained" with "corrupted".

I think I may have fixed my issue. After having done the steps I mentioned above, I then uninstalled ntpmerlin, then unmounted/unplugged the USB flash drive. Then re-installed the latest firmware 386.9 again. Then restarted the router. Then restarted again with the USB flash drive plugged back in. Made sure that the router functioned normally with my other scripts running (Skynet, scMerlin). Finally, I reinstalled ntpmerlin via amtm. All seems back to normal - no more System Logs filled up with "fatal signal" and "tainted" message. Hope it stays this way!

Nice work! You may be lucky, but if these come back at some point in the future, you may need to face having to reformat your flashdrive and reinstall entware/addons. I wish these linux distros had a "SFC /scannow" type of command like Windows does to help fix system file corruption issues. ;(

 

[Martinski]

... I wish these linux distros had a "SFC /scannow" type of command like Windows does to help fix system file corruption issues. ;(

The "fsck" & "e2fsck" commands are available on Linux. More info here:

fsck(8): check/repair file system - Linux man page

fsck is used to check and optionally repair one or more Linux file systems. filesys can be a device name (e.g. /dev/hdc1, /dev/sdb2), a mount point (e.g. ...

linux.die.net

e2fsck(8): check ext2/ext3/ext4 file system - Linux man page

e2fsck is used to check the ext2/ext3/ext4 family of file systems. For ext3 and ext4 filesystems that use a journal, if the system has been shut down ...

linux.die.net

 

[badtoast]

Nice work! You may be lucky, but if these come back at some point in the future, you may need to face having to reformat your flashdrive and reinstall entware/addons. I wish these linux distros had a "SFC /scannow" type of command like Windows does to help fix system file corruption issues. ;(

I might have spoken too soon. I went to bed last night feeling satisfied, but woke up to this morning with more of the same errors mentioned before.

The good new is: earlier this afternoon I picked up a new SanDisk Ultra 32GB USB 3.0 flash drive, took out the old USB flash drive out, and installed entware, Skynet, and then ntpmerlin on the new one. Finally, all is well!

 

[Viktor Jaep]

The "fsck" & "e2fsck" commands are available on Linux. More info here:

It looks like these would just fix errors in the file system, but wouldn't repair actual system files or binaries if they were corrupted, right?

 

[Martinski]

It looks like these would just fix errors in the file system, but wouldn't repair actual system files or binaries if they were corrupted, right?

Yes, AFAIK, that's correct.

 

[jj22038]

Frankly, after going to the AX6000 with a new SSD, I also formatted a newer (more namebrand) flashdrive when I reflashed everything from scratch on my old AC86U. I think it all just needed a fresh start. My old flashdrive was just hobbling along as more and more stuff started getting corrupt over time.


For some reason, when something came up as "tainted" in my logs, when executing these, it would also generate "segmentation faults". Thus I equated "tained" with "corrupted".


Nice work! You may be lucky, but if these come back at some point in the future, you may need to face having to reformat your flashdrive and reinstall entware/addons. I wish these linux distros had a "SFC /scannow" type of command like Windows does to help fix system file corruption issues. ;(

A crash (such as segmentation fault, signal 11) generates a register dump. The register dump contains information about the kernel. Since the kernel is "tainted" (contains closed-source portions), you will see the word "Tainted" in the register dump. This has nothing to do with the actual program that crashed.

Kernel maintainers generally won't look at tainted kernel problems.

 

[Viktor Jaep]

A crash (such as segmentation fault, signal 11) generates a register dump. The register dump contains information about the kernel. Since the kernel is "tainted" (contains closed-source portions), you will see the word "Tainted" in the register dump. This has nothing to do with the actual program that crashed.

Kernel maintainers generally won't look at tainted kernel problems.

Thanks for the additional info on this, @jj22038!