RT AC86U Spontaneously Started Crashing on 386.2_4

[PT25]

Hi,

I have been running 386.2_4 on my AC86U for around a week with no issues. Then suddenly last night around 11:30pm it crashed and rebooted. Proceeded to do this a further 4 times over an hour period. Seemed to be fine overnight but it has just happened again. I have looked at the System Log and I don't understand the coding and wording, I am hoping someone far more qualified here might be able to point me in the right direction or take a look and offer some advise as to why this has started spontaneously happening.

For reference, I have an AC86U as my main router with a Swap File and is running Diversion, SpdMerlin, CakeQOS and an OpenVPN Client with Surfshark VPN, I have 2 x DSL AC68U routers acting as AiMesh nodes both running the GNuton 386.2_4 firmware. I have a 24 port TP Link switch connected to the AC86U which connects 15 CAT6 cables to various network points around my home. I also have a USB HDD connected to the AC86U which is acting as my MacBook TimeMachine. Not sure if any of that might help?

I am a bit panicky as I work from home and need my internet to be reliable, which it has been up until 11:30pm last night.

I have attached the entire Crash Log from the most recent crash around 15 mins ago.

Thanks for any help.

 

[vaboro]

Here’s your problem:

May  5 06:05:08 crashlog: <0>Kernel panic - not syncing: Fatal exception
May  5 06:05:08 crashlog: <2>CPU0: stopping
May  5 06:05:08 crashlog: <4>CPU: 0 PID: 7952 Comm: pidof Tainted: P      D    O    4.1.27 #2
May  5 06:05:08 crashlog: <4>Hardware name: Broadcom-v8A (DT)
May  5 06:05:08 crashlog: <0>Call trace:
May  5 06:05:08 crashlog: <4>[<ffffffc0000876d8>] dump_backtrace+0x0/0x150
May  5 06:05:08 crashlog: <4>[<ffffffc00008783c>] show_stack+0x14/0x20
May  5 06:05:08 crashlog: <4>[<ffffffc00050bff4>] dump_stack+0x90/0xb0
May  5 06:05:08 crashlog: <4>[<ffffffc00008e730>] handle_IPI+0x190/0x1a0
May  5 06:05:08 crashlog: <4>[<ffffffc000080c70>] gic_handle_irq+0x88/0x90
May  5 06:05:08 crashlog: <4>Exception stack(0xffffffc00bb0fa80 to 0xffffffc00bb0fba0)
May  5 06:05:08 crashlog: <4>fa80: 1e03ab80 ffffffc0 1e6ca888 ffffffc0 0bb0fbc0 ffffffc0 001954fc ffffffc0
May  5 06:05:08 crashlog: <4>faa0: 0000416d 00000000 00008124 00000000 1e03ab90 ffffffc0 00000000 00000000
May  5 06:05:08 crashlog: <4>fac0: 6c646d63 00656e69 00000000 00000000 39323831 5a5a5a00 9e370001 00000000
May  5 06:05:08 crashlog: <4>fae0: 13aef0f8 ffffffc0 00000000 00000000 00000000 00000000 fefefeff fefefefe
May  5 06:05:08 crashlog: <4>fb00: 000a0cbc 00000000 ffbdca40 00000000 00083d40 00000000 00000000 00000000
May  5 06:05:08 crashlog: <4>fb20: 00187bc0 ffffffc0 00000000 00000000 00000000 00000000 1e03ab80 ffffffc0
May  5 06:05:08 crashlog: <4>fb40: 1e6ca888 ffffffc0 0bb0fd70 ffffffc0 00000001 00000000 0bb0fce8 ffffffc0
May  5 06:05:08 crashlog: <4>fb60: 191bce60 ffffffc0 00000001 00000000 00000005 00000000 0051a000 ffffffc0
May  5 06:05:08 crashlog: <4>fb80: 00020000 00000000 0bb0fbc0 ffffffc0 001954ec ffffffc0 0bb0fbc0 ffffffc0
May  5 06:05:08 crashlog: <4>[<ffffffc000083da8>] el1_irq+0x68/0xd8
May  5 06:05:08 crashlog: <4>[<ffffffc00014b8e4>] lookup_fast+0x2bc/0x328
May  5 06:05:08 crashlog: <4>[<ffffffc00014d80c>] do_last.isra.13+0x11c/0xc20
May  5 06:05:08 crashlog: <4>[<ffffffc00014e394>] path_openat+0x84/0x5c0
May  5 06:05:08 crashlog: <4>[<ffffffc00014f9c8>] do_filp_open+0x30/0x98
May  5 06:05:08 crashlog: <4>[<ffffffc00013f348>] do_sys_open+0x140/0x228
May  5 06:05:08 crashlog: <4>[<ffffffc000187bdc>] compat_SyS_open+0x1c/0x28

Please, take a look at this post.

 

[Tech9]

Not sure if any of that might help?

You put this router under a lot of stress. Everything you run is processed by the CPU - Cake QoS is incompatible with hardware acceleration. spdMerlin is hitting the WAN hard in regular intervals. OpenVPN client and TimeMachine run on the same CPU core, most likely. AiMesh is keeping your radios busy all the time. This router runs hotter with 386 code firmware compared to 384 code firmware. What is the CPU temperature? Is your external HDD powered by the USB port? When was the router reset to factory defaults and manually configured last time?

 

[PT25]

Thank you both for your replies. @vaboro I have taken a look at the posts you referenced, I am not sure I understand what 'Kernal Panic' is. The 'nvram' reset you mentioned in those posts, that is very similar to the nuclear reset I did when upgrading from 384 to 386, except I read someones comments on here stating it was best to leave the router for at least 24 hours to fully drain the capacitors, which I did.

@Tech9 I didn't realise I was putting the router under stress? I see people on this forum running more scripts than I do and putting the router under more stress than I thought I did, is it just that the AC86U is not that powerful a router? To answer your questions, I opted to use CakeQOS as I work from home and have to simultaneously download/upload content while video and conference calling. Since enabling CakeQOS my connection quality has improved immeasurably. I am not sure what you mean by the hardware acceleration? Is that something I can, and should, disable? The spdMerlin script runs at 12 past and 42 past the hour only. Should I change that to just once an hour?

My TimeMachine only functions when my MacBook is on and plugged in on power, so in reality not all that often. The OpenVPN client is running all the time and protecting all of my WIFI devices. I genuinely have no idea if they both run off the same core or not, but when I bought this router I was told that it was a Dual Core router cable of quite a high end usage. Do I need to disable either TimeMachine of OpenVPN client?

Regarding temperature, I have noticed any temperatures that cause me concern. Right now I have 2.4GHz at 42 degrees, 5GHz at 52 degrees and CPU at 74 degrees. Is that bad? Those temperatures give or take a couple of degrees are what I have always had, even on stock firmware.

My external HDD which has TimeMachine as well as running as a NAS is a 3.5inch caddy which is powered externally with its own adapted. I have never powered any storage from the USB drive on the router.

As I mentioned above, I performed a complete factory/nvram/nuclear reset (the one by leaving it switched on while removing the power cable and leaving for more than 24 hours) in between upgrading from 384 to 386.2 Since then it has purred along with no issues whatsoever. I then did a dirty upgrade from 386.2 to 386.2-4 last week and up until last night again it purred lovely. Then completely out the blue it started crashing.

I really appreciate your advise on this. Am I expecting too much from this router? As I said I was told when I bought it that it was a very robust piece of kit able to handle my 'very small' home office needs. Is there anything that would have caused this crashing to start completely out the blue? Do I need to do another complete factory/nvram/nuclear reset, and if so can I use a backup of my settings or do I need to completely reconfigure my router all over again?

Again thanks for your help

 

[John Fitzgerald]

Thank you both for your replies. @vaboro I have taken a look at the posts you referenced, I am not sure I understand what 'Kernal Panic' is. The 'nvram' reset you mentioned in those posts, that is very similar to the nuclear reset I did when upgrading from 384 to 386, except I read someones comments on here stating it was best to leave the router for at least 24 hours to fully drain the capacitors, which I did.

@Tech9 I didn't realise I was putting the router under stress? I see people on this forum running more scripts than I do and putting the router under more stress than I thought I did, is it just that the AC86U is not that powerful a router? To answer your questions, I opted to use CakeQOS as I work from home and have to simultaneously download/upload content while video and conference calling. Since enabling CakeQOS my connection quality has improved immeasurably. I am not sure what you mean by the hardware acceleration? Is that something I can, and should, disable? The spdMerlin script runs at 12 past and 42 past the hour only. Should I change that to just once an hour?

My TimeMachine only functions when my MacBook is on and plugged in on power, so in reality not all that often. The OpenVPN client is running all the time and protecting all of my WIFI devices. I genuinely have no idea if they both run off the same core or not, but when I bought this router I was told that it was a Dual Core router cable of quite a high end usage. Do I need to disable either TimeMachine of OpenVPN client?

Regarding temperature, I have noticed any temperatures that cause me concern. Right now I have 2.4GHz at 42 degrees, 5GHz at 52 degrees and CPU at 74 degrees. Is that bad? Those temperatures give or take a couple of degrees are what I have always had, even on stock firmware.

My external HDD which has TimeMachine as well as running as a NAS is a 3.5inch caddy which is powered externally with its own adapted. I have never powered any storage from the USB drive on the router.

As I mentioned above, I performed a complete factory/nvram/nuclear reset (the one by leaving it switched on while removing the power cable and leaving for more than 24 hours) in between upgrading from 384 to 386.2 Since then it has purred along with no issues whatsoever. I then did a dirty upgrade from 386.2 to 386.2-4 last week and up until last night again it purred lovely. Then completely out the blue it started crashing.

I really appreciate your advise on this. Am I expecting too much from this router? As I said I was told when I bought it that it was a very robust piece of kit able to handle my 'very small' home office needs. Is there anything that would have caused this crashing to start completely out the blue? Do I need to do another complete factory/nvram/nuclear reset, and if so can I use a backup of my settings or do I need to completely reconfigure my router all over again?

Again thanks for your help

I would check everything that plugs into the router (power bricks and the like) with a Digital Multi Meter and make sure they are outputting the correct Voltage(s).
Most are center-pin + positive, the diagram is on the power adapter.
Start simple.

 

[PT25]

I would check everything that plugs into the router (power bricks and the like) with a Digital Multi Meter and make sure they are outputting the correct Voltage(s).
Most are center-pin + positive, the diagram is on the power adapter.
Start simple.

All I have connected to the router is the power supply, the WAN connection to my modem, the HDD (self powered) on the USB2 port, a SWAP on the USB3 port (set to USB2) and 1 x ethernet cable to the TP Link Switch. Thats it. I am only using 1 of the gigabit ports. What is it that you mean I need to check?

Thanks

 

[John Fitzgerald]

All I have connected to the router is the power supply, the WAN connection to my modem, the HDD (self powered) on the USB2 port, a SWAP on the USB3 port (set to USB2) and 1 x ethernet cable to the TP Link Switch. Thats it. I am only using 1 of the gigabit ports. What is it that you mean I need to check?

Thanks

The hard drive you think is self powered may not be, and might be putting a strain on the system.
The same with the 24 port TP-Link switch.
That's why I said test all connected devices including the one connected to the router.
You have a lot of memory errors and unstable power can be a cause.
You said it was fine for 4 days after the upgrade and then out of the blue it crashes.

 

[vaboro]

Thank you for describing your status in so much detail. This I think is essential.

I am not sure I understand what 'Kernal Panic' is

It's the reason of your router's rebooting from time to time. The kernel tries to execute code which signature has changed and «panics» rebooting your router. The most severe case would be a boot loop.

I then did a dirty upgrade from 386.2 to 386.2-4

And this is the most probable reason of the problem.

Do I need to do another complete factory/nvram/nuclear reset

IMHO this would be a way to solve the kernel panic issue. I have found no other way of resolving kernel panics but a «nuclear reset» as you call it.

can I use a backup of my settings or do I need to completely reconfigure my router all over again?

I am afraid you'd better reconfigure the router manually all over again cause something in the backup might be making the kernel panic.

Right now I have 2.4GHz at 42 degrees, 5GHz at 52 degrees and CPU at 74 degrees. Is that bad?

In think these are very good numbers if you didn't change the thermal pads and don't use any fans.

 

[vaboro]

I read someones comments on here stating it was best to leave the router for at least 24 hours to fully drain the capacitors

IMHO couple of seconds is enough. When I connect an UART interface card to the routher that has been disconnected from all cables and dongles as well as from power, and when I push the power button, the LED's on the UART interface card blink and I get couple of lines of code on the console. After that nothing. No more console output. The router has neither power nor charge for any more output.

in between upgrading from 384 to 386.2

I had a boot loop issue after upgrading from 384 to 386.2 that was solved with «nuclear reset». Do not upgrade to 386.2_4 since the addition of jitterentropy-rngd might make my kernel panic sooner or later without another «nuclear reset».

 

[PT25]

Thank you for describing your status in so much detail. This I think is essential.

It's the reason of your router's rebooting from time to time. The kernel tries to execute code which signature has changed and «panics» rebooting your router. The most severe case would be a boot loop.

And this is the most probable reason of the problem.

IMHO this would be a way to solve the kernel panic issue. I have found no other way of resolving kernel panics but a «nuclear reset» as you call it.

I am afraid you'd better reconfigure the router manually all over again cause something in the backup might be making the kernel panic.

In think these are very good numbers if you didn't change the thermal pads and don't use any fans.

@vaboro thank you for your very detailed explanation. I should take this time to say that I am very much amateur to all of this. 12 months ago I was using my ISP router and had no idea what custom firmware was, or that ASUS even existed. I've come a long way but to get to the standard of you guys I have a very long way to go I really appreciate all the help and advise people like yourself offer.

You believe that the dirty upgrade is the cause? Before going from 386.2 to 386.2-4 I read a lot of posts from LL&D and others about upgrading processes and what I understood the consensus to be was that when going from a large number to another (384 to 386 for example) then you should always factory reset. But when gong from the small numbers to small numbers (so 386.2 to 386.2-4) then a dirty upgrade is perfectly fine. Have I totally misunderstood this? If so, should I always do a full factory reset between ALL firmware upgrades, and if that is the case is it ok to restore a back up of settings when doing that? I have a lot of configurations in my router which take me a long time to do (probably due to my amateur level) and it would be annoying to have to do an entire reconfiguration on a regular basis. But I understand that if that's the case then that is the way I need to do it.

I actually do understand what you mean about not using a backup in this scenario as my current configuration might be causing this panic. What I mean is in the future can I use backups when going from a perfectly stable version to a higher version?

Those temperatures are what I have always had and I haven't made any modifications. Some time ago I actually ordered some USB fans for the router but then I found other posts on here that claimed the dust and debris from those cause more issues so I chickened out and never used them. With regard to thermal pads, I have read about those in this forum but don't totally understand what they are. In any case I know better than to open up my router and start tinkering in there... That is far too advanced for me. But yes, those are my temperatures, and I have never actually ever seen it go about 74 degrees. I'm happy you say they are good numbers as some of the recent threads suggest that any AC86U owners should expect their routers to die very young due to heat issues.

While typing this message I think I may have just noticed something. My MacBook started to do a TimeMachine Backup and a few minutes in the router did the crash and reboot thing. I manually rebooted the router again and it was stable, left it for 15 mins and then tried to manually run a TimeMachine backup again and hey presto - crash and reboot. I have just finished running the Health Check scanner in the GUI for the attached disk and it has found errors in a single sector - the TimeMachine location!! "
** The volume TimeMachine could not be repaired after 3 attempts." Could this all be connected? Is there anything in that Crash Log that refers to the TimeMachine process? Last night when all of this started happening my MacBook was plugged in and was plugged in all night syncing a lot of content with my head office - obviously it would have tried to backup up to TimeMachine on several occasions. And, Last night was the first time I have had my MacBook turned on since upgrading to 386.2-4. Is this all just a coincidence or..?

 

[PT25]

I had a boot loop issue after upgrading from 384 to 386.2 that was solved with «nuclear reset». Do not upgrade to 386.2_4 since the addition of jitterentropy-rngd will likely make my kernel panic sooner or later without another «nuclear reset».

I posted a reply to your previous message but for some reason it says it is awaiting Moderator Approval and is invisible to normal readers? Not sure why

 

[vaboro]

I posted a reply to your previous message but for some reason it says it is awaiting Moderator Approval and is invisible to normal readers? Not sure why

Sorry, I thought I am typing too many messages and decided to merge two of them.

 

[PT25]

Sorry, I thought I am typing too many messages and decided to merge two of them.

No I don't think its you. Its my post just hasn't posted, its stuck in some moderator limbo. There isn't anything suspect about it I don't think. Just replying to what to what you said. Strange. Maybe it'll get accepted soon?

 

[vaboro]

Its my post just hasn't posted

Maybe because the id of the message to which you wanted to post a reply has changed after I merged it with another message since your further messages are visible.

 

[PT25]

@vaboro I have no idea why this is being refused. I will post it now in smaller parts;

@vaboro thank you for your very detailed explanation. I should take this time to say that I am very much amateur to all of this. 12 months ago I was using my ISP router and had no idea what custom firmware was, or that ASUS even existed. I've come a long way but to get to the standard of you guys I have a very long way to go I really appreciate all the help and advise people like yourself offer.

 

[PT25]

You believe that the dirty upgrade is the cause? Before going from 386.2 to 386.2-4 I read a lot of posts from LL&D and others about upgrading processes and what I understood the consensus to be was that when going from a large number to another (384 to 386 for example) then you should always factory reset. But when gong from the small numbers to small numbers (so 386.2 to 386.2-4) then a dirty upgrade is perfectly fine. Have I totally misunderstood this? If so, should I always do a full factory reset between ALL firmware upgrades, and if that is the case is it ok to restore a back up of settings when doing that? I have a lot of configurations in my router which take me a long time to do (probably due to my amateur level) and it would be annoying to have to do an entire reconfiguration on a regular basis. But I understand that if that's the case then that is the way I need to do it.

 

[PT25]

I actually do understand what you mean about not using a backup in this scenario as my current configuration might be causing this panic. What I mean is in the future can I use backups when going from a perfectly stable version to a higher version?

Those temperatures are what I have always had and I haven't made any modifications. Some time ago I actually ordered some USB fans for the router but then I found other posts on here that claimed the dust and debris from those cause more issues so I chickened out and never used them. With regard to thermal pads, I have read about those in this forum but don't totally understand what they are. In any case I know better than to open up my router and start tinkering in there... That is far too advanced for me. But yes, those are my temperatures, and I have never actually ever seen it go about 74 degrees. I'm happy you say they are good numbers as some of the recent threads suggest that any AC86U owners should expect their routers to die very young due to heat issues.

 

[PT25]

While typing this message I think I may have just noticed something. My MacBook started to do a TimeMachine Backup and a few minutes in the router did the crash and reboot thing. I manually rebooted the router again and it was stable, left it for 15 mins and then tried to manually run a TimeMachine backup again and hey presto - crash and reboot.

 

[PT25]

I have just finished running the Health Check scanner in the GUI for the attached disk and it has found errors in a single sector - the TimeMachine location!! It said, "The volume TimeMachine could not be repaired after 3 attempts." Could this all be connected? Is there anything in that Crash Log that refers to the TimeMachine process?

 

[vaboro]

Have I totally misunderstood this?

You understood it absolutely correct. However, everyone's configuration is unique and many things may go wrong. So, it's usually a trial and error path.

should I always do a full factory reset between ALL firmware upgrades

Unfortunately, there is no definitive answer to this question. You should try and see what you get for your configuration.

I have a lot of configurations in my router which take me a long time to do (probably due to my amateur level) and it would be annoying to have to do an entire reconfiguration on a regular basis

This is why I don't like firmware upgrades. I have to be ready for something going completely wrong and reconfiguring everything from scratch.