RT-AC87U - Kernel panic

[BSOD2600]

Recently purchased a new AC87U (use to have AC66U) immediately upgrade to 380.68_4 and get everything reconfigured. last week, for the first time installed AB-Solution (v3.9.2), which so far appears to be working great. Reviewing the system messages last night, I noticed the router rebooted itself apparently from a Kernel panic. Searched and saw some recent threads about possibly a corrupt usb drive could cause this... which would coincide with the recently installed AB.

Thoughts on root cause from the stack? Full log attached.

Jul 31 17:00:12 kernel: <4>[<bf791f38>] (PC is at vtree_destroy+0x48/0x7c [IDP])
Jul 31 17:00:12 kernel: <4>[<bf791f38>] (vtree_destroy+0x48/0x7c [IDP]) from [<bf791f14>] (vtree_destroy+0x24/0x7c [IDP])
Jul 31 17:00:12 kernel: <0>Code: e5951010 e2852014 e1a00005 e5831004 (e5813000)
Jul 31 17:00:12 kernel: ce 959f38688432bc2b ]---
Jul 31 17:00:12 kernel: <0>Kernel panic - not syncing: Fatal exception
Jul 31 17:00:12 kernel: <4>[<c0043ff8>] (unwind_backtrace+0x0/0xf8) from [<c ^Cc5c>] (panic+0x7c/0x1a8)
Jul 31 17:00:12 kernel: <4>[<c02cbc5c>] (panic+0x7c/0x1a8) from [<c00416f0>] (die+0x1a4/0x1dc)
Jul 31 17:00:12 kernel: <4>[<c00416f0>] (die+0x1a4/0x1dc) from [<c004515c>] (__do_kernel_fault+0x6c/0x8c)
Jul 31 17:00:12 kernel: <4>[<c004515c>] (__do_kernel_fault+0x6c/0x8c) from [<c00452cc>] (do_page_fault+0x150/0x1ec)
Jul 31 17:00:12 kernel: <4>[<c00452cc>] (do_page_fault+0x150/0x1ec) from [<c003d3a4>] (do_DataAbort+0x30/0x9c)
Jul 31 17:00:12 kernel: <4>[<c003d3a4>] (do_DataAbo(__dabt_svc+0x4c/0x60)
Jul 31 17:00:12 kernel: <4>Exception stack(0xcab09e58 to 0xcab09ea0)
Jul 31 17:00:12 kernel: <4>9e40:                                                       cee48440 00000000
Jul 31 17:00:12 kernel: <4>9e60: cee48454 cee4844c cee48594 cee484408000
Jul 31 17:00:12 kernel:  00000000 cab09ea0 bf791f14 bf791f38 60000013 ffffffff
Jul 31 17:00:12 kernel: <4>[<c0386f2c>] (__dabt_svc+0x4c/0x60) from [<bf791f38>] (vtree_destroy+0x48/0x7c [IDP])
Jul 31 17:00:12 kernel: <4>[<bf791f38>] (vtree_destroy+0x48/0x7c [IDP]) from [<bf791f14>] (vtree_destroy+0x24/0x7c [IDP])
Jul 31 17:00:12 kernel: <0>Rebooting i<4>[<c0043ff8>] (unwind_backtrac>] (do_IPI+0x114/0x154) from [<c<4>Exception stack(0xcf977da8 to 0xcf977df0)
Jul 31 17:00:12 kernel: <4>7da0:           >7dc0: 00000000 d08b0000 cf977e200000 cf977df0
Jul 31 17:00:12 kernel: <4>7de0: c0053d50<c0386f88>] (__irq_svc+0x48/0xe8ead_page+0x214/0x4cc)
Jul 31 17:00:12 kernel: <4>[<c0053ore_read+0x68/0x8c)
Jul 31 17:00:12 kernel: <4>[<c00540b0>] (nandcore_read+0x68/0x8c) from [<c0055540>] (hndnand_read+0x20/0x28)
Jul 31 17:00:12 kernel: <4>[<c0055540>] (hndnand_read+0x20/0x28) from [<c01b7a20)
Jul 31 17:00:12 kernel: <4>[<c01b7a20>] (_nflash_mtd_read+0x1f0/0x3a0) from [<c01b80b4>] (nflash_mtd_read+0x4c/0x64)
Jul 31 17:00:13 kernel: <4>[<c01b80b4>] (nflash_mtd_read+0x4c/0x64) from [<c019ad28>] (p8>] (part_read+0x64/0xe8) from [<c019d784>] (mtd_blktrans_thread+0x204/0x294)
Jul 31 17:00:13 kernel: <4>[<c019d784>] (m

 

[BSOD2600]

upload not attaching. paste instead
https://pastebin.com/0y657kBm

 

[joegreat]

Towards the end of the kernel panic you see the line with "nflash_mtd_read" which could be the root cause: In that case the NVRAM is becoming faulty...

A factory reset will check the NVRAM and mark bad blocks to not be used anymore.
If this does not help then you router is going south: A hardware problem will need to be repaired or the router replaced.

 

[BSOD2600]

Dying hardware already? sheesh, the reviews online do have validity after all then about many dying within a short period of time. Brand new sealed box unit too.

k, will reset and monitor for repeats...

 

[BSOD2600]

Well it didnt take much longer for a few more repeats. This time they have different stacks though. Appears the commonality is the USB port per chance?

reboot 1

Jul 31 17:00:13 kernel: _ Reboot message ... _______________________________________________________
Jul 31 17:00:13 kernel: <1>Unable to handle kernel paging request at vir�U� address d621b009
Jul 31 17:00:13 kernel: <1>pgd = cf7f4000
Jul 31 17:00:13 kernel: <1>[d621b009] *pgd=07780011, *pte=00000000, *ppte=00000000
Jul 31 17:00:13 kernel: <0>Internal error: Oops: 7 [#1] PREEMPT SMP
Jul 31 17:00:13 kernel: <0>last sysfs file: /sys/devices/pci0000:00/0000:00:0b.1/usb2/2-2/2-2:1.0/host0/target0:0:0/0:0:0:0/model
<snip>
Jul 31 17:00:13 kernel: <4>Modules linked in: ct_notification bw_forward(P) IDP(P) nf_nat_sip nf_conntrack_sip nf_nat_h323 nf_conntrack_h323 nf_nat_rtsp nf_conntrack_rtsp nf_nat_ftp nf_conntrack_ftp ip6table_mangle xt_length sr_mod cdrom cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether asix usbnet mii usblp ohci_hcd ehci_hcd xhci_hcd thfsplus tntfs(P) tfat(P) ext2 ext4 crc16 jbd2 ext3 jbd mbcache usb_storage sg sd_mod scsi_wait_scan scsi_mod usbcore ip6t_LOG ip6table_filter jffs2 zlib_deflate nf_n
Jul 31 17:00:13 kernel: <4>CPU: 0    Tainted: P             (2.6.36.4brcmarm #2)
Jul 31 17:00:13 kernel: <4>PC is at nk_signature_init+0x27c/0x260c [IDP]
Jul 31 17:00:13 kernel: <4>LR is at nk_signature_init+0x220/0x260c [IDP]
Jul 31 17:00:13 kernel: <4>pc : [<bf753ec8>]    lr : [<bf753e6c>]    psr: a0000013
Jul 31 17:00:13 kernel: <4>sp : cf701e98  ip : 00000000  fp : d273d000
Jul 31 17:00:13 kernel: <4>r10: 00000068  r9 : ffffffff  r8 : 00000000
Jul 31 17:00:13 kernel: <4>r7 : bf7b6f3c  r6 : d621b008  r5 : d2740000  r4 : bf79e590
Jul 31 17:00:13 kernel: <4>r3 : 00000c74  r2 : 00000221  r1 : 51eb851f  r0 : bf7bd350
Jul 31 17:00:13 kernel: <4>Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Jul 31 17:00:13 kernel: <4>Control: 10c53c7d  Table: 9f7f404a  DAC: 00000015
Jul 31 17:00:13 kernel: <0>Process bwdpi-rule-agen (pid: 4453, stack limit = 0xcf700270)
Jul 31 17:00:13 kernel: <0>Stack: (0xcf701e98 to 0xcf702000)
Jul 31 17:00:13 kernel: <0>1e80:                                                       cf701f04 00000002
Jul 31 17:00:13 kernel: <0>1ea0: cf701ef0 cfa61000 00000000 00000000 00000221 51eb851f 00000000 c003c57c
Jul 31 17:00:13 kernel: <0>1ec0: cf701f48 bf79e388 00000000 bf79e388 40286b07 bec8abf0 cf700000 00000000
Jul 31 17:00:13 kernel: <0>1ee0: bec8abec bf74f5a4 cf720300 bf74f4ec bf79e388 bf74f108 5959fce7 00000028
Jul 31 17:00:13 kernel: <0>1f00: cf57c0a0 bec8abf0 bec8abf0 00000003 c003dc68 c00d86e8 cf57c5a8 c005a1f0
Jul 31 17:00:13 kernel: <0>1f20: cdda201c 00000000 00002e58 cf57c5a0 00000002 cf5ad3b8 c03d2c78 00000001
Jul 31 17:00:13 kernel: <0>1f40: cf57c0a8 cf57c0a0 ceec3700 00000000 00000000 00000000 00000002 00000003
Jul 31 17:00:13 kernel: <0>1f60: ceec3700 c00c886c cf57c0a0 bec8abf0 40286b07 00000003 c003dc68 cf700000
Jul 31 17:00:13 kernel: <0>1f80: 00000000 c00d8c70 4040c6cc 00000000 00000003 4040c6cc 408b66bc 00000003
Jul 31 17:00:13 kernel: <0>1fa0: 00000036 c003dac0 4040c6cc 408b66bc 00000003 40286b07 bec8abf0 bec8abd4
Jul 31 17:00:13 kernel: <0>1fc0: 4040c6cc 408b66bc 00000003 00000036 00008e30 0000a7d8 bec8ace8 bec8abec
Jul 31 17:00:13 kernel: <0>1fe0: 0003465c bec8abb8 0000d304 4009a308 20000010 00000003 00000000 00000000
Jul 31 17:00:13 kernel: <4>[<bf753ec8>] (PC is at nk_signature_init+0x27c/0x260c [IDP])
Jul 31 17:00:13 kernel: <4>[<bf753ec8>] (nk_signature_init+0x27c/0x260c [IDP]) from [<bf74f5a4>] (rebuild_rulesets+0xb8/0x144 [IDP])
Jul 31 17:00:13 kernel: <4>[<bf74f5a4>] (rebuild_rulesets+0xb8/0x144 [IDP]) from [<bf74f108>] (insp_dep_write_proc+0x448/0x4c0 [IDP])
Jul 31 17:00:13 kernel: <4>[<bf74f108>] (insp_dep_write_proc+0x448/0x4c0 [IDP]) from [<c00d86e8>] (do_vfs_ioct ^C80/0x5d0)
Jul 31 17:00:13 kernel: <4>[<c00d86e8>] (do_vfs_ioctl+0x80/0x5d0) from [<c00d8c70>] (sys_ioctl+0x38/0x60)
Jul 31 17:00:13 kernel: <4>[<c00d8c70>] (sys_ioctl+0x38/0x60) from [<c003dac0>] (ret_fast_syscall+0x0/0x30)
Jul 31 17:00:13 kernel: <0>Code: e34511eb e58d2018 e58d8014 e58d101c (e5d60001)
Jul 31 17:00:13 kernel: <4>---[ end trace 3fc1acab470d0ae5 ]---
Jul 31 17:00:13 kernel: <0>Kernel panic - not syncing: Fatal exception
Jul 31 17:00:13 kernel: <4>[<c0043ff8>] (unwind_backtrace+0x0/0xf8) from [<c02cbc5c>] (panic+0x7c/0x1a8)
Jul 31 17:00:13 kernel: <4>[<c02cbc5c>] (panic+0x7c/0x1a8) from [<c00416f0>] (die+0x1a4/0x1dc)
Jul 31 17:00:13 kernel: <4>[<c00416f0>] (die+0x1a4/0x1dc) from [<c004515c>] (__do_kernel_fault+0x6c/0x8c)
Jul 31 17:00:13 kernel: <4>[<c004515c>] (__do_kernel_fault+0x6c/0x8c) from [<c00452cc>] (do_page_fault+0x150/0x1ec)
Jul 31 17:00:13 kernel: <4>[<c00452cc>] (do_page_fault+0x150/0x1ec) from [<c003d3a4>] (do_DataAbort+0x30/0x9c)
Jul 31 17:00:13 kernel: <4>[<c003d3a4>] (do_DataAbort+0x30/0x9c) from [<c0386f2c>] (__dabt_svc+0x4c/0x60)
Jul 31 17:00:13 kernel: <4>Exception stack(0xcf701e50 to 0xcf701e98)
Jul 31 17:00:13 kernel: <4>1e40:                                     bf7bd350 51eb851f 00000221 00000c74
Jul 31 17:00:13 kernel: <4>1e60: bf79e590 d2740000 d621b008 bf7b6f3c 00000000 ffffffff 00000068 d273d000
Jul 31 17:00:13 kernel: <4>1e80: 00000000 cf701e98 bf753e6c bf753ec8 a0000013 ffffffff
Jul 31 17:00:13 kernel: <4>[<c0386f2c>] (__dabt_svc+0x4c/0x60) from [<bf753ec8>] (nk_signature_init+0x27c/0x260c [IDP])
Jul 31 17:00:13 kernel: <4>[<bf753ec8>] (nk_signature_init+0x27c/0x260c [IDP]) from [<bf74f5a4>] (rebuild_rulesets+0xb8/0x144 [IDP])
Jul 31 17:00:13 kernel: <4>[<bf74f5a4>] (rebuild_rulesets+0xb8/0x144 [IDP]) from [<bf74f108>] (insp_dep_write_proc+0x448/0x4c0 [IDP])
Jul 31 17:00:13 kernel: <4>[<bf74f108>] (insp_dep_write_proc+0x448/0x4c0 [IDP]) from [<c00d86e8>] (do_vfs_ioctl+0x80/0x5d0)
Jul 31 17:00:13 kernel: <4>[<c00d86e8>] (do_vfs_ioctl+0x80/0x5d0) from [<c00d8c70>] (sys_ioctl+0x38/0x60)
Jul 31 17:00:13 kernel: <4>[<c00d8c70>] (sys_ioctl+0x38/0x60) from [<c003dac0>] (ret_fast_syscall+0x0/0x30)
Jul 31 17:00:13 kernel: <2>CPU1: stopping
Jul 31 17:00:13 kernel: <0>Rebooting in 3 seconds..[<c0043ff8>] (unwind_backtrace+0x0/0xf8) from [<c003d334>] (do_IPI+0x114/0x154)
Jul 31 17:00:13 kernel: <4>[<c003d334>] (do_IPI+0x114/0x154) from [<c0386f88>] (__irq_svc+0x48/0xe8)
Jul 31 17:00:13 kernel: <4>Exception stack(0xcec87eb0 to 0xcec87ef8)
Jul 31 17:00:13 kernel: <4>7ea0:                                00000000
Jul 31 17:00:13 kernel: <4>7ec0: 00085888 00000059 c040da10 000003ff 00000000 00000030 c03cffdc 00000000
Jul 31 17:00:13 kernel: <4>7ee0: 00003fff cec87ef8 c0062448 c0386f88>] (__irq_svc+0x48/0xe8) from [<c02ce76c>] (_raw_spin_unlock_irq+0x38/0x40)
Jul 31 17:00:13 kernel: <4>[<c02ce76c>] (_raw_spin_unlock_irq+0x38/0x0bd64>] (proc_reg_read+0x70/0x94t_syscall+0x0/0x30)
Jul 31 17:00:13 kernel: _____________________________________________

 

[BSOD2600]

reboot 2

Jul 31 17:00:13 kernel: _ Reboot message ... _______________________________________________________
Jul 31 17:00:13 kernel: <1>Unable to handle kernel NULL pointer derefere�U�at virtual address 00000000
Jul 31 17:00:13 kernel: <1>pgd = cf654000
Jul 31 17:00:13 kernel: <1>[00000000] *pgd=9f623031, *pte=00000000, *ppte=00000000
Jul 31 17:00:13 kernel: <0>Internal error: Oops: 17 [#1] PREEMPT SMP
Jul 31 17:00:13 kernel: <0>last sysfs file: /sys/devices/pci0000:00/0000:00:0b.1/usb2/2-2/2-2:1.0/host0/target0:0:0/0:0:0:0/model
<snip>
Jul 31 17:00:13 kernel: <4>Modules linked in: IDP(P-) nf_nat_sip nf_conntrack_sip nf_nat_h323 nf_conntrack_h323 nf_nat_rtsp nf_conntrack_rtsp nf_nat_ftp nf_conntrack_ftp ip6table_mangle xt_length sr_mod cdrom cdc_mbim qmi_wwan cdc_wdm cdc_ncm rndis_host cdc_ether asix usbnet mii usblp ohci_hcd ehci_hcd xhci_hcd thfsplus tntfs(P) tfat(P) ext2 ext4 crc16 jbd2 ext3 jbd mbcache usb_storage sg sd_mod scsi_wait_scan scsi_mod usbcore ip6t_LOG ip6table_filter jffs2 zlib_deflate nf_nat_pptp nf_conntrack_pptp nf_
Jul 31 17:00:13 kernel: <4>CPU: 1    Tainted: P             (2.6.36.4brcmarm #2)
Jul 31 17:00:13 kernel: <4>PC is at parse_ip6_header+0x1cb6c/0x1cf24 [IDP]
Jul 31 17:00:13 kernel: <4>LR is at devid_sig_destroy+0x1bc/0x21c [IDP]
Jul 31 17:00:13 kernel: <4>pc : [<bf78e448>]    lr : [<bf78e9bc>]    psr: a0000013
Jul 31 17:00:13 kernel: <4>sp : cf649f00  ip : 00000000  fp : 00000000
Jul 31 17:00:13 kernel: <4>r10: 00000000  r9 : cf648000  r8 : bec0cca8
Jul 31 17:00:13 kernel: <4>r7 : d5ecb000  r6 : cf55cea0  r5 : cf55ce80  r4 : cf55ce68
Jul 31 17:00:13 kernel: <4>r3 : 00000000  r2 : 00000000  r1 : 9a000081  r0 : cf55ce80
Jul 31 17:00:13 kernel: <4>Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Jul 31 17:00:13 kernel: <4>Control: 10c53c7d  Table: 9f65404a  DAC: 00000015
Jul 31 17:00:13 kernel: <0>Process rmmod (pid: 25538, stack limit = 0xcf648270)
Jul 31 17:00:13 kernel: <0>Stack: (0xcf649f00 to 0xcf64a000)
Jul 31 17:00:13 kernel: <0>9f00: cf55ce68 d5ecb420 cf648000 bf78e9bc bf79d7f0 bf79d378 cf648000 cf649f44
Jul 31 17:00:13 kernel: <0>9f20: bec0cca8 bf78f2f4 bf79d7f0 bf751750 bf79d7f0 bf749804 00000000 c008f27c
Jul 31 17:00:13 kernel: <0>9f40: cde27bd8 00504449 2e9f0f75 cf648000 40071000 cf649f84 40071000 00000001
Jul 31 17:00:13 kernel: <0>9f60: cf648000 40070000 cddb5dc0 bec0ca14 00005401 00000001 c003dc68 00648000
Jul 31 17:00:13 kernel: <0>9f80: bf79d378 00000880 cf649f8c 00000000 402103b0 00000880 bec0cca8 00000081
Jul 31 17:00:13 kernel: <0>9fa0: c003dc68 c003dac0 402103b0 00000880 bec0cca8 00000880 00000b3c 00000000
Jul 31 17:00:13 kernel: <0>9fc0: 402103b0 00000880 bec0cca8 00000081 000856f0 0000d05c bec0cde8 00000000
Jul 31 17:00:13 kernel: <0>9fe0: 401b774c bec0cc90 0001544c 401b7758 20000010 bec0cca8 00000000 00000000
Jul 31 17:00:13 kernel: <4>[<bf78e448>] (PC is at parse_ip6_header+0x1cb6c/0x1cf24 [IDP])
Jul 31 17:00:13 kernel: <4>[<bf78e448>] (parse_ip6_header+0x1cb6c/0x1cf24 [IDP]) from [<bf78e9bc>] (devid_sig_destroy+0x1bc/0x21c [IDP])
Jul 31 17:00:13 kernel: <4>[<bf78e9bc>] (devid_sig_destroy+0x1bc/0x21c [IDP]) from [<bf78f2f4>] (devid_exit+0x18/0x24 [IDP])
Jul 31 17:00:13 kernel: <4>[<bf78f2f4>] (devid_exit+0x18/0x24 [IDP]) from [<bf751750>] (detector_remove+0x3c/0x4c [IDP])
Jul 31 17:00:13 kernel: <4>[<bf751750>] (detector_remove+0x3c/0x4c [IDP]) from [<bf749804>] (nk_sys_exit+0xac/0xc8 [IDP])
Jul 31 17:00:13 kernel: <4>[<bf749804>] (nk_sys_exit+0xac/0xc8 [IDP]) from [<c008f27c>] (sys_delete_module+0x16c/0x218)
Jul 31 17:00:13 kernel: <4>[<c008f27c>] (sys_delete_module+0x16c/0x218) from [<c003dac0>] (ret_fast_syscall+0x0/0x30)
Jul 31 17:00:13 kernel: <0>Code: e5902020 e1a05000 e1560002 0a000010 (e5924000) 
Jul 31 17:00:13 kernel: <4>---[ end trace 35eb2b8dfb72b554 ]---
Jul 31 17:00:13 kernel: <0>Kernel panic - not syncing: Fatal excep ^Cce+0x0/0xf8) from [<c02cbc5c>] (panic+0x7c/0x1a8)
Jul 31 17:00:13 kernel: <4>[<c02cbc5c>] (panic+0x7c/0x1a8) from [<c00416f0>] (die+0x1a4/0x1dc)
Jul 31 17:00:13 kernel: <4>[<c00416f0>] (die+0x1a4/0x1dc) from [<c004515c>] (__do_kernel_fault+_kernel_fault+0x6c/0x8c) from [<0x1ec)
Jul 31 17:00:13 kernel: <4>[<c00452cc>] (do_page_fault+0x150/0x1ec) from [<c003d3a4>] (do_DataAbort+0x30/0x9c)
Jul 31 17:00:13 kernel: <4>[<c003d3a4>] (do_DataAbort+0x30/0x9c) from [<c0386f2c>] (__dabt_svc+0x4c/0x60)
Jul 31 17:00:13 kernel: <4>Exception sta9ea0:                           00000000 cf55ce68 cf55ce80 cf55cea0 d5ecb000 bec0cca8 cf648000
Jul 31 17:00:13 kernel: <00 cf649f00 bf78e9bc bf78e448 a0(__dabt_svc+0x4c/0x60) from [<bf6c/0x1cf24 [IDP])
Jul 31 17:00:13 kernel: <4>[<bf78e448>] (parse_ip6_header+0x1cb6c/0x1cf24 [IDP]) from [<bf78e9bc>] (debf78f2f4>] (devid_exit+0x18/0x24xit+0x18/0x24 [IDP]) from [<bf751750>] (detector_remove+0x3c/0x4c [IDP])
Jul 31 17:00:13 kernel: <4>[<bf751750>] (detectys_exit+0xac/0xc8 [IDP]) from [<c008f27c>] (sys_delete_module+0x
Jul 31 17:00:13 kernel: _______

 

[ColinTaylor]

Have you repartitioned and reformatted your USB drive since you first experienced this problem? The reason I ask is because your initial log in pastebin showed an error with the size of partition 1 on the USB. So it could just be that after a while the router is trying to write data beyond the end of the device.

 

[BSOD2600]

No, I have not (kinda a pain to do on Windows). I did format it fresh though when I started to use this router and AB Solution about a month ago.
I'll format it full again. One more chance before RMA'ing it...

 

[BSOD2600]

10 more days of uptime before the panic occurred again. anyone familiar with the code base able to chime in on a more definitive root cause? Seems most of the time the offending process is rmmod, which is used for unload the specified kernel modules from the kernel.... so doesnt seem that would be USB related. Possibly coincidental that the TrendMicro IDPfw was updated right after the router got done rebooting?

Jul 31 17:00:13 kernel: _ Reboot message ... _______________________________________________________
Jul 31 17:00:13 kernel: <1>Unable to handle kernel NULL pointer derefere�U�at virtual address 00000000
Jul 31 17:00:13 kernel: <1>pgd = c9a00000
Jul 31 17:00:13 kernel: <1>[00000000] *pgd=99586031, *pte=00000000, *ppte=00000000
Jul 31 17:00:13 kernel: <0>Internal error: Oops: 17 [#1] PREEMPT SMP
Jul 31 17:00:13 kernel: <0>last sysfs file: /sys/devices/pci0000:00/0000:00:0b.1/usb2/2-2/2-2:1.0/host1/target1:0:0/1:0:0:0/model

Jul 31 17:00:13 kernel: <4>CPU: 0    Tainted: P             (2.6.36.4brcmarm #2)
Jul 31 17:00:13 kernel: <4>PC is at remove_vm_area+0x4c/0x88
Jul 31 17:00:13 kernel: <4>LR is at remove_vm_area+0x38/0x88
Jul 31 17:00:13 kernel: <4>pc : [<c00b9710>]    lr : [<c00b96fc>]    psr: a0000013
Jul 31 17:00:13 kernel: <4>sp : c9611ed0  ip : 00000000  fp : c03ce948
Jul 31 17:00:13 kernel: <4>r10: c9610000  r9 : c03c4080  r8 : c003d434
Jul 31 17:00:13 kernel: <4>r7 : 00000001  r6 : 00000000  r5 : c9c28c00  r4 : cb0606a0
Jul 31 17:00:13 kernel: <4>r3 : 00000000  r2 : 00000000  r1 : 00000001  r0 : c0417a98
Jul 31 17:00:13 kernel: <4>Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Jul 31 17:00:13 kernel: <4>Control: 10c53c7d  Table: 99a0004a  DAC: 00000015
Jul 31 17:00:13 kernel: <0>Process rmmod (pid: 11406, stack limit = 0xc9610270)
Jul 31 17:00:13 kernel: <0>Stack: (0xc9611ed0 to 0xc9612000)
Jul 31 17:00:13 kernel: <0>1ec0:                                     d2e34000 bf7e7fe8 00000000 c00b97a0
Jul 31 17:00:13 kernel: <0>1ee0: bf7e7a80 bf7e7fe8 bf7e7fe8 f9b29b1f c003d434 bf7d7d64 bf7e5af4 00000002
Jul 31 17:00:13 kernel: <0>1f00: c0418cbc bf7d3b60 c0413ef4 064d62ed 00000009 c9611f80 00000002 00000000
Jul 31 17:00:13 kernel: <0>1f20: bf7e7d70 c9610000 c9611f44 be987ca8 c9610000 00000000 00000000 c008f27c
Jul 31 17:00:13 kernel: <0>1f40: cefee1d8 665f7762 6177726f c9006472 400cc000 c9611f84 400cc000 00000001
Jul 31 17:00:13 kernel: <0>1f60: c9610000 400cb000 cef51c80 be987a14 00005401 00000001 c003dc68 00610000
Jul 31 17:00:13 kernel: <0>1f80: bf7e7d70 00000880 c9611f8c 00000000 4017a3b0 00000880 be987ca8 00000081
Jul 31 17:00:13 kernel: <0>1fa0: c003dc68 c003dac0 4017a3b0 00000880 be987ca8 00000880 00000b3c 00000000
Jul 31 17:00:13 kernel: <0>1fc0: 4017a3b0 00000880 be987ca8 00000081 000856f0 0000d05c be987de8 00000000
Jul 31 17:00:14 kernel: <0>1fe0: 4012174c be987c90 0001544c 40121758 20000010 be987ca8 02007f00 00002a00
Jul 31 17:00:14 kernel: <4>[<c00b9710>] (PC is at remove_vm_area+0x4c/0x88)
Jul 31 17:00:14 kernel: <4>[<c00b9710>] (remove_vm_area+0x4c/0x88) from [<c00b97a0>] (__vunmap+0x24/0xf4)
Jul 31 17:00:14 kernel: <4>[<c00b97a0>] (__vunmap+0x24/0xf4) from [<bf7d7d64>] (deinit_ct_extra+0xb0/0x14c [bw_forward])
Jul 31 17:00:14 kernel: <4>[<bf7d7d64>] (deinit_ct_extra+0xb0/0x14c [bw_forward]) from [<bf7d3b60>] (__forward_exit+0x16c/0x1e4 [bw_forward])
Jul 31 17:00:14 kernel: <4>[<bf7d3b60>] (__forward_exit+0x16c/0x1e4 [bw_forward]) from [<c008f27c>] (sys_delete_module+0x16c/0x218)
Jul 31 17:00:14 kernel: <4>[<c008f27c>] (sys_delete_module+0x16c/0x218) from [<c003dac0>] (ret_fast_syscall+0x0/0x30)
Jul 31 17:00:14 kernel: <0>Code: e1540003 02866008 0a000003 e1a06003 (e5933000)
Jul 31 17:00:14 kernel:  ^C--[ end trace ca1120d2a4eb2b6c ]---
Jul 31 17:00:14 kernel: <0>Kernel panic - not syncing: Fatal exception
Jul 31 17:00:14 kernel: <4>[<c0043ff8>] (unwind_backtrace+0x0/0xf8) from [<c02cbc5c>] (panic+0x7c/0x1a8)
Jul 31 17:00:14 kernel: <4>[<c02cbc5c>] (panic+0x7c/0x1a8) from [<c00416f0>] (die+0x1a4/0x1dc)
Jul 31 17:00:14 kernel: <4>[<c00416f0>] (die+0x1a4/0x1dc) from [<c004515c>] (__do_kernel_fault+0x6c/0x8c)
Jul 31 17:00:14 kernel: <4>[<c004515c>] (__do_kernel_fault+0x6c/0x8c) from [<c00452cc>] (do_page_fault+0x150/0x1ec)
Jul 31 17:00:14 kernel: <4>[<c00452cc>] (do_page_fault+0x150/0x1ec) from [<c003d3a4>] (do_DataAbort+0x30/0x9c)
Jul 31 17:00:14 kernel: <4>[<c003d3a4>] (do_DataAbort+0x30/0x9c) from [<c0386f2c>] (__dabt_svc+0x4c/0x60)
Jul 31 17:00:14 kernel: <4>Exception stack(0xc9611e88 to 0xc9611ed0)
Jul 31 17:00:14 kernel: <4>1e80:                   c0417a98 00000001 00000000 00000000 cb0606a0 c9c28c00
Jul 31 17:00:14 kernel: <4>1ea0: 00000000 00000001 c003d434 c03c4080 c9610000 c03ce948 00000000 c9611ed0
Jul 31 17:00:14 kernel: <4>1ec0: c00b96fc c00b9710 a0000013 ffffffff
Jul 31 17:00:14 kernel: <4>[<c0386f2c>] (__dabt_svc+0x4c/0x60) from [<c00b9710>] (remove_vm_area+0x4c/0x88)
Jul 31 17:00:14 kernel: <4>[<c00b9710>] (remove_vm_area+0x4c/0x88) from [<c00b97a0>] (__vunmap+0x24/0xf4)
Jul 31 17:00:14 kernel: <4>[<c00b97a0>] (__vunmap+0x24/0xf4) from [<bf7d7d64>] (deinit_ct_extra+0xb0/0x14c [bw_forward])
Jul 31 17:00:14 kernel: <4>[<bf7d7d64>] (deinit_ct_extra+0xb0/0x14c [bw_forward]) from [<bf7d3b60>] (__forward_exit+0x16c/0x1e4 [bw_forward])
Jul 31 17:00:14 kernel: <4>[<bf7d3b60>] (__forward_exit+0x16c/0x1e4 [bw_forward]) from [<c008f27c>] (sys_delete_module+0x16c/0x218)
Jul 31 17:00:14 kernel: <4>[<c008f27c>] (sys_delete_module+0x16c/0x218) from [<c003dac0>] (ret_fast_syscall+0x0/0x30)
Jul 31 17:00:14 kernel: 3ff8>] (unwind_backtrace+0x0/0xfffff
Jul 31 17:00:14 kernel: <0>Rebooting in 3 seconds..

<snip>
Dec 10 21:15:10 kernel: * Make sure sizeof(struct sw_struct)=160 is consistent
Dec 10 21:15:11 kernel: IDPfw: TrendMicro forward module ver-1.0.34
Dec 10 21:15:11 kernel: IDPfw: Apply module param dev_wan=eth0
Dec 10 21:15:11 kernel: IDPfw: Apply module param sess_num=30000
Dec 10 21:15:11 kernel: [alloc_udb:323] Successfully malloc udb 3084 bytes at ceea8000
Dec 10 21:15:11 kernel: [alloc_app:387] Successfully malloc udb 131088 bytes at d2e2f000
Dec 10 21:15:11 kernel: [alloc_patroldb:836] Successfully malloc devdb 268602 bytes at d2e52000
Dec 10 21:15:11 kernel: IDPfw: Init chrdev /dev/idpfw with major 191
Dec 10 21:15:11 kernel: IDPfw: IDPfw is ready
Dec 10 21:15:11 kernel: sizeof forward param = 160

 

[snbdora]

hi,

Did you find any resolution for this reboot issue? I've had a few random kernel reboot on this router for a while and even on the latest firmware.

Thanks

 

[BSOD2600]

hi,

Did you find any resolution for this reboot issue? I've had a few random kernel reboot on this router for a while and even on the latest firmware.

Nope

latest firmware update 380.69_2 I noticed there was an entry for this model fixing a watchdog timeout, so wonder if related. anyways, just got a crash a few days ago:

Jul 31 17:00:12 kernel: _ Reboot message ... _______________________________________________________
Jul 31 17:00:12 kernel: <1>Unable to handle kernel NULL pointer derefere�U�ule:  ip6table_mangle    bf703000      qmi_wwan    bf6cf000    5780
Jul 31 17:00:12 kernel: <4>mo  usblp    bf694000    10321
Jul 31 17:00:12 kernel: <4>modufs    bf5d2000    454497
Jul 31 17:00:12 kernel: <4>module: 
Jul 31 17:00:12 kernel: <4>module:  ext3    bf509000    1064>module:  wl    bf032000    3938142
Jul 31 17:00:12 kernel: <4>module:  igs    bf02a000    1188367
Jul 31 17:00:12 kernel: <4>Modules linked in: nf_nat crc16 jbd2 ext3 jbd mbcache usbnat_proto_gre nf_conntrack_protoU: 1    Tainted: P             (c004dc10>] (dev_nvram_read+0x6c/ ^C1ec)
Jul 31 17:00:12 kernel: ____________________________________________________________________________

 

[sfx2000]

So it could just be that after a while the router is trying to write data beyond the end of the device.

Interesting - not the first time in the last couple of weeks that things mentioned are running off the end of the world...

Someone should probably look into this.

 

[snbdora]

I don't have any USB drive connected and still has the reboot issue. I've the stock firmware installed. I wish I could get in touch with the product engineer at Asus directly instead of the useless L1 support. This was the last reboot message..

Feb 13 16:00:13 kernel: _ Reboot message ... _______________________________________________________
Feb 13 16:00:13 kernel: <1>Unable to handle kernel paging request at vir?U?e:  nf_conntrack_rtsp    bf714000    4067
Feb 13 16:00:13 kernel: <4>module:  nf_nat_ftp    bf70e000    1144
Feb 13 16:00:13 kernel: <4>module:  nf_conn ext2    bf614000    52776
Feb 13 16:00:13 kernel: <4>module 4599
Feb 13 16:00:13 kernel: <4>module:  usb_storage    b00    4494
Feb 13 16:00:13 kernel: <4>module:  ip6table_fid000    5816
Feb 13 16:00:13 kernel: <4>module:  cdc_wdm    ndis_host    bf4d7000    5216
Feb 13 16:00:13 kernel: <4>mod000    11900
Feb 13 16:00:13 kernel: <4>module:  mii    bf4b1000    3367
Feb 13 16:00:13 kernel: <4>module:  ohci_hcd    20875
Feb 13 16:00:13 kernel: <4>Modules linked in: wl(Pebtable_nat ebtables tdts_udbfw f_conntrack_sip nf_nat_h323 nf_cntrack_rtsp nf_nat_ftp nf_conntrack_ftp ip6table_mangle usblp thhost cdc_ether asix cdc_acm usbn78a40>]    psr: 60000113
Feb 13 16:00:13 kernel: <4>sp : c03bfaf0  ip : 00000000  fp : 0                       00000000 0 00000000 c94aa2a0
Feb 13 16:00:13 kernel: <0>fc20: bf02a52c cfb66a00 00000001 0000000e 00000000 00000000 00000030 c94a6e93124 00000000 00000000
Feb 13 16:00:13 kernel: <0>fce ^C000
Feb 13 16:00:13 kernel: <0>fe20: 00000000 c6e9311e cf 00000000 00000000 c03bffc0 c003a4>] (do_DataAbort+0x30/0x9c) fc0378a40 c0378a44 60000113 fffff
Feb 13 16:00:13 kernel: ____________________________________________________________________________

 

[BSOD2600]

Few more weeks and a slightly new panic message. This time I clicked the reboot button and noticed the following in the logs.

Feb 28 20:09:24 kernel: usbcore: deregistering interface driver usblp

Jul 31 17:00:14 kernel: _ Reboot message ... _______________________________________________________
Jul 31 17:00:14 kernel: <1>Unable to handle kernel NULL pointer derefere�U�at virtual address 00000084
Jul 31 17:00:14 kernel: <1>pgd = cf914000
Jul 31 17:00:14 kernel: <1>[00000084] *pgd=9faef031, *pte=00000000, *ppte=00000000
Jul 31 17:00:14 kernel: <0>Internal error: Oops: 17 [#1] PREEMPT SMP
Jul 31 17:00:14 kernel: <0>last sysfs file: /sys/devices/pci0000:00/0000:00:0b.1/usb2/2-2/2-2:1.0/host0/target0:0:0/0:0:0:0/model

Jul 31 17:00:14 kernel: <4>CPU: 0    Tainted: P             (2.6.36.4brcmarm #2)
Jul 31 17:00:14 kernel: <4>PC is at file_sb_list_add+0x30/0x54
Jul 31 17:00:14 kernel: <4>LR is at file_sb_list_add+0x10/0x54
Jul 31 17:00:14 kernel: <4>pc : [<c00cb8d0>]    lr : [<c00cb8b0>]    psr: 60000013
Jul 31 17:00:14 kernel: <4>sp : cf825e60  ip : cf820000  fp : 00000000
Jul 31 17:00:14 kernel: <4>r10: c97ef398  r9 : 00000000  r8 : cf990140
Jul 31 17:00:14 kernel: <4>r7 : cf51c660  r6 : cef40640  r5 : 00000000  r4 : cef40640
Jul 31 17:00:14 kernel: <4>r3 : c0418cbc  r2 : 081da000  r1 : 00000000  r0 : cef40640
Jul 31 17:00:14 kernel: <4>Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Jul 31 17:00:14 kernel: <4>Control: 10c53c7d  Table: 9f91404a  DAC: 00000015
Jul 31 17:00:14 kernel: <0>Process preinit (pid: 1, stack limit = 0xcf824270)
Jul 31 17:00:14 kernel: <0>Stack: (0xcf825e60 to 0xcf826000)
Jul 31 17:00:14 kernel: <0>5e60: c02d5514 00000000 cef40640 c00c8990 cf825ef0 00000000 00000000 00000024
Jul 31 17:00:14 kernel: <0>5e80: 00000000 ffffff9c cf51c660 c00d6060 00000000 ffffff9c cf824000 00000000
Jul 31 17:00:14 kernel: <0>5ea0: 00000000 cf825ef0 c9743000 00000000 cf824000 c00d64a4 00000000 00000000
Jul 31 17:00:14 kernel: <0>5ec0: cf825f48 cf825f50 cfa65bc0 c8d7f420 c03d1420 00000001 00000000 00000000
Jul 31 17:00:14 kernel: <0>5ee0: 000003ff c0111f60 00000001 cef40f00 cf990140 cf51c660 00b118fa 00000007
Jul 31 17:00:14 kernel: <0>5f00: c974300a cfa3c1e0 cf411000 00000101 00000000 00000000 cef40f08 00000001
Jul 31 17:00:14 kernel: <0>5f20: cef40f08 cf51caa0 c03cd080 c015bd74 00000001 cf51caa0 c03d2c78 00000001
Jul 31 17:00:14 kernel: <0>5f40: 00000000 cef40640 cf816000 cf816008 cf990140 cf51c660 00000000 c9743000
Jul 31 17:00:14 kernel: <0>5f60: ffffff9c 00000008 00000000 00000000 cf824000 00000000 0011bc85 c00c994c
Jul 31 17:00:14 kernel: <0>5f80: 00000024 cf816000 bea14328 bea14328 00010021 00000005 c003dc68 cf824000
Jul 31 17:00:14 kernel: <0>5fa0: 00000000 c003dac0 bea14328 bea14328 bea14784 00000000 00000000 bea14796
Jul 31 17:00:14 kernel: <0>5fc0: bea14328 bea14328 00010021 00000005 bea14338 0000000a bea14328 0011bc85
Jul 31 17:00:14 kernel: <0>5fe0: 4012b7b8 bea142d0 401006ec 4061c954 60000010 bea14784 1bf0e7eb dff4ce4b
Jul 31 17:00:14 kernel: <4>[<c00cb8d0>] (PC is at file_sb_list_add+0x30/0x54)
Jul 31 17:00:14 kernel: <4>[<c00cb8d0>] (file_sb_list_add+0x30/0x54) from [<c00c8990>] (__dentry_open.clone.12+0xe8/0x2a4)
Jul 31 17:00:14 kernel: <4>[<c00c8990>] (__dentry_open.clone.12+0xe8/0x2a4) from [<c00d6060>] (do_last.clone.33+0x34c/0x5f0)
Jul 31 17:00:14 kernel: <4>[<c00d6060>] (do_last.clone.33+0x34c/0x5f0) from [<c00d64a4>] (do_filp_open+0x1a0/0x4fc)
Jul 31 17:00:14 kernel: <4>[<c00d64a4>] (do_filp_open+0x1a0/0x4fc) from [<c00c994c>] (do_sys_open+0x5c/0xfc)
Jul 31 17:00:14 kernel: <4>[<c00c994c>] (do_sys_open+0x5c/0xfc) from [<c003dac0>] (ret_fast_syscall+0x0/0x30)
Jul 31 17:00:14 kernel: <0>Code: e34c3041 e5922014 e584201c e7932102 (e5953084) 
Jul 31 17:00:14 kernel: <4>---[ end trace b0ae0b02c034b3bc ]---
Jul 31 17:00:14 kernel: <0>Kernel panic - not syncing: Fatal exception
Jul 31 17:00:14 kernel: <4>[<c0043ff8>] (unwind_backtrace+0x0/0xf8) from [<c02cbc5c>] (panic+0x7c/0x1a8)
Jul 31 17:00:14 kernel: <4>[<c02cbc5c>] (panic+0x7c/0x1a8) from [<c00416f0>] (die+0x1a4/0x1dc)
Jul 31 17:00:14 kernel: <4>[<c00416f0>] (die+0x1a4/0x1dc) from [<c004515c>] (__do_kernel_fault+0x6c/0x8c ^C>[<c004515c>] (__do_kernel_fault+0x6c/0x8c) from [<c00452cc>] (do_page_fault+0x150/0x1ec)
Jul 31 17:00:14 kernel: <4>[<c00452cc>] (do_page_fault+0x150/0x1ec) from [<c003d3a4>] (do_DataAbort+0x30/0x9c)
Jul 31 17:00:14 kernel: <4>[<c003d3a4>] (do_DataAbort+0x30/0x9c) from [<c0386f2c>] (__dabt_svc+0x4c/0x60)
Jul 31 17:00:14 kernel: <4>Exception stack(0xcf825e18 to 0xcf825e60)
Jul 31 17:00:14 kernel: <4>5e00:                                                       cef40640 00000000
Jul 31 17:00:14 kernel: <4>5e20: 081da000 c0418cbc cef40640 00000000 cef40640 cf51c660 cf990140 00000000
Jul 31 17:00:14 kernel: <4>5e40: c97ef398 00000000 cf820000 cf825e60 c00cb8b0 c00cb8d0 60000013 ffffffff
Jul 31 17:00:14 kernel: <4>[<c0386f2c>] (__dabt_svc+0x4c/0x60) from [<c00cb8d0>] (file_sb_list_add+0x30/0x54)
Jul 31 17:00:14 kernel: <4>[<c00cb8d0>] (file_sb_list_add+0x30/0x54) from [<c00c8990>] (__dentry_open.clone.12+0xe8/0x2a4)
Jul 31 17:00:14 kernel: <4>[<c00c8990>] (__dentry_open.clone.12+0xe8/0x2a4) from [<c00d6060>] (do_last.clone.33+0x34c/0x5f0)
Jul 31 17:00:14 kernel: <4>[<c00d6060>] (do_last.clone.] (do_filp_open+0x1a0/0x4fc)
Jul 31 17:00:14 kernel: <4>[<c00d64a4>] (do_filp_open+0x1a0/0x4fc) from [<c00c994c>] (do_sy<c003dac0>] (ret_fast_syscall+0xfffffff
Jul 31 17:00:14 kernel: <0>Rebooting in 3 seconds..

Feb 28 20:11:48 rc_service: ntp 1071:notify_rc restart_upnp

 

[snbdora]

Not sure if Merlin has a clue or know a possible cause of these panic but I think this issue is something Asus should look into. If you haven't logged a case with Asus support, I would suggest to do that. I did that but I've not heard from them yet. I did ask them to escalate to engineering/product support teams.

 

[snbdora]

FYI - @arthurlien suggested to disable the RTSP passthrough. I did that and I haven't had a reboot since then.

 

[BSOD2600]

FYI - @arthurlien suggested to disable the RTSP passthrough. I did that and I haven't had a reboot since then.

Alrighty, disabled. I dont use it anyways lets see if that makes a difference over the coming weeks...

 

[RMerlin]

If it does, then it's an interesting finding from Asus, especially since I can't see the relation there.

 

[ColinTaylor]

If it does, then it's an interesting finding from Asus, especially since I can't see the relation there.

I'm doubtful about this as a solution as well. But one can always hope.

I suspect the recommendation is because @snbdora's crash dump had nf_conntrack_rtsp as the first module listed. I don't think you can read too much into that as a) the first line of output of the dumps is always corrupted (maybe someone should look into that), and b) @BSOD2600's dumps are different from @snbdora's (mostly referring to USB).

 

[RMerlin]

I'm doubtful about this as a solution as well. But one can always hope.

I suspect the recommendation is because @snbdora's crash dump had nf_conntrack_rtsp as the first module listed. I don't think you can read too much into that as a) the first line of output of the dumps is always corrupted (maybe someone should look into that), and b) @BSOD2600's dumps are different from @snbdora's (mostly referring to USB).

Let's not forget however that there might be different reasons for a kernel to crash, it's not necessarily one single cause for everyone affected. It's not unlikely that at least in that one case, the rtsp module was causing it. We're dealing with fairly old kernel modules here, and those NAT helpers were never stellar examples of coding. Quite a few people saw bogus messages from the sip helper over the years as it was trying to interpret any traffic found on port 5060 as SIP.