WRT1900AC Spontaneous Reboots, lockups

[htismaqe]

Yeah, we don't have any MicroCenter's or Fry's here. Just Best Buy and Radio Shack.

I buy pretty much all of my gear on Amazon anymore. Customer service simply can't be beat.

Oh and FYI for Roger and chadster:

page generated on Sat May 31 21:57:49 UTC 2014

UpTime:
 21:57:49 up 2 days,  1:32, load average: 1.19, 1.25, 1.30

 

[chadster766]

I captured what occurs just before a random reboot in the WRT1900AC.

The attached picture shows the WRT1900AC WiFi configuration.

What took place prior to reboot:

1. Connected a PCI D-Link Wireless N adapter
2. Ran a iperf throughput test
3. Disconnected the PCI D-Link Wireless N adapter
4. Connected a USB D-Link Wireless AC adapter
5. Ran a iperf throughput test

As soon as I hit enter on the iperf command the reboot happened and I was recording the serial debug output.

Serial Output of just before reboot:

AP-STA-CONNECTED c8:d3:a3:fd:52:de
info: Received event notification_event-device-online with value "303538fc-0735-4466-88e8-351f0407189a".
info: Handled event notification_event-device-online.
info: Received event notification_event-device-offline with value "303538fc-0735-4466-88e8-

351f0407189a".
info: Handled event notification_event-device-offline.
Internal error: Oops: 817 [#1] SMP
Modules linked in: ap8x(P) mod_wdtutil(O) nf_nat_ftp nf_conntrack_ftp mod_bdutil(O)
CPU: 0    Tainted: P           O  (3.2.40 #1)
PC is at free_block+0x78/0x148
LR is at cache_flusharray+0x78/0xb4
pc : [<c00ee13c>]    lr : [<c0544174>]    psr: 000f0193
sp : c0735cd0  ip : cce8a000  fp : 00000000
r10: 00000000  r9 : 00200200  r8 : c07ad340
r7 : 00100100  r6 : cf802f40  r5 : 00000002  r4 : cf800840
r3 : cf8025a0  r2 : ceb493e0  r1 : ce286420  r0 : ffffffff
Flags: nzcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 10c53c7d  Table: 0d85c06a  DAC: 00000015
Process swapper/0 (pid: 0, stack limit = 0xc07342f0)
Stack: (0xc0735cd0 to 0xc0736000)
5cc0:                                     00000004 00000000 00000000 00000000
5ce0: d0d09040 00000004 cf802f20 cf8025a0 00000000 cf800840 c0734000 00000001
5d00: 000000f6 c0544174 cf802f20 cea50000 a00f0113 00000000 00000000 c00edfd8
5d20: ce9e9c00 ce9e9c01 cf95d800 c041ef54 00000001 cec77e00 ce9e9c01 c0365d84
5d40: 00000001 cefed7c0 ce2c8000 0000000c 0000000f c0734000 00000000 00000181
5d60: 00000002 ceb2db20 50015803 00000000 cf95d000 cec30c00 c058a778 c073f1a8
5d80: c073f690 c042aa68 c0502e5c ceb2db20 00000004 ceb2db20 ceb2db20 ceeb2200
5da0: ceb2db20 cf95d000 cec30c00 ceeb225c 00000001 c073f1a8 00200200 c043e110
5dc0: cec30c00 ceeb2200 ceb2db20 cf95d000 cec30c00 00000000 c073f1a8 c042ae88
5de0: ceeb225c cc0f4000 ceb2db20 ceb2db20 cc0f40ba ced8d440 ccf3ca00 00000001
5e00: c073f1a8 c0502f14 00000000 c0503cac bf0af48c ceb2db20 00000000 c0735e6c
5e20: cc0f40ba ceb2db20 00000000 c0735e6c cc0f40ba c0504010 00000000 c0503ac0
5e40: 80000000 00000000 ce970000 c0503dc4 c0733a84 c0734000 c073f690 c0426244
5e60: c073f688 00000020 cfb6d860 ceb2db20 000000a0 c073f690 c09c8ad8 00000000
5e80: 00000001 c09c8ad4 c09c8a74 00100100 c09c8a60 c0426580 c09c8ad8 00000000
5ea0: c0736080 c09c8ad8 c09c8a60 00000001 0000012c 00000040 c09c8a68 c07ba7e8
5ec0: 002989f7 c04278c0 c0427848 c0734000 00000001 c073604c 00000100 00000009
5ee0: c073f1a8 c07951c0 c074bf18 c0079bc4 c07385e0 c0738634 00000003 00000000
5f00: c001a578 c0734000 00000000 fbc21000 c0735f74 0000406a 562f5842 00000000
5f20: 00000000 c0079fe4 0000003b c000e150 c0065258 c0065264 200f0013 c054a914
5f40: 000bd573 000003e8 538a5d18 538a5d18 c09c7290 00000000 c074b5a4 00000000
5f60: 0000406a 562f5842 00000000 00000000 2e39ca47 c0735f88 c0065258 c0065264
5f80: 200f0013 ffffffff 538a5d18 000bd560 538a5d18 000bd573 562f5842 c09c7290
5fa0: c0734000 c03fe8bc 00000000 c0734000 c076fca4 c0551b70 c074618c c000e560
5fc0: 00000000 c073ef94 c072b97c c09c4d40 c0746184 c06fd7a8 c06fd1c0 00000bdb
5fe0: 00000100 c072b97c 10c53c7d c073e6c0 c072b978 00008040 00000000 00000000
[<c00ee13c>] (free_block+0x78/0x148) from [<c0544174>] (cache_flusharray+0x78/0xb4)
[<c0544174>] (cache_flusharray+0x78/0xb4) from [<c00edfd8>] (kfree+0x88/0xac)
[<c00edfd8>] (kfree+0x88/0xac) from [<c041ef54>] (__kfree_skb+0x208/0x2e4)
[<c041ef54>] (__kfree_skb+0x208/0x2e4) from [<c0365d84>] (mv_eth_tx+0x6c4/0x984)
[<c0365d84>] (mv_eth_tx+0x6c4/0x984) from [<c042aa68>] (dev_hard_start_xmit+0x3e8/0x510)
[<c042aa68>] (dev_hard_start_xmit+0x3e8/0x510) from [<c043e110>] (sch_direct_xmit+0x74/0x1e8)
[<c043e110>] (sch_direct_xmit+0x74/0x1e8) from [<c042ae88>] (dev_queue_xmit+0x2f8/0x588)
[<c042ae88>] (dev_queue_xmit+0x2f8/0x588) from [<c0502f14>] (br_dev_queue_push_xmit+0xb8/0xc4)
[<c0502f14>] (br_dev_queue_push_xmit+0xb8/0xc4) from [<c0503cac>] (br_handle_frame_finish+0x1ec/0x304)
[<c0503cac>] (br_handle_frame_finish+0x1ec/0x304) from [<c0504010>] (br_handle_frame+0x24c/0x28c)
[<c0504010>] (br_handle_frame+0x24c/0x28c) from [<c0426244>] (__netif_receive_skb+0x204/0x384)
[<c0426244>] (__netif_receive_skb+0x204/0x384) from [<c0426580>] (process_backlog+0x94/0x174)
[<c0426580>] (process_backlog+0x94/0x174) from [<c04278c0>] (net_rx_action+0x78/0x178)
[<c04278c0>] (net_rx_action+0x78/0x178) from [<c0079bc4>] (__do_softirq+0x9c/0x130)
[<c0079bc4>] (__do_softirq+0x9c/0x130) from [<c0079fe4>] (irq_exit+0x44/0x90)
[<c0079fe4>] (irq_exit+0x44/0x90) from [<c000e150>] (handle_IRQ+0x7c/0xbc)
[<c000e150>] (handle_IRQ+0x7c/0xbc) from [<c054a914>] (__irq_svc+0x34/0xe8)
[<c054a914>] (__irq_svc+0x34/0xe8) from [<c0065264>] (armadaxp_enter_idle+0x68/0xa0)
[<c0065264>] (armadaxp_enter_idle+0x68/0xa0) from [<c03fe8bc>] (cpuidle_idle_call+0xac/0x140)
[<c03fe8bc>] (cpuidle_idle_call+0xac/0x140) from [<c000e560>] (cpu_idle+0x78/0xcc)
[<c000e560>] (cpu_idle+0x78/0xcc) from [<c06fd7a8>] (start_kernel+0x2c8/0x324)
Code: e59d2004 e5943048 e7933002 e8910005 (e5802004)
---[ end trace 928fd6c625515c89 ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
[<c0013c88>] (unwind_backtrace+0x0/0xe4) from [<c0012640>] (handle_IPI+0xfc/0x17c)
[<c0012640>] (handle_IPI+0xfc/0x17c) from [<c054a914>] (__irq_svc+0x34/0xe8)
[<c054a914>] (__irq_svc+0x34/0xe8) from [<c0065264>] (armadaxp_enter_idle+0x68/0xa0)
[<c0065264>] (armadaxp_enter_idle+0x68/0xa0) from [<c03fe8bc>] (cpuidle_idle_call+0xac/0x140)
[<c03fe8bc>] (cpuidle_idle_call+0xac/0x140) from [<c000e560>] (cpu_idle+0x78/0xcc)
[<c000e560>] (cpu_idle+0x78/0xcc) from [<0053fa34>] (0x53fa34)

 

[sfx2000]

I captured what occurs just before a random reboot in the WRT1900AC.

The attached picture shows the WRT1900AC WiFi configuration.

What took place prior to reboot:

1. Connected a PCI D-Link Wireless N adapter
2. Ran a iperf throughput test
3. Disconnected the PCI D-Link Wireless N adapter
4. Connected a USB D-Link Wireless AC adapter
5. Ran a iperf throughput test

As soon as I hit enter on the iperf command the reboot happened and I was recording the serial debug output.

Serial Output of just before reboot:

<snip>

I have a good idea what it is, but it's been a while since I've worked on Marvell cores - and I've not worked on the Armada XP in any event, but I've seen something similar.

Chad - send the trace over to the Linksys Engineering group, the breakpoint is in your trace - depending on what JTAG they use (for ARM, Lauterbach is common, but a lot of folks also use the Abatron BDI3000 series...) - might need one or the other. They'll need to use the JTAG to step through the code one instruction at a time, but it should be a simple fix once they find it.

Be specific with the steps going into the crash, and ensure that you can reproduce this consistently...

 

[htismaqe]

Excellent work!

 

[chadster766]

The WRT1900AC seems to have a third radio for the guest network but I could be mistaken. Using my Galaxy and Wifi Analyzer I noticed and confirmed that the guest network uses the same channel and has the same power as the standard 2.4ghz network. I'm concerned that such a configuration will cause interference between the regular network and the guest network.

I also noticed that initially the guest SSID wasn't showing up. Only after I turned it on and off again did the SSID appear.

Screenshots attached and the WRT SSID is the standard 2.4ghz network.

I'm only going to give this to engineering once I get some of SNB's users thoughts on this.

 

[htismaqe]

The guest network is a virtual interface on the same radio as the main 2.4Ghz SSID.

There's no interference.

 

[htismaqe]

By the way, I got a new Blu-ray player today to replace one that got struck by lightning. When I went to add a DHCP reservation, the router crashed. I re-entered the GUI and it crashed again.

It hasn't done it since but I'm also noticing now that I have to change my DHCP reservations and commit them multiple times in order for them to take.

 

[chadster766]

The guest network is a virtual interface on the same radio as the main 2.4Ghz SSID.

There's no interference.

Thanks I'm glad to hear about the no interference.

I'm not familiar with a virtual interface on a radio. Does that create a new radio signal? WiFi Analyzer displays it as that.

 

[htismaqe]

Most wifi analyzers use SSIDs and channels to identify networks so when they see two different SSIDs using the same channel, they declare a conflict.

If you look at the sysinfo.cgi, you can see the two wireless radios - wdev0 (2.4Ghz) and wdev1 (5Ghz). Each of them have 2 interfaces. For example, the 2.4Ghz radio has wdev0ap0 (private SSID) and wdevap1 (guest network).

 

[chadster766]

Most wifi analyzers use SSIDs and channels to identify networks so when they see two different SSIDs using the same channel, they declare a conflict.

If you look at the sysinfo.cgi, you can see the two wireless radios - wdev0 (2.4Ghz) and wdev1 (5Ghz). Each of them have 2 interfaces. For example, the 2.4Ghz radio has wdev0ap0 (private SSID) and wdevap1 (guest network).

Awesome thanks for the clarification that the guest network isn't a concern for interference

 

[htismaqe]

Well, it's not JUST the GUI that is causing this but it just has to be part of it.

First of all, I've noticed the router is completely ignoring my DNS settings. I have DNS1 and DNS2 hard set to my ISPs DNS servers. DNS3 is still set to 0.0.0.0. All of the devices I checked this morning have a DNS server address of the router.

So I tried changing the settings and of course the router rebooted. So I disconnected it, brought it back to my office, and connected to it via ethernet cable.

I reset it to factory defaults and completely reconfigured it from scratch. During that entire period, it didn't reboot a single time.

I took it back in and plugged everything back in and noticed still that it's ignoring my DNS settings. So I tried changing them again and the router immediately rebooted.

 

[htismaqe]

Router rebooted FOUR times while looking at/editing the network map.

Hasn't rebooted once since I exited the GUI.

 

[sfx2000]

Well, it's not JUST the GUI that is causing this but it just has to be part of it.

First of all, I've noticed the router is completely ignoring my DNS settings. I have DNS1 and DNS2 hard set to my ISPs DNS servers. DNS3 is still set to 0.0.0.0. All of the devices I checked this morning have a DNS server address of the router.

So I tried changing the settings and of course the router rebooted. So I disconnected it, brought it back to my office, and connected to it via ethernet cable.

I reset it to factory defaults and completely reconfigured it from scratch. During that entire period, it didn't reboot a single time.

I took it back in and plugged everything back in and noticed still that it's ignoring my DNS settings. So I tried changing them again and the router immediately rebooted.

If I recall correctly, this is a feature of dnsmasq (which replaces BIND (named), nmbd (for SMB), and dhcpd (for client/server address management).

sfx

 

[sfx2000]

The guest network is a virtual interface on the same radio as the main 2.4Ghz SSID.

There's no interference.

agreed, this is normal - and also note that the IEEE OUI is a virtual value, not aligned back to Marvell or Linksys.

The reason for the different strength values - this is an averaged value over time in a rolling window - since the guest SSID is a separate 802.11 Beacon Frame, it'll show slightly different values sometimes.

 

[htismaqe]

I'm going to go ahead and attempt to open a ticket with Linksys.

 

[htismaqe]

If I recall correctly, this is a feature of dnsmasq (which replaces BIND (named), nmbd (for SMB), and dhcpd (for client/server address management).

sfx

So there's no way to force DHCP to assign static DNS addresses?

 

[htismaqe]

So first Linksys support says it sounds like my wireless devices are causing interference. I explained to them that I never have issues unless I enter the router GUI.

Then they tell me that I need to only access the router GUI via a wired PC.

Now they've settled on a "known compatibility issue" between the Linksys GUI and Safari.

I've asked them to go ahead and open a case so I can do more testing.

 

[sfx2000]

So there's no way to force DHCP to assign static DNS addresses?

It should be honoring the manual IP addresses you entered into the router configuration - dnsmasq is proxying the requests to the upstream if the lookup isn't cached locally - dnsmasq I believe is configured as a caching DNS server in the WRT1900ac software.

One of the issues, and one I should check into, is if one has an internal DNS server (for example 192.168.1.20) for internal usage, as one of the entries, along with the external DNS (either manual in router config or assigned as part of DHCP from the operator), that the internal DNS was being ignored...

 

[htismaqe]

It should be honoring the manual IP addresses you entered into the router configuration - dnsmasq is proxying the requests to the upstream if the lookup isn't cached locally - dnsmasq I believe is configured as a caching DNS server in the WRT1900ac software.

One of the issues, and one I should check into, is if one has an internal DNS server (for example 192.168.1.20) for internal usage, as one of the entries, along with the external DNS (either manual in router config or assigned as part of DHCP from the operator), that the internal DNS was being ignored...

Yeah, that's not happening here.

I'm putting my ISPs DSN servers in the 1st and 2nd slots. The WRT1900 simply isn't passing them on for whatever reason.

 

[sfx2000]

So first Linksys support says it sounds like my wireless devices are causing interference. I explained to them that I never have issues unless I enter the router GUI.

Then they tell me that I need to only access the router GUI via a wired PC.

Now they've settled on a "known compatibility issue" between the Linksys GUI and Safari.

I've asked them to go ahead and open a case so I can do more testing.

Find that hard to believe that it's a web client side issue, as reports on the reboots haven't been Mac specific...