Release Asuswrt-Merlin 386.12_6 is now available for AC models

[RMerlin]

I think I should have seen some accumulation in the /tmp/ folder by now if that's where the culprit lies?

Unless it`s a specific event that when it occurs ends up filling it up within a short period of time.

 

[Sixsonic]

chromecast not showing up on RT-AC88U with this latest version, must be something in the firmware? RT-AC68U works fine.

 

[heywire]

Unless it`s a specific event that when it occurs ends up filling it up within a short period of time.

Any other suggestions? Do I just have to let it crash again, and then hopefully the /tmp/ folder will show something? (Will it, considering that an automatic restart occurs?)

How to gather information about what really happened, if the logs don't show anything but what I posted earlier after the last crash? (they're in "debug" message log level, btw)

 

[RMerlin]

Any other suggestions?

Remove any third party addons, and monitor through top.

All I can say is I have two routers here running multiple weeks with no sign of any memory leak.

 

[Martinski]

Any other suggestions? Do I just have to let it crash again, and then hopefully the /tmp/ folder will show something? (Will it, considering that an automatic restart occurs?)

How to gather information about what really happened, if the logs don't show anything but what I posted earlier after the last crash? (they're in "debug" message log level, btw)

I have a shell script that I've used in the past for diagnostics purposes to monitor & check scenarios leading to OOM crashes. The script takes a snapshot of RAM usage stats from a few sources and logs it into a file. When running as a cron job (e.g. logging every hour), it can provide clues about any RAM usage trends & show correlations that might point to the problem causing the OOM condition, especially if an unexpectedly large file is filling up the "tmpfs" filesystem.

If you're interested in trying it for your particular case, let me know and I can put it in my GitHub repository.

 

[heywire]

Remove any third party addons, and monitor through top.

All I can say is I have two routers here running multiple weeks with no sign of any memory leak.

No scripts. I have "Enable JFFS custom scripts and configs" set to Disabled (Administration/system).

Here is a printout of top (not much load on the network atm. only internet use from 2 clients). Nothing looks out of the usual here?:

Mem: 375056K used, 50964K free, 3832K shrd, 0K buff, 53932K cached
CPU: 0.2% usr 0.4% sys 0.0% nic 99.2% idle 0.0% io 0.0% irq 0.1% sirq
Load average: 3.24 3.05 3.02 1/158 8107
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
1264 1 admin S 31000 7.2 1 0.2 networkmap --bootwait
208 2 admin SW 0 0.0 0 0.1 [bcmsw_rx]
1132 1 admin S 10864 2.5 1 0.1 asd
1221 1 admin S 11836 2.7 0 0.0 watchdog
1315 1 admin S 21084 4.9 0 0.0 roamast
1338 1 admin S 13576 3.1 1 0.0 cfg_server
1734 1 admin S 13496 3.1 0 0.0 wred -B
1100 1 admin S 11836 2.7 1 0.0 /sbin/wanduck
1275 1 admin S 11124 2.6 0 0.0 mastiff
1218 1 admin S 6976 1.6 0 0.0 /usr/sbin/infosvr br0
6460 6426 admin R 3328 0.7 0 0.0 top
3 2 admin SW 0 0.0 0 0.0 [ksoftirqd/0]
1317 1 admin S 53836 12.6 1 0.0 conn_diag
285 1 admin S 18508 4.3 1 0.0 /bin/swmdk
6251 1 admin S < 15664 3.6 1 0.0 dcd -i 3600 -p 43200 -b -d /tmp/bwdpi/
1360 1 admin S 15404 3.6 0 0.0 amas_lib
1139 1 admin S 13884 3.2 0 0.0 /sbin/netool
1137 1 admin S 13572 3.1 0 0.0 nt_monitor
1 0 admin S 13528 3.1 0 0.0 /sbin/init
1211 1 admin S 12860 3.0 1 0.0 httpd -i br0
1284 1 admin S 11840 2.7 1 0.0 pctime
1483 1 admin S 11836 2.7 0 0.0 usbled
1793 1 admin S 11836 2.7 0 0.0 bwdpi_wred_alive
1225 1 admin S 11836 2.7 1 0.0 amas_lanctrl
1029 1 admin S 11836 2.7 1 0.0 /sbin/PS_pod
1222 1 admin S 11836 2.7 0 0.0 check_watchdog
1277 1 admin S 11836 2.7 1 0.0 bwdpi_check
12244 1 admin S 11836 2.7 0 0.0 pc_block
1581 1 admin S 11836 2.7 0 0.0 disk_monitor
1161 1 admin S 11836 2.7 1 0.0 wpsaide
1251 1214 admin S 11568 2.7 0 0.0 vis-dcon
1150 1 admin S 11492 2.6 1 0.0 nt_center
1210 1 admin S 11060 2.5 1 0.0 httpds -s -i br0 -p 8443
12802 1 admin S 9768 2.2 1 0.0 /usr/sbin/smbd -D -s /etc/smb.conf
12798 1 admin S 9488 2.2 1 0.0 /usr/sbin/nmbd -D -s /etc/smb.conf
12801 12798 admin S 9464 2.2 0 0.0 /usr/sbin/nmbd -D -s /etc/smb.conf
1184 1 admin S 8100 1.9 1 0.0 nt_actMail
1484 1 admin S 7960 1.8 1 0.0 usbmuxd
1168 1 admin S 6980 1.6 0 0.0 /usr/sbin/bsd
1138 1 admin S 6836 1.6 1 0.0 protect_srv
12667 1 admin S 6576 1.5 1 0.0 /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
12668 12667 admin S 6576 1.5 1 0.0 /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
1214 1 admin S 5424 1.2 0 0.0 vis-dcon
1220 1 admin S 5196 1.2 1 0.0 sysstate
1177 1 admin S 5044 1.1 1 0.0 /usr/sbin/wlceventd
1217 1 admin S 4816 1.1 1 0.0 vis-datacollector
1164 1 admin S 4664 1.0 1 0.0 /usr/sbin/wlc_nt
1224 1 admin S 4284 1.0 0 0.0 rstats
1162 1 admin S 3928 0.9 1 0.0 nas
1529 1 admin S 3680 0.8 1 0.0 /usr/sbin/ntp -t -S /sbin/ntpd_synced -p pool.ntp.org -p time.nist.gov
1165 1163 admin S 3504 0.8 1 0.0 /usr/sbin/wl -i eth5 noise
6426 6405 admin S 3328 0.7 1 0.0 -sh
1209 1 admin S 3324 0.7 1 0.0 crond -l 9
1033 1 admin S 3324 0.7 1 0.0 /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 7
1035 1 admin S 3324 0.7 0 0.0 /sbin/klogd -c 5
1163 1317 admin S 3324 0.7 1 0.0 sh -c /usr/sbin/wl -i eth5 noise
1391 1 admin S < 3324 0.7 1 0.0 sh -c hwinfo hw_check_result
12783 1 admin S 3324 0.7 0 0.0 /sbin/udhcpc -i eth0 -p /var/run/udhcpc0.pid -s /tmp/udhcpc_wan -A5 -O33 -O249
1336 1332 nobody S 3316 0.7 0 0.0 lldpd -L /usr/sbin/lldpcli -I eth1,eth2,eth3,eth4,eth5,eth6,wl0.1,wl0.3,wl1.1,wl1.
1332 1 admin S 3316 0.7 0 0.0 lldpd -L /usr/sbin/lldpcli -I eth1,eth2,eth3,eth4,eth5,eth6,wl0.1,wl0.3,wl1.1,wl1.
1246 1 admin S 3220 0.7 1 0.0 lld2d br0
7110 1 admin S 3132 0.7 0 0.0 /usr/sbin/acsd
1392 1391 admin S < 3120 0.7 1 0.0 hwinfo hw_check_result
1216 1 nobody S 3100 0.7 1 0.0 avahi-daemon: running [RT-AC86U-5268.local]
1283 1 admin S 2968 0.7 1 0.0 lld2c br0
1159 1 admin S 2936 0.6 0 0.0 /bin/eapd
1198 1 admin S 2900 0.6 1 0.0 /usr/sbin/dhd_monitor
12781 1 nobody S 2628 0.6 0 0.0 dnsmasq --log-async
12782 12781 admin S 2628 0.6 0 0.0 dnsmasq --log-async
6405 1369 admin S 2496 0.5 0 0.0 dropbear -p 192.168.2.1:44446 -j -k
12649 1 admin S 2412 0.5 0 0.0 miniupnpd -f /etc/upnp/config
1140 1 admin S 2376 0.5 1 0.0 /usr/sbin/haveged -r 0 -w 1024 -d 32 -i 32
1369 1 admin S 2368 0.5 1 0.0 dropbear -p 192.168.2.1:44446 -j -k
12565 1 admin S 1916 0.4 0 0.0 /bin/mcpd
12800 1 admin S 1856 0.4 1 0.0 /usr/sbin/wsdd2 -d -w -i br0 -b sku:RT-AC86U,serial:244bfe605268
299 1 admin S 1752 0.4 1 0.0 /usr/sbin/envrams
296 1 admin S 1712 0.4 0 0.0 hotplug2 --persistent --no-coldplug
292 1 admin S 1568 0.3 1 0.0 {wdtctl} wdtd
209 2 admin SW 0 0.0 1 0.0 [bcmsw]
14 2 admin SW 0 0.0 1 0.0 [ksoftirqd/1]
362 2 admin SWN 0 0.0 0 0.0 [jffs2_gcd_mtd9]
555 2 admin SW 0 0.0 1 0.0 [dhd_watchdog_th]
561 2 admin SW 0 0.0 1 0.0 [dhd_watchdog_th]
7 2 admin SW 0 0.0 1 0.0 [rcu_preempt]
20 2 admin SW 0 0.0 0 0.0 [kworker/0:1]
7106 2 admin SW 0 0.0 1 0.0 [kworker/1:1]
204 2 admin SW 0 0.0 0 0.0 [bcmFlwStatsTask]
13 2 admin SW 0 0.0 1 0.0 [migration/1]
10 2 admin SW 0 0.0 0 0.0 [migration/0]
11 2 admin SW 0 0.0 0 0.0 [watchdog/0]
12 2 admin SW 0 0.0 1 0.0 [watchdog/1]
170 2 admin SWN 0 0.0 0 0.0 [jffs2_gcd_mtd2]
6 2 admin SW 0 0.0 0 0.0 [kworker/u4:0]
26 2 admin SW 0 0.0 1 0.0 [bcmFapDrv]
2 0 admin SW 0 0.0 0 0.0 [kthreadd]
5 2 admin SW< 0 0.0 0 0.0 [kworker/0:0H]
8 2 admin SW 0 0.0 0 0.0 [rcu_sched]
9 2 admin SW 0 0.0 0 0.0 [rcu_bh]
16 2 admin SW< 0 0.0 1 0.0 [kworker/1:0H]
17 2 admin SW< 0 0.0 1 0.0 [khelper]
18 2 admin SW 0 0.0 1 0.0 [kdevtmpfs]
19 2 admin SW< 0 0.0 1 0.0 [writeback]
21 2 admin SWN 0 0.0 1 0.0 [ksmd]
22 2 admin SW< 0 0.0 1 0.0 [crypto]
23 2 admin SW< 0 0.0 1 0.0 [bioset]
24 2 admin SW< 0 0.0 1 0.0 [kblockd]
25 2 admin DW 0 0.0 1 0.0 [skbFreeTask]
27 2 admin SWN 0 0.0 1 0.0 [kswapd0]
28 2 admin SW 0 0.0 1 0.0 [fsnotify_mark]

 

[heywire]

I have a shell script that I've used in the past for diagnostics purposes to monitor & check scenarios leading to OOM crashes. The script takes a snapshot of RAM usage stats from a few sources and logs it into a file. When running as a cron job (e.g. logging every hour), it can provide clues about any RAM usage trends & show correlations that might point to the problem causing the OOM condition, especially if an unexpectedly large file is filling up the "tmpfs" filesystem.

If you're interested in trying it for your particular case, let me know and I can put it in my GitHub repository.

Thanks for the offer!

@RMerlin do you think this would be helpful in this case?

/tmp/ is rock stable around 2.6-2.9M every time I check, so my uneducated guess is that this is some sort of "flash-crash" event where something goes.... well, "heywire" very fast? Or how long does such an even usually take? Hence, I don't know if logging it every hour would be often enough?

Are there any other ways to catch and gather data from this memory leak event as it happens, that would gain further insight which can help diagnosis?

 

[Martinski]

/tmp/ is rock stable around 2.6-2.9M every time I check, so my uneducated guess is that this is some sort of "flash-crash" event where something goes.... well, "heywire" very fast? Or how long does such an even usually take? Hence, I don't know if logging it every hour would be often enough?

If you use a high-capacity USB-attached drive for the logfile path, you can log the stats every 15 minutes, or whatever you think is appropriate for your case.

Not on routers, but I've diagnosed OOM crashes that have happened within hours of the initial event that leads to the crash. But if it happens faster than that on your router, perhaps logging every 10-15 minutes might work for you.

Just my 2 cents.

 

[Tomo]

Any other suggestions?

Are your temperatures ok? If you have the option, use another power supply.
In 5 years my ac86 with jffs, usb, script ( adguardhome ), imesh has not had any anomalous restarts, i only took care to fix the thermal pads with copper plates and i brought the temperatures back between 50-70 degrees, i think you have some hardware problem on your router.

 

[KrypteX]

@heywire Tomo raised a great question!

 

[heywire]

Are your temperatures ok? If you have the option, use another power supply.
In 5 years my ac86 with jffs, usb, script ( adguardhome ), imesh has not had any anomalous restarts, i only took care to fix the thermal pads with copper plates and i brought the temperatures back between 50-70 degrees,

Temperatures 2.4 GHz: 45°C - 5 GHz: 51°C - CPU: 74°C
CPU Load Average (1, 5, 15 mins) 3.39, 3.03, 2.96

I don't have the exact indoors temperature, but about 22°C (winter in Norway outside, heating by thermostat electric oven inside, so no wild swings to higher temperature).

This doesn't seem out of the ordinary from what I can remember from before when I have had stable firmware versions installed.

(btw. I upgraded to .11 again, because the same crashes after seeming stability for days (unlike .12) were happening with both).

With respect to power supply, are issues with these known to have caused memory leaks? I guess it could destabilize something and lead to it, but I think it would have happened more frequently, and not days and weeks between?

i think you have some hardware problem on your router.

People have been drumming my ears full of that before when I had a hell of a time with intermittent connection drops and crashes, and it turned out to be a firmware issue. So I'm skeptical of such claims..

 

[Tomo]

This doesn't seem out of the ordinary from what I can remember from before when I So I'm skeptical of such claims..

if you are convinced that the problem is the firmware then mount on the original one and go backwards until you find the stability you are looking for, but I doubt you will find it, it seems clear to me that there is something wrong with the hardware, I repeat it is not has ever had a reboot, ever.

 

[bibikalka]

if you are convinced that the problem is the firmware then mount on the original one and go backwards until you find the stability you are looking for, but I doubt you will find it, it seems clear to me that there is something wrong with the hardware, I repeat it is not has ever had a reboot, ever.

This makes a lot of sense! If @heywire believes that there was a magic past firmware that worked forever, it could be chased down by bi-section.

386.7_2 was a multi-month stretch without updates, so good to try that one first. Router temperatures seem OK, so a temperature related issue is probably not the first on the list.

 

[KrypteX]

As @bibikalka said above, 386.7_2 was one of the most stable versions (GPL 386_48966), so it's worth a try, @heywire .

Then stability issues started occurring with newer GPLs, as far as I can see:
386.9 - 386.11 (GPL 386_50757)
386.12+ (GPL 386_51997)

 

[Martinski]

Thanks for the offer!

The script "LogMemoryStats.sh" is available on GitHub:

curl -kLSs --retry 3 --retry-delay 5 --retry-connrefused \
https://raw.githubusercontent.com/Martinski4GitHub/CustomMiscUtils/master/Diags/LogMemoryStats.sh \
-o /jffs/scripts/LogMemoryStats.sh
chmod 755 /jffs/scripts/LogMemoryStats.sh

I'd recommend looking at the script file header for some brief comments/documentation I added for new users. Also, check the "CUSTOMIZABLE PARAMETERS SECTION" section below the script header. You might need to modify the directory path where the log file will be created to suit your own router setup:

scriptLogDPath="/opt/var/log"

For diagnostics purposes, here's a suggestion to run it as a cron job:

cru a LogMemStats "*/20 * * * * /jffs/scripts/LogMemoryStats.sh"

HTH

 

[cebu99]

As @bibikalka said above, 386.7_2 was one of the most stable versions (GPL 386_48966), so it's worth a try, @heywire .

Then stability issues started occurring with newer GPLs, as far as I can see:
386.9 - 386.11 (GPL 386_50757)
386.12+ (GPL 386_51997)

Is there a big security risk to using such an old firmware?

 

[Analog-1]

Is there a big security risk to using such an old firmware?

Yes, if there have been patches and likely there have been many.

 

[learning_curve]

~
Then stability issues started occurring with newer GPLs, as far as I can see:
~
386.12+ (GPL 386_51997)

Or, they did not, depending on your own Asus Router and the specific config you've applied to it.
I sold my AC-68U - C1 to a colleague, way back when it was running 386.7 (Merlin).
He's since regularly upgraded it and is now running it on 386-12_6 (Merlin) without ANY issues on his (quite simple, to be fair) home setup.
The usual caveat YMMV etc but it's very rarely, JUST the firmware, all on its own, that is the sole reason for people having stability issues.
Config (including choice of ISP and their config...) will usually be a factor too

 

[KrypteX]

Or, they did not, depending on your own Asus Router and the specific config you've applied to it.
I sold my AC-68U - C1 to a colleague, way back when it was running 386.7 (Merlin).
He's since regularly upgraded it and is now running it on 386-12_6 (Merlin) without ANY issues on his (quite simple, to be fair) home setup.
The usual caveat YMMV etc but it's very rarely, JUST the firmware, all on its own, that is the sole reason for people having stability issues.
Config (including choice of ISP and their config...) will usually be a factor too

What you are saying here defeats the whole purpose of it. If your colleague's config is one that doesn't bring the issues to light, then that's not evidence, just anecdotes.
For example, let's say your colleague restarts his AC68U every night. He will probably won't report anything of issue to you or us, for that matter. I hope you see my point.

 

[learning_curve]

What you are saying here defeats the whole purpose of it. If your colleague's config is one that doesn't bring the issues to light, then that's not evidence, just anecdotes.
For example, let's say your colleague restarts his AC68U every night. He will probably won't report anything of issue to you or us, for that matter. I hope you see my point.

Not really. I can only see semantics if I'm honest.
Your own post included the phrase; "...as far as I can see" which is quite speculative.
I had no issue with your post, other than it appeared to just be speculative, firmware, finger pointing.

The example I posted, was an AC68U running problem free on 386-12_6 (Merlin) that I have physically seen myself i.e. fact not speculation.
FWIW No he doesn't restart his AC68U every night and / or even regularly.
Why would he or should he, when it works perfectly well (for him) as it is?
Yes others will have problems with their AC68U / 386-12_6 (Merlin) hence the last two lines in my previous post.