Since upgrading to firmware 386.3_2, my Internet will not stay connected for even a day.

[DTS]

Just weird. I can't understand how that file could fail to write. And why it's even empty. There's plenty of storage. It's clearly writing leases to that file given we see the attempt. Can you write anything else to that same storage? Try the following script.

cat << 'EOF' > /var/lib/misc/temp.txt
123
456
789
EOF

cat /var/lib/misc/temp.txt

Sorry, I restarted httpd before I saw your message. I guess this is not relevant now? Or do you still want me to try this?

 

[eibgrad]

Sorry, I restarted httpd before I saw your message. I guess this is not relevant now? Or do you still want me to try this?

Probably irrelevant now. Yes, RAM was lower than usual, but there should still have been plenty of storage to handle the rather small dnsmasq.leases file. Not unless the exhaustion was only temporary at the moment the file was being written. That's why it's hard to diagnose these types of problems w/o being there to watch it.

 

[DTS]

after restarting httpd, the file /var/lib/misc/dnsmasq.leases has the expected content.

 

[eibgrad]

Perhaps it might be worth restarting some services like httpd on a nightly basis. Better than having to reboot the router.

 

[DTS]

Perhaps it might be worth restarting some services like httpd on a nightly basis. Better than having to reboot the router.

Yes, I would like to try restarting httpd weekly (or daily). How would I do it? I know the command (service restart_httpd), but I don't know how to put it into a cron job or similar on the router.

I have to step out for an hour. If anyone wants me to run any other diagnostics before I reboot, leave a post and I will run them when I return.

For example, I could try restarting the vpn tunnel instead of rebooting.

Thanks for all the help! It seems we have gained some important insight into the issue.

 

[eibgrad]

Is the router now fully functional?

 

[DTS]

Is the router now fully functional?

No, the vpn client tunnel is still down. I guess that is related to the earlier power glitch and is separate from this main issue.

pidof vpnclient1

returns nothing

 

[ColinTaylor]

Yes, RAM was lower than usual, but there should still have been plenty of storage to handle the rather small dnsmasq.leases file.

The information shown by df is misleading regarding tmpfs (I've written about this in detail in an old post). It shows the amount of storage that is available if there is free memory to support it. So if you've run out of memory you will be unable to write to tmpfs despite what df says.

 

[eibgrad]

Regarding the nightly restart of the httpd service, make sure JFFS custom configs and scripts are enabled under Administration->System. Then using the shell (ssh), copy/paste the following script into the window. It will automatically create and install the necessary services-start script. When you reboot, the schedule job(s) will automatically be applied.

#!/bin/sh

SCRIPTS_DIR='/jffs/scripts'
SCRIPT="$SCRIPTS_DIR/services-start"

mkdir -p $SCRIPTS_DIR

create_script() {
cat << "EOF" > $SCRIPT
#!/bin/sh
cru a restart_httpd "0 5 * * * /sbin/service restart_httpd"
EOF
chmod +x $SCRIPT
}

if [ -f $SCRIPT ]; then
    echo "error: $SCRIPT already exists; requires manual installation"
else
    create_script
    echo 'Done.'
fi

Note, if it finds an existing services-start script, it will NOT overwrite it. You'll instead have to add the schedule job(s) manually to the existing services-start script.

Of course, this still begs the question as to *why* it's consuming so much memory.

 

[DTS]

Regarding the nightly restart of the httpd service, make sure JFFS custom configs and scripts are enabled under Administration->System. Then using the shell (ssh), copy/paste the following script into the window. It will automatically create and install the necessary services-start script. When you reboot, the schedule job(s) will automatically be applied.

Thank you!

Of course, this still begs the question as to *why* it's consuming so much memory.

Yes, and if anybody wants to try to figure that out, I won't use the restart script and we'll continue to troubleshoot. Let me know.

 

[DTS]

FYI - the only way I found to get my router working properly was to power it off and back on. This is the same behavior I have seen in past incidents.

It was not sufficient to restart the vpnclient, nor even to reboot.

After a reboot, I could not access the GUI or login via SSH.

It took the power cycle before everything was operational, and this is exactly what I had to do each prior time it has happened.

 

[DTS]

Regarding the nightly restart of the httpd service, make sure JFFS custom configs and scripts are enabled under Administration->System. Then using the shell (ssh), copy/paste the following script into the window. It will automatically create and install the necessary services-start script. When you reboot, the schedule job(s) will automatically be applied.

#!/bin/sh

SCRIPTS_DIR='/jffs/scripts'
SCRIPT="$SCRIPTS_DIR/services-start"

mkdir -p $SCRIPTS_DIR

create_script() {
cat << "EOF" > $SCRIPT
#!/bin/sh
cru a restart_httpd "0 5 * * * /sbin/service restart_httpd"
EOF
chmod +x $SCRIPT
}

if [ -f $SCRIPT ]; then
    echo "error: $SCRIPT already exists; requires manual installation"
else
    create_script
    echo 'Done.'
fi

Note, if it finds an existing services-start script, it will NOT overwrite it. You'll instead have to add the schedule job(s) manually to the existing firewall-start script.

Of course, this still begs the question as to *why* it's consuming so much memory.

admin@AC86U-Merlin:/jffs/scripts# cru l
admin@AC86U-Merlin:/jffs/scripts# cru a restart_httpd "0 5 * * * /sbin/service restart_httpd"
admin@AC86U-Merlin:/jffs/scripts# cru l
0 5 * * * /sbin/service restart_httpd #restart_httpd#

I do have an existing services-start script. It had only 1 line in it -- to start YazFi. Would the following be sufficient?

admin@AC86U-Merlin:/jffs/scripts# cat services-start
#!/bin/sh

/jffs/scripts/YazFi startup & # YazFi

cru a restart_httpd "0 5 * * * /sbin/service restart_httpd"

> ... add the schedule job(s) manually to the existing firewall-start script.

Why do you suggests using the firewall-start script instead of services-start?

 

[eibgrad]

admin@AC86U-Merlin:/jffs/scripts# cru l
admin@AC86U-Merlin:/jffs/scripts# cru a restart_httpd "0 5 * * * /sbin/service restart_httpd"
admin@AC86U-Merlin:/jffs/scripts# cru l
0 5 * * * /sbin/service restart_httpd #restart_httpd#

I do have an existing services-start script. It had only 1 line in it -- to start YazFi. Would the following be sufficient?

admin@AC86U-Merlin:/jffs/scripts# cat services-start
#!/bin/sh

/jffs/scripts/YazFi startup & # YazFi

cru a restart_httpd "0 5 * * * /sbin/service restart_httpd"

> ... add the schedule job(s) manually to the existing firewall-start script.

Why do you suggests using the firewall-start script instead of services-start?

You should be able to modify the YazFi script. The reference to firewall-start was just a typo.

 

[DTS]

I'm using cru to restart the httpd service as discussed above. Here's the free memory today.

# free -m
             total       used       free     shared    buffers     cached
Mem:        440324     212496     227828       1572          0      22020
-/+ buffers/cache:     190476     249848
Swap:            0          0          0

Memory usage has gone up a small amount (from 201824 to 212496). Does this look like what one would expect?

 

[eibgrad]

I'm using cru to restart the httpd service as discussed above. Here's the free memory today.

# free -m
             total       used       free     shared    buffers     cached
Mem:        440324     212496     227828       1572          0      22020
-/+ buffers/cache:     190476     249848
Swap:            0          0          0

Memory usage has gone up a small amount (from 201824 to 212496). Does this look like what one would expect?

Looking at memory usage for one specific moment in time isn't all that useful (at least not to me). What I would be monitoring is the trendline over time. I don't really care if memory usage is up for some period of time, as long as the long term trend stays within reasonable bounds.

I will say you seem to have a LOT of active processes, far more than me. So perhaps that makes you more vulnerable to memory issues (e.g., memory leaks).

Frankly, I was surprised how much memory was free'd up the first time you restarted the http server. What in the world could the http server be doing to consume so much memory?

 

[DTS]

Looking at memory usage for one specific moment in time isn't all that useful (at least not to me). What I would be monitoring is the trendline over time. I don't really care if memory usage is up for some period of time, as long as the long term trend stays within reasonable bounds.

I will say you seem to have a LOT of active processes, far more than me. So perhaps that makes you more vulnerable to memory issues (e.g., memory leaks).

How can I find out what could be responsible for this?

At one point during the troubleshooting, I tried multiple VPN clients but that just made the problem worse. All I need is a relatively simple setup, and with your help @eibgrad , I reverted to that simple setup and have been using it since.

Frankly, I was surprised how much memory was free'd up the first time you restarted the http server. What in the world could the http server be doing to consume so much memory?

I hope we can find out what's going on. The amount of time (from Oct 2021 to now) and money I have put into this is crazy.

What I have done so far is:

1. upgrade Asus RT-AC86U router #1 from an earlier version of Merlin fw to 386.3_2. Afterwards, I saw this issue with the GUI becoming unresponsive and clients losing their Internet connection after 1-3 days and I just assumed it was because I had upgraded directly to the latest fw from a fw version several versions back (and had not cleared my config, etc.). So, next step:

2. I installed fw 386.3_2 again being much more careful in my steps, but the same issue occurred. So, next step:

3. I did a full reset and pristine fw install as described by L & LD here: https://www.snbforums.com/threads/merlin-on-asus-ax86u-question.74295/#post-708304 . The same issue occurred.

4. purchased RT-AC86U #2 and installed fw 386.3_2 and all settings from scratch. At the same time I also:
- got another VPN provider (PureVPN)
- got a new ISP (AT&T instead of Comcast)

The same issue occurred, so I started this thread. I certainly understand the problem better now, but I have no idea why my router would have any more active processes than anyone else running a VPN client. And I know of no reason why my web GUI would consume more memory than any other user running this fw and router.

As mentioned, I did add YazFi after starting this thread, but the problems existed before that and YazFi has not, as far as I can tell, had any detrimental effect.

 

[eibgrad]

As I said before, there's only so much that can be done in these forums. Obviously if I was sitting there watching all this happen, and had the ability to just casually browse around the router, I might spot something. But when all I have are what YOU report, that severely limits what I or anyone else can do. All we can do is make educated guesses. But it might be something incredibly subtle that's at play here.

At this point, the best we can do is restart the http server on a regular basis. And if that's insufficient, reboot the router on a regular basis. That shouldn't be necessary, but sometimes that's your best option. Esp. if the problem is progressive. You avoid it by beating it to the punch. At least last time you went some 16 days w/o a problem. I consider that progress. Maybe you'll get lucky and these restarts will do the trick. Maybe you should consider 386.4 (I almost hate saying that because it just feels like we're throwing darts hoping to get lucky, but I'm out of ideas). Or maybe go back before 386.3_2 (you said all these problems began w/ that update).

Frankly, the fact it now takes as much as two weeks to see the problem is a major hindrance too.

 

[DTS]

Thanks for your reply @eibgrad. That all makes sense. Here's the latest news.

The problem came back last night. It was about 13 days this time I'm aware of that the period/duration did not increase from the prior period. Also, this time appears to be different and I can only guess that it might be due to the fact that I'm restarting the httpd service (as per our prior posts). This time we have:

free -m
             total       used       free     shared    buffers     cached
Mem:        440324     227700     212624       1424          0      22452
-/+ buffers/cache:     205248     235076
Swap:            0          0          0

And "/var/lib/misc/dnsmasq.leases" is not empty (since the device has not run out of space).

However, just like in all past cases, the router maintains a working WAN interface as well as a working VPN tunnel. I am able to complete a traceroute thru the tunnel, even while the wired and wireless clients are unable to connect to the internet.

The syslog does not show any errors. In particular, when I do a grep on a copy of the syslog I just saved, the terms "error" and "no space left" are not present in the logs.

I know your prior message said that we are (almost or completely) out of troubleshooting ideas. If that's still true, I will just switch to rebooting the router every day until such time that new options come up. For example, I might purchase a AX86U and start fresh (yet again) and I could share each step of my setup. Or, I could ship the router to you, lol. I don't know... I'm searching for any out-of-the box troubleshooting ideas...

 

[DTS]

@eibgrad - would you like to SSH into my router? If yes, we can discuss in PM. For example, you could share your public key and I could open up WAN access for SSH.

I can also send you all the unredacted logs and diagnostics I just ran (such as traceroute through the VPN tunnel, etc.).

EDIT: The issue is present in the router right now. You could have access to troubleshoot any time this weekend, but before Monday I'll need to get the router functional (such as by rebooting it).

 

[eibgrad]

If you want me to get into the router and take a look, that's fine. Maybe I'll see something.