Update fails while connected to router via OpenVPN

[M4tt]

Hi!

I'm running several RT-AC66U and RT-AC68U at different locations. Until approximately 1 1/2 to 2 years ago I was able to just connect to every location over OpenVPN (every router runs an OpenVPN server) and update the routers firmware as soon as an update was available (I don't remember the last FW-version this still worked...). Currently, updating over an OpenVPN-connection doesn't work anymore, as an active OpenVPN connection seems to disrupt the update process: The router shows the update progress (percentage), restarts but after reboot the old FW-Version is still present.
This behavior is especially troublesome for me, as I maintain routers at locations I hardly every visit in person but I'm now forced to visit in order to update (resulting in some of the routers running very old FW-Versions).

The problem seems to be caused somehow by the OpenVPN server deamon as it only occurs when a incoming connection is active. I came to this conclusion, as updating a router in AP-mode while connecting to another router in the same network via OpenVPN works (using the OpenVPN client's browser).
Updating a router while connected to that router via OpenVPN with any browser running on the OpenVPN-client doesn't work, and, most important, updating via (any) browser running on a client physically "inside" the routers network while connected to this client over VNC using an OpenVPN connection terminating on the router to be updated also doesn't work. As soon as I am at a location in person (and no OpenVPN connection is active), updating works on the first try despite it failed via OpenVPN many times before. As I said, this issue didn't exist 1 1/2 to 2 years ago, but I don't remember the FW-version this behavior started...

Does anyone have any idea how to fix this? Does it need to be fixed/is it possible to fix it on FW-level?

 

[RMerlin]

updating via (any) browser running on a client physically "inside" the routers network while connected to this client over VNC using an OpenVPN connection terminating on the router to be updated also doesn't work

I can confirm that this works for me. I frequently update my router at home while connected to my home desktop through an OpenVPN tunnel, never had any problem.

Try rebooting that router first to free up sufficient RAM.

 

[cybrnook]

I wonder, maybe provide more details to what you are connecting "to" on your remote sites to perform the update. Are you just VPN'ing in, then pulling up the WebUI on your locally VPN'd systems web browser?

@RMerlin I see you carefully said you remote into a Desktop at home, then perform the update from the dekstop. He/She may be connecting to something different, or even directly to the webui from remote PC.

 

[RMerlin]

I wonder, maybe provide more details to what you are connecting "to" on your remote sites to perform the update. Are you just VPN'ing in, then pulling up the WebUI on your locally VPN'd systems web browser?

@RMerlin I see you carefully said you remote into a Desktop at home, then perform the update from the dekstop. He/She may be connecting to something different, or even directly to the webui from remote PC.

He also mentioned using VNC on a remote client, in the portion that I quoted.

 

[cybrnook]

He also mentioned using VNC on a remote client, in the portion that I quoted.

You are right.... my eyes went cross-ways by that point.

 

[JimbobJay]

I update the firmware on my parent's Merlin flashed router that I remotely manage, via SSH through an OpenVPN tunnel, and that has always worked fine. Have done it with several iterations of Merlin's fw, including occasional betas. I usually download the firmware onto my computer first, then initiate the OpenVPN tunnel and scp the .trx file into the router's /tmp/ folder. Then I login to the router via ssh, and issue the mtd-write2 command pointing to the .trx file. Has worked without a hitch. OpenVPN connection becomes unresponsive whilst the firmware is being flashed. I usually disconnect, wait a few minutes, then reconnect and it's fine. I guess this could be risky if anything went wrong with the firmware flash, but so far it's been good. Maybe give it a go on your setup?

 

[Zirescu]

@M4tt - Do you have a USB drive connected to them as well?

 

[Xentrk]

I update the firmware on my parent's Merlin flashed router that I remotely manage, via SSH through an OpenVPN tunnel, and that has always worked fine. Have done it with several iterations of Merlin's fw, including occasional betas. I usually download the firmware onto my computer first, then initiate the OpenVPN tunnel and scp the .trx file into the router's /tmp/ folder. Then I login to the router via ssh, and issue the mtd-write2 command pointing to the .trx file. Has worked without a hitch. OpenVPN connection becomes unresponsive whilst the firmware is being flashed. I usually disconnect, wait a few minutes, then reconnect and it's fine. I guess this could be risky if anything went wrong with the firmware flash, but so far it's been good. Maybe give it a go on your setup?

Thanks for this tip. I have been wondering how best to safely remote flash a firmware upgrade over OpenVPN tunnel at a site I support so I can do it off hours. I will try the recommendation later this week.

 

[JimbobJay]

Thanks for this tip. I have been wondering how best to safely remote flash a firmware upgrade over OpenVPN tunnel at a site I support so I can do it off hours. I will try the recommendation later this week.

No problem. Yes, give it a go. I think it's probably the safest way of updating fw remotely.
Enter the following into ssh

mtd-write2 /tmp/file.trx linux && reboot

The ssh session and openvpn tunnel go down whilst the router is rebooting and flashing so I just disconnect, wait a few minutes, and then reconnect.

I usually switch on webgui access via WAN temporarily before I do this, just in case the update causes a problem with OpenVPN, but this is just a precaution and I've never needed to use it. I always switch off WAN access again once the firmware has updated.

 

[martinr]

But this wouln't work if the update required a reset to factory default settings (RFDS), even if you intended using John's NVRAM save/restore facility, because the RFDS would have destroyed the possibilty of remotely reconnecting. No way round that, is there?

 

[RMerlin]

No problem. Yes, give it a go. I think it's probably the safest way of updating fw remotely.
Enter the following into ssh

mtd-write2 /tmp/file.trx linux && reboot

Never flash remotely using mtd-write. Many times, your router will be unable to reboot, because you are trying to run the whole shutdown process off a firmware partition that just got completely overwritten. When that happens, someone has to physically power cycle the router to complete the reboot.

Flashing through the webui takes certain precautions, such as creating a temporary filesystem that contains what's necessary to complete the reboot.

 

[Xentrk]

Never flash remotely using mtd-write. Many times, your router will be unable to reboot, because you are trying to run the whole shutdown process off a firmware partition that just got completely overwritten. When that happens, someone has to physically power cycle the router to complete the reboot.

Flashing through the webui takes certain precautions, such as creating a temporary filesystem that contains what's necessary to complete the reboot.

With 512MB of memory available on the AC88U, I find I can leave USB drives in when flashing. However, I have been burned one time and the USB drive was corrupt afterwards. For remote firmware upgrades, do you think it would be safe to copy the firmware.trx file to a usb drive connected to the router and choose that as the file location from the web gui?

 

[JimbobJay]

Never flash remotely using mtd-write. Many times, your router will be unable to reboot, because you are trying to run the whole shutdown process off a firmware partition that just got completely overwritten. When that happens, someone has to physically power cycle the router to complete the reboot.

Flashing through the webui takes certain precautions, such as creating a temporary filesystem that contains what's necessary to complete the reboot.

Oh dear That puts paid to my solution then I suppose. Although, I'm a bit confused, as I was following your instructions from an earlier thread (https://www.snbforums.com/threads/how-to-install-firmware-via-telnet.7784/) so had assumed it had your A-OK!

I'd be interested to know what is your most recommended and reliable method for flashing the firmware remotely via an OpenVPN connection then, RMerlin, as I thought I had a pretty good method. Would using the mtd-write2 command using a .trx located on a mounted USB HDD be any better? Guessing not from what you described, but would still be curious to know.

I understand the risks involved, but I also have to say, this was a method that I've had 100% success rate with so far.

 

[RMerlin]

With 512MB of memory available on the AC88U, I find I can leave USB drives in when flashing. However, I have been burned one time and the USB drive was corrupt afterwards. For remote firmware upgrades, do you think it would be safe to copy the firmware.trx file to a usb drive connected to the router and choose that as the file location from the web gui?

No, that won't help. The issue is the flash partition - you are flashing data right on top of the currently running OS. During shutdown, the firmware has to run various commands to shut down services. If the flash cell containing the file you want to run isn't cached in memory and overwritten by different data, then the firmware will crash.

Think of it as if you were trying to format your Windows partition WHILE running Windows. This is pretty much the same.

USB corruption could very well come from that crash occurring during the shutdown.

 

[RMerlin]

Oh dear That puts paid to my solution then I suppose. Although, I'm a bit confused, as I was following your instructions from an earlier thread (https://www.snbforums.com/threads/how-to-install-firmware-via-telnet.7784/) so had assumed it had your A-OK!

Look at the date of that post I stopped flashing this way a few years ago. Those issues only started appearing later on, as the firmware became more complex, running more services during shutdown. Asus themselves only implemented the temp filesystem later on, to deal with that same issue.

If you have physical access to the router, then it's OK - you can manually power cycle the device if it crashes. But remotely, it's a bit suicidal. Might work, might fail.

I'd be interested to know what is your most recommended and reliable method for flashing the firmware remotely via an OpenVPN connection then, RMerlin, as I thought I had a pretty good method. Would using the mtd-write2 command using a .trx located on a mounted USB HDD be any better? Guessing not from what you described, but would still be curious to know.

First, connect to a remote computer that sits on the LAN (VNC, Remote Desktop, TeamViewer, etc...)

Next, go to the webui, and eject any mounted USB disk.

Then, go to the Firmware Upgrade page, and proceed with the upgrade, from the computer that sits on that LAN.

If it fails, reboot the router first, to free up any used RAM, then repeat the same steps: eject, then flash.

Note that this isn't 100% foolproof tho. On more rare occasions, you might potentially still need someone on-site to power cycle the router. For some odd reason, it seems to happen more often with the RT-AC68U than other models. With my RT-AC88U, it might happen once every 15-20 flashes. I suspect it's because something is missing in the temprootfs, so it's a gamble whether or not it will cause any issue.

 

[JimbobJay]

Look at the date of that post I stopped flashing this way a few years ago. Those issues only started appearing later on, as the firmware became more complex, running more services during shutdown. Asus themselves only implemented the temp filesystem later on, to deal with that same issue.

If you have physical access to the router, then it's OK - you can manually power cycle the device if it crashes. But remotely, it's a bit suicidal. Might work, might fail.



First, connect to a remote computer that sits on the LAN (VNC, Remote Desktop, TeamViewer, etc...)

Next, go to the webui, and eject any mounted USB disk.

Then, go to the Firmware Upgrade page, and proceed with the upgrade, from the computer that sits on that LAN.

If it fails, reboot the router first, to free up any used RAM, then repeat the same steps: eject, then flash.

Note that this isn't 100% foolproof tho. On more rare occasions, you might potentially still need someone on-site to power cycle the router. For some odd reason, it seems to happen more often with the RT-AC68U than other models. With my RT-AC88U, it might happen once every 15-20 flashes. I suspect it's because something is missing in the temprootfs, so it's a gamble whether or not it will cause any issue.

Hi Merlin. Thanks for the tip. It was interesting though - I tried out your recommended way with 380.66_2. Connected to the OpenVPN remote server on the AC87U, established a teamviewer connection with a PC on the LAN via the tunnel, and tried to flash the router from web browser on there, following all your instructions. Well, the flash didn't take, and my parents were without internet and I was locked out of the router until they were able to do a manual power cycle onsite.

Logged back into the router via OpenVPN, and was interested to see that the firmware update wasn't even successful, as the firmware is listed as 380.66 still, not .66_2.

Anyway, thought I should let you know. Found it interesting that that method didn't work, especially as up until then I'd had 100% success with the mtd-write2 command. Think I might stick to that, despite the risks.