[R7800] Will no longer save settings after reboot.

[kokishin]

I'll check. What could be done.


P.S.
Guys, I am really sorry. This thread is important. But I am totally overloaded. You know... This isolation. E.g. my parents need my daily help. Tp help with ordering food, to help with order by Internet (how it is possible to pay by credit card in Internet - they do not understand). Kids: remote school conferencing. Zoom, Skype. Viber, Whatsapp, Team Viewer, RDP, ssh, etc. etc. etc...

Voxel.

Don't worry about us. In the overall scheme of things, this 7800 issue is mice nuts.

Your family and you are #1!

Voxel, all the best!

 

[verbage]

Thanks. Makes sense. Vicious cycle.

Yes, it's a vicious cycle.

I mentioned a while back that all the configuration settings that one sees in the web GUI are actually just implemented as text files under the filesystem. The web GUI just puts a pretty face on it. So this also provides a potential path to a solution. Imagine having all the text files on a USB device. And then a simple script copies them over to the appropriate locations on the filesystem, and restarts all the necessary daemons so they reread the new configuration files. This would solve the problem (if is can be implemented). It would also provide another benefit--at least from my perspective. Why? Because the configuration settings files saved by the nvram command or the web GUI interface are binary blobs. And you can't really easily edit a binary blob. So it would be cool if you could instead use simple text files to make the configuration, and avoid the web GUI altogether because it can be rather slow if you add a bunch of DCHP reservations, change computer hostnames so they are intelligible, etc. If you could just manually edit simple text files, it would be much faster and more flexible.

 

[verbage]

Guys, I am really sorry.

As @kokishin has said, completely understandable, and we are all in the same boat regarding the extra stress and responsibilities from work at home nowadays. So no problem at all, indeed, for those whose routers are afflicted with this issue, we would be thrilled that you would even consider it at some future point when things settle down! Thanks, @Voxel, for everything you have already done for the community!

 

[NetBytes]

I'll check. What could be done.


P.S.
Guys, I am really sorry. This thread is important. But I am totally overloaded. You know... This isolation. E.g. my parents need my daily help. Tp help with ordering food, to help with order by Internet (how it is possible to pay by credit card in Internet - they do not understand). Kids: remote school conferencing. Zoom, Skype. Viber, Whatsapp, Team Viewer, RDP, ssh, etc. etc. etc...

Voxel.

Family comes first. We are all in the same boat.
My 7800 had been running since January with no interruptions because I think I would have had a full scale revolt messing with the internet connection. Yesterday the power went out and back about five times during a whopper of a storm so...the router got a reboot workout lol.
Still working here - thanks Voxel.

 

[HELLO_wORLD]

Health and family are #1 priority, no need to be sorry.

About the backup button on web, I am not sure it would work without some kind a reboot. When you change SSID for example...

We have a working fix: restore nvram at boot time.
It would recommend not to touch the NG web GUI, and have a separate GUI page to control backup and restore our way (like in Kamoj addon). This is doable but it it time and work consuming...

For now, I think the focus is to release a simple plugin (no web GUI yet) to simply offer a working fix for people having this problem (yes, manual setup each time is taking a lot of time).
Later, it can be made fancy, with GUI, options, etc...

I will see if I can release something tomorrow.

I'll check. What could be done.


P.S.
Guys, I am really sorry. This thread is important. But I am totally overloaded. You know... This isolation. E.g. my parents need my daily help. Tp help with ordering food, to help with order by Internet (how it is possible to pay by credit card in Internet - they do not understand). Kids: remote school conferencing. Zoom, Skype. Viber, Whatsapp, Team Viewer, RDP, ssh, etc. etc. etc...

Voxel.

 

[oliveryuan]

I suggest contacting support for this issue they would be able to replace the unit if needed.

https://www.netgear.com/support/contact.aspx

 

[verbage]

I will investigate the reset button under OpenWRT this evening so as not to disturb the peace right now!

So I have not actually attempted a full factory reset with the reset button on my R8700 with OpenWRT because it is still under heavy use for leisure activities at the moment. But there is a whole webpage on the OpenWRT website that discusses the various types of resets that one can do. For the R8700 with a physical reset button, holding the button depressed for 10 seconds while the router is powered on will trigger the hard reset function.

Of course, the is the same way that one would do a factory reset while using official Netgear/Voxel firmware. But interestingly, they provide a lot of details about what actually happens during this process (more below). What's clear is that the process is controlled at the OS/firmware level--i.e. how the reset is implemented depends on what the programmers coded for it in the firmware. So it is not an ultra low-level function that is hard-coded for erasing mtd11, for example.

Here's what they say about such a hard reset:

This command will erase and reformat the whole jffs2 partition and create it again. They key for a real “hard reset” is to unmount the overlay partition first and only then issue the jffs2reset (or firstboot) command:

umount /overlay && jffs2reset && reboot now

While in most cases this is producing similar end-result as the “soft reset”, this marks the whole flash area of the JFFS2 (read-write) overlay partition as a empty non-initialised jffs2 partition. Thus the partition will be re-created at the next mount, usually at the next boot. So, this hard reset bypasses the current file system of the overlay.

If you are inclined to know more, you can find the page at:

https://openwrt.org/docs/guide-user/troubleshooting/failsafe_and_factory_reset

I'll see if I can find the appropriate amount of downtime this weekend to to actually try this out when the family is sleeping.

 

[HELLO_wORLD]

Thank you!
No, I don’t think you need to try it as this information you found is giving enough information to know that the button is not operating on the hardware level, but is sending a signal to the os to perform a software based task.
What would be interesting is to hack this task to have it perform a logging to figure out if it is called or not during unsolicited resets at reboot.

So I have not actually attempted a full factory reset with the reset button on my R8700 with OpenWRT because it is still under heavy use for leisure activities at the moment. But there is a whole webpage on the OpenWRT website that discusses the various types of resets that one can do. For the R8700 with a physical reset button, holding the button depressed for 10 seconds while the router is powered on will trigger the hard reset function.

Of course, the is the same way that one would do a factory reset while using official Netgear/Voxel firmware. But interestingly, they provide a lot of details about what actually happens during this process (more below). What's clear is that the process is controlled at the OS/firmware level--i.e. how the reset is implemented depends on what the programmers coded for it in the firmware. So it is not an ultra low-level function that is hard-coded for erasing mtd11, for example.

Here's what they say about such a hard reset:



If you are inclined to know more, you can find the page at:

https://openwrt.org/docs/guide-user/troubleshooting/failsafe_and_factory_reset

I'll see if I can find the appropriate amount of downtime this weekend to to actually try this out when the family is sleeping.

 

[HELLO_wORLD]

I made a little script called nvram-utils.
To install:

1. Connect to the router with ssh or telnet
2. Go to the directory you want to install the script (I recommend /opt/scripts):

   cd /opt/scripts

   If directory does not exist use:

   mkdir -p /opt/scripts; cd /opt/scripts

3. Copy and paste the following code:

   wget https://tinyurl.com/nvram-utils && chmod +x nvram-utils

Now the script is installed.

Usage:

• /opt/scripts/nvram-utils backup
  • To backup the current settings to be able to restore with the boot fix.
  • If you have an external drive labeled ‘optware’, it will save a copy to it with a timestamp in /mnt/optware/nvram_backups/ in binary and text formats.
• /opt/scripts/nvram-utils bootfix install
  • To install the boot fix (init.d and rc.d) to restore saved settings from backup at early boot stage.
  • THIS WILL PREVENT THE ABILITY TO DO A FACTORY RESET as the backup settings will always be restored. To prevent that, you need to access the root file system (telnet or ssh) and either:
    • remove /nvram_backup (rm /nvram_backup and confirm), or
    • uninstall the boot fix (see next command)
• /opt/scripts/nvram-utils bootfix uninstall
  • To uninstall the boot fix

That’s it. I threw that quickly without much testing. It is basic and to help people having the settings reset at reboot or restore from web GUI problem.

Thanks to @kamoj who helped with the boot fix.

 

[verbage]

I made a little script called nvram-utils.
Thanks to @kamoj who helped with the boot fix.

Thank you @HELLO_wORLD and @kamoj!!! This is wonderful progress, and will undoubtedly be greatly appreciated by those whose R7800s are out of warranty, but suffering from this frustrating issue!

 

[RRANet]

@vebage what version of OpenWRT are you using? Are you using 19.07.2 direct from https://openwrt.org/toh/netgear/r7800#installation?

Did you flash from the GUI or did you use the TFTP flash per the instructions listed?

Thanks -

And just some more info... I began to see the same issues after upgrading to V1.0.2.74.4SF. I had some connectivity issues at home and power cycled the router and POOF - settings were gone. I keep my router on a UPS, so I have no idea when the problem started. Router was purchased new from Amazon in December 2019.

 

[verbage]

@vebage what version of OpenWRT are you using? Are you using 19.07.2 direct from https://openwrt.org/toh/netgear/r7800#installation? Did you flash from the GUI or did you use the TFTP flash per the instructions listed?
And just some more info... I began to see the same issues after upgrading to V1.0.2.74.4SF. I had some connectivity issues at home and power cycled the router and POOF - settings were gone. I keep my router on a UPS, so I have no idea when the problem started. Router was purchased new from Amazon in December 2019.

Yes, I installed the latest OpenWRT v19.07.2 using the TFTP flash method. You need a computer with an actual ethernet port--not just a wireless connection--but once you figure it out, it is very simple, and also failsafe.

I am sorry to hear that your R7800 has failed in the same way as so many others. Since it is only from Dec 2019, you should absolutely go the RMA method to get it replaced by Netgear. If for some reason you decide not to do that, yeah, there are some options to return to basic functionality at this point, either OpenWRT or the nvram restore settings script that @HELLO_wORLD recently mentioned.

I would advise going the RMA route first, but I'll be happy to offer some guidance with OpenWRT if you prefer that instead.

 

[HELLO_wORLD]

I figured out a way that would allow to do a factory reset if needed while using the nvram boot fix, and if there is no way to connect to it using telnet or ssh (to nvram_backup or uninstall the boot fix)
Using a thumb drive with autorun script (like Voxel’s Entware), it is possible to remove the /nvram_backup file when the usb drive is plugged, after full boot and then allow a factory reset to happen.
This is also a possible way to restore the router from a specific backup without a way to access it (telnet or ssh).

 

[RRANet]

@vebage what version of OpenWRT are you using? Are you using 19.07.2 direct from https://openwrt.org/toh/netgear/r7800#installation?

Did you flash from the GUI or did you use the TFTP flash per the instructions listed?

Thanks -

And just some more info... I began to see the same issues after upgrading to V1.0.2.74.4SF. I had some connectivity issues at home and power cycled the router and POOF - settings were gone. I keep my router on a UPS, so I have no idea when the problem started. Router was purchased new from Amazon in December 2019.

I originally put this as an ETA to my post, but decided to just do a reply instead...

I just remembered that I reset my router and updated to v1.0.2.74.4SF after the power interrupt/reset happened. I am pretty sure that I was on ~74.2SF before that. So, my problem started sometime after 2/20 (release date of 74/2SF). Originally I thought my issue happened before we went into DST, but I am now almost positive it happened after DST. I was also running Traffic Meter back on my previous version. For what its worth, I have not power cycled my router since.

 

[verbage]

I figured out a way that would allow to do a factory reset if needed while using the nvram boot fix, and if there is no way to connect to it using telnet or ssh (to nvram_backup or uninstall the boot fix)
Using a thumb drive with autorun script (like Voxel’s Entware), it is possible to remove the /nvram_backup file when the usb drive is plugged, after full boot and then allow a factory reset to happen.
This is also a possible way to restore the router from a specific backup without a way to access it (telnet or ssh).

Great, so you could basically have a spare USB drive dedicated to being a failsafe reset drive that removes the /nvram_backup files--so another step forward!

Here is a fundamental question--when the early boot nvram restore happens, does it actually write the configuration to the mtd11, which the firmware subsequently reads back as the device continues to boot? Or is the configuration only held in volatile RAM, and thus, would be lost on a reboot or power cycle (if it were not for the early boot config restoration script that will restore it once again). This will help get at the fundamental question of whether the physical portion that comprises the mtd11 partition of the flash chip is "worn out" or electronically damaged, or whether this is being caused by a perpetual factory reset look because of an other faulty component besides the NAND flash chip.

 

[verbage]

I just remembered that I reset my router and updated to v1.0.2.74.4SF after the power interrupt/reset happened. I am pretty sure that I was on ~74.2SF before that. So, my problem started sometime after 2/20 (release date of 74/2SF). Originally I thought my issue happened before we went into DST, but I am now almost positive it happened after DST. I was also running Traffic Meter back on my previous version. For what its worth, I have not power cycled my router since.

Though there seem to be a recent flurry of reports on this issue, and at least in this forum that time frame can be related to a certain revision of Voxel's firmware because many folks are using it, I do not think that this is a firmware problem at all. The evidence is much stronger to suggest that it is a hardware issue. This can be discerned by the close tracking of a single piece of equipment (more below) that helps remove some of the blurriness that comes in from multiple reports--usually from folks who are exasperated because of the frustrating nature of this problem (i.e. the vicious cycle mentioned above).

When I received my RMA replacement R7800, I immediately installed v1.0.2.75.2SF because it has just been released. And it worked perfectly well for two days before an apparent electrical brownout caused this new, replacement R7800 to show the same issue. In turn, there are many, many other R7800s out that that have been running for years successfully--using the same firmware revisions, and quite likely subject to the same occasional hiccups in power delivery as well.

There is a pattern that emerges, but is quite blurry because of the lack of data points. It seem that the older R7800s (from 2016-2018) seem more robust, and do not seem to show this problem. And newer R7800s (from 2019 onwards), yes, seem to be the ones showing this issue. There is at least one hardware difference between the older and newer R7800s--the older ones seem to use Micron- or SkyHigh-brand NAND flash chips, and the newer ones seem to use Macronix-brand ones. But again, we need more data points.

There may be other subtle hardware differences, too, or it may just be a simple batch of less robust Macronix chips, while good batches of Macronix chips are just as robust as any others. Or it may be another component altogether. But again, it seems that older R7800s are not seeing this problem whiles newer ones are--even when both are using the very same firmwares.

 

[HELLO_wORLD]

Yes, and the beauty of it is that the failsafe can be made from a computer, so if someone has a router not accessible (like no Ethernet access and forgot WiFi password), plugging the failsafe thumb drive would remove the nvram_backup file and the performing a factory reset will work.

For the other point, unfortunately it does not gives an answer.
Without the boot fix, a router having this problem will loose its settings at reboot, so it is required to have the router survive reboots and keep its settings.
But this does not help to understand if mtd11 is corrupted or reset at any loss of power.
The boot fix is restoring the backup into dram and supposedly into mtd11 as well, but I don’t know how the low level works for nvram.
I suppose (without knowing) that nvram

• show, get, set and backup are working with what is in dram.
  
• commit is copying what is in dram to mtd11
  
• restore is copying what is in input file to dram and mtd11

But quite frankly, even if it seems logical, I don’t know for a fact that nvram is using dram at all (I suspect because then what would be the purpose of commit?)
Also, nvram has an uncommit function that I don’t know whqt it is supposed to do (maybe keeping in dram but removing from mtd11)? and a defaults function that could be a way to factory reset nvram?

EDIT: even if it is not directly about our router, this seems to confirm that it is common practice to have nvram settings in dram and flash (MTD11) https://oldwiki.archive.openwrt.org/oldwiki/historic.whiterussian.nvram

Great, so you could basically have a spare USB drive dedicated to being a failsafe reset drive that removes the /nvram_backup files--so another step forward!

Here is a fundamental question--when the early boot nvram restore happens, does it actually write the configuration to the mtd11, which the firmware subsequently reads back as the device continues to boot? Or is the configuration only held in volatile RAM, and thus, would be lost on a reboot or power cycle (if it were not for the early boot config restoration script that will restore it once again). This will help get at the fundamental question of whether the physical portion that comprises the mtd11 partition of the flash chip is "worn out" or electronically damaged, or whether this is being caused by a perpetual factory reset look because of an other faulty component besides the NAND flash chip.

 

[microchip]

Another interesting fact is that a lot of ASUS routers use Macronix for their Flash chip and I haven't seen any reports that they lose settings after reboot. My speculative theory is that some recently produced R7800s have a bad batch of these Flash chips. Macronix is a major player in the NAND flash storage and if they only produce crap, they'd be bankrupt by now. A second theory is that something in the software mistakenly triggers the NVRAM reset

 

[verbage]

My speculative theory is that some recently produced R7800s have a bad batch of these Flash chips.

Yes, if the Macronix NAND flash chip is faulty/less robust/weak, etc., I suspect it is only a batch of them--not all of them. Indeed, since this occurs ACROSS product models (just inspect the 35+ threads on the Netgear forums for proof of this...), it would seem to point to a defective component that is shared between these product models, not just the R7800, i.e. a bad batch of flash chips (or perhaps another component).

We could figure this out at this level if we start to pull together data. For example, if we could document:

1.) The approximate purchase date of the router (if new, and perhaps an estimate of when it was from if not);
2.) The actual production date of a router according to "artmtd -r protime";
3.) Potentially the serial number (minus the last four or five identifying digits) to see if we can limit the issue to a particular range;
4.) The manufacturer ID of the NAND flash chip either obtainable through @kamoj's beta add-on, or via early boot dmesg capture;
5.) Potentially other ancillary information about how/when the problem first appeared like out-of-the-blue, after a flash process, during heavy usage, following a power outage or brownout, etc.

For sure if we have this info, we could really start to narrow in on the problem. I also recognize that not all information would be available about the routers that are affected--some have already been returned for RMAs, some have probably been tossed or discarded, some details might not be so easy for folks to obtain, etc. But if we could fill in some of these details, I think the problem would come into focus, and for sure, we could start to eliminate some of the possibilities that have been discussed.

Not that I have time or desire to do this, but I recognize that actually getting this done requires time and energy. I would be willing to collect and collate information, and put it in a public place for everyone's inspection. Indeed, I will make a first start this evening, and contact those who have already provided information to make sure that they are okay with this before I actually include anything they might have contributed either publicly or via PMs.

 

[Alex Tzonkov]

Just a quick update on my R7800...it is one of the older batches purchased on Amazon Nov 26, 2018. I did flash and am running the latest Voxels FW and Kamojs add-on (5.0). Things have been stable for a couple of weeks. I did end up getting a laptop cooler, which according to Kamojs add-on dropped my Router's CPU temp by 10 degrees celsius. I never re-enabled the Trafic Monitor or plugged the external USB drive back in (still need to figure out how to use it without ReadyShare).

I do plan on trying the latest beta Kamoj add-on when it is available, so I can at least give you some data on which flash chip my router has.

As I am one of the folks whose router is out of Warranty, thanks for creating the scripts/methods to restore settings early on in boot. If I confirm that my NVRAM is indeed bad, I will be working on implementing that.

Last a quick question, for those of you that see the issue, is it intermittent, or once it happens it stays bad?