Development musings

[RMerlin]

All those issues that show up in newer Asus releases (the broken QoS, Dual WAN unstability, broken traffic monitoring, random issues with networkmap, etc...) are simply too much for one single developer to deal with anymore. I can't keep up anymore with the team of 20+ programmers that keep adding/changing code in stock firmware every 2-3 months, with an increasing amount of code being moved to closed source binary blobs.

Extending the public beta period like I've done these past two releases doesn't seem to help that much it seems, judging from the number of new issues being reported only after coming out of beta.

The Asuswrt codebase is increasingly becoming unmanageable, as it changes too much within a short period of time. And things aren't going to slow down anytime soon, as there's a few more things that will most likely appear in the stock firmware over the next couple of months (Asus usually releases major new features sometime around summer time).

 

[jeff288]

Wifi performance is whatever the closed source driver provides. It's outside of my control, and since half of the time people are posting totally opposite experiences with the same driver, I suspect that the vast majority of time, the issues are local to the user's network or the test methodology, and not in the wireless driver itself. It's not like the wireless driver has major changes every new firmware release - it barely gets any change at all between releases, except for the RT-AC87U's troublesome 5 GHz driver that seems to get rewritten every 6 months by Quantenna...

A few years ago, I did a blind test with some users, back when people were complaining about the RT-N66U wifi performance. I compiled three firmwares, and asked users to test them all, and tell me which one gave better results. What I did not tell them was that two of these three had the exact same wireless driver. The third one was using a completely different SDK (and that was what I truly wanted to see tested).

Here's what happened: some people were reporting better results with one of the two firmwares which had the exact same wireless driver.

Bottom line: wifi performance is much more affected by the environment and your configuration than by the wireless driver itself. Having the channel set to "Auto" for instance is a sure way to ensure you WILL get different results between router reboots, as you might be on a completely different channel between reboots. And sometimes that auto mode does pretty silly things (like last night, the RT-AC68U I powered up for some tests ended up using channel 10...) A firmware upgrade is also matched by a router reboot, which means your channel might change then.

Start by doing the usual troubleshooting, use real-life tests and don't rely on RSSI values or reported link rates. And making sure you don't have a computer launching into a Windows Update as soon as it just reconnected to wifi following a router reboot (I ran into a similar issue myself a few weeks ago, only it was my Carbonite backup that decided to launch immediately after the router was rebooted...)

And eliminate the variables. I can see my RSSI change just by the position of the laptop screen here when I test things out (antennas are located in the screen).

That's largely why I stayed on John's fork because I heard good things about wireless performance but after doing some cursory testing this weekend between yours and 374.43 on John's, if anything the newer ones performed better for me. I have my 5ghz set to balanced and I get fine reception upstairs. I went back to your latest and it's great on my 66u.

 

[jeff288]

All those issues that show up in newer Asus releases (the broken QoS, Dual WAN unstability, broken traffic monitoring, random issues with networkmap, etc...) are simply too much for one single developer to deal with anymore. I can't keep up anymore with the team of 20+ programmers that keep adding/changing code in stock firmware every 2-3 months, with an increasing amount of code being moved to closed source binary blobs.

Extending the public beta period like I've done these past two releases doesn't seem to help that much it seems, judging from the number of new issues being reported only after coming out of beta.

The Asuswrt codebase is increasingly becoming unmanageable, as it changes too much within a short period of time. And things aren't going to slow down anytime soon, as there's a few more things that will most likely appear in the stock firmware over the next couple of months (Asus usually releases major new features sometime around summer time).

You're not thinking of leaving us are you? I'd understand, it's unimaginable the work you do here keeping it all together. I don't think I'd buy another Asus router if we didn't have guys like you making the firmware better for us all. It seems Asus is pushing features above bug fixes. If you quit, I'd probably move back to John's or Tomato. Or better yet, buy an OpenWRT compatible router.

 

[RMerlin]

You're not thinking of leaving us are you?

Not in the short term, but I will need to find a way to change some things. Right now, this has gone from a fun hobby into a chore.

 

[kvic]

I will need to find a way to change some things. Right now, this has gone from a fun hobby into a chore.

Perhaps create two branches: one for stable release and another for unstable/development release.

The stable branch gets major releases 2-3 times per year, with minor bug fix releases in between.

The unstable/development branch gets in sync with Asus latest GPL code. Make binary releases as you see fit. I'm sure some helpers will build binaries for early birds too.

More early adopters will attempt to use unstable/dev releases and report issues. For those you can fix, that's great. Or else at least gets Asus attention from this community. Within one or two new GPL releases, Asus might have fixed the issues. For those who can't really wait..perhaps shall let them freely drift to the stock FW.

Perhaps also right time to bring @john9527 to work on the latest repository together with yourself (if his interest on asus FW is still strong). Keep minimal work on his existing fork. As a courtesy, some of his "invented there" features shall be incorporated into the latest repository. Eventually his fork to let wither (except maybe RT-N66) to get focus on a latest and single release for the community.

I think there are a few other people will likely to submit bug fix and small features too.

 

[matthew_eli]

Perhaps create two branches: one for stable release and another for unstable/development release.

The stable branch gets major releases 2-3 times per year, with minor bug fix releases in between.

The unstable/development branch gets in sync with Asus latest GPL code. Make binary releases as you see fit. I'm sure some helpers will build binaries for early birds too.

More early adopters will attempt to use unstable/dev releases and report issues. For those you can fix, that's great. Or else at least gets Asus attention from this community. Within one or two new GPL releases, Asus might have fixed the issues. For those who can't really wait..perhaps shall let them freely drift to the stock FW.

Perhaps also right time to bring @john9527 to work on the latest repository together with yourself (if his interest on asus FW is still strong). Keep minimal work on his existing fork. As a courtesy, some of his "invented there" features shall be incorporated into the latest repository. Eventually his fork to let wither (except maybe RT-N66) to get focus on a latest and single release for the community.

I think there are a few other people will likely to submit bug fix and small features too.

Indeed I'm ok with this proposal of kvic: one stable release of the "most robust" GPL form ASUS and possibily the latest updates for the various components (OpenVPN, Samba, Minidlna, etc...) and an unstable/development release based on the latest ASUS GPL and obviously on the latest updates for the various components. I can say for example that 380.58 was very stable for me: I ran it on my RT-AC88U for almost 45 days without an issue. Using instead the 380.59 I almost immediately noted there was a problem related to traffic monitor statistics, due to the buggy implementation on the latest ASUS GPL code. I decided to revert to 380.58, but I'd like to have it updated for the various components, mostly for the security updates on Samba

I think this could be a really good idea!

 

[charlie2alpha]

All those issues that show up in newer Asus releases (the broken QoS, Dual WAN unstability, broken traffic monitoring, random issues with networkmap, etc...) are simply too much for one single developer to deal with anymore. I can't keep up anymore with the team of 20+ programmers that keep adding/changing code in stock firmware every 2-3 months, with an increasing amount of code being moved to closed source binary blobs.

Extending the public beta period like I've done these past two releases doesn't seem to help that much it seems, judging from the number of new issues being reported only after coming out of beta.

The Asuswrt codebase is increasingly becoming unmanageable, as it changes too much within a short period of time. And things aren't going to slow down anytime soon, as there's a few more things that will most likely appear in the stock firmware over the next couple of months (Asus usually releases major new features sometime around summer time).

I understand how you feel, I think you're burned out basically. The only way I can think of to handle it is to devise some way to slow down development, limit the public releases at least. Keep pushing stuff at github whenever you work on something but do no release builds unless you feel it's time. Just food for thought for you, you are going to ultimately decide how to handle this, because chores are no fun!

 

[charlie2alpha]

Wifi performance is whatever the closed source driver provides. It's outside of my control, and since half of the time people are posting totally opposite experiences with the same driver, I suspect that the vast majority of time, the issues are local to the user's network or the test methodology, and not in the wireless driver itself. It's not like the wireless driver has major changes every new firmware release - it barely gets any change at all between releases, except for the RT-AC87U's troublesome 5 GHz driver that seems to get rewritten every 6 months by Quantenna...

A few years ago, I did a blind test with some users, back when people were complaining about the RT-N66U wifi performance. I compiled three firmwares, and asked users to test them all, and tell me which one gave better results. What I did not tell them was that two of these three had the exact same wireless driver. The third one was using a completely different SDK (and that was what I truly wanted to see tested).

Here's what happened: some people were reporting better results with one of the two firmwares which had the exact same wireless driver.

Bottom line: wifi performance is much more affected by the environment and your configuration than by the wireless driver itself. Having the channel set to "Auto" for instance is a sure way to ensure you WILL get different results between router reboots, as you might be on a completely different channel between reboots. And sometimes that auto mode does pretty silly things (like last night, the RT-AC68U I powered up for some tests ended up using channel 10...) A firmware upgrade is also matched by a router reboot, which means your channel might change then.

Start by doing the usual troubleshooting, use real-life tests and don't rely on RSSI values or reported link rates. And making sure you don't have a computer launching into a Windows Update as soon as it just reconnected to wifi following a router reboot (I ran into a similar issue myself a few weeks ago, only it was my Carbonite backup that decided to launch immediately after the router was rebooted...)

And eliminate the variables. I can see my RSSI change just by the position of the laptop screen here when I test things out (antennas are located in the screen).

Wi-Fi is a messy thing, I myself encountered issues that for all I know should not be happening! My Nexus 9 decided to keep disconnecting a few days ago. But my Zenfone 2 was perfectly stable at the same 5Ghz band. Turns out something did not initialize properly on my AC56U ( I think dnsmasq ) and google connectivity services was mistakenly thinking I have no internet and kept disconnecting the WiFi attempting to find another. Fixed with a reboot of the router (should have restarted dnsmasq, was too lazy XD ).

Also, the last few days my Chromecast 2 decided it doesn't like my WIFI, keeps disconnecting. I factory reset it this morning, again it's a device that it's known to have issues with its 5GHz radio. Too many variables, and too many implementations that ignore the standards. Also, some devices attempt to smart, only to make things worse (like my stupid Nexus).

I would suggest to limit your personal support for the WIFI portion at least, it's all closed source for a while now and with the new lockdown measures even more of it will. If it works, it works. If not, sorry, downgrade your firmware and report the issue to ASUS.

 

[hellboy2]

Not in the short term, but I will need to find a way to change some things. Right now, this has gone from a fun hobby into a chore.

If anything, Merlin, you have clearly pushed ASUS to make their firmware better, more secure and more robust. Even if you stopped today, your effort would not have been in vain, but your spirit lives on in ASUS' code and practice. Your experience from this project should also make it easy for you to get a job as IT professional in any company, if you don't already have that.

I have been very impressed with your tireless efforts, almost outperforming 20-30 professional workers, but also wondered how long you could really keep up the enthusiasm and support so many different ASUS models?
Perhaps reduce the ambition level, and carefully choose what to "do better than the best"

I for one have been asking myself "why am I using Merlin, and not official ASUS?"
My router usage is rather simple, and can be divided in three levels of importance:-
- basic router function (wireless, switch, DHCP, NAT, firewall, system log)
- router management (parental controls, DNS proxy, virtual servers)
- other stuff (IPv6, NTP, Samba, DLNA, dual WAN, DNShost client, VPN, Torrent client)

Q: Merlin has better UI, especially network map, DHCP, and Wireless clients
A: that is why I still prefer Merlin, but ASUS are slowly catching up

Q: Merlin says "some performance improvements"
A: I get about same performance with both, so I am not sure what the difference really would be (maybe there was big difference in the past?)

Q: newer software packages
A: Some of the "other stuff" software packages were more up to date than official ASUS, but ASUS seems to get their act together.

Q: Persistent connection history?
A: Neither ASUS nor Merlin seem to keep this for more than 12-24 hours.

Q: Trustworthy traffic thruput monitor and usage per client
A: results were often very slow, confusing, and incomplete (can't see stats for clients outside top 5?) I gave up and decided to concentrate on using the router's essential "raison d'etre" ... partly because the ISP also stopped metering the network usage.

Q: DLNA and Samba are served from 5-year old NAS, not from 1-year old Router.
A: If I should start today, I would buy USB-disk, and use DLNA/Samba from Router.

NB: I came from an 8-year old D-link DIR-655, which was only retired because problems with N-type connection with several mobiles. It had good N-class performance. ASUS router (RT-AC68U) is technically a solid update, but ASUS' (and Merlin) UI still leaves a lot to be desired compared to what I was used to in D-link's router. After 18 months experience with ASUS, I should probably have bought TP-link C9, but now I'll keep ASUS with Merlin firmware for at least some years still.
NB: About 4-5 years ago, I tried an highly praised Netgear router, but returned it, when it was no better than the D-link I already had. ASUS kept me as one satisfied customer primarily because of your work.
NB: I bought ASUS because it had best results in reviews, like Netgear had years ago. Next time, I will not read reviews, because live experience has been quite dissapointing several times now.

 

[hellboy2]

... one stable release of the "most robust" GPL form ASUS and possibily the latest updates for the various components (OpenVPN, Samba, Minidlna, etc...) and an unstable/development release based on the latest ASUS GPL and obviously on the latest updates for the various components. I can say for example that 380.58 was very stable for me: I ran it on my RT-AC88U for almost 45 days without an issue. Using instead the 380.59 I almost immediately noted there was a problem ...

I think this could be a really good idea!

I disagree... those who even dares to use any non-official firmware will either use almost-ready (beta) code, or they will have a misconception of bug-free "final" version of Merlin. I get a free product at very high quality "best effort" level, and I pay by participating in beta and report observations. If I wanted "supported", I'd use ASUS's official firmware, which has warranty and more. I use Merlin, because it offers more than ASUS themselves do, but then my warranty is void, until I restore ASUS official firmware.

The idea of "beta" vs. "final" has already been abandoned by many professional organisations, where they are moving to "agile" project practice. The idea is to move faster, slightly less quality, but react even faster if bugs show up. No beta, but more frequent deliverables. Another trend is products that stay "beta" forever, even with very high quality code.

Now, if Merlin really wanted something "not quite ready", he could call it "experimental" or "alpha", so people know what to look out for. He already does that.

 

[Grievous Angel]

Not in the short term, but I will need to find a way to change some things. Right now, this has gone from a fun hobby into a chore.

I've been using your builds for a few years now. First on the RT-AC66R and now the AC3100.

I had myself suspicious your firmware was the culprit of some unstable upload speeds--when in fact, my default position should have been: blame Charter. I went back and forth on the official firmware and yours, back and forth between my old router and the new, and of course plugging right into the modem.

I should know better. I don't do anything exotic, but I've always found your firmware to be very stable for me as the routers matured. You get blamed for every bleeding edge wrinkle ASUS introduces (that you can't control) and for many blips in the various ISPs.

I'm in I/T myself and spent a lot of years in development. I don't know how you stay patient and keep your cool. Much less put out all these variants at the quality that you do (though I admit I'm ignorant when it comes to knowing the scope of what is involved and the differences between the routers--I assume it is huge).

Anyway--to do this for free as a hobby, and put up with the BS, and have ASUS add some of your stuff back in? They (ASUS) should compensate you IMO. I for one appreciate the effort and have found I in fact do not like to live without the extra Merlin niceties . . so thanks again.

I have a machine running Ubuntu and have a decade plus in the web java world plus c/c++ way back in the day . . you ever thought of recruiting/allowing help? Or would that create more headaches than it's worth?

I'll end this with a question (out of pure curiosity), though: Everyone says the RT-AC3100 is basically the AC88U minus the red accents and the extra ports. If that's true, why does it require different firmware? Are they actually different or is this just to reduce confusion?

Another question--probably not the best thread for it but I don't want to start another thread--Merlin's always cautioning us to not do frequent writes to NVRAM (be it for traffic statistics or whatever) because it's flash memory and will wear out. Does this not apply to flashing firmware updates? Or is that even relevant?

 

[RMerlin]

Perhaps create two branches: one for stable release and another for unstable/development release.

I considered it ( I was thinking about having a LTS release at one point), and it won't help, for two reasons:

1) Two branches to maintain = even more work
2) When Asus releases a new GPL, it's next to impossible for me to backport any specific fixes, as I don't have access to their specific commits. That means I can't make the stable branch any more stable by cherry-picking fixes, so GPL merges are usually all-or-nothing affairs. Since each new release has its share of both fixes and new issues, I can't point at a single specific GPL release that has been 100% positive so far. The networkmap issues for instance has been there in one form or another for over two years. Fix this, break this...

I tried a somewhat similar approach these past three months by releasing early alpha and beta releases. It showed that very few people are actually willing to run an unstable/development release and actively submit feedback. So that doesn't help either.

 

[RMerlin]

I have a machine running Ubuntu and have a decade plus in the web java world plus c/c++ way back in the day . . you ever thought of recruiting/allowing help? Or would that create more headaches than it's worth?

The project is open sourced, and I keep my Github repo up-to-date throughout development. So the door is already open to pull requests. I occasionally get one from time to time.

One thing I'd love to have right now is someone to take over the MIPS models. A lot of things have to be implemented and/or tested twice due to the difference in architecture. So far I've never had any luck in finding anyone willing to devote the time and efforts toward it.

I'll end this with a question (out of pure curiosity), though: Everyone says the RT-AC3100 is basically the AC88U minus the red accents and the extra ports. If that's true, why does it require different firmware? Are they actually different or is this just to reduce confusion

The Realtek switch that adds 4 extra ports on the RT-AC88U. That requires the firmware to be compiled with different options.

 

[RMerlin]

I would suggest to limit your personal support for the WIFI portion at least,

I already do. Notice that I ignore most wifi-related reports - been the case for a year or two now. I just had to address it in this thread because the number of wifi-related complains had been increasing.

 

[RMerlin]

The idea of "beta" vs. "final" has already been abandoned by many professional organisations, where they are moving to "agile" project practice. The idea is to move faster, slightly less quality, but react even faster if bugs show up. No beta, but more frequent deliverables. Another trend is products that stay "beta" forever, even with very high quality code.

This is a trend in the computer industry that's been bothering me. It's as if they realized that customers have grown used to software having bugs, so they say "to Hell with it", and no longer worry about pushing out buggy code, just telling themselves that "we can release a fixed release in two weeks if something's badly broken".

I have been involved in computers in one way or another for roughly 30 years now. And I see a tendency for software to become increasingly complex, and also increasingly buggy. We all know a bug or two in Windows that has been there for years, yet has never been fixed by Microsoft because "it's not a priority / it's not a deal breaker". The max filename length issues for instance - those have existed since the Windows 98 days. Every few months I get a customer who runs into it. Windows has allowed to create a file with a full pathname > 255 characters, but you are unable to access the file until you shorten its path.

I always prided myself in pushing out stable code. That's something I'm unable to achieve to a satisfying level with this project, because I don't have complete control over the code base. And that's a source of frustration for me.

 

[RMerlin]

Another question--probably not the best thread for it but I don't want to start another thread--Merlin's always cautioning us to not do frequent writes to NVRAM (be it for traffic statistics or whatever) because it's flash memory and will wear out. Does this not apply to flashing firmware updates? Or is that even relevant?

I've flashed some of my routers more times than 90% of the users in this forum ever will flash all of their routers in their lifetime. 50-60 flashes are not significant when the NAND chip is rated for many thousands of P/E cycles.

 

[kvic]

I considered it ( I was thinking about having a LTS release at one point), and it won't help, for two reasons:

1) Two branches to maintain = even more work
2) When Asus releases a new GPL, it's next to impossible for me to backport any specific fixes, as I don't have access to their specific commits.

I did a quick sketch[1] to illustrate... Will it work better for you? Perhaps a bit more merge work but seems could be done in similar amount of time but more enjoyable..

Along the unstable branch (development/main trunk), you can pick when to make a release for tests and feed back.

It showed that very few people are actually willing to run an unstable/development release and actively submit feedback. So that doesn't help either.

That's because most users take a position of wait and see as they expect final release will be out soon.

With major releases on stable branch separated from a quart to half year, more time for feedback and testings on the development/main trunk. Also give yourself room to work on new features between major releases.

Some people won't be happy with development releases, and they always have fallback to a well supported stable branch. Only bug fix after its major release there..so work is minimal after the initial release on a stable branch.

[1]

 

[hellboy2]

I see a tendency for software to become increasingly complex, and also increasingly buggy.

Increasingly complex: Yes
Increasingly buggy: no...

Modern day "Buggy" perception is coming from people, who weren't there 30 years ago, where code was simpler, but still had more bugs per line of code than it has today. Who wants "quality" of Win95, where you had to boot your PC several times every day? My 80-year old mother was not there at the time (no PC). Today she complains "all software is buggy" when it takes a little too long to connect something.

Modern day users just won't put up with yesterday's "quality" any more ...

 

[Broken Hope]

It's a shame that Asus don't incorporate more of Merlin's code into the official firmware, that would limit the amount of work he'd have to do.

Stuff like the Wireless log is vastly superior in Merlin's firmware and I really missed it when I was briefly on the official firmware to get the updated wireless driver.

 

[RMerlin]

Modern day users just won't put up with yesterday's "quality" any more ..

The Internet has made developers lazy. Look at computer/console games as another example. Day-zero patches is not something you would have heard about 10 years ago. Now, it's becoming the norm.