Development musings

[RMerlin]

I did a quick sketch[1] to illustrate... Will it work better for you? Perhaps a bit more merge work but seems could be done in similar amount of time but more enjoyable..

The problem is those bugfixes are often impossible for me to isolate and extract out of a new GPL release and reapply on what would be a stable branch. So, the stable branch wouldn't really get more stable than a development branch. A lot of the issues can only be fixed by merging the whole new GPL.

That forked development model works great when 100% of the code is under your control, and everything gets merged as separate commits. It makes it easy then to cherry pick fixes and merge them back into a separate branch. I can't do that with the all-or-nothing GPL releases.

 

[cowst]

The problem is those bugfixes are often impossible for me to isolate and extract out of a new GPL release and reapply on what would be a stable branch. So, the stable branch wouldn't really get more stable than a development branch. A lot of the issues can only be fixed by merging the whole new GPL.

That forked development model works great when 100% of the code is under your control, and everything gets merged as separate commits. It makes it easy then to cherry pick fixes and merge them back into a separate branch. I can't do that with the all-or-nothing GPL releases.

Didn't Asus acknowledge your work if I remember correctly?
If for a while they would shift their efforts to port your code into theirs (it would be an admirable humble move), both you and them would probably take advantage of each other in the long run.
If they could even open their code, jackpot

Sent from my Nexus 4 using Tapatalk

 

[cowst]

The web server is crashing, no idea why (Asus has been stuffing so much code into it that it's no longer "just a web server".)

Try accessing it over HTTP instead of HTTPS - there's been some issues with SSL in the latest GPL release, and I'm unable to track them down so far.

If it helps, a few posts back I could reproduce the httpd restart consistently in the dhcp page across different browsers.
If that's not in the secret Asus code, it seems like a good place to start debugging in a controlled environment.

Sent from my Nexus 4 using Tapatalk

 

[kvic]

The problem is those bugfixes are often impossible for me to isolate and extract out of a new GPL release and reapply on what would be a stable branch. So, the stable branch wouldn't really get more stable than a development branch. A lot of the issues can only be fixed by merging the whole new GPL.

You don't have get every bug fixed on a stable branch. Or else what's the point of next major stable release? Prioritise and fix on a best effort basis. Some by yourself. Some by ppl here. Some hopefully by identified codes from subsequent GLP merge on the development/main trunk.

Don't take it arbitrary the timing for branching a stable release. In retrospect, e.g. 378.55 timeframe was a good time for a stable branch. With yourself and ppl here spending more time working and testing on the development branch, your chance of getting the right moment of branching is significantly increased...

That forked development model works great when 100% of the code is under your control, and everything gets merged as separate commits. It makes it easy then to cherry pick fixes and merge them back into a separate branch. I can't do that with the all-or-nothing GPL releases.

Look at it differently. Assume all codes open and developed in house. Believe it or not. No single moment all codes are under your control. In a state that you like. Treat binary blobs and Asus GPL schedule like that. But still you can devise a schedule for your delivery of the end product.

Perhaps time to think out of box. Hope you find a sustainable way. Or else soon you will burn out along the current trajectory..

 

[RMerlin]

Don't take it arbitrary the timing for branching a stable release. In retrospect, e.g. 378.55 timeframe was a good time for a stable branch.

378.55 would have meant no support for the RT-AC88U, RT-AC3100 or RT-AC5300. So, the switch to the 380 codebase was necessary. The security improvements to httpd were also too extensive to be easily backported (a lot of code changed between the last 378 and the first 380 httpd release, with the switch to ticket-based authentication).

I've been looking for quite some time already for that single release that could have been used as a potential LTS release. Couldn't find a single one in the past year. I thought 380.59 was it, as the beta test feedback was quite positive. Turned out to have its own share of issues as well (half of which I can't even reproduce here).

Maybe if I can get that one release based on one solid GPL with no major issue, I will revisit the possibility of branching out an LTS release. That simply hasn't happened yet.

 

[kvic]

378.55 would have meant no support for the RT-AC88U, RT-AC3100 or RT-AC5300. So, the switch to the 380 codebase was necessary. The security improvements to httpd were also too extensive to be easily backported (a lot of code changed between the last 378 and the first 380 httpd release, with the switch to ticket-based authentication).

Ah, since you're on this point. I found it very interesting a while back. You don't encourage people to purchase the latest and greatest hardware from vendors. But you prioritise FW development on such models.

From a developer perspective, I could understand. It's fun. From a user base perspective probably not. Personally I would take it 2nd priority..important but not critical. So I get more time for 380 GPL to mature for example

On securities...I don't know about the specific bugs patched in httpd. So can't comment. In general though, I think we have to judge if the change is critically important or not...On this project on Asus touched codes, back porting anything more than 50 lines I won't go ahead...

I've been looking for quite some time already for that single release that could have been used as a potential LTS release. Couldn't find a single one in the past year. I thought 380.59 was it, as the beta test feedback was quite positive. Turned out to have its own share of issues as well (half of which I can't even reproduce here).

Maybe if I can get that one release based on one solid GPL with no major issue, I will revisit the possibility of branching out an LTS release. That simply hasn't happened yet.

Depends on the time span of LTS. Any LTS longer than six month is unrealistic IMO. Personally I think two major releases per year are very good...plus one more is perfect.

 

[shpankey]

Not in the short term, but I will need to find a way to change some things. Right now, this has gone from a fun hobby into a chore.

Hi Merlin, first, THANK YOU, for all you have done for us over the years. Amazing dedication for no return on your time. I think I speak for us all here when I say that.

EDIT: never mind, i see you've already thought of this.

Anyhow, might I suggest something? Team up w/ the likes of John, and perhaps you could only work on certain firmware versions... ones that Asus releases and users respond well to. Do your magic on a known good release. In the meantime John could fill the in-betweens, like he does now. Just upgrading your version with the new Asus versions security fixes, until after awhile, another new, good, quality, stable version gets released and you make a new base version for which John could work from to add security fix additions along w/ other extremely minor fixes, if needed.

This would be the best of both worlds for all involved, and you could limit yourself to 3 or 4 base version per year... or whatever you feel like handling whenever you get the bug, so to speak. Just a thought. Anyhow, please and thanks again.

 

[vaszago]

I always prided myself in pushing out stable code. That's something I'm unable to achieve to a satisfying level with this project, because I don't have complete control over the code base. And that's a source of frustration for me.

Have you Asked Asus to see the Source Code.
That you sign a declaration, to see the source.

And Asus give feedback when you find a bug in Code.
Because an external sees the thing may be somewhat different.

But I think that's what you not necessarily want!

 

[punkinduster]

RMerlin:
This is probably something that has been talked about many times but have you or ASUS ever talked about having you sign an agreement to give you access to the ENTIRE code base. Would you have to be an employee to have access to the proprietary stuff or sign a nondisclosure agreement. Instead of having your hands tied at certain issues that are out of your control you might be able to tweak any or all lines of code as long as you follow their/FCC rules. At minimum to maintain their control if they prevented you from releasing code changes to the proprietary stuff you could at least view it to see where the problem lies or how to perform a temporary work-a-round until they can release the fix.

 

[Tekneek]

It never stops blowing my mind how people can gripe and complain about something that is free. Bug reports are one thing, but demanding that something be done for them on a completely free project makes absolutely no sense to me. If it doesn't work for you, file the bug reports and/or enhancement requests and either put up with the current state of affairs or move along. Even if you are donating, that does not exactly imply that you get to have a say in the way things go.

 

[L&LD]

RMerlin:
This is probably something that has been talked about many times but have you or ASUS ever talked about having you sign an agreement to give you access to the ENTIRE code base. Would you have to be an employee to have access to the proprietary stuff or sign a nondisclosure agreement. Instead of having your hands tied at certain issues that are out of your control you might be able to tweak any or all lines of code as long as you follow their/FCC rules. At minimum to maintain their control if they prevented you from releasing code changes to the proprietary stuff you could at least view it to see where the problem lies or how to perform a temporary work-a-round until they can release the fix.

If I understand how things are working correctly, RMerlin (or anyone else that downloads the source files Asus provides) does have full access already. Except for the proprietary Broadcom drivers that even Asus themselves have not control over.

 

[RMerlin]

Ah, since you're on this point. I found it very interesting a while back. You don't encourage people to purchase the latest and greatest hardware from vendors. But you prioritise FW development on such models.

I don't necessarily priorize it, but adding support for the new high-end models (I stay away from any mid/low-range models) is just as important for me. First, because I do often get early access to the new models. Also, because otherwise I get a daily avalanche of email/tweets/forum posts asking me if/when I will support it. And third because newer models open the door to improvements not possible on previous models. The ARM models for instance opened the door to much better VPN performance, a large, dedicated JFFS partition, etc...

The only reason I don't actively push people toward buying new models is the price. Price-aside, the RT-AC88U is their best router in a while, from the point of performance, stability and features. It's what I use as my main router at home (another reason for me to support it as soon as possible, for selfish reasons).

The original mandate of this project has been to enhance the original firmware, rather than truly fork away from it. That's another reason for me to try to keep in sync with Asus as much as possible. User feedback has shown that this is one of the main reason for people using my firmware rather than Tomato or DD-WRT. I sometime get people asking me about merging new versions before Asus even releases the GPL code for it... To stop staying in-sync with them would be a major change of philosophy that I wouldn't take lightly, and would require a lot of thinking before making such a decision.

On securities...I don't know about the specific bugs patched in httpd. So can't comment. In general though, I think we have to judge if the change is critically important or not...On this project on Asus touched codes, back porting anything more than 50 lines I won't go ahead...

Look back at the Asus changelog for the past 18 months. Almost each new release has one or two XSS or other webui-related flaw fixed. A lot of additional flaws were avoided by the switch from basic auth to session-based tickets. Having stuck to the original basic auth model would have meant become responsible for finding and fixing any other XSS flaw still present in that old auth scheme.

 

[RMerlin]

This is probably something that has been talked about many times but have you or ASUS ever talked about having you sign an agreement to give you access to the ENTIRE code base. Would you have to be an employee to have access to the proprietary stuff or sign a nondisclosure agreement. Instead of having your hands tied at certain issues that are out of your control you might be able to tweak any or all lines of code as long as you follow their/FCC rules. At minimum to maintain their control if they prevented you from releasing code changes to the proprietary stuff you could at least view it to see where the problem lies or how to perform a temporary work-a-round until they can release the fix.

This would have certain advantages, but it also comes with its share of problems, such as being unable to know which part of the code is confidential, which part is work-in-progress, and which part is freely open.

And this is also the kind of thing I, personally, don't feel comfortable to ask, as it could be seen as presumptuous. If they were to offer it, I would definitely consider it. But I don't want to make anyone uncomfortable by directly asking for such a thing.

 

[RMerlin]

Sorry for derailing the original thread anyway, I was just thinking out loud last night, after spending a lot of time trying to merge the newest OpenVPN release (Asus's status/error report code wasn't compatible with it, and had to be re-engineered).

 

[RMerlin]

It never stops blowing my mind how people can gripe and complain about something that is free. Bug reports are one thing, but demanding that something be done for them on a completely free project makes absolutely no sense to me.

Truth be told, this is rarely the case. The vast majority of people are being very polite when reporting issues.

 

[fiasko]

The Internet has made developers lazy. Look at computer/console games as another example. Day-zero patches is not something you would have heard about 10 years ago. Now, it's becoming the norm.

I dont' think the developers are lazier than before, but the quality control is hasn't developed with the size and complexity of the software. Internet has only lowered the bar to release software, since it's so easy and fast to send an updated software to custommers if they report bugs. Because the software is much more complex, it's getting harder and harder to spot the problems and test all the corner cases. And very ofter there comes a case that the company can't even test some features by them self (been there few times) and they need to resort more and more for end-users to do the final testing. Sad but true.

And I'm quite sure you are the victim of this same problem than I am. More and more software gets out, more and more hardware variants get out, and developers like you and me have less and less controll over the quality of the software, due to it's size, complexity or some other reasons (like byrocrasy). This has built quite a bit of frustration during the years for me.

 

[chamberc]

20 years of tier 1 app development (you've used what I work on). It's not the developers, it's customer demands. More features more features more features.

 

[RMerlin]

I dont' think the developers are lazier than before, but the quality control is hasn't developed with the size and complexity of the software. Internet has only lowered the bar to release software, since it's so easy and fast to send an updated software to custommers if they report bugs.

That's a good point.

20 years of tier 1 app development (you've used what I work on). It's not the developers, it's customer demands. More features more features more features.

And marketing dept asking that the code ships on a specific date, regardless of the state (as long there's no critical blockers left).

I know someone who works at Symantec in the QA department (not at their lowest level). I've heard a few choice stories over the years...

 

[RMerlin]

A couple of things I've been thinking about these past few months, and which could (potentially) help:

Having someone take over support & development for the two MIPS routers (RT-N66U and RT-AC66U). These models (especially the N66U) are far too popular to completely drop support for, however having someone take care of maintaining the MIPS side of things would free up some of my time. No more hunts for MIPS-specific GPLs (for the MIPS binary blobs), two less routers to test.

Extending development cycles. I've already started on this, extending from the former one month cycle (where I was generally alternating between feature releases and bugfix releases) into a longer 2-3 months cycle, with occasional beta (and even alpha) builds. This did help in getting some issues spotted earlier during development, thanks to a few users willing to run even alpha level code, and providing really good feedback. I will most likely continue in this direction.

Re-enabling the issue tracker on Github. I disabled it a few years ago because the vast majority of new issues were really support requests rather than genuine bug reports, and it was becoming a good amount of work sorting through my mailbox notifications. I might be considering re-opening it once again.

Getting a more reliable access to new GPLs. That one isn't under my control unfortunately, but issues like the 2695/2697 corrupted and/or missing GPL drops do cost me a lot of time and efforts.

Enforcing a strict "if you don't mention your router model then your bug report will be ignored" policy. This isn't about being a jerk to people, just that I often waste a lot of time trying to reproduce an issue, only to learn later on that it only happens on a very specific model. Take the RT-AC87U for instance, which accounts for a lot of "My 5 GHz no longer works" reports. Maybe setup (at least informally) some type of template of info that MUST be included whenever reporting any issue. Router model and whether or not it can be reproduce on the stock firmware would be good starting points.

More eyeballs on the code on the debugging front. So far, I've noticed that most people actively contributing are more interested in feature addition/improvements than actual debugging of existing issues. Once again, not something I have any control over. The learning curve isn't necessarily easy for anyone's first attempt at diving into this jungle. There are large parts of the code where I simply don't step foot myself, like the Dual WAN code, or any of the non-router operation modes...


Once again, just thinking out loud here.

 

[martinr]

.... Right now, this has gone from a fun hobby into a chore.

The only surprise is that it's taken this long.

And you haven't mentioned the time and work you put in to the forum postings - I worked it out as an average of 12 postings a day over the 4 years the project has been running, though I still can't believe my maths.

It was brought home to us recently when you had to take time out for health problems that this can't go on for ever, and certainly not at your current workload. With only the best of intentions and care for your welfare, some might say "Get a life, Eric".

You've always made a point of stability being the priority (with new bells and whistles being way down the list, if indeed it was even on the list. I don't understand enough about this stuff to know whether or not it's even practical, but if you "froze" the project at this point and only brought out updates to counter security and stability flaws, I'd still happily stay with your firmware. But I doubt life is that simple.