[Dev] Q&A for Merlin Addons API

[Jack Yaz]

I know FreshJR had some unique standards for how he coded the terminal sequences, and I'm not much of a terminal coder. I'd welcome the rest of the community to point out issues or simplifications. If we don't get any feedback from Fresh, I would propose to adapt the UI to the new Addon API and add a table of rows for rules, similar to DNS Privacy, DHCP reservations, etc. I would also propose to make it a separate page from Merlin's original Classification page, to minimize the number of bind mounts of ROM pages.

I'm only prepared to make general changes to the structure of the page/script. I don't game, so I don't plan to entertain complaints from gamers who are sad because of their ping times (apologies in advance if you're a gamer who whines about ping times ).

Yet I still get waaaayyy ahead of myself. It's daunting to try to improve someone else's code when they're not around to explain their choices. Hence my encouragement for community involvement (pull requests!).

Those 10 user mounts are running out fast haha. The Addon API would remove the limit on number of rules that can be added, at the very least. I can probably chip in if you or others get stuck, but my existing script base development and support (and the non stop feature requests..) have got me pretty busy.

 

[SomeWhereOverTheRainBow]

Those 10 user mounts are running out fast haha. The Addon API would remove the limit on number of rules that can be added, at the very least. I can probably chip in if you or others get stuck, but my existing script base development and support (and the non stop feature requests..) have got me pretty busy.

you may have to move onto using other pages

 

[dave14305]

Those 10 user mounts are running out fast haha. The Addon API would remove the limit on number of rules that can be added, at the very least. I can probably chip in if you or others get stuck, but my existing script base development and support (and the non stop feature requests..) have got me pretty busy.

The custom_settings.txt file is limited to 8K, but I’ve not seen many devs choose to use it, so it’s a start. Then I would need to steal from other devs who manage their own settings - once it proves too small. I’m guessing IPv6 addresses will bloat things a lot (if I can figure all that out).

I would like to think of this as “Stone soup” QoS.

 

[Jack Yaz]

The custom_settings.txt file is limited to 8K, but I’ve not seen many devs choose to use it, so it’s a start. Then I would need to steal from other devs who manage their own settings - once it proves too small. I’m guessing IPv6 addresses will bloat things a lot (if I can figure all that out).

I would like to think of this as “Stone soup” QoS.

since my scripts have their own config files, i cycle my settings through custom_settings.txt to get things back and forth between the WebUI and the script config in /jffs/addons/scriptdir

 

[thelonelycoder]

since my scripts have their own config files, i cycle my settings through custom_settings.txt to get things back and forth between the WebUI and the script config in /jffs/addons/scriptdir

Ah, that’s how you do it. I was wondering and meant to look into your code.
@dave14305 I’m using it but can’t tell at the moment how many entries go into it as it’s still a work in progress. One function at a time.

 

[thelonelycoder]

since my scripts have their own config files, i cycle my settings through custom_settings.txt to get things back and forth between the WebUI and the script config in /jffs/addons/scriptdir

The Merlin wiki says to use your own settings file for larger configs but does not say how to address it in the code. Are we missing something or is it just not implemented?

 

[dave14305]

Are we missing something or is it just not implemented?

There seems to be very little flexibility (due to security concerns) to write to the filesystem ad-hoc. Maybe we can get some clarification from @RMerlin or @john9527.

 

[Jack Yaz]

The Merlin wiki says to use your own settings file for larger configs but does not say how to address it in the code. Are we missing something or is it just not implemented?

Not sure, but I came up with something to bridge the gap and it seems to work!

 

[dave14305]

There seems to be very little flexibility (due to security concerns) to write to the filesystem ad-hoc. Maybe we can get some clarification from @RMerlin or @john9527.

Based on how Merlin implemented the amng_custom as a dummy nvram variable of size 8192, it’s seeming unlikely to write anything out to the file system not already defined in the httpd code. So using nvram and custom_settings is the likely only way to egress data from the UI. Flexibility is only in reading it from a config like Jack’s that is soft-linked to /www/user directory.

Maybe if we can come up with a compelling use-case we can lobby for 10 custom_settings files (1 for each addon page allowed). If you don’t ask, the answer is always no.

 

[dave14305]

This will be a general thread for developers to discuss questions and answers about the Addon API and how to best implement certain features, or how to play nicely together within the constraints of the router UI.

Wiki: https://github.com/RMerl/asuswrt-merlin.ng/wiki/Addons-API

 

[thelonelycoder]

Based on how Merlin implemented the amng_custom as a dummy nvram variable of size 8192, it’s seeming unlikely to write anything out to the file system not already defined in the httpd code. So using nvram and custom_settings is the likely only way to egress data from the UI. Flexibility is only in reading it from a config like Jack’s that is soft-linked to /www/user directory.

Maybe if we can come up with a compelling use-case we can lobby for 10 custom_settings files (1 for each addon page allowed). If you don’t ask, the answer is always no.

The custom_settings file would have to have a defined name like diversion_custom_settings.txt. One then would set that as a var custom_settings_file in the page.asp.

Better still one could simply add the location to custom_settings.txt and that would be it. No more values are stored there but in the referenced file.

am_settings_set diversion_settings_file_loc = /jffs/addons/diversion/ui_settings.txt

 

[RMerlin]

There seems to be very little flexibility (due to security concerns) to write to the filesystem ad-hoc. Maybe we can get some clarification from @RMerlin or @john9527.

I never found a way to interface external config files with the webui. Too many technical limitations, like the size of the POST buffer used by httpd. The idea of an external settings file was only intended for the script's own use, not for webui integration.

 

[dave14305]

I never found a way to interface external config files with the webui. Too many technical limitations, like the size of the POST buffer used by httpd. The idea of an external settings file was only intended for the script's own use, not for webui integration.

I wouldn’t mind posts 4193 and on (except 4204) being moved to my Q&A thread:
https://www.snbforums.com/threads/dev-q-a-for-merlin-addons-api.63621/

 

[RMerlin]

Those 10 user mounts are running out fast haha. The Addon API would remove the limit on number of rules that can be added, at the very least. I can probably chip in if you or others get stuck, but my existing script base development and support (and the non stop feature requests..) have got me pretty busy.

It's not an arbitrary limit, it's because mount points need to be already created at firmware build time. So, the script will limit it based on how many mount points exist.

I can add more mount points in the future based on how things evolved, but I'm not going to create 99 separate entries making the whole www/ folder a complete mess. So, new entries will be added incrementally over time - for example, I might add 5 or 10 more in a future release.

 

[Jack Yaz]

It's not an arbitrary limit, it's because mount points need to be already created at firmware build time. So, the script will limit it based on how many mount points exist.

I can add more mount points in the future based on how things evolved, but I'm not going to create 99 separate entries making the whole www/ folder a complete mess. So, new entries will be added incrementally over time - for example, I might add 5 or 10 more in a future release.

The limit i was referring to was the way FreshJR implemented custom rules - they're currently saved to an unused nvram variable, which have set limits. I of course don't expect you to add an unlimited number of mount points!

 

[dave14305]

Today we end up with an 8KB repository that can store entries up to 2999 characters each, per the wiki. So in the end, more than 2 large settings could overrun the file. 2999 is not small, per se, but similar to existing nvram var constraints.

 

[dave14305]

I’ve tried to build a “check for update/perform update” in the UI of my sandbox Unbound project. It’s not foolproof yet, but I’d like others’ feedback on if you’ve considered something similar and how you might go about it.

https://github.com/dave14305/Unbound-Merlin-UI

My steps:

1. Write the script’s current version to custom_settings.txt
2. By default, show a Check button.
3. When clicked, run an update function via the script called in service-event.
4. If a new version found, write that version in a newversion setting in custom_settings.
5. On page reload, if the current version is different than newversion, hide the check button and show an update button with a text span showing the new version number available.
6. When clicking update, run the update function to unmount the asp page, download new files, remount the ASP page (and pray for the same userx.asp), and update the current version number.
7. When the page reloads (hopefully), the versions match so the update button is switched back to Check.

Please share your ideas or thoughts for improvement.

 

[elnash]

I’ve tried to build a “check for update/perform update” in the UI of my sandbox Unbound project. It’s not foolproof yet, but I’d like others’ feedback on if you’ve considered something similar and how you might go about it.

https://github.com/dave14305/Unbound-Merlin-UI

My steps:

1. Write the script’s current version to custom_settings.txt
2. By default, show a Check button.
3. When clicked, run an update function via the script called in service-event.
4. If a new version found, write that version in a newversion setting in custom_settings.
5. On page reload, if the current version is different than newversion, hide the check button and show an update button with a text span showing the new version number available.
6. When clicking update, run the update function to unmount the asp page, download new files, remount the ASP page (and pray for the same userx.asp), and update the current version number.
7. When the page reloads (hopefully), the versions match so the update button is switched back to Check.

Please share your ideas or thoughts for improvement.

This is great

 

[Dabombber]

am_get_webui_page() {
   for i in 1 2 3 4 5 6 7 8 9 10; do
       page="/www/user/user$i.asp"
       if [ ! -f "$page" ] || [ "$(md5sum < "$1")" = "$(md5sum < "$page")" ]; then
           am_webui_page="user$i.asp"
           return
       fi
   done
   am_webui_page="none"
}

I mentioned this before, but this will return an empty slot if there's a gap before your page.

It seems like cab files are the only type which has auth and doesn't get processed by ej (the <% %> stuff). That would make it the best for raw web data, ideally there'd be something like

{ "**.txt", "text/plain", no_cache_IE7, NULL, do_file, do_auth },

The Merlin wiki says to use your own settings file for larger configs but does not say how to address it in the code. Are we missing something or is it just not implemented?

I guess the best way is linking to the config file from /www/user and reading it from there. As for writing, you'd need to pass the data back using a service event.

ln -s '/jffs/addons/myaddon/custom_settings.txt' '/www/user/mysettings.cab'

$.ajax({
   url: '/user/mysettings.cab',
   dataType: 'text',
   success: function(response) {
       console.log(response);
   }
});

 

[Jack Yaz]

I’ve tried to build a “check for update/perform update” in the UI of my sandbox Unbound project. It’s not foolproof yet, but I’d like others’ feedback on if you’ve considered something similar and how you might go about it.

https://github.com/dave14305/Unbound-Merlin-UI

My steps:

1. Write the script’s current version to custom_settings.txt
2. By default, show a Check button.
3. When clicked, run an update function via the script called in service-event.
4. If a new version found, write that version in a newversion setting in custom_settings.
5. On page reload, if the current version is different than newversion, hide the check button and show an update button with a text span showing the new version number available.
6. When clicking update, run the update function to unmount the asp page, download new files, remount the ASP page (and pray for the same userx.asp), and update the current version number.
7. When the page reloads (hopefully), the versions match so the update button is switched back to Check.

Please share your ideas or thoughts for improvement.

I like this idea. I'm thrashing out some code to see if I can use your framework with spdmerlin as we speak!