[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[FreshJR]

I got the table generation down to 150ms per 500 entries, but I still feel like that can be improved !!!

Gotta look up whats the fastest way to generate and append html elements!

Quick question, how is the modified page being implemented? Are you mounting/binding a duplicate page over the top of the stock page?

I ask since i use the kvic ntpd and have a copy of /www mounted from USB.

Yes. Implemented via the following:

mount -o bind /jffs/FreshJR_QoS_Stats.asp /www/QoS_Stats.asp
umount /www/QoS_Stats.asp && rm /jffs/FreshJR_QoS_Stats.asp

--

Update:
Table generation time down to ~30ms for 300 entries. Need to reimplement sort.

No issues/delays when running torrents and scrolling on the Stats page

 

[Milan]

off topic: can you rewrite the whole GUI?

 

[heysoundude]

I got the table generation down to 150ms per 500 entries, but I still feel like that can be improved !!!

Gotta look up whats the fastest way to generate and append html elements!



Yes. Implemented via the following:

mount -o bind /jffs/FreshJR_QoS_Stats.asp /www/QoS_Stats.asp
umount /www/QoS_Stats.asp && rm /jffs/FreshJR_QoS_Stats.asp

--

Update:
Table generation time down to ~30ms for 300 entries. Need to reimplement sort.

No issues/delays when running torrents and scrolling on the Stats page

So you cut it in half (300ms-150), and then hacked that back by another 80% (150ms-30)...a 90% improvement...in a couple of hours (judging from post times)?
what was that donation link again?

 

[Asad Ali]

I got the table generation down to 150ms per 500 entries, but I still feel like that can be improved !!!

Gotta look up whats the fastest way to generate and append html elements!



Yes. Implemented via the following:

mount -o bind /jffs/FreshJR_QoS_Stats.asp /www/QoS_Stats.asp
umount /www/QoS_Stats.asp && rm /jffs/FreshJR_QoS_Stats.asp

--

Update:
Table generation time down to ~30ms for 300 entries. Need to reimplement sort.

No issues/delays when running torrents and scrolling on the Stats page

That's great, why not push the changes to the upstream, so Eric @RMerlin can implement them directly in the source.

 

[Jack Yaz]

I got the table generation down to 150ms per 500 entries, but I still feel like that can be improved !!!

Gotta look up whats the fastest way to generate and append html elements!



Yes. Implemented via the following:

mount -o bind /jffs/FreshJR_QoS_Stats.asp /www/QoS_Stats.asp
umount /www/QoS_Stats.asp && rm /jffs/FreshJR_QoS_Stats.asp

--

Update:
Table generation time down to ~30ms for 300 entries. Need to reimplement sort.

No issues/delays when running torrents and scrolling on the Stats page

Great, I'll just drop the page into my USB folder and it can mount from there. I'm happy to manually modify the script to comment out the mount line

 

[Vexira]

This is most likely how the "modification" interface will look.

98% done (this does not include testing that I will be performing prior to release).

would the band with reserved values be filled in by default, and also will there be an option to set the bandwidth % eg 95% total bandwidth in the gui since ive seen other routers have a slider in the settings for qos

 

[Kingp1n]

It's so quiet the suspense is killing us....haha ...take your time @FreshJR...should be an awesome experience once it's completed. Def donating for all your hard work!!!

 

[maxsteel]

It's so quiet the suspense is killing us....haha ...take your time @FreshJR...should be an awesome experience once it's completed. Def donating for all your hard work!!!

+1

 

[FreshJR]

It's so quiet the suspense is killing us....haha

Currently just fixing up random odds and ends while testing. Should be out within next 24 hours.

I was also looking into an issue that was bugging me. I noticed that the initial page load takes an insane 1-2 sec!! (While a page refresh is taking 50ms MAX)

On page load, I grab connected devices an nvram variable (static devices), from an arp query (currently connected), and from the DHCP table (not connected but still have active lease).

I narrowed down the drastic load time to the WebUI function that pulling data from the DHCP table.
That function is restarting dnsmasq to repopulate remaining lease time. This process takes the 1-2sec!!

This approach bothers me but it is out of my control to fix since this function is defined during compile time.

I will try to work with RMerlin to either
1) see if we can just simply query the current time && creation of "/var/lib/misc/dnsmasq.leases" and subtract that different from the timestamps in the leases file
or
2) see if hes simply willing to add another version of the "get_leases_array" function that will not restart dnsmasq (I dont need remaining lease time parameter)

--

For this release I can disable querying the devices in the DHCP table to remove the intial load time, but I think I will leave it included.

 

[skeal]

Currently just fixing up random odds and ends while testing. Should be out within next 24 hours.

I was also looking into an issue that was bugging me. I noticed that the initial page load takes an insane 1-2 sec!! (While a page refresh is taking 50ms MAX)

On page load, I grab connected devices an nvram variable (static devices), from an arp query (currently connected), and from the DHCP table (not connected but still have active lease).

I narrowed down the drastic load time to the WebUI function that pulling data from the DHCP table.
That function is restarting dnsmasq to repopulate remaining lease time. This process takes the 1-2sec!!

This approach bothers me but it is out of my control to fix since this function is defined during compile time.

I will try to work with RMerlin to either
1) see if we can just simply query the current time && creation of "/var/lib/misc/dnsmasq.leases" and subtract that different from the timestamps in the leases file
or
2) see if hes simply willing to add another version of the "get_leases_array" function that will not restart dnsmasq (I dont need remaining lease time parameter)

--

For this release I can disable querying the devices in the DHCP table to remove the intial load time, but I think I will leave it included.

It feels like Christmas!!

 

[RMerlin]

Currently just fixing up random odds and ends while testing. Should be out within next 24 hours.

I was also looking into an issue that was bugging me. I noticed that the initial page load takes an insane 1-2 sec!! (While a page refresh is taking 50ms MAX)

On page load, I grab connected devices an nvram variable (static devices), from an arp query (currently connected), and from the DHCP table (not connected but still have active lease).

I narrowed down the drastic load time to the WebUI function that pulling data from the DHCP table.
That function is restarting dnsmasq to repopulate remaining lease time. This process takes the 1-2sec!!

This approach bothers me but it is out of my control to fix since this function is defined during compile time.

I will try to work with RMerlin to either
1) see if we can just simply query the current time && creation of "/var/lib/misc/dnsmasq.leases" and subtract that different from the timestamps in the leases file
or
2) see if hes simply willing to add another version of the "get_leases_array" function that will not restart dnsmasq (I dont need remaining lease time parameter)

--

For this release I can disable querying the devices in the DHCP table to remove the intial load time, but I think I will leave it included.

It doesn't restart dnsmasq, httpd just sends it a SIGUSR2, wait 1 second to let it write the lease file to disk, then httpd reads it, and processes it.

 

[FreshJR]

After more reflection, this 1sec wait is unacceptable!

I will be cutting back on the accuracy of the local IP <--> device name resolution to eliminate the initial wait on page load.

Filtering by IP will STILL work as expected. The only drawback is that this method
1) wont resolve devices into assigned nicknames
2) some device names may resolve into "*" instead of their advertised device name.

The slow approach was rock solid in resolving local IP's into corresponding device names but it did lack the sweet sweet speed I personally expect from webpages.

Maybe @RMerlin will make an exception and make available an identical function (copy/paste of the existing one ) that will grab the DHCP data as available but lacking the 1sec wait that makes sure it returns refreshed data.

Technical:

Spoiler

The data I needed from the DHCP table was:

device IPS corresponding to device MACS.​

I then opened ( /jffs/nmp_cl_json.js ) and grabbed:

device MACS corresponding to device NAMES
​

I aggregated the two bits of data together and got a rock solid Local IP <--> device name correlation.
The issue was that grabbing the DHCP table was too slow.

--

My new workaround will be to ONLY load:

devices IPS & NAMES from the DCHP table (no MAC as ideally desired to then make a correlation with the “nmp” data)​

You may think that we already have device names in the dhcp table so what is the issue. The down side is that the device names in the DHCP table do not correlate assigned nicknames and they are also not always populated correctly (some are listed as "*").

The positive side of the inaccurate method is that the function responsible for only grabbing those two columns of data does not incur a 1second wait when called.

By introducing this "more fuzzy" local IP <--> device NAME resolution, we can gain speed at the sake of minor accuracy in terms of device name.

This can be resolved in the future if RMerlin will decide to supply a function to get the DHCP table data without a hardcoded 1sec wait.

The method of Local IP (DHCP) --> MAC (DHCP) --> Device name(nmp_cl) was rock solid, but the middle link was a bottleneck.

 

[RMerlin]

You could retrieve the lease list through an Ajax call, so while you are rendering/sorting/etc... the lease list will be refreshed and pushed to you, async from the main page. This is how I implemented the VPN status page update.

If you don't need the actual lease time, then look into what's provided through client_functions.js.

 

[FreshJR]

You could retrieve the lease list through an Ajax call, so while you are rendering/sorting/etc... the lease list will be refreshed and pushed to you, async from the main page. This is how I implemented the VPN status page update.

If you don't need the actual lease time, then look into what's provided through client_functions.js.

I will look into running the function via an ajax call instead of an < ASP CODE NUGGET> (sounds solid).

I have already tried most of the raw ASP code snippets available/defined in web.c to try and get the DHCP table without the the 1second wait as you stated.

Unfortunately non of the functions return the data I want (IP + MAC from DHCP table)
Hopefully I didn't miss one!!
client_functions.js is actually using these code snippets itself, so I really don't need it acting as the middle man.

<% get_clientlist_from_json_database(); %>           (returns /jffs/nmp_cl_json.js as JSON --> has offline list of every device ever connected (Name + MAC + Misc data) and online status
WORSE VERSION: <% get_basic_clientlist(); %>           (Above with less misc data)
WORSE VERSION: <% get_all_basic_clientlist(); %>       (Above with Mac + Name Only)
WORSE VERSION: <% get_basic_clientlist_count(); %>     (Above data parsed for counts of Wired + Wireless devices connected)


<% get_leases_array(); >                            (returns /var/lib/misc/dnsmasq.leases IP + Mac + Device Name + Remainder of DHCP Table)
WORSE VERSION: <% dhcpLeaseInfo(); %>                 (Mac + Device Name)
WORSE VERSION: <% dhcpLeaseMacList(); %>              (Mac + Device Name)
WORSE VERSION: <% IP_dhcpLeaseInfo(); %>              (IP + Device Name


<% get_allclientlist(); %>                        (returns /tmp/clientlist.json -- > currently connected clients (IP + MAC) with disappearing client bug)
WORSE VERSION: <% get_wclientlist(); %>             (wireless clients of the above data)
WORSE VERSION: <% get_wiredclientlist(); %>         (wired clients of the above data)

<% dhcp_leases(); %>                             (does nothing - .... o_O)

EDIT: Just for my own sanity checked web.c again for any function related with opening "/var/lib/misc/dnsmasq.leases" and found 4.

<% dhcpLeaseInfo(); %>        --> ej_dhcpLeaseInfo
<% dhcpLeaseMacList(); %>     --> ej_dhcpLeaseMacList
<% IP_dhcpLeaseInfo();    %>  --> ej_IP_dhcpLeaseInfo
<% get_leases_array(); >%     --> ej_get_leases_array
<% nvram_dump(); %>           --> ej_dump
no direct web access          --> ej_lan_leases  ( ran by nvram_dump)
mp direct web access          --> ej_wl_status_qtn_array ( ran by ej_wl_status_5g_array)
no direct web access          --> ej_wl_status_array     ( ran by ej_wl_status_2g_array)

---

I need get_leases_array(); which translates --> ej_get_leases_array(...) but without the

    killall("dnsmasq", SIGUSR2);
    sleep(1);

if I cant get the ajax call working as expected. Looking it up now and have high hopes!!

EDIT:

ajax blackhat magic trickery has given fruits to my labor.
This is after googling for wayyyy too long in the wrong direction with the the wrong keywords. Unfortunately I then also proceeded to spend way to much time making the most simple of regexs.

Initially with your suggestion, I was trying to execute the ASP code block itself via an ajax call instead of loading it from another page

If you google information about ASP anything, it brings about insanity. If you then couple ajax + asp in the keywords and it turns to madness

 

[FreshJR]

Check out the first post.

The wait is over. Hope its bug free (Testing was rushed)

 

[RMerlin]

Unfortunately non of the functions return the data I want (IP + MAC from DHCP table)

How are you handling clients with static IPs that won't possess any DHCP lease?

That's why I suggested tying into the networkmap data instead of relying on dnsmasq's incomplete list. It's the only client list that will be complete (well, it will be once Asus sort out its random behaviour), since it relies on ARP rather than on DHCP leases.

If you google information about ASP anything, it brings about insanity. If you then couple ajax + asp in the keywords and it turns to madness

The VPNStatus page is probably a good sample if you want a working example specific to Asuswrt. The tricky part to remember is that those activescript tags are processed server-side, not client-side. So, once the HTML is sent to the browser, all of these tags have been replaced by the data generated by httpd - you cannot change that afterward. when it has reached your browser Took me a while to fully understand it myself. Having an Ajax call requesting an external page allows the httpd to fill that page with the latest data on every queries.

A web-centric way of looking at it: treat it as if it was PHP. The browser never receives any PHP code, it only gets the output generated server-side by the PHP code.

 

[skeal]

Whoop whoop, it's Christmas time!! Thanks @FreshJR I have a new toy for the weekend.

 

[Correlor]

Check out the first post.

The wait is over. Hope its bug free (Testing was rushed)

@FreshJR,

Nice work,

but /jffs/scripts/FreshJR_QOS -menu option 2 does not go to update but to custom rules.

Thank you for your script.

 

[heysoundude]

updated....watching the charts change as people on the LAN make calls and surf...super cool! thank you...but you forgot to include an updated date on the first post...please, after you take a nap.

EDIT: YOWZA, This is a YUUUUUGE improvement in snappiness. thank you!!!

 

[skeal]

@FreshJR One quick question, how can you set more than 4 custom rules? See image below.

I'm probably not doing something right, sorry for the dumb question.