pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[kvic]

rc3 rebuild, 9.5MB

We have two interesting profiling here, @Protik and @jrmwvu04. About the same amount of slh, the equilibrium point of memory use is quite different, 5MB vs 9.5MB.

The underlying cause to my guess is that protik is non Apple/Safari user while jrmwvu04 is a primary Apple/Safari user. (am I right?)

Safari doesn't support latest session resumption technology (for an unknown design decision) while Chrome/Firefox do. This gives extra memory pressure on pixelserv-tls.

While that's part of the story, the other half is that (I suspect) OpenSSL is not efficient in memory use/cleanup in its implementation on the older session resumption.

I encourage Mac users to try a day or two solely on Chrome/Firefox and check (and report) pixelserv-tls memory use.

 

[kvic]

rc3 feels slower than rc2. No real change in browsing habits, but my tav is in the 40-60 range; before it had been in the 10-30 range.

Ad domains on a given set of websites could vary across days or weeks. Hence, mindful readers will recall I said it's quite a challenge to bypass DNS-based adblock for a given website.

From @elorimer's observation, I couldn't be sure it's due to such not unexpected variation or indeed due to one of the variables adjusted in rc.3.

So in next release candidate, rc.4, I'm going to revert one bit back to rc.2 that I thought could contribute to longer tav in a *busy* pixelserv-tls. We can decide later if it does make a difference.

rc.4 shall be out in a few days. Before that, please harness rc.3 as regularly as you can with your typical daily usage. And jot down peak memory use for comparison later with rc.4.

 

[Protik]

We have two interesting profiling here, @Protik and @jrmwvu04. About the same amount of slh, the equilibrium point of memory use is quite different, 5MB vs 9.5MB.

The underlying cause to my guess is that protik is non Apple/Safari user while jrmwvu04 is a primary Apple/Safari user. (am I right?)

Safari doesn't support latest session resumption technology (for an unknown design decision) while Chrome/Firefox do. This gives extra memory pressure on pixelserv-tls.

While that's part of the story, the other half is that (I suspect) OpenSSL is not efficient in memory use/cleanup in its implementation on the older session resumption.

I encourage Mac users to try a day or two solely on Chrome/Firefox and check (and report) pixelserv-tls memory use.

You are quite right. Of more or less 10 active devices, 2 of them are iOS devices. Among these two, only one stays active at a time.

 

[kvic]

You are quite right. Of more or less 10 active devices, 2 of them are iOS devices. Among these two, only one stays active at a time.

Great news!

I shall re-iterate that 10-13MB is about norm on a 250MB RAM system. I get that myself as I'm a Mac user. And OS will take back the memory when in need. So nothing to worry about here.

I'm still delighted to see a light footprint as little as 5MB. Just to prove a point that pixelserv-tls' design decisions haven't made a wrong move (yet). touch wood.

 

[elorimer]

From @elorimer's observation, I couldn't be sure it's due to such not unexpected variation or indeed due to one of the variables adjusted in rc.3.

So in next release candidate, rc.4, I'm going to revert one bit back to rc.2 that I thought could contribute to longer tav in a *busy* pixelserv-tls. We can decide later if it does make a difference.

I've shifted locations for a short while from the 87U at home base to a 56U now on rc.3, and these tav's are down in the teens. So I think my earlier observation can be discounted.

 

[kvic]

I've shifted locations for a short while from the 87U at home base to a 56U now on rc.3, and these tav's are down in the teens. So I think my earlier observation can be discounted.

Okay, thanks for the follow-up. Then rc.4 will stick to rc.3 with respect to optimization.

I've drafted a new sub-section TLS HANDSHAKE to be added under SERVSTATS COUNTER section in the man page. Recommended for everyone since it's useful for hunting down clients connecting to rogue servers.

 

[M@rco]

I particularly like the average processing time per request

 

[jrmwvu04]

We have two interesting profiling here, @Protik and @jrmwvu04. About the same amount of slh, the equilibrium point of memory use is quite different, 5MB vs 9.5MB.

The underlying cause to my guess is that protik is non Apple/Safari user while jrmwvu04 is a primary Apple/Safari user. (am I right?)

Safari doesn't support latest session resumption technology (for an unknown design decision) while Chrome/Firefox do. This gives extra memory pressure on pixelserv-tls.

While that's part of the story, the other half is that (I suspect) OpenSSL is not efficient in memory use/cleanup in its implementation on the older session resumption.

I encourage Mac users to try a day or two solely on Chrome/Firefox and check (and report) pixelserv-tls memory use.

One question I would like to pose, as it relates to the wiki/previous versions..

Next insert PRECMD="ulimit -s 64 in the middle. pixelserv-tls will run with tighter memory resource with this change.

Are other folks here doing this, and what if any effect does this have on the recent Km versions? Academically of course I could test it on my own and probably will, but I’m curious about others.

 

[M@rco]

One question I would like to pose, as it relates to the wiki/previous versions..

Are other folks here doing this, and what if any effect does this have on the recent Km versions? Academically of course I could test it on my own and probably will, but I’m curious about others.

I just checked the other day and noticed it was already in

/opt/etc/init.d/S80pixelserv-tls

I assume this is written during pixelserv setup from AB-Solution by @thelonelycoder and I've installed the current rc by the oneliner provided by @kvic earlier this thread. As it's still in the wiki, I assume it's still more memory efficient so I have left it as is.

 

[jrmwvu04]

I’ve rebooted and re-enabled the stack size limit. Will compare to saved prior results after a couple days.

I ran with that setting up until somewhere early in the Km test builds. My reasoning was that since it was going to start caching all these certificates, I wasn’t going to limit it. That, and it could use many times the memory it does and wouldn’t be a problem. Just something I have been wondering on.

I started looking up info about it back then and restarted my search yesterday. I feel like I’m in way over my head on all this stack size stuff. @kvic do you have any reasoning for 64, for specifying it at all? Or have any decent sources to point me to? You seem to have taken a liking to writing man pages lately.

 

[penguin22]

I'm not sure if this has been posted elsewhere on the forums, but I didn't see it myself and felt it worthwhile enough to ask here and see if anyone else is willing to test and confirm.

I was going through my AC86U configs for various filters (OpenVPN client w/ PIA, DNSCrypt, AB-Solution, Skynet) as Groupon isn't working while on WiFi (chalking it up to Groupon blocking from PIA). I upgraded to 384.4 shortly after released and did a factory reset. After going through all settings, I also enabled AIProtection and took note that the only item in Router Security Assessment is UPNP, which I leave enabled intentionally.

Today, when going through the router config, I saw AIProtection showing 2, which was unexpected. Digging into it, Web access from WAN had become enabled. This is something I have never done and the setting is far removed from enabling by accident. This led me to think of any actions I performed recently; over the past week, I wiped my USB drive to move from the arm7 to arm8 (x64) install of the mentioned items above. In the process I also followed the steps to install the PixelServ CA cert for use when logging into the router. All worked well and I'm able to get to the router login page without the security warning, but I'm wondering if this could have possibly triggered WAN access to enabled? Anyone willing to give it a try on a router without the certificate installed to see if the setting changes?

 

[jrmwvu04]

Today, when going through the router config, I saw AIProtection showing 2, which was unexpected. Digging into it, Web access from WAN had become enabled. This is something I have never done and the setting is far removed from enabling by accident.

Have you by any chance used the mobile app at all? I’ve seen a number of mentions recently that using the app silently enables the access from WAN setting. That’s my only idea.

Edit: the Asus router config app - I could have been a little clearer

 

[Butterfly Bones]

There is an entire thread about getting hacked. The common point seems that using the ASUS mobile app automatically enables WAN access without warning. I had the same happen when I installed and tested the mobile app. It is uninstalled now and WAN access off. If you use the ASUS mobile app on a phone or tablet that is likely what happened.

I've imported the crt into three mobile devices and three computers with no issues with WAN access being enabled.

 

[penguin22]

Have you by any chance used the mobile app at all? I’ve seen a number of mentions recently that using the app silently enables the access from WAN setting. That’s my only idea.

Edit: the Asus router config app - I could have been a little clearer

I just deleted the access in the mobile app and found that after re-adding, low and behold, the access from WAN is re-enabled. Quickest App removal in history.

I could swear that an earlier version of the mobile app did not have this nasty side-effect. Thanks for the quick response and for verifying that it is not the CA cert to router that makes the change.

Edit: I also found out that the mobile app not only enabled remote WAN, but also DDNS as I had that disabled as well and it was re-enabled in testing.

 

[Butterfly Bones]

It has been about 6 - 8 months since I had the mobile app installed before this experience, and can confirm that it did not enable WAN access in the previous version I used.

 

[penguin22]

It has been about 6 - 8 months since I had the mobile app installed before this experience, and can confirm that it did not enable WAN access in the previous version I used.

Yes, to my perspective, this is a significant potential security risk that ASUS does not advertise adequately in the application notes. In a sick humor way, they do specify that, "AP mode and Remote Connection (DDNS) is supported now!!!" with the "!!!" seriously part of their update. There is no indication that this will occur or requested consent via the app. I'll be sending a nasty-gram tomorrow after calming down and getting some rest.

 

[penguin22]

There is an entire thread about getting hacked. The common point seems that using the ASUS mobile app automatically enables WAN access without warning. I had the same happen when I installed and tested the mobile app. It is uninstalled now and WAN access off. If you use the ASUS mobile app on a phone or tablet that is likely what happened.

I've imported the crt into three mobile devices and three computers with no issues with WAN access being enabled.

In my case, I hadn't been compromised, just had the WAN access enabled unexpectedly. Since I have been using the mobile app for ages without having it enabled, it didn't trigger the IT rule of what changed in my mind. In fairness, an app update is indeed a change and turns out to be the likely culprit. With more recent changes of the PixelServ CA cert getting applied for router access, that's what made me come here first.

Great sleuthing and I think this might be worth a sticky post, which I suggested in the other thread, due to the security implications.

 

[kvic]

Next insert PRECMD="ulimit -s 64 in the middle. pixelserv-tls will run with tighter memory resource with this change.

As it's still in the wiki, I assume it's still more memory efficient so I have left it as is.

I looked up the wiki, and indeed it was there. Github ranks the page as second mostly read in the past two weeks btw...

It was written back in 2016. While the content is still valid and applicable to the latest beta except this item.

In v2.0, I had made changes to the code so that memory management was way more efficient than before. It could also run with as little as 32KB stack size. "ulimit -s 64" is no longer required nor it will hurt performance in anyway. In fact, since v2.0 pixelserv-tls had advanced to a point no "ulimit -s" is required from users.

I feel like I’m in way over my head on all this stack size stuff. @kvic do you have any reasoning for 64, for specifying it at all?

In Linux, default stack size is 8MB which might not be a good value for small systems like a router, NAS etc.

But note that this 8MB value is what an application will ask for, it's not fully mapped by OS into RAM. So you'll end up perhaps with 128KB or 256KB or another actual value initially depending on your system setup (which users have little say).

By limiting to 64KB (as proven in pixelserv-tls version KK and before), we give a hand to OS (and glibc) to do a simpler and more efficient task on allocating RAM for stack use.

 

[kvic]

Great sleuthing and I think this might be worth a sticky post, which I suggested in the other thread, due to the security implications.

Latest ASUS mobile app openning up WAN access without user consent (that potentially had reported cases being hacked due to this) is definitely worth a sticky. But politically I won't think it's going to happen for the obvious reason.

Just like I could quickly tell people how to remove the nagging "banner" but I won't do so for being politically or perhaps ethnically correct.

 

[kvic]

Update on my two instances after near 3 days uptime.

n1 at 5%. n2 at 4.8%. Both 250MB RAM systems. I believe I've seen the peak RAM usage in my setup.

As I mentioned before, I'm a primary Mac/Safari user, if I switch to use Chrome as default browser, I think my pixelserv-tls memory footprint will be way smaller (as demonstrated in @Protik's case above).