pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[Raphie]

Htop 2.9

pixelserv-tls 2.1.0-rc.3 (compiled: Mar 30 2018 17:10:59) options: 192.168.1.2
246714 uts, 1 log, 5 kcc, 42 kmx, 1.19 kvg, 328 krq, 10802 req, 3912 avg, 797849 rmx, 21 tav, 8284 tmx, 9592 slh, 38 slm, 0 sle, 370 slc, 81 slu, 100 sct, 6361 sch, 53 scm, 28 scp, 6 sst, 2896 ssh, 69 ssm, 0 ssp, 2465 nfe, 6 gif, 1 ico, 1329 txt, 1 jpg, 6 png, 0 swf, 8 sta, 1 stt, 131 ufe, 18 opt, 2494 pst, 303 hed, 759 rdr, 0 nou, 1 pth, 0 204, 299 bad, 56 tmo, 378 cls, 2428 cly, 0 clt, 0 err

 

[M@rco]

In v2.0, I had made changes to the code so that memory management was way more efficient than before. It could also run with as little as 32KB stack size. "ulimit -s 64" is no longer required nor it will hurt performance in anyway. In fact, since v2.0 pixelserv-tls had advanced to a point no "ulimit -s" is required from users.

Thanks for clarifying. Might be useful to @thelonelycoder as well for AB4.

 

[SMS786]

Latest stats for rc3 running on a 68U. Memory usage at 20mb.

 

[esco]

I particularly like the average processing time per request

@M@rco, do you have a PS certificate installed on your browser/device? Your SLH equals zero, or am I missing the point here?

 

[M@rco]

@M@rco, do you have a PS certificate installed on your browser/device? Your SLH equals zero, or am I missing the point here?

I have only imported ca.crt in Firefox in Linux Mint, which is a fresh install. Haven't had time yet/didn't bother to install it on any other devices yet...

 

[DonnyJohnny]

I have only imported ca.crt in Firefox in Linux Mint, which is a fresh install. Haven't had time yet/didn't bother to install it on any other devices yet...

I think the new version has simplified the process of importing.
Just open the browser and http://pixelserv’s ip/ca.crt

Do it at your own time

 

[quant88]

After 24hrs, AC88U MEM% = 6.3

87376 uts, 2 log, 40 kcc, 73 kmx, 1.26 kvg, 119 krq, 432403 req, 1532 avg, 62034 rmx, 21 tav, 4407 tmx, 14572 slh, 16 slm, 0 sle, 2967 slc, 413283 slu, 242 sct, 427006 sch, 17 s cm, 0 scp, 3 sst, 11681 ssh, 1305 ssm, 0 ssp, 1757 nfe, 9 gif, 7 ico, 226 txt, 7 jpg, 0 png, 0 swf, 11 sta, 0 stt, 626 ufe, 0 opt, 12965 pst, 0 hed, 309 rdr, 0 nou, 0 pth, 0 204, 79 bad, 127 tmo, 2980 cls, 0 cly, 0 clt, 1 err

 

[thelonelycoder]

In v2.0, I had made changes to the code so that memory management was way more efficient than before. It could also run with as little as 32KB stack size. "ulimit -s 64" is no longer required nor it will hurt performance in anyway. In fact, since v2.0 pixelserv-tls had advanced to a point no "ulimit -s" is required from users.

I dropped PRECMD="ulimit -s 64" in AB4.
For AB3.x I'll wait until 2.1.0 is available through opkg as then it will be generally available for all users still running pre 2.x (aka Kk or Kl) versions.

 

[kvic]

I dropped PRECMD="ulimit -s 64" in AB4.
For AB3.x I'll wait until 2.1.0 is available through opkg as then it will be generally available for all users still running pre 2.x (aka Kk or Kl) versions.

For AB3.x, you could simply leave it as-is wrt this PRECMD because its existence has no impact on version v2.X

 

[kvic]

For people with memory usage much higher than 10MB, could you pls try with option "-c 99", and see if it makes a huge difference?

If not, also worth trying option "-c 50" to see if it makes an impact.

From my latest tests, "-c 99" vs "-c 100" makes a remarkable different on Entware (i only have armv7 to test).

On x86, regardless of option -c X, RAM usage is very stable.

Weird phenomenon indeed. Wait to hear back from some of you.

 

[Spydawg]

This look normal? Installed beta yesterday..

pixelserv-tls 2.1.0-rc.3 (compiled: Mar 30 2018 17:10:59) options: 192.168.1.2

uts    1d 02:53    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    39    maximum number of service threads
kvg    1.00    average number of requests per service thread
krq    5    max number of requests by one service thread
req    9665    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1184 bytes    average size of requests
rmx    57353 bytes    largest size of request(s)
tav    29 ms    average processing time (per request)
tmx    1020 ms    longest processing time (per request)
slh    0    # of accepted HTTPS requests
slm    1715    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    4496    # of dropped HTTPS requests (client disconnect without sending any request)
slu    3328    # of dropped HTTPS requests (other TLS handshake errors)
sct    44    cert cache: # of certs in cache
sch    5717    cert cache: # of reuses of cached certs
scm    0    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    14    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    485    sess cache: # of reuses of cached TLS sessions
ssm    1783    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session
nfe    41    # of GET requests for server-side scripting
gif    0    # of GET requests for GIF
ico    2    # of GET requests for ICO
txt    21    # of GET requests for Javascripts
jpg    0    # of GET requests for JPG
png    0    # of GET requests for PNG
swf    0    # of GET requests for SWF
sta    7    # of GET requests for HTML stats
stt    0    # of GET requests for plain text stats
ufe    0    # of GET requests /w unknown file extension
opt    0    # of OPTIONS requests
pst    22    # of POST requests
hed    0    # of HEAD requests (HTTP 501 response)
rdr    3    # of GET requests resulted in REDIRECT response
nou    0    # of GET requests /w empty URL
pth    0    # of GET requests /w malformed URL
204    0    # of GET requests (HTTP 204 response)
bad    2    # of unknown HTTP requests (HTTP 501 response)
tmo    14    # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls    4511    # of dropped requests (client disconnect without sending any request)
cly    0    # of dropped requests (client disconnect before response sent)
clt    0    # of dropped requests (reached maximum service threads)
err    0    # of dropped requests (unknown reason)

 

[Spydawg]

Oh oh getting loads of this error ..


Apr 2 18:15:42 pixelserv-tls[981]: generate_cert: failed to open file for write: /opt/var/cache/pixelserv/_.segment.io

 

[Butterfly Bones]

Oh oh getting loads of this error ..


Apr 2 18:15:42 pixelserv-tls[981]: generate_cert: failed to open file for write: /opt/var/cache/pixelserv/_.segment.io

The code in this post should fix it.
https://www.snbforums.com/threads/p...bserver-for-adblock.26114/page-70#post-390236

 

[Spydawg]

The code in this post should fix it.
https://www.snbforums.com/threads/p...bserver-for-adblock.26114/page-70#post-390236

Thanks

 

[kvic]

Some interesting tidbits

(all numbers on 1.2GHz Cortex-A9 i.e. RT-AC56U)

I rewrote a small part of cert cache lookup. Now a cache hit takes ~60us (microseconds). Previously was ~100us. All with "-c 100". The rewrote shall perform much better when cache size is large.

These are substantially less than a cache miss that takes 10,000us or ~10ms to load from disk.

A full TLS handshake takes ~40ms. A session resumption takes 150us or 0.15ms.

 

[kvic]

On the latest and local build, Entware saga continues to play out. Now the magic number is "-c 98" !

With "-c 98" (or smaller cache size) memory usage very stable on RT-AC56. 99 or beyond could use substantially more.

On Intel X86, any number is stable.

I believe if we could get to the bottom of this we may resolve the "issue" or else we might have a known "issue" on man page. lol

This thing had driven me crazy..

Tests

To conduct the tests, I wrote a script as clients, issue access to 100 different domains. The first 100 access will populate the cache with 100 different certs. The next accesses divide into two cases (in separate round of tests): 1) the next access will always trigger a cache miss and a purge. 2) the next access will always cache hit. Run it long enough to reach stable state wrt memory use. Occasionally bend in some manually browsing. cnn.com is really good for this test.

To manually simulate the above tests, the closest possible is to browse as may websites as you can: first, set option "-O 5" or a larger timeout that your patience could tolerate. Browse some website. Wait for 5 seconds or longer. Browse more websites, and repeat. After you repeat tens of times, it's about running over the cache a few times.

Or can check sct and sch on servstats page (using HTTP not to interference with HTTPS tests) to monitor your progress in the above steps.

 

[SMS786]

This look normal? Installed beta yesterday..

pixelserv-tls 2.1.0-rc.3 (compiled: Mar 30 2018 17:10:59) options: 192.168.1.2

uts    1d 02:53    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    39    maximum number of service threads
kvg    1.00    average number of requests per service thread
krq    5    max number of requests by one service thread
req    9665    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1184 bytes    average size of requests
rmx    57353 bytes    largest size of request(s)
tav    29 ms    average processing time (per request)
tmx    1020 ms    longest processing time (per request)
slh    0    # of accepted HTTPS requests
slm    1715    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    4496    # of dropped HTTPS requests (client disconnect without sending any request)
slu    3328    # of dropped HTTPS requests (other TLS handshake errors)
sct    44    cert cache: # of certs in cache
sch    5717    cert cache: # of reuses of cached certs
scm    0    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    14    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    485    sess cache: # of reuses of cached TLS sessions
ssm    1783    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session
nfe    41    # of GET requests for server-side scripting
gif    0    # of GET requests for GIF
ico    2    # of GET requests for ICO
txt    21    # of GET requests for Javascripts
jpg    0    # of GET requests for JPG
png    0    # of GET requests for PNG
swf    0    # of GET requests for SWF
sta    7    # of GET requests for HTML stats
stt    0    # of GET requests for plain text stats
ufe    0    # of GET requests /w unknown file extension
opt    0    # of OPTIONS requests
pst    22    # of POST requests
hed    0    # of HEAD requests (HTTP 501 response)
rdr    3    # of GET requests resulted in REDIRECT response
nou    0    # of GET requests /w empty URL
pth    0    # of GET requests /w malformed URL
204    0    # of GET requests (HTTP 204 response)
bad    2    # of unknown HTTP requests (HTTP 501 response)
tmo    14    # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls    4511    # of dropped requests (client disconnect without sending any request)
cly    0    # of dropped requests (client disconnect before response sent)
clt    0    # of dropped requests (reached maximum service threads)
err    0    # of dropped requests (unknown reason)

Be sure to install the Pixelserv certs on all the clients attached to your wifi network. On those clients, visiting "http://YourPixelservIP/ca.crt" should do the trick.

 

[kvic]

I believe if we could get to the bottom of this we may resolve the "issue" or else we might have a known "issue" on man page. lol

This thing had driven me crazy..

Can't wait to pop up and break a piece of news. I might have spotted the culprit in Entware! Need time to do experiments later today. This is perhaps one of my biggest tech eureka moment in recent weeks!

So to let everyone know I might be wrong but if some silent helpers are working hard in assisting on this issue. Please take a coffee break for now..

 

[kvic]

Need time to do experiments later today.

Remarkable difference. See top:

17496 is latest build of pixelserv-tls running on a rebuild of Entware's openssl library. This instance has been abused to the harshest abuse possible. Peak out ~6.5MB. Idle long enough fall back to ~3.5MB. Very sweet.

2412 is rc.3 running on stock Entware library. Organically grew to its current state. Let's not mention the *peak* number under the same abuse...horrible.

Here is the plan: rc.4 will be out in a day or two. Going to default cert cache to 50.

At the same time, hopefully people will get new library to test. For these people, you can manually adjust option "-c X" to higher values.

 

[visortgw]

Remarkable difference. See top:

17496 is latest build of pixelserv-tls running on a rebuild of Entware's openssl library. This instance has been abused to the harshest abuse possible. Peak out ~6.5MB. Idle long enough fall back to ~3.5MB. Very sweet.

2412 is rc.3 running on stock Entware library. Organically grew to its current state. Let's not mention the *peak* number under the same abuse...horrible.

Here is the plan: rc.4 will be out in a day or two. Going to default cert cache to 50.

At the same time, hopefully people will get new library to test. For these people, you can manually adjust option "-c X" to higher values.

How does one obtain/install the rebuilt Entware openssl library? Can I assume that the new library is compatible with other openssl functions on the device?