pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[XIII]

Beta 6 on SanDisk Ultra-Fit 32 GB USB 3.0 drive in RT-AC86U:

CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
 1. generate cert to disk: 32.492 ms    load from disk: 4.781 ms
 2. generate cert to disk: 80.824 ms    load from disk: 4.768 ms
 3. generate cert to disk: 51.650 ms    load from disk: 4.761 ms
 4. generate cert to disk: 74.175 ms    load from disk: 4.773 ms
 5. generate cert to disk: 43.837 ms    load from disk: 4.759 ms
 6. generate cert to disk: 44.809 ms    load from disk: 4.764 ms
 7. generate cert to disk: 41.737 ms    load from disk: 4.763 ms
 8. generate cert to disk: 54.682 ms    load from disk: 4.760 ms
 9. generate cert to disk: 54.286 ms    load from disk: 4.780 ms
10. generate cert to disk: 53.130 ms    load from disk: 4.767 ms
generate to disk average: 53.162 ms
  load from disk average: 4.768 ms

 

[Makaveli]

Beta 6 on SanDisk Ultra-Fit 32 GB USB 3.0 drive in RT-AC86U:

CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
 1. generate cert to disk: 32.492 ms    load from disk: 4.781 ms
 2. generate cert to disk: 80.824 ms    load from disk: 4.768 ms
 3. generate cert to disk: 51.650 ms    load from disk: 4.761 ms
 4. generate cert to disk: 74.175 ms    load from disk: 4.773 ms
 5. generate cert to disk: 43.837 ms    load from disk: 4.759 ms
 6. generate cert to disk: 44.809 ms    load from disk: 4.764 ms
 7. generate cert to disk: 41.737 ms    load from disk: 4.763 ms
 8. generate cert to disk: 54.682 ms    load from disk: 4.760 ms
 9. generate cert to disk: 54.286 ms    load from disk: 4.780 ms
10. generate cert to disk: 53.130 ms    load from disk: 4.767 ms
generate to disk average: 53.162 ms
  load from disk average: 4.768 ms

based on the results i'm seeing it looks like cpu speed is just as important as the drive speed.

@kvic

What processes should one be looking for when trying to view how much memory its using with the Top command in ssh?

 

[kvic]

I'm not sure if it is related but there are 366 files under "\var\cache\pixelserv" and sst is now 444

Tonight I need to keep my laptop turned on but my other devices will be turned off I'll let you know tomorrow morning. here it's 15:30 so I have lots of time

Thanks.

Km-test.6 runs well on your router. Just be aware of that "slc" is very high. That indicates some devices on your LAN (or VPN server) have intensive browsing activities but without the CA cert (generated your AB-Solution) imported.

I've to correct myself. What I meant is "slu" not "slc". In @pattiri's servstats, "slc" is huge when compared to "req". That raises a very interesting question. It means there are some clients that passed TLS handshake but disconnects without sending a request. The high slc means many such connections.

Hmm..what could be the possible explanation?

 

[kvic]

based on the results i'm seeing it looks like cpu speed is just as important as the drive speed.

@kvic

What processes should one be looking for when trying to view how much memory its using with the Top command in ssh?

Absolutely. CPU power is way more important for crypto stuff.

You've to install htop from Entware. In SSH,

opkg install htop

 

[Protik]

Same for me. slc is almost twice compared to slh. Most of my devices has certificate installed. Usually my slc is less than slh.

pixelserv-tls 2.1.0-test.6 (compiled: Mar 13 2018 03:09:35) options: 192.168.2.2 -c 150

uts    0d 14:33    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    34    maximum number of service threads
kvg    1.13    average number of requests per service thread
krq    16    max number of requests by one service thread
req    2873    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1352 bytes    average size of requests
rmx    61368 bytes    largest size of request(s)
tav    57 ms    average processing time (per request)
tmx    2082 ms    longest processing time (per request)
slh    517    # of accepted HTTPS requests
slm    2    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    1131    # of dropped HTTPS requests (client disconnect without sending any request)
slu    455    # of dropped HTTPS requests (unknown error)
sct    121    cert cache: # of certs in cache
sch    2097    cert cache: # of reuses of cached certs
scm    21    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    825    sess cache: # of TLS sessions in cache
ssh    122    sess cache: # of reuses of cached TLS sessions
ssm    45    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

 

[esco]

PS beta 6, RT-AC86U, Sandisk Cruzer Blade USB 2.0

CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
 1. generate cert to disk: 70.464 ms    load from disk: 4.798 ms
 2. generate cert to disk: 44.058 ms    load from disk: 4.814 ms
 3. generate cert to disk: 39.931 ms    load from disk: 4.778 ms
 4. generate cert to disk: 43.126 ms    load from disk: 4.897 ms
 5. generate cert to disk: 67.056 ms    load from disk: 4.802 ms
 6. generate cert to disk: 31.249 ms    load from disk: 4.787 ms
 7. generate cert to disk: 33.800 ms    load from disk: 4.766 ms
 8. generate cert to disk: 39.006 ms    load from disk: 4.770 ms
 9. generate cert to disk: 76.681 ms    load from disk: 4.834 ms
10. generate cert to disk: 40.053 ms    load from disk: 4.784 ms
generate to disk average: 48.542 ms
  load from disk average: 4.803 ms

 

[kvic]

Same for me. slc is almost twice compared to slh. Most of my devices has certificate installed. Usually my slc is less than slh.

pixelserv-tls 2.1.0-test.6 (compiled: Mar 13 2018 03:09:35) options: 192.168.2.2 -c 150

uts    0d 14:33    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    34    maximum number of service threads
kvg    1.13    average number of requests per service thread
krq    16    max number of requests by one service thread
req    2873    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1352 bytes    average size of requests
rmx    61368 bytes    largest size of request(s)
tav    57 ms    average processing time (per request)
tmx    2082 ms    longest processing time (per request)
slh    517    # of accepted HTTPS requests
slm    2    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    1131    # of dropped HTTPS requests (client disconnect without sending any request)
slu    455    # of dropped HTTPS requests (unknown error)
sct    121    cert cache: # of certs in cache
sch    2097    cert cache: # of reuses of cached certs
scm    21    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    825    sess cache: # of TLS sessions in cache
ssh    122    sess cache: # of reuses of cached TLS sessions
ssm    45    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

Thanks, I need to think about the slc.. I notice sst is 825. I wonder if it's really using 40MB RAM? (825 * 50 / 1024 + 3)

Pls also help to check sst after a prolonged quiet period on your LAN. I expect it to drop a bit in the morning.

 

[Protik]

Thanks, I need to think about the slc.. I notice sst is 825. I wonder if it's really using 40MB RAM? (825 * 50 / 1024 + 3)

Pls also help to check sst after a prolonged quiet period on your LAN. I expect it to drop a bit in the morning.

For the RAM, the highest I saw was 20MB out of 250MB.

I will report you back sst tomorrow morning.

 

[Makaveli]

20 hours in on Test 6

There was less browsing over that time than usual so numbers will be alittle lower.

pixelserv-tls 2.1.0-test.6 (compiled: Mar 13 2018 03:09:35) options: 192.168.1.3

uts    0d 20:11    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    13    maximum number of service threads
kvg    1.09    average number of requests per service thread
krq    10    max number of requests by one service thread
req    1194    total # of requests (HTTP, HTTPS, success, failure etc)
avg    408 bytes    average size of requests
rmx    786 bytes    largest size of request(s)
tav    7 ms    average processing time (per request)
tmx    80 ms    longest processing time (per request)
slh    112    # of accepted HTTPS requests
slm    6    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    728    # of dropped HTTPS requests (client disconnect without sending any request)
slu    192    # of dropped HTTPS requests (unknown error)
sct    42    cert cache: # of certs in cache
sch    910    cert cache: # of reuses of cached certs
scm    5    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    0    sess cache: # of TLS sessions in cache
ssh    50    sess cache: # of reuses of cached TLS sessions
ssm    2    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

My usb thumb drive is a no name one I picked up at work so the numbers aren't bad considering it 2GB usb 2.0 and just ancient.

CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.googletagservices.com
1. load cert from disk: 10.895 ms
2. load cert from disk: 9.742 ms
3. load cert from disk: 10.028 ms
4. load cert from disk: 9.829 ms
5. load cert from disk: 9.935 ms
6. load cert from disk: 9.882 ms
7. load cert from disk: 9.811 ms
8. load cert from disk: 9.944 ms
9. load cert from disk: 9.998 ms
10. load cert from disk: 9.972 ms
average: 10.053 ms

Memory usage about 2.1% for both processes.

 

[Protik]

After 1 day,

pixelserv-tls 2.1.0-test.6 (compiled: Mar 13 2018 03:09:35) options: 192.168.2.2 -c 150

uts    0d 23:52    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    13    number of active service threads
kmx    34    maximum number of service threads
kvg    1.65    average number of requests per service thread
krq    45    max number of requests by one service thread
req    3997    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1052 bytes    average size of requests
rmx    61368 bytes    largest size of request(s)
tav    26 ms    average processing time (per request)
tmx    2082 ms    longest processing time (per request)
slh    1136    # of accepted HTTPS requests
slm    3    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    1216    # of dropped HTTPS requests (client disconnect without sending any request)
slu    469    # of dropped HTTPS requests (unknown error)
sct    130    cert cache: # of certs in cache
sch    2780    cert cache: # of reuses of cached certs
scm    30    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    944    sess cache: # of TLS sessions in cache
ssh    175    sess cache: # of reuses of cached TLS sessions
ssm    61    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

The slc did not increase much after last time and slc is catching up. So it is possible the sudden increase of slc happened due to a rogue website trying to load a blocked domain again and again. RAM usage is 2.4% so far.

 

[kvic]

Thank you @Protik and @Makaveli

In the next test version, I've put in improvements to gracefully recover from some of the situations that contribute to both slc and slu. In other words, such situations will end up as good requests and increase slh. Practically it means a smoother browsing experience.

I didn't see sst number dropping overnight. That worries me a bit. However, it might simply be my misunderstanding since RAM usage doesn't grow hand in hand with sst. Let's continue to have an eye on sst or anything else interesting people could spot.

 

[pattiri]

@kvic I have bad news but don't panic not that bad

Just after I woke up, there was a power loss at my home so I couldn't check servstats but I woke up at night about 4 hours ago checked on my phone, sst was 687 so it didn't dropped. for now it's up for 14 minutes and here the servstats.

http://prntscr.com/iqxyrp

 

[kvic]

Thank you @pattiri. Hope everything is fine on your end

Sad to hear sst doesn't drop but the good news is I may have finally nailed it. In next test version, there will also be improvement in SSL cache management!

 

[pattiri]

Thank you @pattiri. Hope everything is fine on your end

Sad to hear sst doesn't drop but the good news is I may have finally nailed it. In next test version, there will also be improvement in SSL cache management!

Yes all fine. Thank you. After 4 hours sst is 41 on my end. I think it's not getting higher today like yesterday.

 

[Raphie]

I just upgraded my USB but can't seem to overwrite my new cert. with the old one?

 

[kvic]

I just upgraded my USB but can't seem to overwrite my new cert. with the old one?

In SSH, issue

chown -R nobody /opt/var/cache/pixelserv

 

[Raphie]

doesn't work, still can't delete or overwrite the ca.crt in /entware/var/cache/pixelserv

Just to be clear:
I reinstalled USB with AMTM
I have previous ca.crt and ca.key on my Windows desktop
I'm logging in with FTP to /Sandisk/entware/var/cache/pixelserv
I cannot overwrite or delete the ca.crt or ca.key in that directory
file chmod properties are 666

 

[kvic]

New beta version Km-test.7 aka v2.1.0-test.7

Thanks again for all the GREAT feedback at an ever FASTER pace. All good except my little spare time!

Notably we now have URI "/ca.crt" to download and import your Pixelserv CA certificate on client devices. Major improvement in SSL cache management as well as handling SSL connections for troubled clients.

As usual, you can read all the changes and details on kazoo.ga/pixelserv-tls.

Entware (ARMv7, mipsel, ARMv8) users can use the one liner below as usual or otherwise to install.

sh -c "$(wget -qO - https://kazoo.ga/pixelserv-tls/install-beta.sh)"

Will appreciate any feedback.

 

[jrmwvu04]

That's it for test 6 then.

uts    0d 21:42    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    11    maximum number of service threads
kvg    1.98    average number of requests per service thread
krq    72    max number of requests by one service thread
req    914    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1028 bytes    average size of requests
rmx    21507 bytes    largest size of request(s)
tav    11 ms    average processing time (per request)
tmx    627 ms    longest processing time (per request)
slh    353    # of accepted HTTPS requests
slm    136    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    8    # of dropped HTTPS requests (client disconnect without sending any request)
slu    66    # of dropped HTTPS requests (unknown error)
sct    45    cert cache: # of certs in cache
sch    343    cert cache: # of reuses of cached certs
scm    45    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    116    sess cache: # of TLS sessions in cache
ssh    36    sess cache: # of reuses of cached TLS sessions
ssm    19    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

 

[kvic]

That's it for test 6 then.

uts    0d 21:42    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    11    maximum number of service threads
kvg    1.98    average number of requests per service thread
krq    72    max number of requests by one service thread
req    914    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1028 bytes    average size of requests
rmx    21507 bytes    largest size of request(s)
tav    11 ms    average processing time (per request)
tmx    627 ms    longest processing time (per request)
slh    353    # of accepted HTTPS requests
slm    136    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    8    # of dropped HTTPS requests (client disconnect without sending any request)
slu    66    # of dropped HTTPS requests (unknown error)
sct    45    cert cache: # of certs in cache
sch    343    cert cache: # of reuses of cached certs
scm    45    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    116    sess cache: # of TLS sessions in cache
ssh    36    sess cache: # of reuses of cached TLS sessions
ssm    19    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

Thanks, we're near the closing of Km cycle.

One reason Km-test.7 is accelerated for testing because of a newly discovered way of managing the SSL cache. Much more effective and sst should be vibrant and responsive to expired sessions.