pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[kvic]

I figured that was probably the case.

As I said, it's just a habit formed many years ago to reduce the surface area for problems. There's a reason that 'turn it off and back on' is in the first lessons of tech support 101.

uts    1d 07:45    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    30    maximum number of service threads
kvg    2.69    average number of requests per service thread
krq    105    max number of requests by one service thread

req    6702    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1706 bytes    average size of requests
rmx    21740 bytes    largest size of request(s)
tav    9 ms    average processing time (per request)
tmx    10003 ms    longest processing time (per request)

slh    2584    # of accepted HTTPS requests
slm    301    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    235    # of dropped HTTPS requests (client disconnect without sending any request)
slu    140    # of dropped HTTPS requests (unknown error)
sct    97    ssl cache: # of cached cert
sch    1482    ssl cache: # of cache hit
scm    97    ssl cache: # of cache miss
scp    0    ssl cache: # of purge to free up slots

End users are always right. I don't do end users for work and I'll leave the authority opinion to others Instead, let's check your latest servstats numbers. Always impressive!

 

[kvic]

Here is my servstats on two nodes after 36 hrs. Node2 was restarted a few times for diagnosis.

Ratio of SSL requests (slh/req) is ~57% on Node2 and ~54% on Node1 and yet Node2 retains lower tav. For light workload higher clock fewer cores wins. Node2 is overclocked RT-AC56U.

In next test version, I've changed HTTP_KEEPALIVE (option -O) default to 300s. The original 120s was intended for a balance between performance and the largest population of concurrent clients. 300s shall provide better experience for SOHO users. Even longer is okay for smoother experience at the expense of using more system resources. Beyond 600s (i.e. 10mins) is very likely to provide no additional gain.

 

[Protik]

Interesting experiment!

115KB per cert is certainly an over-estimate. Half of that amount perhaps is fair. During KL cycle, I found pixelserv-tls used about 50KB per SSL connection. That's about the RAM usage when a session is to be cached in RAM for speedy resume later.

To have a better ballpark estimate, take the number (sct) you set for SSL cache size. Then estimate average number (say, avg_conn) of connections per cert.

• RAM usage = sct x avg_conn x 50KB

This will still be an overestimate but closer. Two sources of overestimate: 1) 50KB per session 2) avg_conn is hard to guess in version Km

Note that another factor avg_conn is hard to guess is that SSL sessions' valid lifetime is set for 2hrs in version Km. When a session expired after 2hr, a browser will not be able to do a fast resume. I made the call that 2hr is fair assumption because

1. people's average browsing sessions perhaps last that long (or shorter) and then they shutdown browsers completely.
2. extra, say 20ms, every two hour is fair (rather than caching sessions for longer using more RAM that aren't going to be re-used).

Apparently people could see there are a couple of variables that SSL caching could be fined tuned on an ongoing basis in future updates of pixelserv-tls. Just to provide more food for thought for interested users. Hopefully hadn't confused others.

That's better!

Can I safely assume that, when the cache is full, the memory usage should remain more or less the same?

If that's the case, there may be memory leak in the current test.5. I set my cache to 150 which is full now. But the memory usage I am seeing from htop has increased from 7.2% to 7.4% (of 250MB total RAM) even though there is not much of a load on the network. I will keep an eye on it to see if it keeps increasing.

 

[kvic]

That's better!

Can I safely assume that, when the cache is full, the memory usage should remain more or less the same?

Should remain stable in most cases.

If that's the case, there may be memory leak in the current test.5. I set my cache to 150 which is full now. But the memory usage I am seeing from htop has increased from 7.2% to 7.4% (of 250MB total RAM) even though there is not much of a load on the network. I will keep an eye on it to see if it keeps increasing.

0.2% is like 500KB which should be considered normal UNLESS it keeps growing. This could be a show stopper...pls help to monitor.

 

[Protik]

Should remain stable in most cases.



0.2% is like 500KB which should be considered normal UNLESS it keeps growing. This could be a show stopper...pls help to monitor.

It's hard to conclude based on two data points. I will keep an eye and report back.

 

[Raphie]

pixelserv-tls 2.1.0-test.5 (compiled: Mar 10 2018 22:10:47) options: 192.168.1.2
65550 uts, 1 log, 4 kcc, 22 kmx, 1.69 kvg, 1516 krq, 4886 req, 1535 avg, 30056 rmx, 22 tav, 4377 tmx, 3571 slh, 15 slm, 0 sle, 849 slc, 60 slu, 94 sct, 2303 sch, 28 scm, 0 scp, 2307 nfe, 5 gif, 1 ico, 268 txt, 0 jpg, 7 png, 0 swf, 1 sta, 1 stt, 38 ufe, 0 opt, 332 pst, 100 hed, 19 rdr, 0 nou, 0 pth, 0 204, 8 bad, 28 tmo, 821 cls, 547 cly, 0 clt, 0 err

 

[kvic]

.

Looks nice with tav 22 ms. Just curious, what's your average time from load cert benchmark?

Btw, what's your %MEM reading for the 'pixelserv-tls' process on top. Simply type 'top' in SSH to lookup. Once in type 'q' to quit.

 

[Protik]

Should remain stable in most cases.



0.2% is like 500KB which should be considered normal UNLESS it keeps growing. This could be a show stopper...pls help to monitor.

I think it is confirmed. Now it is 7.6%. I have 4 data points so far,

 4.19pm, 11 March - 7.2%
11:00pm, 11 March - 7.4%
11:56pm, 11 March - 7.5%
12.34am, 12 March - 7.6%

 

[Raphie]

1102 1 nobody S 37960 7.3 1 0.0 pixelserv-tls 192.168.1.2
1520 1 nobody S 31664 6.1 1 0.0 dnsmasq --log-async

 

[kvic]

I think it is confirmed. Now it is 7.6%. I have 4 data points so far,

 4.19pm, 11 March - 7.2%
11:00pm, 11 March - 7.4%
11:56pm, 11 March - 7.5%
12.34am, 12 March - 7.6%

Between 11pm and 12.34am, I expect there should be one flush. However, the effect of flush could be offset by new browsing activities during the same period.

The log will register flush events at log LEVEL 3. Can be turned on without restarting pixelserv-tls with issuing in SSH;

curl http://listenip/log=3

I would suggest leaving it running with log=3 for the next 8hrs or so if you may. Then check what's %MEM..

 

[Raphie]

Looks nice with tav 22 ms. Just curious, what's your average time from load cert benchmark?

Btw, what's your %MEM reading for the 'pixelserv-tls' process on top. Simply type 'top' in SSH to lookup. Once in type 'q' to quit.

how do i get the benchmark stats, do i need to add the -B flag in the ps menu in ab-solution
or can i stop and restart pixelserv manually from the command line?

 

[kvic]

1102 1 nobody S 37960 7.3 1 0.0 pixelserv-tls 192.168.1.2
1520 1 nobody S 31664 6.1 1 0.0 dnsmasq --log-async

I made the wrong suggestion...because I just recall the 'top' in ASUSWRT is very much handicapped to a point being useless. The percentage you see is %VSZ which is certainly not a good indicator useful for pixelserv-tls.

I'm afraid you've to install 'htop' from Entware (opkg install htop) and then lookup the MEM% column.

how do i get the benchmark stats, do i need to add the -B flag in the ps menu in ab-solution
or can i stop and restart pixelserv manually from the command line?

Let pixelserv-tls the server continue to run. Need not interrupt it.
Simple issue in SSH

$ pixelserv-tls -B _.googletagservices.com

 

[Raphie]

that's 2.6

 

[SMS786]

tav is one of the counters that are exponential moving averages ie recent values carrying higher weight. Hence, might be more affected by your recent browsing activities.

The exact curve used in pixelserv-tls is shown below. At each point on x axis, the value on y axis is the weighting. x = 0 is the latest measurement, x = 1 is previous measurement so and so forth.

View attachment 12247

Very interesting. Just curious as to what's the reason for the recent queries carrying higher weight in the tav calculation?

 

[Raphie]

CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.googletagservices.com
1. load cert from disk: 9.020 ms
2. load cert from disk: 8.834 ms
3. load cert from disk: 8.817 ms
4. load cert from disk: 8.850 ms
5. load cert from disk: 8.809 ms
6. load cert from disk: 8.104 ms
7. load cert from disk: 8.521 ms
8. load cert from disk: 9.044 ms
9. load cert from disk: 8.885 ms
10. load cert from disk: 8.734 ms
average: 8.842 ms

That's with a CORSAIR Slider USB3 64gb stick on the USB3 port (very fast stick)

 

[kvic]

Very interesting. Just curious as to what's the reason for the recent queries carrying higher weight in the tav calculation?

Server like pixelserv-tls is expected to run for months if not years. Hence, a simple moving average across the whole time horizon is pretty much useless. We're interested in more 'recent measurements' where EMA comes to help.

To look at the problem in a simpler way, consider that the most recent 500 measurements of request processing shall preferably weigh more than the previous 500 measurements. That's more interesting average value for people taking an observation of tav now.

 

[kvic]

Update on my servstats. Finally I made it to 100 sch organically..

For both nodes RAM usage is 10-11MB. Looks good for me. However, knowing the reported usage from @Protik and @Raphie, I think Km-test.5 is still too aggressive on RAM. Will turn down a bit in next test version.

 

[kvic]

CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.googletagservices.com
1. load cert from disk: 9.020 ms
2. load cert from disk: 8.834 ms
3. load cert from disk: 8.817 ms
4. load cert from disk: 8.850 ms
5. load cert from disk: 8.809 ms
6. load cert from disk: 8.104 ms
7. load cert from disk: 8.521 ms
8. load cert from disk: 9.044 ms
9. load cert from disk: 8.885 ms
10. load cert from disk: 8.734 ms
average: 8.842 ms

That's with a CORSAIR Slider USB3 64gb stick on the USB3 port (very fast stick)

<9ms is fast indeed!

 

[MasterBash]

having a weird issue with chtome on my phone and pixelserv.

for some reasons when i load a page i often get part of the page displayed and 30 secs later it finishes displaying. it happens on snbforums.whenever i disable it everything is fine again. tested multiple times with the same result.

its kind of weird because it does not happen on my other devices (pc/laptop with chrome). so im not sure what could be causing that

 

[Butterfly Bones]

<9ms is fast indeed!

AC86u, SanDisk Ultra Fit 3.0 in ASUS 3.0 USB port

CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.googletagservices.com
 1. load cert from disk: 4.829 ms
 2. load cert from disk: 4.793 ms
 3. load cert from disk: 4.892 ms
 4. load cert from disk: 4.776 ms
 5. load cert from disk: 4.819 ms
 6. load cert from disk: 4.780 ms
 7. load cert from disk: 4.812 ms
 8. load cert from disk: 4.795 ms
 9. load cert from disk: 4.767 ms
10. load cert from disk: 4.774 ms
average: 4.821 ms

tav 8 ms average processing time (per request)
tmx 167 ms longest processing time (per request)