pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[RMerlin]

Just as an FYI....with the debian patch applied on my fork, I don't see any difference in memory use (running one VPN Server, one VPN Client, NOT running ABSolution/pixelserv)

Can you see what impact it has on openssl speed?

 

[kvic]

Can you see what impact it has on openssl speed?

Zero from my tests.

btw unrelated this change. It used to be FW openssl speed faster than Entware. Now it's the other way round. But my FW version is 380.66. So probably not fair comparison.

 

[RMerlin]

Zero from my tests.

btw unrelated this change. It used to be FW openssl speed faster than Entware. Now it's the other way round. But my FW version is 380.66. So probably not fair comparison.

It's possible that Entware eventually picked most of the optimizations I've made over the years (switching to -O3 for instance - the upgrade from 1.0.0 to 1.0.2 also leveraged much more optimized ASM code for AES/SHA). Might be worth I had another look at their build recipe for it, in case there might be further optimizations I could pick up.

I wonder too if they wouldn't be gaining some more performance by using glibc instead of the antique uclibc used by pre-HND Asuswrt.

 

[DonnyJohnny]

I can see my reply to you. This thread is getting busier.. easy to lose track.. even for myself but after v2.1 release it shall go quieter I believe.

A few subsequent discussions also apply to your situation.

I'm interested in hearing back from people trying two instances of pixelserv-tls (either for load balancing or two IP approach for filtering known "unknown cert" servers).

I just changed my dns server from cloudflare to Google and I see that it affect the uce. I have no issue with uca but during my usage of cloudflare doh, the uce number will go up occasionally. When Changed to Google doh, it remain 0. There are still small number of slu but i think is small issue.
Site I used for testing
https://pi-hole.net/pages-to-test-ad-blocking-performance/



Question, those choice of dns affect this?

Using rc4 and rebuilted OpenSSL lib

 

[kvic]

A way to test tav

A stripped down version of test in #1618. Set to "-O 5" and restart pixelserv-tls.

Browse website_1, wait for 6 seconds, browse webiste_2, wait for 6s and repeat.

website 1 & 2 could be foxnews and cnn.com. This shall provide a worst scenario tav (for a potential issue I could think of).

I'm also going to do some comparison between rc.3 and rc.4 (both using the new lib) later today.

tested websites

foxnews.com (first four sections - click on each as one website in above description)
cnn.com (first four sections - ditto)
dailymail.co.uk (first four sections - ditto)

Other procedures and settings are described as above.

Results

         tav (old lib)  tav (new lib)
rc.3     8ms           7ms
rc.4     8ms           7ms

Surprise (!!) new lib is faster (by 12.5%) for pixelserv-tls in the above described workload pattern.

I repeated a few times for each case. The results are consistently reproducible.

(Note that...the improved cert cache lookup in rc.4 doesn't have chance to perform in this test 'cos we don't fill up the cache)

edit:

single instance of pixelserv-tls running on 1.2GHz RT-AC56.

 

[kvic]

I just changed my dns server from cloudflare to Google and I see that it affect the uce. I have no issue with uca but during my usage of cloudflare doh, the uce number will go up occasionally. When Changed to Google doh, it remain 0. There are still small number of slu but i think is small issue.
Site I used for testing
https://pi-hole.net/pages-to-test-ad-blocking-performance/



Question, those choice of dns affect this?

Using rc4 and rebuilted OpenSSL lib

If the problem as described proven true, it'll be interesting to understand why so. I can't explain nor I believe choice of dns will have any impact.

Is it repeatable? have you tried multiple times? Multiple reasons could lead to this 'coincidence' (if it's a coincidence).

One of them could be that each time you hit a test website..the advert/tracker servers are slightly different. I cannot *emphasise* this one more time (as I did twice recently in this thread and once in the AB-S thread)

For comparable data, you shall change log LEVEL to 4. Keep track of all advert/tracker servers you visited (well..roughly) after you hit all test pages. Perhaps focus on one page as test that gives you most slu to make your life easier.

 

[Asad Ali]

Seems to me you're over thinking here.

When you manually override browser warning (and if it actually allow you to do it), then from TLS handshake perspective, no difference to having your Pixelserv CA installed.

iOS 8...I don't have a device to test. Seems to me more like an OS issue though

I was just trying to understand how slh is counting and looks like you're right it's OS specific so no further doubts needs to be cleared. Thanks.

One other thing in the new rc4 did you change the debug trace log level to "debug" ?
If yes then will the entries "handshake failed...." will keep on showing in level 2?

Can you just move the missing certificate log entry a level up ( i.e in log level 1 ) because honestly that's the only log entry I'm interested in at all times and everything else I can enable on demand when I suspect something is wrong. Only if it's not too much work

 

[kvic]

tested websites

foxnews.com (first four sections - click on each as one website in above description)
cnn.com (first four sections - ditto)
dailymail.co.uk (first four sections - ditto)

Other procedures and settings are described as above.

Results

         tav (old lib)  tav (new lib)
rc.3     8ms           7ms
rc.4     8ms           7ms

Surprise (!!) new lib is faster (by 12.5%) for pixelserv-tls in the above described workload pattern.

I repeated a few times for each case. The results are consistently reproducible.

(Note that...the improved cert cache lookup in rc.4 doesn't have chance to perform in this test 'cos we don't fill up the cache)

edit:

single instance of pixelserv-tls running on 1.2GHz RT-AC56.

@ryzhov_al @zyxmon In case, you missed the above comparison of new vs old libopenssl..

New lib (with the flag on) is actually *faster* by 12.5% for SSL workload on armv7/kernel 2.6.x.

The above result is consistently reproducible.

 

[sfx2000]

It's possible that Entware eventually picked most of the optimizations I've made over the years (switching to -O3 for instance - the upgrade from 1.0.0 to 1.0.2 also leveraged much more optimized ASM code for AES/SHA). Might be worth I had another look at their build recipe for it, in case there might be further optimizations I could pick up.

I wonder too if they wouldn't be gaining some more performance by using glibc instead of the antique uclibc used by pre-HND Asuswrt.

Yeah - always worry about -03, as it can go off the deep end across different sub-archs - gcc can over optimize and get one into a bad place where things are inconsistent. ARM and x86/AMD64 included...

Not saying that -03 is bad, just needs comprehensive testing before things go out as production level code.

glibc and musl vs. uclib - rather not go down that rat-hole, other than to say that glibc and musl have been kept more current - going there, well, OpenWRT has made the big-bet on musl on their tip - glibc is going to be easier to port things over with less work. On a past project, we started down the musl path, and ended up reverting to glibc - musl performed better, it was more the porting effort once outside of the initial SoC we supported, hence glibc and a rebuild.

 

[kvic]

@sfx2000 I thought that was your virgin debut in this thread or perhaps lost many pages (and years ago)..I can't remember..



I've been hesitating to highlight the following result... in order not to promote for ASUS/Broadcom..

:-

# pixelserv-tls -B
CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
 1. generate cert to disk: 51.049 ms    load from disk: 4.848 ms
 2. generate cert to disk: 39.019 ms    load from disk: 5.251 ms
 3. generate cert to disk: 50.609 ms    load from disk: 5.494 ms
 4. generate cert to disk: 65.390 ms    load from disk: 5.211 ms
 5. generate cert to disk: 66.189 ms    load from disk: 4.798 ms
 6. generate cert to disk: 46.024 ms    load from disk: 4.802 ms
 7. generate cert to disk: 77.728 ms    load from disk: 4.785 ms
 8. generate cert to disk: 46.584 ms    load from disk: 4.833 ms
 9. generate cert to disk: 70.946 ms    load from disk: 4.798 ms
10. generate cert to disk: 59.312 ms    load from disk: 4.796 ms
generate to disk average: 57.285 ms
  load from disk average: 4.962 ms

But the new processor in RT-86 is much faster in doing crypto as people can see in the above pixelserv-tls benchmark.

~60ms is about one tenth of my usual 600ms on 1.2GHz RT-AC56

But note that certificate generations are rare events. Hence, don't go out and buy a new router just for pixelserv-tls.

 

[kvic]

More comparisons..

New lib vs old lib and/or rc.3 vs rc.4 on 1.2GHz RT-AC56

Spoiler: new lib + rc.4

$ /jffs/bin/pixelserv-tls-2.1.0-rc.4 -B
CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
1. generate cert to disk: 514.124 ms load from disk: 10.248 ms
2. generate cert to disk: 483.394 ms load from disk: 10.052 ms
3. generate cert to disk: 539.303 ms load from disk: 10.039 ms
4. generate cert to disk: 269.952 ms load from disk: 10.068 ms
5. generate cert to disk: 603.598 ms load from disk: 10.397 ms
6. generate cert to disk: 457.263 ms load from disk: 10.210 ms
7. generate cert to disk: 1048.259 ms load from disk: 10.118 ms
8. generate cert to disk: 646.139 ms load from disk: 10.057 ms
9. generate cert to disk: 653.628 ms load from disk: 10.083 ms
10. generate cert to disk: 733.011 ms load from disk: 10.098 ms
generate to disk average: 594.867 ms
load from disk average: 10.137 ms

Spoiler: new lib + rc.3

$ /jffs/bin/pixelserv-tls-2.1.0-rc.3 -B
CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
1. generate cert to disk: 502.859 ms load from disk: 10.069 ms
2. generate cert to disk: 569.967 ms load from disk: 10.000 ms
3. generate cert to disk: 587.073 ms load from disk: 10.076 ms
4. generate cert to disk: 489.608 ms load from disk: 9.999 ms
5. generate cert to disk: 447.631 ms load from disk: 10.191 ms
6. generate cert to disk: 535.646 ms load from disk: 10.067 ms
7. generate cert to disk: 876.914 ms load from disk: 10.056 ms
8. generate cert to disk: 323.225 ms load from disk: 10.146 ms
9. generate cert to disk: 450.457 ms load from disk: 9.993 ms
10. generate cert to disk: 814.364 ms load from disk: 9.973 ms
generate to disk average: 559.774 ms
load from disk average: 10.057 ms

Spoiler: old lib + rc.4

$ /jffs/bin/pixelserv-tls-2.1.0-rc.4 -B
CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
1. generate cert to disk: 517.869 ms load from disk: 10.108 ms
2. generate cert to disk: 595.706 ms load from disk: 10.086 ms
3. generate cert to disk: 561.119 ms load from disk: 10.046 ms
4. generate cert to disk: 485.617 ms load from disk: 10.033 ms
5. generate cert to disk: 591.404 ms load from disk: 10.075 ms
6. generate cert to disk: 405.380 ms load from disk: 9.968 ms
7. generate cert to disk: 767.195 ms load from disk: 10.051 ms
8. generate cert to disk: 460.263 ms load from disk: 10.033 ms
9. generate cert to disk: 835.340 ms load from disk: 10.013 ms
10. generate cert to disk: 789.632 ms load from disk: 10.058 ms
generate to disk average: 600.952 ms
load from disk average: 10.047 ms

Spoiler: old lib + rc.3

$ /jffs/bin/pixelserv-tls-2.1.0-rc.3 -B
CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
1. generate cert to disk: 757.421 ms load from disk: 10.061 ms
2. generate cert to disk: 453.094 ms load from disk: 10.196 ms
3. generate cert to disk: 520.816 ms load from disk: 10.066 ms
4. generate cert to disk: 756.035 ms load from disk: 10.050 ms
5. generate cert to disk: 738.381 ms load from disk: 10.103 ms
6. generate cert to disk: 584.394 ms load from disk: 10.118 ms
7. generate cert to disk: 296.894 ms load from disk: 10.079 ms
8. generate cert to disk: 630.724 ms load from disk: 10.088 ms
9. generate cert to disk: 691.679 ms load from disk: 10.000 ms
10. generate cert to disk: 562.374 ms load from disk: 10.015 ms
generate to disk average: 599.181 ms
load from disk average: 10.078 ms

TLDR;

Certificate generations are one of the most computationally intensive task (crypto workload) in pixelserv-tls.

No measurable difference between old lib and new lib (as well as between rc.3 and rc.4)

 

[kvic]

While assisting one EdgeRouter X user on pixelserv-tls, I found my blog's comment section become cluttered very rapidly..

Then I saw Discus allows users to create "discussion board" (!!)...so here we go: pixelserv-tls discussion board on Discus

In case, people want to try, feel free to drop a line.

Frankly I still prefer Discourse more..but I never attempted to find a host for that. Discus is just out there that I wasn't aware of

 

[Butterfly Bones]

Very happy with this!
htop shows 1.9% mem used (9.7Mb of 512Mb on AC86U)!!!
tav. slu / req, uce, kcc all good with rc.4 and new lib
And your instructions to remove "handshake error: unknown cert" entries in syslog and these stats.

pixelserv-tls 2.1.0-rc.4 (compiled: Apr 5 2018 21:02:10 flags: tfo) options: 192.168.1.2 -l 2

uts    0d 08:58    process uptime
log    2    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    92    number of active service threads
kmx    105    maximum number of service threads
kvg    20.56    average number of requests per service thread
krq    88938    max number of requests by one service thread
req    110110    total # of requests (HTTP, HTTPS, success, failure etc)
avg    546 bytes    average size of requests
rmx    35869 bytes    largest size of request(s)
tav    3 ms    average processing time (per request)
tmx    5357 ms    longest processing time (per request)
slh    107294    # of accepted HTTPS requests
slm    55    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    1104    # of dropped HTTPS requests (client disconnect without sending any request)
slu    9    # of dropped HTTPS requests (other TLS handshake errors)
uca    0    slu break-down: # of unknown CA reported by clients
uce    8    slu break-down: # of unknown cert reported by clients
sct    50    cert cache: # of certs in cache
sch    1560    cert cache: # of reuses of cached certs
scm    278    cert cache: # of misses to find a cert in cache
scp    265    cert cache: # of purges to give room for a new cert
sst    3    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    605    sess cache: # of reuses of cached TLS sessions
ssm    1169    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session
nfe    905    # of GET requests for server-side scripting
gif    29    # of GET requests for GIF
ico    25    # of GET requests for ICO
txt    104785    # of GET requests for Javascripts
jpg    80    # of GET requests for JPG
png    21    # of GET requests for PNG
swf    0    # of GET requests for SWF
sta    26    # of GET requests for HTML stats
stt    0    # of GET requests for plain text stats
ufe    45    # of GET requests /w unknown file extension
opt    0    # of OPTIONS requests
pst    275    # of POST requests
hed    0    # of HEAD requests (HTTP 501 response)
rdr    1562    # of GET requests resulted in REDIRECT response
nou    0    # of GET requests /w empty URL
pth    0    # of GET requests /w malformed URL
204    0    # of GET requests (HTTP 204 response)
bad    31    # of unknown HTTP requests (HTTP 501 response)
tmo    773    # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls    1490    # of dropped requests (client disconnect without sending any request)
cly    0    # of dropped requests (client disconnect before response sent)
clt    0    # of dropped requests (reached maximum service threads)
err    0    # of dropped requests (unknown reason)

 

[kvic]

Very happy with this!
htop shows 1.9% mem used (9.7Mb of 512Mb on AC86U)!!!
tav. slu / req, uce, kcc all good with rc.4 and new lib
And your instructions to remove "handshake error: unknown cert" entries in syslog and these stats.

That's awesome! Here is my two instances after 12 hrs.

I was back to two instances in load balance mode. Also after I took care of the peculiar "graph.instagram.com", my slu "issue" has basically gone and remains clean.

So far exceptions are rare for me, hence, I simply redirect "graph.instagram.com" to 0.0.0.0.

If people with more exceptions and want take full benefits of pixelserv-tls at the moment, I strongly recommend you to try the "two instances in primary/second mode" discussed a few pages back in a response to @penguin22. Look forward to hearing the first field report.

happy pixelserv'ing

 

[Protik]

After 1day 14hours, using 1% of 250MB.

pixelserv-tls 2.1.0-rc.4 (compiled: Apr 5 2018 21:02:08) options: 192.168.2.2 -c 150 -l 2

uts    1d 14:05    process uptime
log    2    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    26    maximum number of service threads
kvg    1.34    average number of requests per service thread
krq    45    max number of requests by one service thread
req    2852    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1326 bytes    average size of requests
rmx    62993 bytes    largest size of request(s)
tav    38 ms    average processing time (per request)
tmx    2933 ms    longest processing time (per request)
slh    881    # of accepted HTTPS requests
slm    1    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    635    # of dropped HTTPS requests (client disconnect without sending any request)
slu    1274    # of dropped HTTPS requests (other TLS handshake errors)
uca    133    slu break-down: # of unknown CA reported by clients
uce    114    slu break-down: # of unknown cert reported by clients

 

[kvic]

After 1day 14hours, using 1% of 250MB.

Your signature is outdate..lol

Awesome memory usage consistently from your servstats btw.
I realise it also heavily depends on people's browsing habit.

 

[RMerlin]

Not saying that -03 is bad, just needs comprehensive testing before things go out as production level code.

I've been using -O3 for close to four years now on my OpenSSL build, I'd say that's more than "comprehensively tested"...

 

[kvic]

pixelserv-tls discussion board on Discus

After poking around for a little while, I found it's quite intuitive to use. Layout is clean too (I only tested with adblock on).

I suggest EdgeRouter X & Linux users could raise questions there for support on pixelserv-tls. Single sign-on Login choices: Disqus, Facebook, Google and Twitter.

While this thread will remain primarily focussed on ASUSWRT-merlin & development announcement, people may treat the above board as a contingency backup.

 

[Sebastien Bougie]

Hello,

I have a question, When i turn pixelserv-tls on in AB-Solution, I'm getting error on my Amazon Android Apps the Oops message, someone know how to fix this?


Thanks

 

[visortgw]

Hello,

I have a question, When i turn pixelserv-tls on in AB-Solution, I'm getting error on my Amazon Android Apps the Oops message, someone know how to fix this?

Thanks

Check the AB-Solution thread. A week or two back, @thelonelycoder added several additional whitelist entries to resolve SNB Forums ad issues. These entries also resolve Amazon Android app access.