pixelserv pixelserv - A Better One-pixel Webserver for Adblock

DonnyJohnny · Mar 20, 2018

kvic said:
New beta version 2.1.0-rc.2

Thanks again for all the GREAT test effort. Without many servstats from you, we can't spot the issue in slu. We may settle it finally in this RC!

Manage to squeeze in another new feature as well since the change is minimal. Now pixelserv-tls could run without a CA cert. Pls read kazoo.ga/pixelserv-tls for what it means and how to make use of it.

Entware (ARMv7, mipsel, ARMv8) users can use the one liner below as usual or otherwise to install.

Code:

sh -c "$(wget -qO - https://kazoo.ga/pixelserv-tls/install-beta.sh)"

Will appreciate any feedback.

Omg! Where all the slu gone to??.!!?

kvic · Mar 20, 2018

quant88 said:
Thank you for the new RC
To run without CA cert, do I have to set any flag to pixelserv-tls or its a default behaviour?

To run without a CA cert, no extra CLI options required. pixelserv-tls won't complain when it cannot find ca.crt/ca.key in /opt/var/cache/pixelserv. When there are requests coming in, it'll lookup certs in the same directory.

One example that I use for conducting SSL Labs test.

Say you get a Let's Encrypt certificate for your domain "mypixel.serv". Simply place both the certificate together with its private key (issued to you by Let's Encrypt) into a file named "mypixel.serv" in the same directory.

Then when you access through "https://mypixel.serv." Everything will just work. pixelserv-tls will respond with empty content or servstats, depending on the URI.

Note that without your Pixelserv CA cert, no new certs for blocked domain will be automatically generated. That'll defeat its original purpose for adblock.

kvic · Mar 20, 2018

DonnyJohnny said:
Omg! Where all the slu gone to??.!!?

Flushed and come back as slh perhaps..

Butterfly Bones · Mar 20, 2018

Before I install the rc2. My slu/req = 34% and slc/req = 60.9%. slc seems way to high. I have certs imported into all 5 clients and no ad blocking other than AB-Solution in the router, no clients with any ad blocking apps or browser extensions.

Code:

uts    1d 10:50    process uptime
log    2    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    2    number of active service threads
kmx    25    maximum number of service threads
kvg    1.03    average number of requests per service thread
krq    6    max number of requests by one service thread
req    8335    total # of requests (HTTP, HTTPS, success, failure etc)
avg    714 bytes    average size of requests
rmx    30929 bytes    largest size of request(s)
tav    11 ms    average processing time (per request)
tmx    2955 ms    longest processing time (per request)
slh    225    # of accepted HTTPS requests
slm    13    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    5072    # of dropped HTTPS requests (client disconnect without sending any request)
slu    2836    # of dropped HTTPS requests (other TLS handshake errors)
sct    100    cert cache: # of certs in cache
sch    7900    cert cache: # of reuses of cached certs
scm    129    cert cache: # of misses to find a cert in cache
scp    104    cert cache: # of purges to give room for a new cert
sst    0    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    89    sess cache: # of reuses of cached TLS sessions
ssm    4905    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session
nfe    86    # of GET requests for server-side scripting
gif    2    # of GET requests for GIF
ico    46    # of GET requests for ICO
txt    60    # of GET requests for Javascripts
jpg    0    # of GET requests for JPG
png    0    # of GET requests for PNG
swf    0    # of GET requests for SWF
sta    50    # of GET requests for HTML stats
stt    1    # of GET requests for plain text stats
ufe    10    # of GET requests /w unknown file extension
opt    2    # of OPTIONS requests
pst    21    # of POST requests
hed    0    # of HEAD requests (HTTP 501 response)
rdr    10    # of GET requests resulted in REDIRECT response
nou    0    # of GET requests /w empty URL
pth    0    # of GET requests /w malformed URL
204    0    # of GET requests (HTTP 204 response)
bad    0    # of unknown HTTP requests (HTTP 501 response)
tmo    98    # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls    5101    # of dropped requests (client disconnect without sending any request)
cly    0    # of dropped requests (client disconnect before response sent)
clt    0    # of dropped requests (reached maximum service threads)
err    0    # of dropped requests (unknown reason)

kvic · Mar 20, 2018

Butterfly Bones said:

Before I install the rc2. My slu/req = 34% and slc/req = 60.9%. slc seems way to high. I have certs imported into all 5 clients and no ad blocking other than AB-Solution in the router, no clients with any ad blocking apps or browser extensions.

Code:

uts    1d 10:50    process uptime
log    2    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    2    number of active service threads
kmx    25    maximum number of service threads
kvg    1.03    average number of requests per service thread
krq    6    max number of requests by one service thread
req    8335    total # of requests (HTTP, HTTPS, success, failure etc)
avg    714 bytes    average size of requests
rmx    30929 bytes    largest size of request(s)
tav    11 ms    average processing time (per request)
tmx    2955 ms    longest processing time (per request)
slh    225    # of accepted HTTPS requests
slm    13    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    5072    # of dropped HTTPS requests (client disconnect without sending any request)
slu    2836    # of dropped HTTPS requests (other TLS handshake errors)
sct    100    cert cache: # of certs in cache
sch    7900    cert cache: # of reuses of cached certs
scm    129    cert cache: # of misses to find a cert in cache
scp    104    cert cache: # of purges to give room for a new cert
sst    0    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    89    sess cache: # of reuses of cached TLS sessions
ssm    4905    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session
nfe    86    # of GET requests for server-side scripting
gif    2    # of GET requests for GIF
ico    46    # of GET requests for ICO
txt    60    # of GET requests for Javascripts
jpg    0    # of GET requests for JPG
png    0    # of GET requests for PNG
swf    0    # of GET requests for SWF
sta    50    # of GET requests for HTML stats
stt    1    # of GET requests for plain text stats
ufe    10    # of GET requests /w unknown file extension
opt    2    # of OPTIONS requests
pst    21    # of POST requests
hed    0    # of HEAD requests (HTTP 501 response)
rdr    10    # of GET requests resulted in REDIRECT response
nou    0    # of GET requests /w empty URL
pth    0    # of GET requests /w malformed URL
204    0    # of GET requests (HTTP 204 response)
bad    0    # of unknown HTTP requests (HTTP 501 response)
tmo    98    # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls    5101    # of dropped requests (client disconnect without sending any request)
cly    0    # of dropped requests (client disconnect before response sent)
clt    0    # of dropped requests (reached maximum service threads)
err    0    # of dropped requests (unknown reason)

I've to break a piece of sad news for Android users. From my carefully crafted tests, intrinsically android systems will have higher slu in pixelserv-tls because I suspect some system components and/or 3rd party services choose not to honour user root CA certificates.

Your numbers seems a bit high indeed. Hope rc.2 will improve the situation. Or else you've to get prepared to run tcpdump/wireshark to see what's going on. Not trying to scare you..also likely some worms on your devices

kvic · Mar 20, 2018

kvic said:
I've to break a piece of sad news for Android users. From my carefully crafted tests, intrinsically android systems will have higher slu in pixelserv-tls because I suspect some system components and/or 3rd party services choose not to honour user root CA certificates.

I feel like I may sound a bit negative. Allow me to add one positive note. The good news is that whatever system components/3rd party services trying to secretly access over HTTPS get blocked by your adblocker, and get exposed to certain extent by pixelserv-tls. So nothing secretly gets uploaded!

Butterfly Bones · Mar 20, 2018

kvic said:
I've to break a piece of sad news for Android users. From my carefully crafted tests, intrinsically android systems will have higher slu in pixelserv-tls because I suspect some system components and/or 3rd party services choose not to honour user root CA certificates.

Your numbers seems a bit high indeed. Hope rc.2 will improve the situation. Or else you've to get prepared to run tcpdump/wireshark to see what's going on. Not trying to scare you..also likely some worms on your devices

Just to note, my main device is android 8.1 Oreo, and I do not see that listed in the SSL Labs listings. Here are the SSL tests from that device in Chrome 65.

The percent adoption rate as of March is very low since it is new and has not been out long. Many manufacturers have not updated. My devices are all Nexus / Pixel running vanilla android and I know and trust the apps I use.

I'm more than a normal android user, and more than a regular user on XDA Forums. No worms or bad junk in my phones, guaranteed. I know my way around Android Debug Bridge and have other tools available that I use so I'll dig deeper into wireshark - I've used tcpdump on android.

I'm surprised that root CA certificates are not accepted with Googles' concentration on security, trying hard to get into the Enterprise market.

Butterfly Bones · Mar 20, 2018

kvic said:
I feel like I may sound a bit negative. Allow me to add one positive note. The good news is that whatever system components/3rd party services trying to secretly access over HTTPS get blocked by your adblocker, and get exposed to certain extent by pixelserv-tls. So nothing secretly gets uploaded!

Not negative to me, more of a fun challenge.

One more note, I run a full time VPN for all clients on my AC86U and a VPN on my android clients when not connected to my home network. I wonder if that effects the slc or slu, though I can not see how, but I still have much to learn.

kvic · Mar 20, 2018

Butterfly Bones said:
Not negative to me, more of a fun challenge.

One more note, I run a full time VPN for all clients on my AC86U and a VPN on my android clients when not connected to my home network. I wonder if that effects the slc or slu, though I can not see how, but I still have much to learn.

It's possible that full-time VPN may have impact..but I don't know how since I haven't run this way... I assume LAN addresses are exempted from VPN routes in your setup since you could access WebGUI as well as '/serverstats'

I'm more on the opposite end. I have VPN server on RT-AC56U. WAN users VPN into my LAN. And recently pixelserv-tls logging does reveal some creepy traffic I still haven't figured out why. They come from the VPN users I suspect but are not valid HTTPS requests which they're expected to be when hitting pixelserv-tls on port 443.

jrmwvu04 · Mar 20, 2018

Shutting down to load rc2.

kvic · Mar 20, 2018

@jrmwvu04 ...did you go to 538 right before screenshooting?! The numbers look very neat. If you set the standard a bit high, the rest of us can't catch up!

DonnyJohnny · Mar 20, 2018

kvic said:
I've to break a piece of sad news for Android users. From my carefully crafted tests, intrinsically android systems will have higher slu in pixelserv-tls because I suspect some system components and/or 3rd party services choose not to honour user root CA certificates.

Your numbers seems a bit high indeed. Hope rc.2 will improve the situation. Or else you've to get prepared to run tcpdump/wireshark to see what's going on. Not trying to scare you..also likely some worms on your devices

Yes I notice android now is the problem and it seems worse than previous version. The number is jumping.... no issue with iOS.

Protik · Mar 20, 2018

kvic said:
Look forward to people setting up a cluster of Rasp Pi, and run one instance of pixelserv-tls on each Pi that will act as a node.

Believe me it'll be SUPER fast and a SUPERIOR browsing experience.

This sounds interesting! I have only one router. Is there any way to load balance or transfer some of the traffic to the Rasp Pi?

Makaveli · Mar 20, 2018

Here is my RC.1 before updating to the new release.

Code:

pixelserv-tls 2.1.0-rc.1 (compiled: Mar 17 2018 21:02:50) options: 192.168.1.3

uts    3d 07:45    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    19    maximum number of service threads
kvg    1.48    average number of requests per service thread
krq    197    max number of requests by one service thread
req    11962    total # of requests (HTTP, HTTPS, success, failure etc)
avg    898 bytes    average size of requests
rmx    43251 bytes    largest size of request(s)
tav    3 ms    average processing time (per request)
tmx    3929 ms    longest processing time (per request)
slh    1786    # of accepted HTTPS requests
slm    40    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    8848    # of dropped HTTPS requests (client disconnect without sending any request)
slu    1179    # of dropped HTTPS requests (other TLS handshake errors)
sct    68    cert cache: # of certs in cache
sch    10460    cert cache: # of reuses of cached certs
scm    22    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    4    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    615    sess cache: # of reuses of cached TLS sessions
ssm    21    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

kvic · Mar 20, 2018

DonnyJohnny said:
Yes I notice android now is the problem and it seems worse than previous version. The number is jumping.... no issue with iOS.

The "android problem" isn't that serious. Users main browsing activities should happen in Chrome or Firefox I assume. Both honours user root CA certs. It's the other android system components and/or 3rd party services that choose not to do so for whatever reasons in their design. These are minorities though.

I believe if you can observe slu increasing *rapidly* while browsing internet on Android. It has to do with other causes. Just note that Firefox by default requires an import of your Pixelserv CA inside itself (on all platforms).

kvic · Mar 20, 2018

Protik said:
This sounds interesting! I have only one router. Is there any way to load balance or transfer some of the traffic to the Rasp Pi?

One way to do it (in my opinion the best way) is to change your script for preparing blocking files a bit. So that some blocked domains will return <pixelserv ip one> and others return <pixelserv ip two>.

For Dnsmasq, it initiliazes with the blocking files on startup once. Then it's the same amount of work as before after startup.

kvic · Mar 21, 2018

Makaveli said:

Here is my RC.1 before updating to the new release.

Code:

pixelserv-tls 2.1.0-rc.1 (compiled: Mar 17 2018 21:02:50) options: 192.168.1.3

uts    3d 07:45    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    19    maximum number of service threads
kvg    1.48    average number of requests per service thread
krq    197    max number of requests by one service thread
req    11962    total # of requests (HTTP, HTTPS, success, failure etc)
avg    898 bytes    average size of requests
rmx    43251 bytes    largest size of request(s)
tav    3 ms    average processing time (per request)
tmx    3929 ms    longest processing time (per request)
slh    1786    # of accepted HTTPS requests
slm    40    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    8848    # of dropped HTTPS requests (client disconnect without sending any request)
slu    1179    # of dropped HTTPS requests (other TLS handshake errors)
sct    68    cert cache: # of certs in cache
sch    10460    cert cache: # of reuses of cached certs
scm    22    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    4    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    615    sess cache: # of reuses of cached TLS sessions
ssm    21    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

Thanks for posting.

slc is a bit too high. I wonder what Apps/clients might have caused that.

slc is after clients having passed TLS handshake. So that means these clients already have CA cert imported. But then somehow they decide to disconnect without sending any requests. And this behaviour repeats frequently in your environment based on the total requests, req and the slu numbers. *puzzled*

Butterfly Bones · Mar 21, 2018

kvic said:
The "android problem" isn't that serious. Users main browsing activities should happen in Chrome or Firefox I assume. Both honours user root CA certs. It's the other android system components and/or 3rd party services that choose not to do so for whatever reasons in their design. These are minorities though.

I believe if you can observe slu increasing *rapidly* while browsing internet on Android. It has to do with other causes. Just note that Firefox by default requires an import of your Pixelserv CA inside itself (on all platforms).

On the "android problem" I ran tcpdump and ngrep and found lots of interesting entries (to geeky me), but nothing valuable with many searches, so I'm going to ignore it since I don't need another road to get lost down right now.

Raphie · Mar 21, 2018

18h and counting...

Code:

pixelserv-tls 2.1.0-rc.2 (compiled: Mar 20 2018 21:11:49) options: 192.168.1.2
67182 uts, 1 log, 5 kcc, 43 kmx, 28.00 kvg, 39121 krq, 142742 req, 1001 avg, 18374 rmx, 1 tav, 7056 tmx, 141662 slh, 36 slm, 0 sle, 371 slc, 280 slu, 93 sct, 3223 sch, 18 scm, 0 scp, 2 sst, 1779 ssh, 4 ssm, 0 ssp, 139483 nfe, 7 gif, 0 ico, 756 txt, 0 jpg, 7 png, 0 swf, 7 sta, 2 stt, 15 ufe, 34 opt, 525 pst, 0 hed, 80 rdr, 0 nou, 1 pth, 0 204, 14 bad, 99 tmo, 371 cls, 1026 cly, 0 clt, 0 err

kvic · Mar 21, 2018

Looks very nice @Raphie. Did you go to 538 for a photo shoot?

Any android devices in use during the same period btw?

Thread starter	Title	Forum	Replies	Date
C	Diversion Pixelserv replacement	Asuswrt-Merlin AddOns	2	Mar 25, 2024
L	Is Diversion better than NextDNS, PiHole or AdGuard Home?	Asuswrt-Merlin AddOns	10	Jul 7, 2024

pixelserv pixelserv - A Better One-pixel Webserver for Adblock

Very Senior Member

Part of the Furniture

Part of the Furniture

Very Senior Member

Part of the Furniture

Part of the Furniture

Very Senior Member

Very Senior Member

Part of the Furniture

Very Senior Member

Part of the Furniture

Very Senior Member

Senior Member

Very Senior Member

Part of the Furniture

Part of the Furniture

Part of the Furniture

Very Senior Member

Senior Member

Part of the Furniture

Similar threads

Similar threads

Support SNBForums w/ Amazon

Sign Up For SNBForums Daily Digest