pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[jrmwvu04]

One other question for @kvic or @thelonelycoder regarding the initial PKI setup. I see in the FAQ/Wiki there are instructions for setting up the default self-signed pixelserv-tls certificates. However, contrary to instruction this is all done for you automatically. I'm just interested if that is old info now for pixelserv as a whole, or if ab-solution has just added scripts into its proprietary install routine to do those steps for you?

That’s automated by AB. That’s actually what the magic of AB is - it doesn’t do any blocking, but it does set up all the pieces required to do that with very minimal user input (or understanding) required.

 

[bengalih]

Did you check the home page/changelogs and pixelserv man page

Actually I did (as I mentioned in a previous post). Let me clarify:
Yes the man page has some good info on the counters - though not all are there, like "slc."
The changelog does give a brief description for slc as:

Added a new counter 'slc' that counts the number of clients who disconnect

I'm interested in knowing a bit more like for instance, slu states:

If TLS handshakes fail other than above reasons, this counter is
incremented by one. slu is further classified into two common
types, uca and uce. Note that slu includes more than these two
types.

So...what other reasons might it fail?
Also, I'm interested in knowing how to examine the logs to find the offending clients based on each specific counter.
This is mostly just an exercise in understanding a bit more what is going on under the hood. I'm sure that some of this information can be obtained by combing through various RFCs on SSL, etc - but I'm looking for more of a practical/hands-on route of examining what's going on.

I'm not asking for a huge write-up, but would be interested in any example or tricky issues people have tracked down through examining the logs, etc.

 

[bengalih]

Hmmm...I've got one more issue...maybe... with my certificate authority.

In practice things seem to be working. I don't get any cert errors in the log (debug) when I start up and presumably all communication is taking place properly but I see this issue and I'm not exactly sure why it is happening and if it means something isn't working right.

So basically I have my CA set up like this:

BH Root CA (a)
- BH Intermediate Signing CA 01 (b)
- Intermediate Pixelserv CA (c)

Obviously (a) is my root and (b) is what I use to sign other server certs for my network. As I mentioned above I created (c) because pixelserv requires that either the password be stripped from the private key of the CA or else that the key password is stored in a file. Neither one is secure and I didn't want to compromise my main signing CA, so I created (c) solely to function as the issuing CA for pixelserv.

On my client I have installed the "BH Root CA" which, based on what I know, should be the only certificate I need to install on my clients. This is a general statement - not specific to pixelserv. Only the root CA should be required to validate any certificates issued by the root or subordinate. I believe it is the job of the server to send this certificate chain info over to the client for validation.

Here is an example of a web server I have which is using a certificate signed by (b). As you can see it shows the server cert being validated all the way up through (a). This is the case with *only* (a) installed on my client.

So I believe the above is validating that this basic configuration works and my CA and certs (as a whole) are functioning.

The FAQ states that one can use an intermediate CA as the CA for pixelserv:

If it's an intermediate CA certificate issued by a self-signed CA certificate, then save it together with the self-signed CA certificate in ca.crt, and the private key of the intermediate CA in ca.key under CERT_PATH.

I have done exactly that. The inside of my ca.crt looks like:

-----BEGIN CERTIFICATE-----
MIIFcDCCA1igAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwRTELMAkGA1UEBhMCVVMx
....
ZtySaI6u7qALFWiKnLeabGezpuVxj6G7FZ/1NnmANRW5NLQNhKiwoLEuEDmoz9wX
YHXpkw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFcDCCA1igAwIBAgIJAPMKSNEq5Uq3MA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV
...
IVj9bulWofhQ9HdnSYPRFdmIDcKR/9xmwljJ7WnG2ebgZ/m79ZxBjwEI9ICh16xk
10LLMg==
-----END CERTIFICATE-----

That's my intermediate CA cert at the top and my root CA as the second cert listed.
I also have my ca.key:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,AA3FDEA0EF1E840EF18E45087BE14F80

...
lJNY5rYKHgtimwT9bb6LRYWT7enm/DzP9/ztcmESP5R71NEK9K6ZLHmp+niASDbZ
...
-----END RSA PRIVATE KEY-----

As mentioned I don't get any errors about certificates loading, etc when initializing through the debug logs (I did at first and used them to help figure out what was going on...).

What I'm seeing is a problem validating the certificate path from the auto generated certificates from the cache.
I pull an example one out of the cache and it looks like this on my client:

As can be seen the pixelserv Intermediate CA has generated this wildcard cert for appsflyer.com. However my system can't verify this certificate because it knows nothing about its certification path.

I can resolve this issue by installing the pixelserv Intermediate CA certificate onto my client. If I do so, the above now looks like this:

That's all well and good...but I don't want to have to install my intermediate CA certs on my clients - that's not how things are supposed to work.

Now I can only assume one of a few things is happening here:
1) I have something setup wrong with my certs. I don't think this is the case, but maybe someone better with PKI than myself can explain it.

2) Something is wrong with pixelserv. Even if functionality is working on the outside, something may be implemented in a way that isn't exactly optimal. If this is the case - I'm ok with it. As long as I know that everything that needs to function is functioning - I just don't know enough of the guts of pixelserv yet to say.

3) I'm interpreting the facts wrong. Because I am looking at the raw certificate here pulled from the cache. It is not being *delivered* to me by the pixleserv web server. Therefore in normal operations this cert goes through the server which also sends along the required information to validate the certificate chain.

I'm hoping #3 is the correct one. I do have some other data points on some non-pixelserver related things that could back that up.
But...I'm not sure.

@kvic or another expert.... can you assist?

Thanks.

 

[jrmwvu04]

 

[kvic]

@bengalih

To be frank, I just skimmed through your long posts...but I could guess what you're trying to get at (hopefully).

As you've already figured out, you could use an immediate CA cert in pixelserv-tls in place of generating a new root CA certificate. That's been available since two years ago but I only know one user who make use of it. lol

pixelserv-tls is like Apache & etc in terms of treating CA certs. You could strip the password or keep it there. If you use password, you've to supply the password in a text file, named ca.key.passphrase under CERT_PATH (i.e. /opt/var/cache/pixelserv on Entware systems or /var/cache/pixelserv on standard Linux).

And if you're even more paranoid, after pixelserv-tls starts up, you could safely delete the ca.key.passphrase. But then when pixelserv-tls restarts, you need to supply the file again as usual like people key in a password when asked.

 

[kvic]

Thanks to the few posted servstats in the past month. I was trying to see if we need a refinement in the slu & co breakdowns. From what I've seen perhaps we shall keep it as-is for now. I'll find time to conclude 2.1.3 testing shortly.

I got a very long run recently. Unfortunately I think my 56U auto reboots into its 70-ish days of run. Here is my latest stats. Personally I've seen my HTTPS ad share has gone up from 30% to its current 80% over the last few years.

 

[kvic]

Now I can only assume one of a few things is happening here:
1) I have something setup wrong with my certs. I don't think this is the case, but maybe someone better with PKI than myself can explain it.

Re-visiting the post and looking closer, none of the screenshots pop up in full resolution..

You aren't expected as usual to import the immediate CA cert generated for use in pixelserv-tls. I can't recall doing that in my development tests two years ago. And I didn't receive complaints from the gent who deploys it to his one million users..

But there is a chance he might be deploying the intermediate cert as well. I need to find time to re-test and confirm it there is an issue.

 

[JimbobJay]

Hi. Not sure if this is the right place to ask this. I notice that in the latest 4.0 release of Pi-Hole, they now have an option of specifying where blocked domains are pointed to (https://docs.pi-hole.net/ftldns/blockingmode/), with null (0.0.0.0) being the new default, and the pi-hole ip if you want the blocking page with stats etc to be displayed. Could you instead, in theory, use the ip of a pixelserv server to get the best of both pi-hole and properly blocked https domains? Currently I use pixelserv in combination with AB-solution on my AC-87U, but I think the processing power of a 3B+ Pi would beat that of the ASUS router.

 

[kvic]

Could you instead, in theory, use the ip of a pixelserv server to get the best of both pi-hole and properly blocked https domains?

You could run pixelserv-tls anywhere, not necessarily on the router. Simply update "pixelserv ip" to reflect where it's run. Then ad requests will go there.

So for example, you run ABS on 87U. pixelserv-tls on Rasp Pi. Change "pixelserv ip" in ABS to that of the Rasp PI.

 

[SMS786]

Latest stats for my 68U

pixelserv-tls 2.1.3-test.4 (compiled: Jul 17 2018 01:44:19)

uts    15d 09:56    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    46    maximum number of service threads
kvg    2.37    average number of requests per service thread
krq    2704    max number of requests by one service thread
req    107459    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1258 bytes    average size of requests
rmx    128091 bytes    largest size of request(s)
tav    23 ms    average processing time (per request)
tmx    7258 ms    longest processing time (per request)
slh    35312    # of accepted HTTPS requests
slm    261    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    7953    # of dropped HTTPS requests (client disconnect without sending any request)
slu    61444    # of dropped HTTPS requests (other TLS handshake errors)
uca    18492    slu break-down: # of unknown CA reported by clients
uce    33440    slu break-down: # of unknown cert reported by clients
ush    5996    slu break-down: # of shutdown by clients after ServerHello
sct    50    cert cache: # of certs in cache
sch    73405    cert cache: # of reuses of cached certs
scm    1418    cert cache: # of misses to find a cert in cache
scp    1405    cert cache: # of purges to give room for a new cert
sst    5    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    3134    sess cache: # of reuses of cached TLS sessions
ssm    1867    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session
nfe    20574    # of GET requests for server-side scripting
gif    466    # of GET requests for GIF
ico    92    # of GET requests for ICO
txt    3173    # of GET requests for Javascripts
jpg    301    # of GET requests for JPG
png    142    # of GET requests for PNG
swf    0    # of GET requests for SWF
sta    21    # of GET requests for HTML stats
stt    0    # of GET requests for plain text stats
ufe    1833    # of GET requests /w unknown file extension
opt    4529    # of OPTIONS requests
pst    4866    # of POST requests
hed    0    # of HEAD requests (HTTP 501 response)
rdr    1370    # of GET requests resulted in REDIRECT response
nou    0    # of GET requests /w empty URL
pth    1    # of GET requests /w malformed URL
204    0    # of GET requests (HTTP 204 response)
bad    8    # of unknown HTTP requests (HTTP 501 response)
tmo    320    # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls    7989    # of dropped requests (client disconnect without sending any request)
cly    0    # of dropped requests (client disconnect before response sent)
clt    0    # of dropped requests (reached maximum service threads)
err    0    # of dropped requests (unknown reason)

 

[bengalih]

Re-visiting the post and looking closer, none of the screenshots pop up in full resolution..

You aren't expected as usual to import the immediate CA cert generated for use in pixelserv-tls. I can't recall doing that in my development tests two years ago. And I didn't receive complaints from the gent who deploys it to his one million users..

But there is a chance he might be deploying the intermediate cert as well. I need to find time to re-test and confirm it there is an issue.

Not sure what you mean about full resolution...the pics I posted? They appear in normal res on my end, plenty large/clear enough to see what I am indicating. Apologies if you have some type of visual impairment and have requested a specific type of shot (being totally real here...not facetious). In any event, I think my 3rd explanation in my post is probably the correct one and there is not an issue. I've done several tests with both the self signed certs and my PKI and the stats look similar so I think I have the PKI setup correctly.

I am however having another issue which is pretty problematic. At first I thought it was AB-solution, but my testing shows I don't have the problem with just AB-solution, only when I use pixelserv with it. I have done my testing both with my PKI and the original certs (OOB config) and got the same results. I originally posted this as an issue in the AB thread (issue #2 here), but am going to move it here based on further testing.

The summation is:

I'm not sure how much more info I can give other than pixelserv On = horrible lag on snbforums, pixelserv Off = seemingly normal usage. I can basically recreate this at will. Any ideas?

Now that I've done more testing on this and I can definitively conclude that pixelserv is causing this problem - or at least exacerbating it. It's almost a show stopper for me, and I really don't want it to be as I'd like to remain a disciple in the pixelserv fold

My testing was done with Chrome (my preferred browser), IE, and Edge. When I visit this a forum thread with each with pixelserv turned on, I get the following differences in how blocked ads are displayed. I'm not sure how important this is, but it is interesting (especially IE's behavior) and may provide a data point to my real issues:

Links to pics:
- Chrome
- IE
- Edge

In addition to these viewing differences, both Chrome and IE are unbearable laggy. If I try to scroll the page using the right-most scroll bar (or arrow keys), if I try to select text, or if I am typing it is very jumpy. For instance, at least every other sentence I type right now I am having a delay of 3-5 seconds for text to catch up. It takes sometimes 3+ seconds for an incorrectly spelled word to be highlighted, etc. All around just a bad experience that makes it seem I'm working on a 15 year old computer, not an i7 with 32GB of RAM.

Additionally, Chrome seems to never fully finish even loading the page (or at least it takes several minutes, after which time the lagginess remains). What I see in Chrome's address bar is the loading wheel constantly spinning and addresses flashing in the lower left corner as if it is trying to access those resources. Here is an example of the two areas:

If I go into AB-solution and disable pixelserv, this issue goes away.

I have seen similar behavior on at least a couple other sites over the past week or so as I've been setting up this solution, unfortunately I can't remember which ones. However based on my ability to recreate this at will with SNBforums I think that's enough to work with.

I could really use some advice on this as it is pretty unbearable.

thanks.

UPDATE:

I've got some more damning evidence showing that pixelserv is definitely negatively influencing Chrome's behavior (in my setup).

I started Chrome up with no extension/plugins and visited this site.
One time with pixelserv enabled (and exhibiting the issues), and one time without.
Each time I let chrome settle for about 30 seconds to finish loading the page and any chrome initial resources.

This is what Chrome's task manager shows when pixelserv is disabled:

Relatively small cpu and mem footprint. But this is the exact same test with pixelserv enabled on the router:

That was after about 30 seconds. Within another 30 it had climbed to 1.5 GB memory footprint and continued to climb.

Obviously something isn't right here.

It is entirely possible that I have a misconfiguration somewhere - but this is all a pretty straight forward setup. As mentioned I have these issues with using the default PKI setup as well (as my PKI is the only special setup I've done).

Note that most websites don't seem to exhibit this behavior - but seemingly some do.

 

[M@rco]

I'm not sure what's happening at your side @bengalih...

I just tried to replicate your issue, but I can't. I opened Chrome Taskmanager and loaded the Pixelserv Stats page to see what happens. I left it open for at least 5 minutes, and this is about as high as it came, with pixelserv-tls 2.1.3-test 4 enabled:

CPU load never came higher than 1.5, memory footprint started at 21.762 K and varied up to 24.092 K over the 5 minute (+) time frame. GPU load in the screenshot is high because of the screenshot taken. All extensions and Chrome apps have been disabled prior to testing.

So, I don't know what's causing issues, but I sure can't replicate them on my system. By the time I'm finished uploading the image to imgur and typing this post, it's still at 24.092 K.

 

[bengalih]

I'm not sure what's happening at your side @bengalih...

I just tried to replicate your issue, but I can't. I opened Chrome Taskmanager and loaded the Pixelserv Stats page to see what happens. I left it open for at least 5 minutes, and this is about as high as it came, with pixelserv-tls 2.1.3-test 4 enabled:

It isnt the pixelserv stats page causing the issue...it is snbforums. Load this thread and see.

 

[DonnyJohnny]

It isnt the pixelserv stats page causing the issue...it is snbforums. Load this thread and see.

Other than chrome, ie, and edge. Have you tried on fresh installation of Firefox?
Sometime lag could be caused by duplicate certification? Have you check the browser certification management to remove all pixelstats cert that you could have imported from http://[pixelstats ip]/ca.crt
Then try reimport them again.
Same for windows certification management. Remove and reimport?

I don’t have any issue with my chrome/edge when loading the forum.

 

[M@rco]

It isnt the pixelserv stats page causing the issue...it is snbforums. Load this thread and see.

As requested:

Screenshot taken again after 5 minutes for comparison.

 

[jrmwvu04]

The other thing that matters is what all you’re sending to pixelserv. If you’ve got a wildly aggressive block list it can be causing problems others don’t see. Didn’t see that mentioned in skimming through.

 

[DonnyJohnny]

Yes, that makes some sense. I have 2 Rokus online and at least one other phone and possibly some tablet use that don't have any certs on them. I plan to install them on what I can, but obviously you can't get them on the Rokus, etc.

I've actually got the certificates setup nicely now. I redid my PKI last night and created an Intermediate CA solely for pixelserv.
I did this because you have either remove the password from the signing cert or else place it in a "ca.key.passphrase" file on the system. While I probably have much greater problems if my router is compromised, I wouldn't want my main RootCA credentials to be taken.
But, now that I have it running off of my own CA, I don't need to distribute any certificates to my existing systems since they already are configured with my RootCA as trusted (or at least those devices that are capable).
I also created my own web server certificate for the /servstats page signed by my main Intermediate CA so that I don't get any warnings when accessing that.



Yeah...heh... I was doing some searching and looked through some older posts in this thread last night and found a post by @kvic explaining how to manually generate each wildcard cert for blocking and install it. I was really confused like "doesn't it do this automatically??" when I realized the thread was from 2-3 years ago and realized that development has come a long way

One other question for @kvic or @thelonelycoder regarding the initial PKI setup. I see in the FAQ/Wiki there are instructions for setting up the default self-signed pixelserv-tls certificates. However, contrary to instruction this is all done for you automatically. I'm just interested if that is old info now for pixelserv as a whole, or if ab-solution has just added scripts into its proprietary install routine to do those steps for you?

Thanks again guys - I'll be monitoring this thread moving forward so hopefully will keep up to date and be able to contribute something back!

The one last thing I'm interested in (for now ) is to get some more information on each of the servstats, particularly the sxx and uxx ones and how to interpret the log files. For instance, when debugging the logfile yesterday I was seeing some failures from my wife's phone (with no cert) that were logging her IP address. But, when I removed the cert from my laptop it looked like I was seeing some hits from my machine, but no IP was shown. If anyone knows of a good resource that explains these things in more detail I'd love to take a look!

By the way, please clear your browser cache to see that helps.

 

[kvic]

Not sure what you mean about full resolution...the pics I posted? They appear in normal res on my end, plenty large/clear enough to see what I am indicating. Apologies if you have some type of visual impairment and have requested a specific type of shot (being totally real here...not facetious). In any event, I think my 3rd explanation in my post is probably the correct one and there is not an issue. I've done several tests with both the self signed certs and my PKI and the stats look similar so I think I have the PKI setup correctly.

For people can't see full pic, check if "postimg.cc" is in your blind list.

About your new issue...DNS based adblock to some extent is playing a Russian roulette really. For example, when you visit https://sample.webiste, it loads, say,

• https://resource.me/abc
• https://yet.another.resource/xyz

If both do not load (e.g. hitting 0.0.0.0 blindly), you maybe lucky to see a loaded sample.website. Other times, you might not.

If both loads (i.e. hitting pixelserv-tls), you're lucky again. Everything just works.

If one of them doesn't load (doesn't matter hitting 0.0.0.0 or pixelserv-tls), you maybe lucky to get stunted i.e. getting some weird issue. Sounds like you run into this case. For many reasons, I don't want to discuss further since it's regarding this forum's website.

What you can do is open your browser's Web Inspector, and then follow the below guide to figure out what domains to whitelist in your script:

• https://github.com/kvic-z/pixelserv...ut-domain(s)-to-whitelist-on-a-broken-website

Btw, about your issue on intermediate CA, what's the content of your ca.crt?

 

[DonnyJohnny]

How to resolve the error?
slu break-down: # of unknown cert reported by clients
slu break-down: # of shutdown by clients after ServerHello

Why so many ServerHello error and unknown cert error?
The ServerHello is from Ipad.
The unknown cert from a android phone.

Aug 26 21:14:36 pixelserv-tls[948]: handshake failed: reached max retries. client 192.168.2.9:56555 server www.googleadservices.com
Aug 26 21:14:36 pixelserv-tls[948]: handshake failed: reached max retries. client 192.168.2.9:56559 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56560 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56561 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56563 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56564 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56565 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56568 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56569 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56570 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56571 server www.googleadservices.com
Aug 26 21:14:37 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56572 server www.googleadservices.com
Aug 26 21:14:54 pixelserv-tls[948]: handshake failed: unknown cert. client 192.168.2.6:38418 server ssl.google-analytics.com
Aug 26 21:15:17 pixelserv-tls[948]: handshake failed: unknown cert. client 192.168.2.6:38426 server api.mixpanel.com
Aug 26 21:15:17 pixelserv-tls[948]: handshake failed: unknown cert. client 192.168.2.6:38428 server api.mixpanel.com
Aug 26 21:15:17 pixelserv-tls[948]: handshake failed: unknown cert. client 192.168.2.6:38430 server decide.mixpanel.com
Aug 26 21:15:20 pixelserv-tls[948]: handshake failed: unknown cert. client 192.168.2.6:38432 server e.crashlytics.com
Aug 26 21:27:39 pixelserv-tls[948]: handshake failed: shutdown after ServerHello. client 192.168.2.9:56910 server metrics.icloud.com

 

[kvic]

How to resolve the error?
slu break-down: # of unknown cert reported by clients
slu break-down: # of shutdown by clients after ServerHello

Why so many ServerHello error and unknown cert error?
The ServerHello is from Ipad.
The unknown cert from a android phone.

Both Android and iOS, especially a few system components but also expanding into vendor apps, check certificates against their hardcoded fingerprints. In the of better security (there is some truth there) but more likely to prevent other people from seeing the content of transfer between apps and servers.

There is no way to avoid such problems. Hence, pixelserv-tls reports them with best effort of classification. All these fall into the category of suspicious connections, and you shall check if they're okay to ignore/suppress. I recall @SMS786 caught one app with a malware attempting to connect to a remote coin miner a few months ago.

Now with the beta report script, the process of checking and monitoring shall be lot easier. Don't forget to make good use of the daily report.