pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[mstombs]

Most of it is over my head, but it did seem to increment disconnect data by parsing the text of 'syslog' events. Without proper understanding of https connections though, I can't say whether commenting out some piping would make servstats more 'sensible'? Perhaps my lack of https knowledge is my real problem.

pixelserv can write to syslog, but doesn't parse it. MYLOG is a macro which is null defined in util.h unless compiled with -DVERBOSE

Do clients without a imported Pixelserv root certificates result in a very high cls? ...

good catch, the "cls" will be incremented if the the SSL_accept fails or rv from SSL_read (for https) or recv is zero, the description of "cls" is not necessarily correct, when rv == 0 the client may have cleanly disconnected just not sent any data.

if @kvic is watching we could have more categories of result added, from the ssl section rv==0 could be from default because ssl_err not > 0, OR no data from SSL_read. From the normal connection section both errors rv<0 AND no data rv==0 are lumped together, sorry I probably did that in a much earlier version (before pipes or https!).

Current code starts here https://github.com/kvic-z/pixelserv-tls/blob/master/socket_handler.c#L601

cls counts the number of FAIL_CLOSED pipedata.status returns via the pipe from connection handling process back to main process (.... somehow), SSL status returned via pipedata.ssl .

My current stats from servstats.txt - need to review the certs on my devices!

pixelserv-tls version: v35.HZ12.Kj compiled: May 31 2017 04:30:23 options: 192.168.66.254 -p 80 -p 81 -p 8080 -p 8081 -k 443 -o 2
2042236 uts, 0 log, 65537 req, 1266 avg, 9097 rmx, 89 tav, 2748 tmx, 60669 slh, 158 slm, 0 sle, 479 slu, 697 nfe, 29 gif, 1 ico, 1190 txt, 0 jpg, 0 png, 0 swf, 3 sta, 1 stt, 73 ufe, 1111 rdr, 0 nou, 0 pth, 0 204, 116 pst, 0 hed, 0 bad, 0 err, 784 tmo, 61532 cls

 

[truglodite]

It's annoying this forum doesn't have a 'quote button' I could use on the above post. This quote took a lot of work...

mstombs said:
pixelserv can write to syslog, but doesn't parse it. MYLOG is a macro which is null defined in util.h unless compiled with -DVERBOSE

I was probably not accurate in describing the problem with syslog + serverstats. To be clear, the bug I noticed:
The servstats SSL fields, slh, slm, sle, and slu, do not increment while 'log to syslog' is enabled.

My initial theory as to why was incorrect as you pointed out. More digging gives these facts...

In sockethandler.c, there are some flags that result in incrementing servstats ssl stuff (SSL_MISS, SSL_ERR...). The flags are first stored in tlsext_cb_arg->status (example in line 438). Then they're saved to pipedata.ssl (line 604), which is then used to actually increment the stats values in pixelserv.c (line 579). Before they are allowed to increment, pixelserv.c has a few layers of if/else() to pass through.

With all of those possible breaking points, debugging this is beyond me. Hopefully this helps those who have the time/skills to fix it.

Cheers,
Kevin

 

[thelonelycoder]

It's annoying this forum doesn't have a 'quote button' I could use on the above post. This quote took a lot of work...

Mark text to quote, the rest is mindblowing...

 

[truglodite]

mindblowing

Oh heck yeah... mindblown! I didn't see that in the middle of my frustration while manual quoting, lol. Thanks!

 

[Popov]

@kvic , I think I've nailed one source for potential pixelserv-tls crashes :

Jul 5 16:23:57 box user.info kernel: pixelserv.x86.p[5979]: segfault at 0 ip 0815a211 sp b6bfdedc error 4 in pixelserv.x86.performance.static[8048000+245000]

Version :

/mnt/sda1/server/pixelserv/pixelserv/pixelserv.x86.performance.static
 version: v35.HZ12.Kj
 compiled: May 11 2017 20:06:54
 options: -u pixelserv -z /mnt/sda1/server/pixelserv/conf

It can be replicated at will using openssl :

$ openssl s_client -connect pixelserv:443
CONNECTED(00000003)
140579554252440:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:656:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1499264374
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

note : the ssl3_write_byte is always called, even when using the " -tls1_2 " flag or another -tls1_? version

It occured before, but I didn't make the link until a while with a solution for network monitoring and mapping, based on nmap I was testing. Had to activate the /var/log/message to the micro VM dedicated to pixelserv to get a trace of the situation.
Seems pixelserv tls doesn't survive such a connection

Also, not related to a crash, but this message can generate a lot of logs :

Jul  2 12:31:02 box daemon.warn pixelserv[2828]: Sending HTTP 501 response for unknown HTTP method or non-SSL, non-HTTP request: OPTIONS

Would be nice if pixelserv at least skip the message in the log (maybe a -silent setting ? didn't see one)

 

[mstombs]

Also, not related to a crash, but this message can generate a lot of logs :

Jul  2 12:31:02 box daemon.warn pixelserv[2828]: Sending HTTP 501 response for unknown HTTP method or non-SSL, non-HTTP request: OPTIONS

Would be nice if pixelserv at least skip the message in the log (maybe a -silent setting ? didn't see one)

That message goes to the log because it was expected to be very rare! It seems pixelserv should be enhanced to handle the "http OPTIONS" request:-

https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS

What client browser/website tends to trigger it?

 

[Popov]

this one :
=> network monitoring and mapping, based on nmap
use nmap itself, or any network scanning tool : it's the usual behavior when a webserver is scanned, as the OPTION method given informations about the webserver capabilities.

 

[jrmwvu04]

That message goes to the log because it was expected to be very rare!

I get those. Usually a handful a day, sometimes more, sometimes none. I assume no big deal.

 

[Alfred]

Hello kvic,

Would you be able and willing to change this pixelserv-tls syslog behavior:
[Release] AB-Solution 3.8 - The Ad Blocking Solution (3.8.1)

 

[Popov]

That message goes to the log because it was expected to be very rare! It seems pixelserv should be enhanced to handle the "http OPTIONS" request:-

https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS

What client browser/website tends to trigger it?

At first, I assumed it was due to the scanning tools I'm using, but the information was missing because the used VM didn't do any tracing to /var/log/. Since it was actived, I'm also seeing a lot of OPTIONS, and given the time, I could trace it back to Firefox.
Here's the information related to this behavior : https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS

Basically, allowing an ajax query to target a different domain is done through CORS, and the first step is to use an OPTIONS call.
As ads are usually on a (chain of) different domains, I suppose having a lot of call to OPTIONS is expected

 

[jrmwvu04]

Anyone having issues with safari on iOS 10.3.3 complaining about the certificates? Behavior is like what you'd see if the certificate wasn't imported on the client. After a very long time of it working fine, it suddenly doesn't and the only change I can think of that lines up with it is the 10.3.3 update.

 

[jrmwvu04]

Well, I figured it out.

In iOS 10.3 and later, when you manually install a profile that contains a certificate payload, that certificate isn't automatically trusted for SSL.

https://support.apple.com/en-us/HT204477

 

[Lost Dog]

Pixelserv-tls seems to be breaking the Android Amazon app (I've got version 12.4.0.200). I thought it was adblock however with a stock Medium block file with pixelserv-tls on Amazon throws an error ("Sorry, there's a problem with this page right now. We're working on the fixing it. CS3"). Once I turn pixelserv-tls off Amazon works perfectly.

I tried logging but saw nothing to indicate what is going on.

 

[Popov]

It's might be related to a website address the app must access, preventing it to continue if it wasn't able to do so. In this case, you need to check the latest queries in your dns server log after trying to load the app, and compare with the sites you're blocking. There could be one you have completly blocked at the domain level, instead of a particular subdomain, thus preventing the use of other ressources.

 

[superjet]

possible to pregenerate certs for the whole block list?

 

[superjet]

I've cloned the git repository onto a linux box I use for a dns server already. It is amd64. After linking all the libraries and doing the make amd64 im not sure what to do next. make install doesnt work. Help?

 

[mstombs]

possible to pregenerate certs for the whole block list?

pixelserv-tls generates the certificate if it does no exist of first need.
I guess you could write a script to try to access every blocked domain using https, but leaving as is ensures the certificate properly matches the browser request when first generated.

 

[mstombs]

I've cloned the git repository onto a linux box I use for a dns server already. It is amd64. After linking all the libraries and doing the make amd64 im not sure what to do next. make install doesnt work. Help?

You have made the pixelserv-tls binary, what you now need is an adblock script to create a valid ip address for pixelserv to listen on, start pixelserv with appropriate vars, and then to poison the dns entries for the adblock domains in your dns server so that the clients browsers communicate with pixelserv as a dummy 'null' target. There must be generic linux box scripts - @thelonelycoder 's ADB solution is very Asuswrt-Merlin specific. I am not sure what dns server or pixelserv version Pi-hole uses, I did help out with old pixelserv on a raspberrypi at one point.

 

[kvic]

Version double K is available from Entware-ng repository.

From RTM to available for download, it took less than two weeks this time around. Great team work, Entware-ng!

For a complete list of changes, visit here.

Some of the recent feedback in this thread went into this release. Honourable mentions (but not limitied to)

• the reproducible crash case from @Popov , which led to discovering and fixing a couple of crash bugs
• support for HTTP OPTION method is also added
• description of a few counters updated and some new counters added

README.md on Github - the official introductory guide to pixelserv-tls - has update about trusting CA certificates on iOS. That solves a puzzle I also had for a while since iOS 10. Thanks to @jrmwvu04.

Wiki on Github will carry supplementary documentation.

Official announcement will be available here, and followed by posting in this thread when suitable.

I'm sorry that I had not (and won't be) actively reacting to posts here. Nevertheless, every bit of feedback is read and considered upon its merits.

One issue not long ago accidentally sent me through the re-discovery of the original benefit of pixelserv. In 2017, ads become so complicated that sinking requests to 0.0.0.0 is considered bad practice (that's from me!). It becomes the source of ending spinning circles on loading certain websites.

That re-kindles me and everyone's desire in pixelserv-tls. With Version double K, pthread is enabled in all Entware-ng editions. This paves the way for some exciting performance leap under the hood in next version, Version KL..

Start using double K, providing feedback and look forward to a butter smooth browsing experience in next version

 

[jrmwvu04]

$ chown nobody /opt/var/cache/pixelserv

I did have to do that. Noticed it wasn't working and the syslog was full of errors and put two and two together, but otherwise seem to be good to go now.