pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[kvic]

Next week there will be a test build of next version KL for download on Github.

Changes include:

• Support HTTP/1.1 persistent connections
• Add option 'O' for specifying timeout of persistent connections
• Add new counters kcc, kmx and krq related to service threads (of persistent connections)
• Add support for TCP Fast Open (require Linux kernel >= 3.16)
• Support HTTP POST method for graceful processing
• Double message buffer for more efficient processing of lengthy requests
• Correct/enhance ring buffer handling in certificate generations

For feedback to current version Kk or future test versions, ppl have a choice to leave comments here, on the pixelserv-tls page or issuing a Github ticket.

All feedback will be read and thoroughly considered.

 

[eclp]

After the kk update comes this message:

pixelserv[1302]: invalid file path de-de

 

[kvic]

After the kk update comes this message:

pixelserv[1302]: invalid file path de-de

Safe to ignore. It just reports a malformed URL by the ad scripts embedded in a website you visit.

BTW, forgot to mention "more granular control over logging" is planed for KL. May appear in later test versions.

 

[elorimer]

Trying .KL. So far so good. 87U on 68_4.

Out of curiosity, is there a guide to the messages pixelserv is now logging?

Oct 10 07:52:46 pixelserv[28676]: Exit recv loop fd:23 rv:0 errno:0 wait_cnt:100 num_req:1
Oct 10 07:52:46 pixelserv[28676]: ( 5) 192.168.0.107 ost: lptag.liveperson.net GET /tag/tag.js?site=10216802 HTTP/1.1
Oct 10 07:53:12 pixelserv[28676]: Exit recv loop fd:24 rv:0 errno:0 wait_cnt:100 num_req:1
Oct 10 07:53:48 pixelserv[28676]: Exit recv loop fd:24 rv:0 errno:0 wait_cnt:100 num_req:1
Oct 10 07:53:48 pixelserv[28676]: ( 5) 192.168.0.107 ost: lptag.liveperson.net GET /tag/tag.js?site=10216802 HTTP/1.1
Oct 10 07:54:01 pixelserv[28676]: Exit recv loop fd:24 rv:0 errno:0 wait_cnt:100 num_req:1
Oct 10 07:54:08 pixelserv[28676]: POST Content-Length: 94
Oct 10 07:54:08 pixelserv[28676]: POST expect length: 0
Oct 10 07:54:08 pixelserv[28676]: (19) 192.168.0.151 ost: ads.nexage.com POST /admax/sdk/handshake/1 HTTP/1.1
Oct 10 07:54:08 pixelserv[28676]: (15) 192.168.0.151 ost: ad.crwdcntrl.net GET /5/c=991/e=app/mid=bac2789e-c504-4c14-9236-cfb374fea6de/pe=y HTTP/1.1
Oct 10 07:54:08 pixelserv[28676]: (15) 192.168.0.151 ost: ad.crwdcntrl.net GET /5/c=991/e=app/mid=bac2789e-c504-4c14-9236-cfb374fea6de/pe=y HTTP/1.1
Oct 10 07:54:08 pixelserv[28676]: (15) 192.168.0.151 ost: ad.crwdcntrl.net GET /5/c=991/e=app/mid=bac2789e-c504-4c14-9236-cfb374fea6de/pe=y HTTP/1.1
Oct 10 07:54:08 pixelserv[28676]: Exit recv loop fd:25 rv:-1 errno:104 wait_cnt:100 num_req:3
Oct 10 07:54:16 pixelserv[28676]: POST Content-Length: 6856
Oct 10 07:54:16 pixelserv[28676]: POST expect length: 2512
Oct 10 07:54:16 pixelserv[28676]: POST recv length: 2512; errno: 0
Oct 10 07:54:16 pixelserv[28676]: (19) 192.168.0.151 ost: udm.scorecardresearch.com POST /offline?c2=3005403&s=be743faf24367505afe9d75338b025c6 HTTP/1.1
Oct 10 07:54:16 pixelserv[28676]: (15) 192.168.0.151 ost: b.scorecardresearch.com GET /p2?c1=19&c2=3005403&ns_ap_an=NYTimes&ns_ap_pn=android&ns_ap_pv=6.0.1&c12=67231acc8548961802a81dc32450938d-cs31&ns_ak=D%2FnCqRCPvh9aHjh59%2F0MoYgcxvhicKPK6PTQrPuNV9MGoqTlIm2WgbqF84ajIjw4eu89cM6eeF8Zjg1E4Wz%2B7xeQ1TQSv%2FZRKz3ft9k9ooQyRd2dclEwciXsbU73ZWSZsKs1R9%2FtgkxVG2jl1dtiUx6UescuJYQLPX1hEnuKxV8%3D&name=start&ns_ap_ec=1&ns_ap_ev=start&ns_ap_device=flo&ns_ap_id=1507636450765&ns_ap_csf=1&ns_ap_bi=com.nytimes.android&
Oct 10 07:54:16 pixelserv[28676]: Exit recv loop fd:26 rv:0 errno:0 wait_cnt:100 num_req:1
Oct 10 07:54:18 pixelserv[28676]: Exit recv loop fd:26 rv:0 errno:0 wait_cnt:100 num_req:1
Oct 10 07:54:42 pixelserv[28676]: Exit recv loop fd:9 rv:-1 errno:11 wait_cnt:1 num_req:2
Oct 10 07:54:42 pixelserv[28676]: Exit recv loop fd:12 rv:-1 errno:11 wait_cnt:1 num_req:2
Oct 10 07:54:42 pixelserv[28676]: Exit recv loop fd:10 rv:-1 errno:11 wait_cnt:1 num_req:2

 

[thelonelycoder]

Same as @elorimer, I see a lot of these on both ARM routers, using pixelserv-tls.Kl-test1.Entware-ng.armv7softfloat:

Oct 10 09:01:00 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:00 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:11 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:12 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:11 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:15 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:15 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:02:07 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:02:07 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:02:07 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1

 

[elorimer]

kl does seem to be faster than kj. Average processing time is 30ms, maximum number of threads 67. I forget what my average processing time was before.

I'm guessing these Exit recv loop messages are timeouts?

 

[kvic]

Both the "POST" and "Exit recv loop" are debug messages that will be removed in the final release or moved to only appear in a higher level of verbosity of logging.

Oct 10 07:54:08 pixelserv[28676]: POST Content-Length: 94
Oct 10 07:54:08 pixelserv[28676]: POST expect length: 0

The first line indicates a POST request is received and the client indicates a message of length 94 bytes. The second line indicates all 94 bytes are received in the very first message, and hence we expect no more from the client.

Another example:
Oct 10 07:54:16 pixelserv[28676]: POST Content-Length: 6856
Oct 10 07:54:16 pixelserv[28676]: POST expect length: 2512
Oct 10 07:54:16 pixelserv[28676]: POST recv length: 2512; errno: 0

The first line indicates a POST message of length 6856 bytes. We received more than 3kb in the first message, and only expect 2512 bytes from the client. The third line indicates all 2512 bytes received successfully (hence errno = 0 = success).

The Exit recv loop is logged when a "service thread" finishes and kills itself.

Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:11 rv:0 errno:0 wait_cnt:30 num_req:1
Oct 10 09:01:14 pixelserv[28396]: Exit recv loop fd:10 rv:0 errno:0 wait_cnt:30 num_req:1

fd is the socket descriptor which are very likely re-used and repeated. "errno: 0" indicates the client proactively closes the connection. "num_req" indicates how many requests are serviced by this thread aka known as HTTP/1.1 persistent connection.

 

[kvic]

kl does seem to be faster than kj. Average processing time is 30ms, maximum number of threads 67. I forget what my average processing time was before.

I'm guessing these Exit recv loop messages are timeouts?

HTTP/1.1 persistent connections help in speeding up the processing. This is especially apparent for HTTPS connections as TLS handshakes are no longer repeated for every single request. Nevertheless it helps both HTTP/S.

Max number of threads 67 indicates at its highest load 67 ad requests are being simultaneously serviced. Some websites are just crazy!

From my run so far only between 1 and 2 requests are serviced by one service thread or http/1.1 connection on average. It's registered in the kvg counter. This though varies with websites people visit.

 

[kvic]

Version KL-test2 is available.

Changes:

• added new facility of tiered logging
• reviewed/refined tiers of all logged messages
• expanded the CLI option '-l' to '-l level'
• expanded the function of dynamically switching log tiers through URL
• expanded display of counter 'log' on servstats page
• cleaned up further logics/code in connection handler
• fixed measurement of POST processing time

Also check out https://kazoo.ga/pixelserv-tls/

 

[thelonelycoder]

Version KL-test2 is available.

Changes:

• added new facility of tiered logging
• reviewed/refined tiers of all logged messages
• expanded the CLI option '-l' to '-l level'
• expanded the function of dynamically switching log tiers through URL
• expanded display of counter 'log' on servstats page
• cleaned up further logics/code in connection handler
• fixed measurement of POST processing time

Also check out https://kazoo.ga/pixelserv-tls/

Do you plan to make any more switch changes for Kl or is '-l level' the last you add before stable release?
I ask because I add them to the AB options.

 

[kvic]

Do you plan to make any more switch changes for Kl or is '-l level' the last you add before stable release?
I ask because I add them to the AB options.

I think that should be it for KL..

 

[thelonelycoder]

I think that should be it for KL..

Thanks, I'll push an update for the addon as soon as time permits.
The log facility is solely the syslog, correct?

 

[kvic]

Thanks, I'll push an update for the addon as soon as time permits.
The log facility is solely the syslog, correct?

In KL, it's only for syslog. People shall upgrade to syslog-ng from Entware-ng so that they can better manage with dedicated files. There is a thread on syslog-ng on this forum.

The new log facility will allow adding options of writing to a text file or maybe a sql db in a future version.

 

[elorimer]

Having a go now with T2 and default logging. Started up fine.

Weather channel starts 24 threads
AARP and DailyMail.co.uk start 18.
NYTimes starts 12 threads.

 

[kvic]

possible to pregenerate certs for the whole block list?

Yes, I mentioned in this thread before. But not recommended so I won't repeat here. Because ppl only need a small set of domains for daily use. It's a waste of resources to pre-generate certificates for close to 100K domains.

For a new domain, only first requests fails. It only takes 500ms to generate a certificate on a slow machine such as Cortext A9 800MHz CPU.

I've cloned the git repository onto a linux box I use for a dns server already. It is amd64. After linking all the libraries and doing the make amd64 im not sure what to do next. make install doesnt work. Help?

make won't install but just build the binaries. Ideally we shall have a configure script, and include "install" as an option on supported platforms.

If anyone has a working "configure" script, feel free to contribute.

 

[kvic]

Hello kvic,

Would you be able and willing to change this pixelserv-tls syslog behavior:
[Release] AB-Solution 3.8 - The Ad Blocking Solution (3.8.1)

It'll be available in Version KL. A test version KL-test2 is available for trial.

If you specify "-l 0" in CLI, it's close to dead silence.

 

[elorimer]

No issues so far with KL2.

The log is showing a few bursts of

conn_handler reported unknown ssl state: 5

 

[kvic]

No issues so far with KL2.

The log is showing a few bursts of

conn_handler reported unknown ssl state: 5

This error indicates that some devices on your LAN are talking to port 443 (or a secure port you specified at CLI) in non https fashion.

However, the message was put at wrong logging tier. I'll fix it in next test version.

Edit: The number of appearances of this error message shall be equal to counter "slu". They're equivalent.

 

[kvic]

One of the new features in KL will be full support of HTTP POST method.

See how much data about you trackers and advertisers are grabbing. In the below screenshot, I suspect it was close to 120KB of mine.

To see what's intended to be POST'ed to them, enable "-l 4" (log tier INFO; very chatty..be warned) .

 

[kvic]

plain old pixelserv

Every request is answered by spawning a process. A process is a heavy weight beast which requires significantly more resource (memory and time) to create and destroy.

in pixelserv-tls

When pthread was instroduced in pixelserv-tls, every request is answered by spawning a thread. A thread is also known as light weight process which is much more efficient to work with.

The question is why do we have to create a new process/thread to serve one single request and then kill off the process/thread?

We don't have to, and it's also not HTTP/1.1 compliant.

in version KL

pixelserv-tls will introduce HTTP/1.1 persistent connection. _cough_. A single thread can serve as many requests in burst as clients insist.

The screenshot below registers 102 in counter 'krq' - one single thread serves 102 ad requests (i.e. to the same ad server). That totally blows my mind away. Ads on some websites are simply gone mad.