pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[mstombs]

@mstombs used to say he found some ad networks use non-standard ports. I believe that's why option -p was added to cover such edge cases. I added -k in the same spirit. I believe ad networks using non-standard ports are extremely rare today.
.

I'm still using "-p 80 -p 81 -p 8080 -p 8081 -k 443 -o 2", although I note that IANA defines 591, 8008 and 8080 as http_alt or HTTP Alternate, probably just an academic exercise to check the multiport code!

 

[Dixiewrecked]

I fail to see why it cannot to start in the previous test where entware's init.d script can do. I'll save your commands for later testing when I start debugging the crash issue.

http/2 is interesting. pixelserv already supports HTTP pipelining. A quick check says http/2 benefits pixelserv little. I know very little about http/2...so ready to be enlightened One optimisation that I wish to do in pixelserv-tls is to re-use connections that'll make SSL requests even faster.

Pixelserv is still going strong. I also had it generate about a hundred sixty-three certs for ads and no issues so far.
Would you mind taking a look at this article about http/2? http://searchengineland.com/everyone-moving-http2-236716
It discusses many benefits from going from http/1 to http/2. I'm sure eventually, that will be the norm like ipv6. Speaking about ipv6. Will ipv6 be supported in the future by pixelserv? Similar to how ipv4 is handled where we would forward ipv6 requests with a ipv6 listen address.

 

[thelonelycoder]

For AB-Solution users, please read

The Entware package repository now offers the latest version of pixelserv-tls v35.HZ12.Kj for both MIPS and ARM routers.
This update or upgrade replaces the start file /opt/etc/init.d/S80pixelserv-tls with a standard version that will no longer work in AB-Solution.
The solution is as simple as turning ad-blocking (a) off and back on after the update. This writes a new compatible start file and pixelserv-tls will work again. Then purge the old certificates.

See this post for update/upgrade instructions in AB-Solution.

 

[Dixiewrecked]

Okay @kvic,
A week has gone by and pixelserv is still doing its job. No crashes. Only thing I noticed is that a lot of sites when I look at the logs show something like this
doubleclickbygoogle.com GET /favicon.ico HTTP/1.0 secure. That's fine it works, but curious why it says http/1.0 and for other sites its http/1.1. Also, if I use inspect element in the network tab on firefox I see for that exact same site http/1.1. Strange how the logs show one thing, but browser shows another. Not that big of a issue. Its still blocked. Just curious why that is. Also, side note google does use http/2 for some of their sites. Not sure about ads just yet, but google.com is.

 

[jrmwvu04]

.Kj has been up for 3 weeks with only a handful of those errors I mentioned before, so it's rather infrequent on the whole.

May 20 09:31:04 pixelserv[685]: Failed to create conn_handler thread
May 21 13:31:03 pixelserv[685]: Failed to create conn_handler thread
May 21 13:31:03 pixelserv[685]: Failed to create conn_handler thread
May 25 17:20:20 pixelserv[685]: Failed to create conn_handler thread
May 25 17:20:20 pixelserv[685]: Failed to create conn_handler thread
May 25 22:16:55 pixelserv[685]: Failed to create conn_handler thread
Jun  3 21:32:49 pixelserv[685]: Failed to create conn_handler thread
Jun  3 21:32:49 pixelserv[685]: Failed to create conn_handler thread
Jun  3 22:45:53 pixelserv[685]: Failed to create conn_handler thread
Jun  3 22:45:53 pixelserv[685]: Failed to create conn_handler thread
Jun  6 14:27:04 pixelserv[685]: Failed to create conn_handler thread

 

[kvic]

Okay @kvic,
A week has gone by and pixelserv is still doing its job. No crashes.

Good to hear

Only thing I noticed is that a lot of sites when I look at the logs show something like this
doubleclickbygoogle.com GET /favicon.ico HTTP/1.0 secure. That's fine it works, but curious why it says http/1.0 and for other sites its http/1.1. Also, if I use inspect element in the network tab on firefox I see for that exact same site http/1.1. Strange how the logs show one thing, but browser shows another. Not that big of a issue. Its still blocked. Just curious why that is.

To make sure it's not typo in pixelserv, I checked the code. The string shall be from the header sent by your client instead. You can tell from the logs which IP on your LAN making those requests for doubleclickbygoogle.com. I suspect it's a very old client hence HTTP/1.0. But why only this domain...?

Also, side note google does use http/2 for some of their sites. Not sure about ads just yet, but google.com is.

facebook, twitter and cloudflare also support http/2 as well as other major internet firms and CDNs. That's really good news. No wonder my browsing experience has been very vibrant and responsive on a 6-year old Sandy bridge.

 

[kvic]

.Kj has been up for 3 weeks with only a handful of those errors I mentioned before, so it's rather infrequent on the whole.

May 20 09:31:04 pixelserv[685]: Failed to create conn_handler thread
May 21 13:31:03 pixelserv[685]: Failed to create conn_handler thread
May 21 13:31:03 pixelserv[685]: Failed to create conn_handler thread
May 25 17:20:20 pixelserv[685]: Failed to create conn_handler thread
May 25 17:20:20 pixelserv[685]: Failed to create conn_handler thread
May 25 22:16:55 pixelserv[685]: Failed to create conn_handler thread
Jun  3 21:32:49 pixelserv[685]: Failed to create conn_handler thread
Jun  3 21:32:49 pixelserv[685]: Failed to create conn_handler thread
Jun  3 22:45:53 pixelserv[685]: Failed to create conn_handler thread
Jun  3 22:45:53 pixelserv[685]: Failed to create conn_handler thread
Jun  6 14:27:04 pixelserv[685]: Failed to create conn_handler thread

Searching my past month's log on "conn_handler", I do find one occasion:

pixelserv.log.3:May 11 22:53:38 Phaeo pixelserv[931]: [truncated] (13) 192.168.1.100: collector.githubapp.com GET /github/page_view?dimensions[page]=https%3A%2F%2Fgithub.com%2Fkvic-z%2Fpixelserv-tls%2Fblob%2F5018ef352e4e6ab9a258b8166ce7d337a4221bbd%2Fpixelserv.c&dimensions[title]=pixelserv-tls%2Fpixelserv.c%20at%205018ef352e4e6ab9a258b8166ce7d337a4221bbd%20%C2%B7%20kvic-z%2Fpixelserv-tls%20%C2%B7%20GitHub&dimensions[referrer]=https%3A%2F%2Fgithub.com%2Fkvic-z%2Fpixelserv-tls%2Fsearch%3Futf8%3D%25E2%259C%2593%26q%3DFailed%2Bto%2Bcreate%2Bconn_handler%26type%3D&dimensions[user_agent]=Mozilla%2F5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_12_4)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F58.0.3029.96%20Safari%2F537.36&dimensions[screen_resolution]=2560x1440&dimensions[pixel_ratio]=1&dimensions[browser_resolution]=1099x1343&dimensions[tz_seconds]=28800&dimensions[timestamp]=1494496418824&dimensions[request_id]=D747%3A2623%3A6E6B71E%3AA8434FE%3A591434A1&dimensions[user_id]=13570418&dimensions[user_login]=kvic

This is a tracker by GitHub but intercepted and stopped by DNS adblocker.

"Fail to create conn_handler thread" indicates at that moment a new pthread cannot be created and hence the ad request fails to be served. Would be interesting to understand why that happens. It could be due to heavy system load or a bug in the ageing pthread library in ASUSWRT. Either way not much could be done.

I'm finding time to review pthread codes in pixelserv-tls. First to iron out any potential instability bugs. Next to re-use SSL connections for lighning fast responses.

 

[kvic]

I've created a page on my blog for pixelserv-tls. You can visit by going to: https://kazoo.ga/pixelserv-tls/.

Its purpose is to receive feedback on a timely fashion as I found the email notification by Disqus is very reliable. So consider that page as back-up to this thread and vice versa.

Hopefully more info will find time to be add there too.

 

[thelonelycoder]

I've created a page on my blog for pixelserv-tls. You can visit by going to: https://kazoo.ga/pixelserv-tls/.

Its purpose is to receive feedback on a timely fashion as I found the email notification by Disqus is very reliable. So consider that page as back-up to this thread and vice versa.

Hopefully more info will find time to be add there too.

As far as I can tell, your contribution to pixelserv and other projects, and your continued support here with detailed answers to questions looks like more than just a hobby to me.
But I understand, as in my case with AB-Solution, you treat it like a hobby and write off the many hours you spend on it as relaxation and following your passion and the willingness to contribute to the open source community.
Some users are more grateful than just say "Thanks for what you do" and would like to express their gratitude with a small monetary contribution.
I'd be the first to donate if you were to add a link on your website.
Just a hint from a happy user.

 

[kvic]

As far as I can tell, your contribution to pixelserv and other projects, and your continued support here with detailed answers to questions looks like more than just a hobby to me.
But I understand, as in my case with AB-Solution, you treat it like a hobby and write off the many hours you spend on it as relaxation and following your passion and the willingness to contribute to the open source community.
Some users are more grateful than just say "Thanks for what you do" and would like to express their gratitude with a small monetary contribution.
I'd be the first to donate if you were to add a link on your website.
Just a hint from a happy user.

I don't have plan to accept donation which might limit my freedom of choices. However, thanks for your warm words.


Separately I have a plan to look into offering a service for bypassing GFW. To allow people freedom of access to open information in restricted countries. Start with China. People living/working there can drop me a line to express your interest for a trial when it's ready.

 

[truglodite]

kvic, seems donations would have the opposite affect on your freedom of choice.

OT, I naively posted a pixelserv-tls question in lonelycoder's thread (fittingly n00bish). Anyhow, I would like to expand my understanding of the values shown in "/servstats", beyond the few words already typed in your script.

Below are my stats after a recent router reset (before with several days uptime, the numbers were of course much larger, but the relative proportions don't change much). The server appears to be working perfectly on the clients end (no ads, nor grey squares with pixelated frowns). The http/https doubleclick.net test also shows proper response.

Is there a reason why my cls (~=req?) seems much higher than the other examples posted here? To my unskilled brain, that reads "my pixelserv isn't correctly connecting and serving it's pixels". Considering everything seems to work as it should, should I be worried about this, or is this just a meaningless side effect resulting from my setup and content use? FWIW, I've got a pretty basic ab-solution setup (with script installed dnscrypt)... no other issues really... not even sure this is an 'issue'.

Thanks,
Kev

 

[jrmwvu04]

Is there a reason why my cls (~=req?) seems much higher than the other examples posted here? To my unskilled brain, that reads "my pixelserv isn't correctly connecting and serving it's pixels". Considering everything seems to work as it should, should I be worried about this, or is this just a meaningless side effect resulting from my setup and content use? FWIW, I've got a pretty basic ab-solution setup (with script installed dnscrypt)... no other issues really... not even sure this is an 'issue'.

Nothing to be concerned about. I used to have a very similar cls% as well, a few moons ago. One thing I might suggest you try to see if it works for you is running with the "-o [number]" option to cap the timeout of a request, as I see your tmx is 10 seconds and your average as such is higher than I'm used to. I use "-o 1" myself. I got a more appreciated result with that. If you are using ab solution I think you have to maybe set the option through there, though I'm not sure. I'm not a user myself.

 

[slobodan]

Separately I have a plan to look into offering a service for bypassing GFW. To allow people freedom of access to open information in restricted countries. Start with China. People living/working there can drop me a line to express your interest for a trial when it's ready.

Just inquiring about such software could land someone in jail. You're playing with human lives.

 

[mstombs]

Is there a reason why my cls (~=req?)

I started looking at that some time ago but didn't get too far. I'm sure it is not a coincidence the number of cls is very close to the total number of https (all types), I suspected it must be part of the handover from initial connection to the secure one? There are actually 2 routes to get the FAIL_CLOSED status, maybe the second one - "no data received" should not always be included in stats?

https://github.com/kvic-z/pixelserv-tls/blob/master/socket_handler.c#L614

 

[truglodite]

Nothing to be concerned about. I used to have a very similar cls% as well, a few moons ago. One thing I might suggest you try to see if it works for you is running with the "-o [number]" option to cap the timeout of a request, as I see your tmx is 10 seconds and your average as such is higher than I'm used to. I use "-o 1" myself. I got a more appreciated result with that. If you are using ab solution I think you have to maybe set the option through there, though I'm not sure. I'm not a user myself.

I appreciate the advice. I did try playing with the -o option. Values ranging from 1 to 15 didn't reduce the cls%.

Mstombs, that sheds a little more light... I know near zero about how https works. I skinned over the code and was also attracted to socket_handler.c... seems that's where the data gets piped to /servstats. Most of it is over my head, but it did seem to increment disconnect data by parsing the text of 'syslog' events. Without proper understanding of https connections though, I can't say whether commenting out some piping would make servstats more 'sensible'? Perhaps my lack of https knowledge is my real problem.

Not sure if it makes a diff... but I am running an n66u on a 25/35mbit connection (fiber in home... speedtests match/exceed what I pay for 24/7)... I know... I really need to upgrade. Maybe this config may lead to timing issues?

Kev

[edit: I figured this out, I think.

Do clients without a imported Pixelserv root certificates result in a very high cls? ...

I just imported root certs from my router (/opt/var/cache/pixelserv/ca.crt after installing ab-solution) to a couple of test clients and started hammering some ad servers. The cls was pretty much static during this test (req and slh climbed thousands without a single cls++). I think this is the proverbial nail in the coffin.

I've got lots of clients on my home network (different os's, many 'internet only guests' hanging out). I don't mind importing the cert on my family's devices just to keep the bits choochin' like they're supposed to. Would guest devices (without imported certs) make the system work harder in any way? In other words, do 'client disconnect before connection accepted' events result in more cpu overhead?

I also will play with lowering -o... since it seems killing it early might make some pages load faster, with (I think) no real consequence other than more disconnects.]

 

[jrmwvu04]

Do clients without a imported Pixelserv root certificates result in a very high cls? ...

I just imported root certs from my router (/opt/var/cache/pixelserv/ca.crt after installing ab-solution) to a couple of test clients and started hammering some ad servers. The cls was pretty much static during this test (req and slh climbed thousands without a single cls++). I think this is the proverbial nail in the coffin.

I've got lots of clients on my home network (different os's, many 'internet only guests' hanging out). I don't mind importing the cert on my family's devices just to keep the bits choochin' like they're supposed to. Would guest devices (without imported certs) make the system work harder in any way? In other words, do 'client disconnect before connection accepted' events result in more cpu overhead?

I also will play with lowering -o... since it seems killing it early might make some pages load faster, with (I think) no real consequence other than more disconnects.]

That’s an interesting idea. I imported the certs on all the rest of the devices on my network that allow for that around the same time I played with the timeout values. There was something on my network routinely taking 10s and timing out so my thought was might as well kill the connection quickly if it’s going to time out anyway, but you may be on to something there.

 

[truglodite]

'...devices on my network that allow'

I didn't know how challenging it would be to add certs to all my usual clients. Older hardware can handle user certs in weird ways. For example you have to have a pin screen lock for user certs to work on my a couple tab4's my kids use (kit kat). The devices were swipe open, and of course they cause the most ad hits (kids twitchy browsing habits + aggressive marketing to youths = tons of dns queries). Anyhow, for me changing those devices from swipe to pin wasn't worth it... still works fine other than illogical disconnect values in servstats.

On a side note... got my first ab-solution weekly email... friggin' awesome!

Kev

 

[thelonelycoder]

On a side note... got my first ab-solution weekly email... friggin' awesome!

 

[jrmwvu04]

I figured this out, I think.

Do clients without a imported Pixelserv root certificates result in a very high cls?

It seems they do. Good catch.

On a lark, I decided to reboot the router and remove the cert from my machine and browse around a bit. Results would seem to suggest you found the cause.

 

[truglodite]

@thelonelycoder ...well deserved beverage money sent via paypal.

@jrmwvu04 I got my kids over the fact that they have to type a pin... actually they got all excited about the new "secret codes" to lock their devices, LOL! After doing that, our cls is less than 1/10 of slh. So I am also very confident cert-less clients do something (?) to contribute to cls. I suppose the clients reject the secure connection to pixelserv since it doesn't match the real ad server's cert... or something to that effect (like an slm, except we already have the stored cert on the router). I'm learning about https, and still have hardly a clue as to what goes on under pixelserv's hood when a cert-less client is at play... cool how it seems to still work the same (for the client) with or without the cert. I vaguely recall reading about 'connections fail gracefully' in the pixelserv docs somewhere; swear it had to do with what we are discussing here. I'll link to it if I find it.

Kevin