pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[swetoast]

Fairly easy just install it and add optional paramaters to the /opt/etc/init.d/S80pixelserv-tls file and either use AB Solutions or my script (ublockr) for making a hostfile for all those pesky ad sites

 

[kvic]

I decided to finally set up the tls version last night since it was included in the Entware repository. There's also a package for easyrsa. Getting it set up was a little different from the guide in the OP but it was close enough that I was able to figure it out. Very pleased with it so far.

I haven't tried the easyrsa package from Entware-ng. I believe it's the older 2.x version.

The new EasyRSA 3.0 is even simpler to use. For crypto newbies, it's fool proof IMO. For semi/pro's, data files/directories are better organised and more coherent.

The package is only a few KiB mainly one shell script driving command line tools from OpenSSL libraries which perform heavy lifting.

For Asuswrt/Merlin users, OpenSSL comes preinstalled with the firmware. Only have to download EasyRSA 3.0 to your router (e.g. /tmp since it's tiny). Here is one guide to generate ca.crt and ca.key for pixelserv-tls.

Once finished, free to trash EasyRSA 3.0

Just a quick bootcamp for potential users.

 

[kvic]

Fairly easy just install it and add optional paramaters to the /opt/etc/init.d/S80pixelserv-tls file and either use AB Solutions or my script (ublockr) for making a hostfile for all those pesky ad sites

Indeed. Let's also not under estimate the challenge of the community here..

 

[bayern1975]

i opened ten links after reboot router and see at serverstats very high usage req:279, cls:2017 and slh:215.....is this normal?

 

[kvic]

i opened ten links after reboot router and see at serverstats very high usage req:279, cls:2017 and slh:215.....is this normal?

Some clients on your LAN are "abnormally" aggressive in accessing adservers.

They're blocked and exposed.. I won't worry too much.

If people are curious, you can turn on access log in /opt/etc/init.d/S80pixelserv-tls with command line option ( -l ). Then reboot..You'll be able to see who (both which clients and what ad servers) are the offenders in syslog.

Check README for how-to on Github.

 

[mstombs]

@bayern1975 the stats are cumulative, not difficult to block hundreds in minutes!

@kvic in the stats above the numbers do not seem to add up - is there overlap between cls and slh which explains why the sum doesn't add up to req?

An old feature is that max and average times are distorted by those requests that are abandoned due to timeouts, would be more interesting to know timings of requests that are completed successfully - including those that need certificate generation?

 

[kvic]

@kvic in the stats above the numbers do not seem to add up - is there overlap between cls and slh which explains why the sum doesn't add up to req?

Recalling from my memory: after a SSL request is decrypted, I pass it to process just like a regular non-SSL request.

Hence, a good SSL request is counted in both slh plus another sub-categories.

Looking at bayern1975's example. 279 != 7 + 207 + 13 + 10 + 21 + 1 + 6 + 2 + 10.

We're missing 2 not accounted..

An old feature is that max and average times are distorted by those requests that are abandoned due to timeouts, would be more interesting to know timings of requests that are completed successfully - including those that need certificate generation?

Ah. No wonder tmx is always 10s. I'll find time to re-visit. I think we shall exclude the processing time of the tmo requests.

 

[Nullity]

Chrome offers an "Allow invalid certificates for resources loaded from localhost." option in chrome://flags

(Sorry if this was well-known...)

Edit: "flags" not "flag"

 

[jrmwvu04]

I haven't tried the easyrsa package from Entware-ng. I believe it's the older 2.x version.

The new EasyRSA 3.0 is even simpler to use. For crypto newbies, it's fool proof IMO. For semi/pro's, data files/directories are better organised and more coherent.

The package is only a few KiB mainly one shell script driving command line tools from OpenSSL libraries which perform heavy lifting.

For Asuswrt/Merlin users, OpenSSL comes preinstalled with the firmware. Only have to download EasyRSA 3.0 to your router (e.g. /tmp since it's tiny). Here is one guide to generate ca.crt and ca.key for pixelserv-tls.

Once finished, free to trash EasyRSA 3.0

Just a quick bootcamp for potential users.

I had some issues with mktemp, as I saw pointed out earlier in this thread. Rather than modify anything, I just installed mktemp in Entware and was able to otherwise proceed according to the information you provided. The process was a bit smoother in 3.0, you were right. Really great work on pixelserv-tls.

 

[kvic]

Chrome offers an "Allow invalid certificates for resources loaded from localhost." option in chrome://flag

I didn't know this flag. Thanks for sharing.

When I was running WebUI over HTTPS long ago, I kept a set of key.pem and cert.pem (backup from /etc). Re-apply after every firmware upgrade.

Since this thread had gone through lot of effort of creating ca.crt and ca.key for pixelserv-tls. Why not make good use of it for WebUI?

Share the same server certificate between Pixelserv and WebUI

(Assume people already have enabled WebUI over HTTPS)
$ cp /opt/var/cache/pixelserv/ca.crt /etc/cert.pem
$ cp /opt/var/cache/pixelserv/ca.key /etc/key.pem
$ nvram set https_crt_save=1
$ nvram set https_crt_file=""
$ service restart_httpd​

This will switch WebUI to use the pixelserv CA certificate. Persistent across reboots and firmware upgrades.

Now on client side, we only have to import ca.crt once as outlined previously in the thread. Save lot of hassle, especially for people running Kaspersky or the like.

I haven' tried myself. Let's know the result if someone is going to try..

 

[kvic]

I had some issues with mktemp, as I saw pointed out earlier in this thread. Rather than modify anything, I just installed mktemp in Entware and was able to otherwise proceed according to the information you provided. The process was a bit smoother in 3.0, you were right. Really great work on pixelserv-tls.

You're right. mktemp isn't included in Merlin. EasyRSA needs it as well as OpenSSL. People can download from Entware-ng. The package name is coreutils-mktemp.

 

[bayern1975]

i miss adblock database hosts here with pixelserv-tls. now i have to run ab-solution and pixelserv-tls to working ok.....

 

[RMerlin]

Note that Asuswrt-Merlin includes EasyRSA in the /rom folder. It's not v3, but it still works.

 

[kvic]

An old feature is that max and average times are distorted by those requests that are abandoned due to timeouts, would be more interesting to know timings of requests that are completed successfully - including those that need certificate generation?

mstombs, thanks for pointing out the timeout.

Change made: exclude tmo from tav and tmx.

The average and max process time look more meaningful:

About SSL requests...

SSL are included already as decrypted requests are treated just like non-SSL requests.

Cert gents are handled by an async thread/process. Current treatment gracefully fail such requests when cert generation submitted to queue.

It takes ~500ms on a 1GHz ARM7 for generating one cert. It's possible to wait until completion, reload and reply back. It's not done this way ATM. To keep it simple..

 

[kvic]

For people running pixelserv, really shall find time to have top or htop open on your router. Go to your favourite sites (not snbforums of coz..). Look at the number of pixelserv processes on top/htop.

Each ad request spawns off a pixelserv (and dies when completes). On an ad-busy site, the list could stay a few seconds or longer. Spectacular moments.

For every bit of information, people are probably pulling 5 bits of ads. Even with browser plugins, bandwidth is still wasted because ads are retrieved first and then filtered locally.

Only DNS based adblock saves bandwidth (and the world)..

 

[mstombs]

Hopefully the number of pixleserv processes does drop back to one... BTDTGTTS. Real web servers normally have a fixed number of threads and farm out requests between them avoiding the overhead of the OS creating and killing processing (without leaving zombies) you also notice the PID of new processes increases a lot, the first master low number one will stay.
Not unusual to catch processes sending MB of data to hosts diverted to pixelserv, they either use incredibly long request lines (with data after ?) or are blindly html POSTing or whatever methods are used to transfer cookie data... Not necessarily malicious, one I had was a Cyanogen mod Android phone sending crash dump data, presumably with the best of intentions!

 

[kvic]

Hopefully the number of pixleserv processes does drop back to one... BTDTGTTS.

Depends. lol..

On a platform that support native pthread, it's. I think that's the case for Entware-ng. One main thread + one thread for cert generation.

On a platform with old linuxthread, there will be three processes minimum. One main process. Another for cert gent. Third is the dummy culprit of linuxthread.

 

[mstombs]

Using mips Entware version on N66 it is 2 main threads consecutive PID the cert generation runs as admin, the old main as unprivileged 'nobody'.

Have tried built-in easyrsa and openvpn/easyRSA-3.0.1 on router, latter needed the mktemp package as above.

So far have only managed to get Firefox to use the generated cert on win7 laptop. Google Chrome/IE refuses to play. Might be issue with corporate/domain/antivirus PC with some locked down settings. Is there a definitive guide for each OS/browser (Trusted root authoritative stores etc)?

pixelserv-tls version: V35.HZ12.Kh compiled: Mar 26 2016 12:16:28 options: 192.168.66.254 -p 80 -p 81 -p 8080 -p 8081 -k 443 -o 2
982 uts, 43 req, 501 avg, 808 rmx, 70 tav, 1993 tmx, 0 err, 1 tmo, 15 cls, 0 nou, 0 pth, 4 nfe, 1 ufe, 1 gif, 0 bad, 1 txt, 1 jpg, 2 png, 1 swf, 5 ico, 22 slh, 0 slm, 0 sle, 0 slu, 2 sta, 2 stt, 0 204, 7 rdr, 0 pst, 0 hed, 0 log

That's

slh: # of HTTPS /w a good cert 24

 

[kvic]

So far have only managed to get Firefox to use the generated cert on win7 laptop. Google Chrome/IE refuses to play. Might be issue with corporate/domain/antivirus PC with some locked down settings. Is there a definitive guide for each OS/browser (Trusted root authoritative stores etc)?

Except Firefox, everything else uses the certificates from Windows security vault. Here is a guide to import the CA cert into there:

https://support.comodo.com/index.php?/Default/Knowledgebase/Article/View/636/17/

 

[mstombs]

Except Firefox, everything else uses the certificates from Windows security vault. Here is a guide to import the CA cert into there:

https://support.comodo.com/index.php?/Default/Knowledgebase/Article/View/636/17/

Thanks I didn't use the method to access, can already export/import via Google chrome advanced settings or IE settings - but it now works - I think the key step was to import the certificate into both root and intermediate as detailed in your link, had previously also tried personal and auto... I have also used the same certs for the https web gui as above - and have got rid of broken padlock in several combinations of PC/browser.

One curiosity, can collect pixelserv stats using https and a blocked domain i.e.

https://doubleclick.net/servstats

but not direct by IP address

https://192.168.66.254/servstats

This site can’t provide a secure connection
192.168.66.254 uses an unsupported protocol.

ERR_SSL_VERSION_OR_CIPHER_MISMATCH