pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[kvic]

I'm not sure what the ARCH define does, it is actually used in the Makefile before defined so good job it is correct by default!

It's an ugly trick of mine for linking openssl lib of different architectures. "=" and ":=" assignments are different. One is like a constant. The other is like a pointer whose value is only determined at evaluation upon actual use. I think there is a better way to get this done.

STRIP still breaks the dynamic build (noted in the Makefile), so commented out, UPX works but is slow and not really needed! Too many warnings for my like

strip broke again for amd64 too. No issue for arm & mips. Haven't looked into it yet..I had the exact sentiment on UPX. It bloats up the size of both binary image and runtime RAM use. I don't get why it was added in the first place. So commented out in all the builds I created.

asprint warning in certgen is a recent one. It's one of the only two warnings I saw in arm, mips & amd64. I did a sweep to clean up all the warnings at the very beginning. The other warning is on LINE_NUMBER defined not used.

The binaries run, but I still don't know how to use the ssl certs...

See if this gets you started..."Prepare your Root CA cert" section from https://github.com/kvic-z/pixelserv-tls

 

[kvic]

Or do I have to manually import EVERY cert that's generated? Any thoughts?

We don't need to important every single cert. If we have to, then the whole purpose of my design is defeated. The design is to import the Root CA cert once on the client OS and works for all certs generated by pixelserv for all ad domains.

It's also worth pointing out another mis-conception. Under normal browsing, users won't be prompted to accept new certs when HTTPS connections are made to ad servers in background _even if_ the Root CA is not imported into client OS.

Such HTTPS requests to pixelserv will silently fail like as-before. So for example guests visiting you and connecting to your Wifi.. there is no way they will be troubled to import a Root CA cert or else face constant bombing to accept new certs. Let's stop this FUD...

all sites still give NET::ERR_CERT_AUTHORITY_INVALID in Chrome and a blank page in internet explorer. I have the cert imported as a Trusted Root CA authority, chrome even shows it as the certificate it flagged as bad.

This indicates that the Root CA cert is not properly imported into your client OS.

To import a Root CA cert in Windows, have to be done from Management Console. Here is a walkthrough. Worth pointing out that we cannot import a Root CA cert from within the browser like Chrome. I think the browser simply imports a Root CA cert as a user cert.

 

[mstombs]

It's an ugly trick of mine for linking openssl lib of different architectures. "=" and ":=" assignments are different. One is like a constant. The other is like a pointer whose value is only determined at evaluation upon actual use. I think there is a better way to get this done.

Neat - I see that now, thanks!

strip broke again for amd64 too. No issue for arm & mips. Haven't looked into it yet..I had the exact sentiment on UPX. It bloats up the size of both binary image and runtime RAM use. I don't get why it was added in the first place. So commented out in all the builds I created.

Its the removal of GNU info by STRIP that breaks the binaries, this seems to work

# aggressive strip command
# note that this breaks the x86 build on x86-64 for some reason
STRIP     := strip -s -R .note -R .comment
# -R .gnu.version -R .gnu.version_r

presumably single architecture router firmwares do not need the info in these sections, and it saves a few bytes, I probably copied this STRIP command from elsewhere in Tomato build, but not much effect here

Without strip

-rwxrwx--- 1 root vboxsf 34668 Nov 18 23:13 pixelserv.x86.performance.dynamic*
-rwxrwx--- 1 root vboxsf 2085456 Nov 18 23:13 pixelserv.x86.performance.static*

With

-rwxrwx--- 1 root vboxsf   34544 Nov 19 09:14 pixelserv.x86.performance.dynamic*
-rwxrwx--- 1 root vboxsf 2085328 Nov 19 09:14 pixelserv.x86.performance.static*

So no point in bothering!

UPX works, only use is to make the smallest binary size, by more than 50% where used

-rwxrwx--- 1 root vboxsf   15876 Nov 19 09:16 pixelserv.x86.performance.dynamic*
-rwxrwx--- 1 root vboxsf  846748 Nov 19 09:16 pixelserv.x86.performance.static*

 

[shooter40sw]

Thanks for the info, I hace ssh on port 443, and I cant move that port, I was reading that it was possible to move the pixelserv to another port, but I dont know if this is possible with this version. About how much performance is gained with this compared with the host only version?
Thanks

A step by step guide will take much time to draft and make it near faultless...so personally I don't plan to do so. However, I can sketch high-level steps (without going into exact details upfront):

1. Move your WebUI to run on HTTPS with port 8443. This will free up port 80 for pixelserv.
2. Rename ip address of a few select entries (e.g. doubleclick.net is a good candidate; t's for pilot test) in your block list to point to your.router.ip.address. Restart dnsmasq.
3. Start pixelserv from command line with "/path/to/pixelserv your.router.ip.address"
4. Test with "http://your.router.ip.address/servstats". This will show you a page similar to the running stats in post #1.
5. Test with "http://doubleclick.net/" . This will show you a blank page without error.
6. Up to this point everything looks good then. Proceed to setup auto run of pixelserv on startup by deploying the init.d script in post #1.
7. Then modify your dnsmasq block list scripts to use your.router.ip.address instead of 0.0.0.0.
8. Follow the guide in post #1 to setup a CA cert. After that certs for ad domain will be auto generated by pixelserv.
9. Up to this point, I believe you shall be all set.

I hope you can work out the details. Feel free to ask us if you run into errors.

 

[mstombs]

pixelserv can be run on specific ports "-p 81 -p8080" etc BUT you would then need to use a different IP and have an iptables rule to divert https requests to the other port. The Asus web gui hogs all router IP addresses and port 443 which is the default from your browser for https. Asuswrt merlin supports changing the web ports.

 

[Scacht]

We don't need to important every single cert. If we have to, then the whole purpose of my design is defeated. The design is to import the Root CA cert once on the client OS and works for all certs generated by pixelserv for all ad domains.

It's also worth pointing out another mis-conception. Under normal browsing, users won't be prompted to accept new certs when HTTPS connections are made to ad servers in background _even if_ the Root CA is not imported into client OS.

Such HTTPS requests to pixelserv will silently fail like as-before. So for example guests visiting you and connecting to your Wifi.. there is no way they will be troubled to import a Root CA cert or else face constant bombing to accept new certs. Let's stop this FUD...



This indicates that the Root CA cert is not properly imported into your client OS.

To import a Root CA cert in Windows, have to be done from Management Console. Here is a walkthrough. Worth pointing out that we cannot import a Root CA cert from within the browser like Chrome. I think the browser simply imports a Root CA cert as a user cert.

Walk-through solved it, didn't know it needs to both be trusted and intermediate. Additionally, I was probably putting it as a user cert. Secondary issue I had was the system generating the cert was 3 seconds faster than my network (make sure they're all using the same timeserver), so I got invalid date on cert generations. Everything is as smooth as butter now and absolutely seamless. Thanks!

 

[kvic]

Thanks for the info, I hace ssh on port 443, and I cant move that port, I was reading that it was possible to move the pixelserv to another port, but I dont know if this is possible with this version.

Yes, it's possible to use other ports pixelserv-tls. However, the intended use of port options might not be what you thought in any pixelserv..

Your SSHD need 443 because of enabling on WAN? I never tried SSHD on WAN so I'm not sure if it is as silly as WebUI that binds to all interfaces. As long as it's not, it won't interfere with any version of pixelserv. Otherwise, some workaround is required.

About how much performance is gained with this compared with the host only version?
Thanks

In older browsers, I think the performance boost (i.e. reduction in page load time) is significant and very noticeable. In modern browsers, less obvious, especially to users with a powerful PC.

To more sophisticated users, a few less socket open and shutdown, a few less HTTP requests, a few milliseconds saved here and there, all matter. That's the bottom line of what you can get from using pixelserv.

pixelserv-tls also has a feature to log all ad requests. This helps to easily identify what hosts to whitelist in situation where a website malfunctions due to excessive blocking.

Statistics of ad requests may also matter to some users. Like how frequent your browser is sending info to ad servers, what size of the payload they send. Isn't it alarming if one tracker is uploading 30KiB of data collected from you?

Personally I'm also interested in the growth of SSL ad. E.g. from my browsing habit, SSL ad represents ~10% of the volume at the moment...

EDIT:

Forgot to mention that..I found on client OS without browser adblock plugins to help perfecting page layout as a result of blocked ad, with pixelserv gives a cleaner presentation.

For example, even though iOS 9 comes with ability of client side adblock, I still don't need one with my setup of dnsmasq based blocking+pixelserv.

 

[shooter40sw]

Hi, yes I got sshd open on WAN side, it seems like I have to make a lot of stuff to implement this, , I´ll need to do this on a weekend, I have a relatively modern pc, but my ISP is high latency with a 4mbit

Yes, it's possible to use other ports pixelserv-tls. However, the intended use of port options might not be what you thought in any pixelserv..

Your SSHD need 443 because of enabling on WAN? I never tried SSHD on WAN so I'm not sure if it is as silly as WebUI that binds to all interfaces. As long as it's not, it won't interfere with any version of pixelserv. Otherwise, some workaround is required.



In older browsers, I think the performance boost (i.e. reduction in page load time) is significant and very noticeable. In modern browsers, less obvious, especially to users with a powerful PC.

To more sophisticated users, a few less socket open and shutdown, a few less HTTP requests, a few milliseconds saved here and there, all matter. That's the bottom line of what you can get from using pixelserv.

pixelserv-tls also has a feature to log all ad requests. This helps to easily identify what hosts to whitelist in situation where a website malfunctions due to excessive blocking.

Statistics of ad requests may also matter to some users. Like how frequent your browser is sending info to ad servers, what size of the payload they send. Isn't it alarming if one tracker is uploading 30KiB of data collected from you?

Personally I'm also interested in the growth of SSL ad. E.g. from my browsing habit, SSL ad represents ~10% of the volume at the moment...

EDIT:

Forgot to mention that..I found on client OS without browser adblock plugins to help perfecting page layout as a result of blocked ad, with pixelserv gives a cleaner presentation.

For example, even though iOS 9 comes with ability of client side adblock, I still don't need one with my setup of dnsmasq based blocking+pixelserv.

 

[Jarnootje]

I've f*cng done it! As a noob with this stuff it has cost me a great couple of hours to read and do on the forums and githubs. But my AC68u is flashed with Merlin @316mw persistent over boots/new updates, usb ext2 with entware, dnsmasq host file setup incl whitelist (updatescript changed from 0.0.0.0 to match 10.8.10.8 pixelserv, no | sort for speedup, same applies for dnsmasq-add entries), iptables prerouting for pixelserv, pixelserv alternative port (in firstpost -p 8080 is missing, and my binary from was not renamed ). And CHMOD'ed 777 everything (actually, winscp >properties > )

But the HTTPS ad redirecting/certificates is where I stop. Hopefully read later on the forums if changes are made and it is easier to setup and no root crt's are necessary for devices (I know I get a error on this https ads and no 1px , but absolute no problem)

Great work all! Try to make first post more 'newb'-friendly.

 

[kvic]

It's encouraging to see more ppl give it a try!

Here is Step-by-Step guide to create the Root CA cert right on your Merlin routers

• mkdir -p /jffs/tmp_dir
  
• cd /jffs/tmp_dir
  
• wget --no-check-certificate -O - https://github.com/OpenVPN/easy-rsa/releases/download/3.0.1/EasyRSA-3.0.1.tgz | tar xzf -
  • Note that the above is one single line...
• cd EasyRSA-3.0.1
  
• ./easyrsa init-pki
• ./easyrsa build-ca nopass
  • when prompted for Common Name, simply put "Pixelserv CA" without quotes

Now you can find "ca.crt" and "ca.key" under "pki" and "pki/private" sub-dir respectively. Simply move them to your Pixelserv-tls dir e.g

• mv pki/ca.crt /opt/var/cache/pixelserv
• mv pki/private/ca.key /opt/var/cache/pixelserv

pixelserv-tls will start autogen certs upon next HTTPS requests. Once it's working, let's clean up the tmp_dir by

• rm -rf /jffs/tmp_dir

That's it.

EDIT:

Fixed EasyRSA link. They changed their repository name from "easyrsa" to "easy-rsa" on Github.

 

[Jarnootje]

It's encouraging to see more ppl give it a try!
.....
That's it.

With this the alerts in router_log like:
pixelserv[1893]: Sending HTTP 501 response for unknown HTTP method or non-SSL, non-HTTP request: ^V^C^A
will be gone? And about importing the root certificate on clients (Android, Windows etc), is this still 'necessary' or is Root CA on router sufficient and doing everything now? /jffs/tmp_dir can be another folder on usb_ext2 mount right or will it give problems with EasyRSA and generating Ca's?

If autogen certs on https doesn't slow browsing noticeable (like >10ms) than I will try this to

Need the router a reboot or pixelserv a restart / kill-restart? (I know how to restart/kill etc now )

Thanks in advance! (using pixelserv-tls V35.HZ12.Kf binary already)

 

[kvic]

With this the alerts in router_log like:
pixelserv[1893]: Sending HTTP 501 response for unknown HTTP method or non-SSL, non-HTTP request: ^V^C^A
will be gone?

501 is standard HTTP/1.1 response which indicates that your browser is sending a request of unknown method in the HTTP header.

If it happens frequently and you want to know what it's. You can enable logging with -l to see what's the requests when this error is generated.

Ad agency/trackers could be doing all sorts of weird things in background...

And about importing the root certificate on clients (Android, Windows etc), is this still 'necessary' or is Root CA on router sufficient and doing everything now?

The same Root CA cert (/opt/var/cache/pixelserv/ca.crt) need to be imported into client OS. Only once. That's how PKI works.

/jffs/tmp_dir can be another folder on usb_ext2 mount right or will it give problems with EasyRSA and generating Ca's?

That's right. /jffs/tmp_dir is a temporary working area. I use /jffs out of convenience. It can be on attached USB disk.

If autogen certs on https doesn't slow browsing noticeable (like >10ms) than I will try this to

The first HTTPS request for a new ad server will fail right away in order not to slow down page loading on your browser. Certgen starts at the same time and will finish in less than 500ms on a 1400MHz Cortex-A9 e.g. my overclocked RT-AC56U.

Subsequent requests to the same ad server will succeed. So certgen only happens once for each ad domain. Generated certs are stored for reuse.

From time to time, it's fun to look at the ad domains in /opt/var/cache/pixelserv that you actually attempted to visit...

Need the router a reboot or pixelserv a restart / kill-restart?

Restart of pixelserv shall be sufficient in rare occasions of resolving weird issues.

As I mentioned before, pixelserv will automatically pick up new Root CA cert...and certs it generated for ad domains. No restart of pixelserv is needed.

 

[Jarnootje]

It's encouraging to see more ppl give it a try!

Here is Step-by-Step guide to create the Root CA cert right on your Merlin routers
That's it.

The github link didn't work. I manually downloaded from: https://github.com/OpenVPN/easy-rsa/tree/master/easyrsa3. And had to edit all lines with "mktemp"

(line 457) out_key_tmp="$(mktemp -u "$out_key.XXXXXXXXXX")"; ...

(to example)
out_key_tmp="$out_key.XXXXXXXXXX"; ...

/opt/var/cache/pixelserv/ wasn't created already (also had to do manually). So moved CA files to it and imported on this laptop to test. Rebooted router, but still:
slh: # of HTTPS /w a good cert 0
slm: # of HTTPS w/o a cert 0
sle: # of HTTPS /w a bad cert 0

I get also error on https ads (ssl_error_rx_record_too_long), so it is blocked, but not replaced by pixelsrv.

Pixelsrv runs and works. I also see no Entware running or logs from it on the usb_ext. Could you give some clues?

 

[kvic]

The github link didn't work. I manually downloaded from: https://github.com/OpenVPN/easy-rsa/tree/master/easyrsa3.

Interesting... EasyRSA guys renamed their Github repository from "easyrsa" to "easy-rsa" in the past week or so. That's why it didn't work.

I fixed the guide in my original post. I think it shall be good now.

And had to edit all lines with "mktemp"

(line 457) out_key_tmp="$(mktemp -u "$out_key.XXXXXXXXXX")"; ...

(to example)
out_key_tmp="$out_key.XXXXXXXXXX"; ...

It should work out of the box without any change...if you run on Merlin routers.
The error relates to the shell you're using...

/opt/var/cache/pixelserv/ wasn't created already (also had to do manually).

/opt/var/cache/pixelserv need to be created by users.

I can get pixelserv to create it automatically but I thought it would be more flexible to leave it to users. As not everyone has Entware/Optware installed, they do not have /opt mount.

So moved CA files to it and imported on this laptop to test. Rebooted router, but still:
slh: # of HTTPS /w a good cert 0
slm: # of HTTPS w/o a cert 0
sle: # of HTTPS /w a bad cert 0

I get also error on https ads (ssl_error_rx_record_too_long), so it is blocked, but not replaced by pixelsrv.

Pixelsrv runs and works. I also see no Entware running or logs from it on the usb_ext. Could you give some clues?

If you check syslog.log, what errors from pixelserv do you see?

 

[Jarnootje]

I ran it from AC68u putty commands, gave still errors on mktemp, so had to fix it manually all.

syslog, when https://ad-emea.doubleclick.net/ (Firefox: ssl_error_rx_record_too_long)

Nov 24 13:03:58 pixelserv[3361]: Sending HTTP 501 response for unknown HTTP method or non-SSL, non-HTTP request: ^V^C^A
Nov 24 13:03:58 pixelserv[3361]: 192.168.1.53null) ^V^C^A​

syslog, when http://ad-emea.doubleclick.net/

Nov 24 13:06:41 pixelserv[3365]: 192.168.1.53: ad-emea.doubleclick.net GET / HTTP/1.1
​

My dnsmasq.conf.add
# AdBlocking
address=/10.8.10.8/10.8.10.8
ptr-record=10.8.10.8.in-addr.arpa,10.8.10.8
addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked
addn-hosts=/tmp/mnt/sda1/hosts/blacklist.txt

host.blocked (example line)
10.8.10.8 ad-universe.com

firewall-start
#!/bin/sh
iptables -t nat -A PREROUTING --dest 10.8.10.8 -p tcp -m multiport --dport 80,443 -j DNAT --to-dest 192.168.1.1:8080

DNS from google (only on WAN, not on LAN dhcp server of course)
NAT Acceleration auto
NAT loopback Merlin

Just browsing, couple of minutes:

./pixelserv.arm.performance.static version: V35.HZ12.Kf compiled: Nov 10 2015 16:54:04 options: 192.168.1.1 -p 8080 -u admin -l
926 uts, 268 req, 342 avg, 1952 rmx, 533 tav, 6075 tmx, 0 err, 0 tmo, 15 cls, 0 nou, 0 pth, 8 nfe, 0 ufe, 4 gif, 172 bad, 36 txt, 0 jpg, 0 png, 0 swf, 0 ico, 0 slh, 0 slm, 0 sle, 0 slu, 10 sta, 1 stt, 0 204, 22 rdr, 0 pst, 0 hed

Syslog just browsing, couple of minutes:
https://www.dropbox.com/s/7gtlt2ecnkemcu4/Syslog.txt?dl=0

 

[kvic]

firewall-start
#!/bin/sh
iptables -t nat -A PREROUTING --dest 10.8.10.8 -p tcp -m multiport --dport 80,443 -j DNAT --to-dest 192.168.1.1:8080

Can't forward both http and https to the same port 8080. I suggest u start pixelserv with -p 8080 -k 8088

Then forward 80 to 8080. Forward 443 to 8088. This shall fix ur issue.

In general, I recommend ppl not to do port forward. Instead move webui to run on https with port 8443. Let pixelserv run on port 80 and 443 default.

The rationale is that your ad server access is way more than you'll spend time on webui..

 

[Jarnootje]

Can't forward both http and https to the same port 8080. I suggest u start pixelserv with -p 8080 -k 8088

Then forward 80 to 8080. Forward 443 to 8088. This shall fix ur issue.

In general, I recommend ppl not to do port forward. Instead move webui to run on https with port 8443. Let pixelserv run on port 80 and 443 default.

The rationale is that your ad server access is way more than you'll spend time on webui..

Ahh ok? I read this in firstpost: "...There is a way out. We can keep WebUI as-is and continue to run on port 80. We run pixelserv on an arbitrary port e.g. 8080. But ad requests in general are sent to port 80 & 443. So we need to redirect these requests to pixelserv. This can be achieved with a iptables rule:..."
and "pixelserv can serve both http & https requests over the same port. This is another benefit over lighttpd."

So this would fix it (can it be one statement?):
iptables -t nat -A PREROUTING --dest 10.8.10.8 -p tcp -m multiport --dport 80 -j DNAT --to-dest 192.168.1.1:8080

iptables -t nat -A PREROUTING --dest 10.8.10.8 -p tcp -m multiport --dport 443 -j DNAT --to-dest 192.168.1.1:8088

EDIT -it is working now:
slh: # of HTTPS /w a good cert 106
slm: # of HTTPS w/o a cert 25
sle: # of HTTPS /w a bad cert 0

 

[kvic]

Ahh ok? I read this in firstpost: "...There is a way out. We can keep WebUI as-is and continue to run on port 80. We run pixelserv on an arbitrary port e.g. 8080. But ad requests in general are sent to port 80 & 443. So we need to redirect these requests to pixelserv. This can be achieved with a iptables rule:..."
and "pixelserv can serve both http & https requests over the same port. This is another benefit over lighttpd."

Nice catch. That was written for the old pixelserv where no true https support included. Hence, ppl can do redirection both to the same port.

Later I thought it would be fun to enable https support. So I forked the code and played with opernssl. The end result is the pixelserv-tls on my github.

Not to confuse ppl further. I'm going to remove the spoiler altogether in the first post soon...

 

[Jarnootje]

Nice catch. That was written for the old pixelserv where no true https support included. Hence, ppl can do redirection both to the same port.

Later I thought it would be fun to enable https support. So I forked the code and played with opernssl. The end result is the pixelserv-tls on my github.

Not to confuse ppl further. I'm going to remove the spoiler altogether in the first post soon...

Shame on me, too much read about this and didn't understand everything down to the last detail, so missed this also.

You did a great job. It is working fine now, without GUI changing to https (I don't like sluggish rendering) Thanks in advance for help also. I only hope this 'messing around ' with CA's and iptables and https doesn't make things here more vulnerable for hackers, spy-ware or something. But I think and hope it's just the other way around now, more secure )

 

[bayern1975]

how to install this and for what you using this?