pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[kvic]

but not direct by IP address

https://192.168.66.254/servstats

Ad requests shall not go by ip addresses. One key presumption in DNS based adblock. Or else the whole thing break apart..

The issue happens due to a wild card cert is generated. For an IP address, it shouldn't be a wild card cert but there is no logics to catch and handle this.

You hit an edge case (good catch btw!). It only applies to accessing pixelserv itself for server stats over https.

Hmm..we can add logics to handle this but will run check for all requests. Since all of them aren't this type except this edge case, we probably shouldn't do...

 

[kvic]

Revised post #1 in lieu of pixelserv-tls availability on Entware-ng.

Also updated documentation here: https://github.com/kvic-z/pixelserv-tls#import-cacrt-into-clients.

Particularly, added how to import ca.crt into various client platforms.

 

[jrmwvu04]

This will switch WebUI to use the pixelserv CA certificate. Persistent across reboots and firmware upgrades.

Now on client side, we only have to import ca.crt once as outlined previously in the thread. Save lot of hassle, especially for people running Kaspersky or the like.

I haven' tried myself. Let's know the result if someone is going to try..

Works perfectly. I don't know if it's relevant or not, but I didn't use your CLI method of importing the certificate into OS X 10.11, I used Keychain Access. I assume it achieves a similar effect, though.

 

[kvic]

Works perfectly. I don't know if it's relevant or not, but I didn't use your CLI method of importing the certificate into OS X 10.11, I used Keychain Access. I assume it achieves a similar effect, though.

I use the command line with intent to automate it after every OS update.. I still haven't figured out a way but the command line stays with me. Using Keychain Access indeed achieves the same result.

Good to hear everything works for you.

 

[kvic]

A heads-up.

Adblock space is getting rapport recently with more people dedicating time on developing innovative solutions for a wide range of audience. I feel excited! lol

In light with that, I changed the tagline of pixelserv a bit last night:

pixelserv-tls is a fork of pixelserv with added support for HTTPS - the tiny webserver that responds to all requests with "nothing" and particularly useful for whitelisting hosts on troubled websites, and for mining "big data" on adservers and trackers.
​

Currently pixelserv-tls provides enough info to perform an easy job on the first functionality. See here: http://www.snbforums.com/threads/pi...ebserver-for-adblock.26114/page-7#post-244782

Syslog will have client, adserver, and the exact request URL to judge what blocked adservers possibly giving trouble and to whitelist.

On the second functionality (data mining), currently parsing syslog is considered okay of getting the job done. I think it would be much more interesting if the data is stored in a database. It might be sqlite3 or MongoDb. Will see which one can do a better job with a tiny footprint and perhaps more fun (for me..). And it'll take time..

So just a heads up if more people are going to mine the "big data" from the access log. You might want to design your solution in a way that it can quickly adapt to a different data structure underneath.

The goal of pixelserv is to empower interested people to mine the big data, and present stats in ways that entertain a wide range of users. Pixelserv itself isn't going to do the mining and presentation.

(To manage people's expectation, if it's not emphasised enough, it takes time as I've to find some time..)

 

[mstombs]

There was a report of a pixelserv fork that fed all the info into a remote mysql database - I never saw any sourcecode though.

You do get interesting info on blocked https sites on cert creation, without full syslog - I currently have 40 certificates in the cache!

These are my stats after putting the current Entware mips pixelserv-tls into production here, I just use the small pgl.yoyo dnsmasq domain blocklist as I have done for years! http://pgl.yoyo.org/adservers/news.php#dnsmasq :-

pixelserv-tls version: V35.HZ12.Kh compiled: Mar 26 2016 12:16:28 options: 192.168.66.254 -p 80 -p 81 -p 8080 -p 8081 -k 443 -o 2
uts: Uptime 1 days 18:12
req: Total # of requests 5963
avg: Avg size of reqs 791 bytes
rmx: Max size of reqs 23295 bytes
tav: Avg process time 630 ms
tmx: Max process time 3540 ms
err: # of error reqs 0
tmo: # of client timeout 505
cls: # of client shutdown 2324
nou: # of reqs w/o URL 0
pth: # of invalid URL 0
nfe: # of missing file ext 837
ufe: # of unknown file ext 46
gif: # of GIF reqs 47
bad: # of unknown HTTP methods 3
txt: # of TXT reqs 1360
jpg: # of JPG reqs 3
png: # of PNG reqs 4
swf: # of SWF reqs 2
ico: # of ICO reqs 112
slh: # of HTTPS /w a good cert 2683
slm: # of HTTPS w/o a cert 28
sle: # of HTTPS /w a bad cert 0
slu: # of unrecognized HTTPS 57
sta: # of HTML stats 153
stt: # of text stats 6
204: # of HTTP/204 (no content) 0
rdr: # of redirects 528
pst: # of POST method 7
hed: # of HEAD method 0
log: access log enabled (0=no 1=yes) 0

I did investigate the cls count a bit, using wireshark on PC - I suspect these are mainly https requests from systems/browsers that do not have the certificate imported - no complaints received though!

 

[kvic]

There was a report of a pixelserv fork that fed all the info into a remote mysql database - I never saw any sourcecode though.

Remote database? Sounds like a good idea of some kind of crowd sourced research project!

When I do it, it'll be local database. So that people need not worry about any privacy issue. lol

I did investigate the cls count a bit, using wireshark on PC - I suspect these are mainly https requests from systems/browsers that do not have the certificate imported - no complaints received though!

Thanks for sharing the info. In such cases, clients will gracefully fail as I found most ad/tracker requests are made in the background. Seems to me only foreground requests in browsers will show explicit warnings of failed requests to users...

 

[kvic]

You do get interesting info on blocked https sites on cert creation, without full syslog - I currently have 40 certificates in the cache!

These are my stats after putting the current Entware mips pixelserv-tls into production here, I just use the small pgl.yoyo dnsmasq domain blocklist as I have done for years! http://pgl.yoyo.org/adservers/news.php#dnsmasq :-

40 certificates in one day were fast.

For me only 2 new certs in the past 2 days. Over six months, I only accumulated 283 certs...

$ ll /opt/var/cache/pixelserv | grep -v 'ca.*' | wc -l
283​

After excluding timeouts, the average process time (41 ms) looks much more impressive.

Max process time still 10s. Upon checking that was contributed by one instance of client disconnect. The other 96 cls took much shorter time.

pixelserv-tls version: V35.HZ12.Kh compiled: Apr 17 2016 00:13:59 options: 192.168.1.3
uts: Uptime 2 days 00:04
req: Total # of requests 13789
avg: Avg size of reqs 833 bytes
rmx: Max size of reqs 7051 bytes
tav: Avg process time 41 ms
tmx: Max process time 10001 ms
err: # of error reqs 0
tmo: # of client timeout 161
cls: # of client shutdown 97
nou: # of reqs w/o URL 0
pth: # of invalid URL 0
nfe: # of missing file ext 1737
ufe: # of unknown file ext 1979
gif: # of GIF reqs 1001
bad: # of unknown HTTP methods 4
txt: # of TXT reqs 6514
jpg: # of JPG reqs 0
png: # of PNG reqs 4
swf: # of SWF reqs 0
ico: # of ICO reqs 4
slh: # of HTTPS /w a good cert 1347
slm: # of HTTPS w/o a cert 2
sle: # of HTTPS /w a bad cert 0
slu: # of unrecognized HTTPS 7
sta: # of HTML stats 93
stt: # of text stats 0
204: # of HTTP/204 (no content) 0
rdr: # of redirects 1884
pst: # of POST method 285
hed: # of HEAD method 0
log: access log enabled (0=no 1=yes) 0 ​

 

[bayern1975]

It would be fine if pixelserv have adblock hosts database inside...we still need to have ab solution or similar installed to work pixelserv....

sent from Kodi 17 Krypton

 

[tijaune]

Except Firefox, everything else uses the certificates from Windows security vault. Here is a guide to import the CA cert into there:

https://support.comodo.com/index.php?/Default/Knowledgebase/Article/View/636/17/

This works for me. In Chrome, I clicked on the red X padlock where you can dive in for current router login page certificate, there are some additional drill down to export to a file (example: pixelserv.cer). Then this file can be imported to windows trusted root certification authorities as highlighted by the guide.

 

[tijaune]

Hi kvic,

I just discovered https://outlook.live.com can't sign in anymore. It failed after too many redirect (I don't think ublockr has anything to do). Gmail and Yahoo mail are ok. Any idea to where I can look?

Thanks,

 

[kvic]

Hi kvic,

I just discovered https://outlook.live.com can't sign in anymore. It failed after too many redirect (I don't think ublockr has anything to do). Gmail and Yahoo mail are ok. Any idea to where I can look?

Thanks,

In general, most of the so called hosts files include some entries which perhaps okay for some people but causing issues for others. This is what happened to you. Some hosts are blocked and give trouble to proper functioning of a website.

The action shall be to figure out what theses hosts are, and white list them in the script that aggregate the hosts files. For one website, usually only one or two hosts to white list.

People can use pixelserv to spot potential candidates of such hosts by inspecting the access log. In the end, it'll be a trial and error game. But with info such as hostname and URL (..very indicate IMO..) the job is easier with pixelserv.

Step 1: Turn on pixelserv's access log (need not to restart pixelserv)

$ wget http://doubleclick.net/log=1

Step 2: Display the log with relevant entries

$ grep pixelserv /tmp/syslog.log

Step 3: Go to your client browser and launch the troubled website e.g. https://outlook.live.com

Step 4: Look at the output of Step 2, and find what hosts are blocked during your engagement with the website in Step 3.

Step 5: Turn off pixelserv's access log

$ wget http://doubleclick.net/log=0

Now you can white list all hosts found in Step 4 which certainly will make the website work, but then some ads will creep in. So by trial and error, you can further narrow down the hosts to white list.

For actual white listing a host, need to go back to the script that prepare the hosts file. In your case, it'll be ublockr. After that, restart dnsmasq. Then flush DNS cache and browser cache on your client machine.

 

[swetoast]

Hi kvic,

I just discovered https://outlook.live.com can't sign in anymore. It failed after too many redirect (I don't think ublockr has anything to do). Gmail and Yahoo mail are ok. Any idea to where I can look?

Thanks,

found a issue it gets login.live.com from one of the blocklists gonna research whats causing it and make a whitelist for domains like that

nice catch ill try to fix it as soon as possible

 

[tijaune]

well thanks to both of you, i'll certainly try out kvic detailed process as exercise. swetoast, last night I modified ublockr to not use no.list and hotmail worked again, I did not have time to single out that list.

btw when I was using thelonelycoder solution before, I have to omit 1 source from http://hosts-file.net/ad_servers.txt ... Expedia fetch their hotel search from some ads providers probably and their site just keep spinning after a search.

 

[swetoast]

i modified ublockr so there is an updated script that omits just that line gonna make a whitelist when im feeling better so if you or anyone else finds more domains just add em on my bugtracker at gitlab

 

[mstombs]

Bug report: pixelserv-tls has stopped generated certs for new ad domains. The second task that handles this is no longer running. Checking back through the log I think I know why - I have a reconnect script that pokes pixelserv to put the stats into the log. Without looking at the code I reckon the "killall -USR1 pixelserv-tls" hits both tasks and no handler defined for the second so it terminates.

Apr 19 10:14:44 pixelserv[523]: 240617 uts, 12553 req, 806 avg, 23295 rmx, 632 tav, 4422 tmx, 0 err, 712 tmo, 3368 cls, 0 nou, 0 pth, 2238 nfe, 436 ufe, 61 gif, 3 bad, 3760 txt, 22 jpg, 28 png, 2 swf, 116 ico, 6792 slh, 45 slm, 0 sle, 124 slu, 157 sta, 6 stt, 0 204, 1609 rdr, 9 pst, 0 hed, 0 log
Apr 19 10:14:45 admin: wan_check: monitoring WAN connection using ping
Apr 19 18:49:57 pixelserv[1867]: h.online-metrix.net _.online-metrix.net missing
Apr 19 18:49:58 pixelserv[1869]: h.online-metrix.net _.online-metrix.net missing
Apr 19 18:50:27 pixelserv[1874]: swrap.tradedoubler.com _.tradedoubler.com missing
Apr 19 18:50:29 pixelserv[1882]: h.online-metrix.net _.online-metrix.net missing
Apr 19 18:50:29 pixelserv[1881]: h.online-metrix.net _.online-metrix.net missing
Apr 19 18:50:29 pixelserv[1880]: h.online-metrix.net _.online-metrix.net missing
Apr 19 18:50:52 pixelserv[1892]: h.online-metrix.net _.online-metrix.net missing
Apr 19 18:51:15 pixelserv[1894]: h.online-metrix.net _.online-metrix.net missing
Apr 19 18:51:15 pixelserv[1895]: h.online-metrix.net _.online-metrix.net missing
Apr 19 18:51:22 pixelserv[1901]: swrap.tradedoubler.com _.tradedoubler.com missing
Apr 19 21:59:21 pixelserv[3037]: sync.adap.tv _.adap.tv missing
Apr 19 21:59:21 pixelserv[3039]: match.adsrvr.org _.adsrvr.org missing
...
Apr 23 10:11:41 pixelserv[523]: 586028 uts, 20959 req, 806 avg, 23295 rmx, 632 tav, 4422 tmx, 0 err, 1405 tmo, 8130 cls, 0 nou, 0 pth, 3156 nfe, 484 ufe, 138 gif, 3 bad, 5037 txt, 33 jpg, 38 png, 2 swf, 116 ico, 11450 slh, 45 slm, 0 sle, 619 slu, 160 sta, 6 stt, 0 204, 2154 rdr, 9 pst, 0 hed, 0 log

 

[swetoast]

@mstombs is that bug present in the latest released version or in dev ?

 

[mstombs]

I should have said - Entware mips version

pixelserv-tls version: V35.HZ12.Kh compiled: Mar 26 2016 12:16:28

 

[swetoast]

Havent notice that behavior in the ARMV7 version wonder if it could be permission problems that it cant write to dir if it could be that easy (perhaps stupid to assume that)

Apr 23 10:30:21 pixelserv[19259]: connect.ekomi.de _.ekomi.de missing
Apr 23 10:30:22 pixelserv[19260]: cert _.ekomi.de generated and saved

pixelserv-tls version: V35.HZ12.Kh compiled: Mar 28 2016 07:50:11

 

[mstombs]

The cert generation works until the command line "killall -USR1 pixelserv" (in your case) issued. Looks to me like the code has an old bug in that any child process active at the time also gets killed, but they are transient so easy to miss (I originally put the USR1 stuff in, so my bug!). I have over 50 certs in the cache, but have missed a few from past few days, no complaints from domestic users mind!