pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[punkinduster]

I have been using AB-Solution on a RT-AC66U for a while now and like it a lot. I recently upgraded my router to a RT-AC86U and installed AB-Solution again with success. I also added a few more scripts like amtm, DNSCrypt, and SkyNet with success. I also added pixelserv-tls and Entware with no errors. I thought everything was fine until my wife said she could no longer watch Fox News videos. It would error with a file not found message when she tried to watch a clip. If I turn OFF pixelserv-tls she can watch the videos/clips but as soon as I turn ON pixelserv-tls the videos say they can't play because they can't find the file. Any ideas? I figured if I was blocking something that I should white list it wouldn't work even with pixelserv-tls off but it works OK. I admit I don't know anything about troubleshooting pixelserv-tls so any help would be appreciated. Thanks in advance.

 

[Makaveli]

Thanks for the detailed update. This one should be easy to spot. You have to install and run tcpdump on your router.

Install:
$ opkg install tcpdump

Then run. replace 192.168.1.3 with your <pixelserv ip>
$ tcpdump -i br0 "tcp[tcpflags] & (tcp-syn) != 0 and host 192.168.1.3 and tcp port 443"

Sample outputs:
12:20:42.493524 IP myclient.ukei.64599 > n1.pixelserv.tls.https: Flags , seq 449919520, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 3158194437 ecr 0,sackOK,eol], length 0

"myclient.ukei" from port 64599 is making the connection at timestamp 12:20:42.493524.

You can make a good guest if this connection is slc. When you see slc rapidly increasing during browsing activities, and many recurring outputs from the tcpdump at the same time. Must be it.

Thanks for this.

Here is the output

The 192.168.1.5 is the IP of the HTPC with no CA.

192.168.1.4 is my current desktop with the CA

 

[kvic]

I figured if I was blocking something that I should white list it wouldn't work even with pixelserv-tls off but it works OK. I admit I don't know anything about troubleshooting pixelserv-tls so any help would be appreciated. Thanks in advance.

Interesting. I visited foxnews often in the past weeks for testing but I seldom read. Long time ago I got the impression the video problem for me was due to geolock. So didn't bother to look closer even though I did see broken video display

pixelserv-tls has logging functionality. We all could try and follow this Mini-Howto to figure out who is the 'wrongly blocked' domain and who is the 'rogue' domain:

https://www.snbforums.com/threads/p...bserver-for-adblock.26114/page-69#post-390044

 

[john9527]

Don't know if it's related, but some foxnews pages have a nasty habit of putting multiple entries for the same page in the browser history (if you look at the back button history you'll see it). Seems related to pages that accept comments and if you leave the page open some comments generate a new entry. On firefox, this can cause what looks like a memory leak, running the memory use into multiple GB.

 

[kvic]

@john9527 thanks for chime in on this.

With a bit debugging and help from pixelserv-tls logging, I think I locate the 'wrongly blocked' domain. No more broken video on foxnews after whitelisting it (yay!):

Mar 23 09:22:52 Phaeo pixelserv-tls[1789]: 192.168.1.104 79423.analytics.edgesuite.net GET /html5/akamaihtml5-min.js HTTP/1.1

@punkinduster pls try whitelisting the domain in bold.

happy pixelserv'ing everyone!

 

[Makaveli]

I finally enabled loggin with -l 2 can be done very quickly from AB-Solution.

if I go to https://www.androidcentral.com/

I see this error in the log

Mar 22 22:43:58 pixelserv-tls[20532]: client  ssl error:14094412:lib(20):func(148):reason(1042)
Mar 22 22:43:58 pixelserv-tls[20532]: client  ssl error:14094412:lib(20):func(148):reason(1042)

And now abit more


This IP belongs to my Blackberry Keyone running nougat 7.1.1 has the ca cert imported

Mar 22 22:51:09 pixelserv-tls[20532]: client 192.168.1.11 ssl error:140760FC:lib(20):func(118):reason(252)
Mar 22 22:51:09 pixelserv-tls[20532]: client 192.168.1.11 ssl error:140760FC:lib(20):func(118):reason(252)
Mar 22 22:51:09 pixelserv-tls[20532]: client 192.168.1.11 ssl error:140760FC:lib(20):func(118):reason(252)
Mar 22 22:51:09 pixelserv-tls[20532]: client 192.168.1.11 ssl error:14094418:lib(20):func(148):reason(1048)
Mar 22 22:51:09 pixelserv-tls[20532]: client 192.168.1.11 ssl error:14094418:lib(20):func(148):reason(1048)

 

[kvic]

Any idea why?

This is from a debug trace added in an update to rc.1 and still preserved in rc.2. However, it won't be available in final release.

The debug trace help to understand people's concern of huge slu counts. 1048 as mentioned by @Butterfly Bones one or two pages back means UNKNOWN CA. That's expected if the Sonos does not have Pixelserv CA imported.

This IP belongs to my Blackberry Keyone running nougat 7.1.1 has the ca cert imported

252 means UNKNOWN PROTOCOL. Pretty bad since some client is talking to pixelserv-tls not in TLS nor HTTP.

1048 is UNKNOWN CA... shall not happen if your client has Pixelserv CA imported but as discussed one or a few pages back, some Android clients do not honour user root CA cert.

To all: note that if the debug trace doesn't repeat itself successively in hundreds or thousands of lines, nothing to worry about.

 

[Makaveli]

This is from a debug trace added in an update to rc.1 and still preserved in rc.2. However, it won't be available in final release.

The debug trace help to understand people's concern of huge slu counts. 1048 as mentioned by @Butterfly Bones one or two pages back means UNKNOWN CA. That's expected if the Sonos does not have Pixelserv CA imported.



252 means UNKNOWN PROTOCOL. Pretty bad since some client is talking to pixelserv-tls not in TLS nor HTTP.

1048 is UNKNOWN CA... shall not happen if your client has Pixelserv CA imported but as discussed one or a few pages back, some Android clients do not honour user root CA cert.

To all: note that if the debug trace doesn't repeat itself successively in hundreds or thousands of lines, nothing to worry about.

Thanks I will leave it running over the course of the next few days to see what the log shows.

Any idea what this is in the tcpdump

03:08:44.698580 IP 2znp09oa.com.https > 192.168.1.4.62894: Flags [S.], seq 3018621002, ack 4078863213, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 4], length 0

 

[kvic]

Any idea what this is in the tcpdump

03:08:44.698580 IP 2znp09oa.com.https > 192.168.1.4.62894: Flags [S.], seq 3018621002, ack 4078863213, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 4], length 0

There should be one more related line above this one that shows "192.168.1.4.62894 > 2znp09oa.com.https"

• "2znp09oa.com" is your blocked domain. Since you're running pixelserv-tls that takes a the role of serving as "2znp09oa.com"
• The first line (not shown in your post) is your client 192.168.1.4 connects to "2znp09oa.com"
• The 2nd line (shown in your post) pixelserv-tls acknowledges the connection attempt.

In TCP/IP terminology, the first line is a SYN packet, the 2nd line is ACK to the SYN.

 

[kvic]

1 day 19h...

My two instances of 2.1.0-rc.2 near 64 hours...

RAM usage
33hrs: n1 8.2MB n2 6.5MB
64hrs: n1 13.9MB n2 11MB

Note that SSL cache is freed. The increased RAM usage represents more the peak demand in the past ~32 hrs. Excess shall be freed from pixelserv-tls when the system needs RAM.

 

[SMS786]

Rc2 buzzing along quite well here for almost 3 days..(68u)

 

[punkinduster]

With a bit debugging and help from pixelserv-tls logging, I think I locate the 'wrongly blocked' domain. No more broken video on foxnews after whitelisting it (yay!):

Mar 23 09:22:52 Phaeo pixelserv-tls[1789]: 192.168.1.104 79423.analytics.edgesuite.net GET /html5/akamaihtml5-min.js HTTP/1.1

@punkinduster pls try whitelisting the domain in bold.

happy pixelserv'ing everyone!

Thank you for your help. I whitelisted the domain above and now foxnews is working again.

Now I need to try a bit of debugging myself to address another similar issue. breitbart.com videos fail except for the ones coming from foxnews!

 

[kvic]

Now I need to try a bit of debugging myself to address another similar issue. breitbart.com videos fail except for the ones coming from foxnews

I tried a couple of videos on breibart. cnn, msnbc, foxnews, in-house videos all no problem for me. Maybe due to my block list is only about 110K entries.

Some general guideline for catching 'wrongly blocked' domains by using pixelserv-tls logging:

• turn on log LEVEL 4. You can use CLI "curl http://<your pixelserv ip>/log=4" or by other means e.g. from within AB-Solution.
• open Developer Tools for Chrome/Firefox or Web Inspector for Safari
• reload the problematic page in browser
• inspect webpage elements around the problematic 'block' usually a DIV these days
• quickly glimpse URLs inside the problematic 'block' and check up against pixelserv-tls logging inside /tmp/syslog.log
• pixelserv-tls captures full URLs. Pay attention to javascript resources ending with *.js extension. Often missing javascript are the culprit !!!
• a quick compare shall lead to discovery of the 'wrongly blocked' domain(s)
• whitelist the domain(s) and test.
• turn log LEVEL back to default.

 

[heysoundude]

Interesting. I visited foxnews often in the past weeks for testing but I seldom read. Long time ago I got the impression the video problem for me was due to geolock. So didn't bother to look closer even though I did see broken video display

pixelserv-tls has logging functionality. We all could try and follow this Mini-Howto to figure out who is the 'wrongly blocked' domain and who is the 'rogue' domain:

https://www.snbforums.com/threads/p...bserver-for-adblock.26114/page-69#post-390044

thanks for this - my al-jazeera app doesn't work and I can't access zerohedge.com when pixelserv is on...I hope this will clear all that up

 

[kvic]

thanks for this - my al-jazeera app doesn't work and I can't access zerohedge.com when pixelserv is on...I hope this will clear all that up

I turned post #1455 into a mini how-to on Github's Wiki.

HOW-TO: Figure out domain(s) to whitelist on a broken website

Enjoy guys!

 

[kvic]

Use your Pixelserv CA cert for WebGUI

I'm far falling behind on asuswrt and/or merlin variants. I've heard that AC88/AC5300 and its variants include support for Let's Encrypt in FW. In case, your models do not include this feature, as a pixelserv-tls user, you have a similar feature for free, and debatably in a more elegant way.

You could make use of Pixelserv CA cert that is generated by yourself with this instruction (or in AB-Solution upon installation of pixelserv-tls). We could use this CA cert to issue a certificate for WebGUI access. Simply SSH into your router, and run the following one-liner script (and a running pixelserv-tls):

sh -c "$(wget -qO - https://kazoo.ga/pixelserv-tls/config-webgui.sh)"

The script will guide you through the process. Use your Pixelserv CA to issue a certificate to a domain for WebGUI access. You have a choice of using your DDNS domain or router.asus.com.

Sample output of a successful run: https://pastebin.com/AyF9Q7Km

The benefit: no more invalid certs or adding exceptions to invalid certs in every browser after every FW upgrade. Simply re-run the above one-liner.

All clients with your Pixelserv CA imported will recognise the WebGUI connection as valid HTTPS. Better yet your Pixelserv CA is good for a very long time - import once good for ten years!

(This script was developed out of discussion in this thread: https://www.snbforums.com/threads/firefox-https-connection.44787/page-2#post-389381)

edit:
adding a screenshot
View attachment 12350

The quoted post is also transformed into GitHub wiki and its latest content will be always available on this link: [ASUSWRT] Use Pixelserv CA to issue a certificate for WebGUI

 

[thelonelycoder]

The quoted post is also transformed into GitHub wiki and its latest content will be always available on this link: [ASUSWRT] Use Pixelserv CA to issue a certificate for WebGUI

That looks like a worthy menu addition in amtm.
I have no time atm to code it but will add it to the future script link list.

 

[kvic]

Rc2 buzzing along quite well here for almost 3 days..(68u)

Also update my two instances of 2.1.0-rc.2 near 87 hours...

RAM usage
33hrs: n1 8.2MB n2 6.5MB
64hrs: n1 13.9MB n2 11MB
87hrs: n1 15.2MB n2 13MB

Careful eyes might notice that my slu stop growing rapidly (unlike in the first 33hrs). This is because I've removed domain "graph.instagram.com" from block list.

A very weird phenomenon that I can't explain, when users VPN into my router and this domain in block list, most of the time requests from Instagram (or its affiliate apps) from VPN users do not talk TLS nor HTTP on port 443. Whenever it happens, an avalanche of slu counts added (in thousands!!).

Original issue described here:

I have VPN server on RT-AC56U. WAN users VPN into my LAN. And recently pixelserv-tls logging does reveal some creepy traffic I still haven't figured out why. They come from the VPN users I suspect but are not valid HTTPS requests which they're expected to be when hitting pixelserv-tls on port 443.

 

[kvic]

I have no time atm to code it but will add it to the future script link list.

Take your time. I believe ABS users are anxiously waiting for v4.

Not sure if you have charting feature planned like in Pi-Hole. I believe the '/servstats.txt' counters could be well illustrated as charts. With pixelserv-tls logging full-time, further charts could be drawn. You'll then beat Pi-Hole by miles as Pi-Hole cannot properly classify more than half of its traffic (in HTTPS)...

Not sure if Pi-Hole users are in this thread and aware of that. It's high time to advocate pixelserv-tls to your beloved Pi-Hole authors. Or get over taken by other better solutions...

 

[Protik]

After 3d 15 hours,

pixelserv-tls 2.1.0-rc.2 (compiled: Mar 20 2018 21:11:49) options: 192.168.2.2 -c 150 -l 2

uts    3d 15:11    process uptime
log    2    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    42    maximum number of service threads
kvg    1.72    average number of requests per service thread
krq    210    max number of requests by one service thread
req    13134    total # of requests (HTTP, HTTPS, success, failure etc)
avg    985 bytes    average size of requests
rmx    64473 bytes    largest size of request(s)
tav    30 ms    average processing time (per request)
tmx    9553 ms    longest processing time (per request)
slh    3550    # of accepted HTTPS requests
slm    6    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    3425    # of dropped HTTPS requests (client disconnect without sending any request)
slu    5354    # of dropped HTTPS requests (other TLS handshake errors)
sct    150    cert cache: # of certs in cache
sch    10431    cert cache: # of reuses of cached certs
scm    93    cert cache: # of misses to find a cert in cache
scp    55    cert cache: # of purges to give room for a new cert
sst    5    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    489    sess cache: # of reuses of cached TLS sessions
ssm    601    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

Using 15.5MB of RAM out of 250MB.