pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[Butterfly Bones]

Bingo!
https://www.abine.com/blog/2013/mixpanel/
https://www.appsflyer.com/
https://www.trustradius.com/compare-products/appsflyer-vs-mixpanel

I'll experiment with blocking this in AB-Solution and see what app stops working. I cannot tell since many of these keep coming up while the phone is on my desk, on, but idle.

appsflyer.com and mixpanel.com are the most frequent.

Well, !@#$%^&* that was unproductive.

appsflyer.com
 was found in the blocking file.

mixpanel.com
 was found in the blocking file.

So I whitelisted them in AB-S instead to see if I that would stop the ssl errors. Nope, still the same entries in syslog. Seems I can not identify apps using them. I'm very careful with what I install, well known apps, nothing borderline. Thanks @kvic for the info to try and nail this down, I know much more now, I just wanted to stop the ssl errors, but it appears that ain't gonna happen.

I did run the benchmark again and here is what I get. AC86U all router add ons are on a Sandisk Ultra Fit drive, USB 3.0, 16Gb formatted ext4, on the router USB 3.0 port

# pixelserv-tls -B
CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
 1. generate cert to disk: 51.049 ms    load from disk: 4.848 ms
 2. generate cert to disk: 39.019 ms    load from disk: 5.251 ms
 3. generate cert to disk: 50.609 ms    load from disk: 5.494 ms
 4. generate cert to disk: 65.390 ms    load from disk: 5.211 ms
 5. generate cert to disk: 66.189 ms    load from disk: 4.798 ms
 6. generate cert to disk: 46.024 ms    load from disk: 4.802 ms
 7. generate cert to disk: 77.728 ms    load from disk: 4.785 ms
 8. generate cert to disk: 46.584 ms    load from disk: 4.833 ms
 9. generate cert to disk: 70.946 ms    load from disk: 4.798 ms
10. generate cert to disk: 59.312 ms    load from disk: 4.796 ms
generate to disk average: 57.285 ms
  load from disk average: 4.962 ms

 

[punkinduster]

I installed the updated libopenssl. Mem percentage only went up to 1.5.

 

[Butterfly Bones]

kmx 65 (!!!!!!!) 1.9% mem used (of 512 Mb) in htop AC86u

pixelserv-tls 2.1.0-rc.4 (compiled: Apr 5 2018 21:02:10 flags: tfo) options: 192.168.1.2 -l 2

uts 0d 04:22 process uptime
log 2 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc 65 number of active service threads
kmx 86 maximum number of service threads
kvg 6.07 average number of requests per service thread
krq 2031 max number of requests by one service thread
req 27457 total # of requests (HTTP, HTTPS, success, failure etc)
avg 599 bytes average size of requests
rmx 6537 bytes largest size of request(s)
tav 2 ms average processing time (per request)
tmx 3531 ms longest processing time (per request)
slh 23673 # of accepted HTTPS requests
slm 139 # of rejected HTTPS requests (missing certificate)
sle 0 # of rejected HTTPS requests (certificate available but bad)
slc 572 # of dropped HTTPS requests (client disconnect without sending any request)
slu 206 # of dropped HTTPS requests (other TLS handshake errors)
uca 0 slu break-down: # of unknown CA reported by clients
uce 205 slu break-down: # of unknown cert reported by clients
sct 50 cert cache: # of certs in cache
sch 970 cert cache: # of reuses of cached certs
scm 88 cert cache: # of misses to find a cert in cache
scp 38 cert cache: # of purges to give room for a new cert
sst 12 sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh 426 sess cache: # of reuses of cached TLS sessions
ssm 491 sess cache: # of misses to find a TLS session in cache
ssp 0 sess cache: # of purges to give room for a new TLS session
nfe 308 # of GET requests for server-side scripting
gif 10 # of GET requests for GIF
ico 11 # of GET requests for ICO
txt 23659 # of GET requests for Javascripts
jpg 8 # of GET requests for JPG
png 2 # of GET requests for PNG
swf 0 # of GET requests for SWF
sta 9 # of GET requests for HTML stats
stt 0 # of GET requests for plain text stats
ufe 71 # of GET requests /w unknown file extension
opt 0 # of OPTIONS requests
pst 125 # of POST requests
hed 0 # of HEAD requests (HTTP 501 response)
rdr 1726 # of GET requests resulted in REDIRECT response
nou 0 # of GET requests /w empty URL
pth 0 # of GET requests /w malformed URL
204 0 # of GET requests (HTTP 204 response)
bad 12 # of unknown HTTP requests (HTTP 501 response)
tmo 483 # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls 689 # of dropped requests (client disconnect without sending any request)
cly 0 # of dropped requests (client disconnect before response sent)
clt 0 # of dropped requests (reached maximum service threads)
err 0 # of dropped requests (unknown reason)

 

[Butterfly Bones]

@kvic Here is a collection for you.
https://photos.app.goo.gl/xg1y9dyobBSVEIMF2

edit - and web pages load in a flash now with rc.4 and libopenssl updates, zowie!

 

[punkinduster]

I notice a bunch of these as well as others in my log:

Apr  5 15:49:34 pixelserv-tls[19594]: handshake failed: unknown cert. client 192.168.2.69:46948 server data.flurry.com
Apr  5 15:49:34 pixelserv-tls[19594]: handshake failed: unknown cert. client 192.168.2.69:46949 server data.flurry.com
Apr  5 15:49:48 pixelserv-tls[19594]: handshake failed: unknown cert. client 192.168.2.69:46958 server analytics.query.yahoo.com
Apr  5 15:49:48 pixelserv-tls[19594]: handshake failed: unknown cert. client 192.168.2.69:46959 server analytics.query.yahoo.com
Apr  5 15:49:48 pixelserv-tls[19594]: handshake failed: unknown cert. client 192.168.2.69:46960 server analytics.query.yahoo.com
Apr  5 15:49:48 pixelserv-tls[19594]: handshake failed: unknown cert. client 192.168.2.69:46961 server analytics.query.yahoo.com
Apr  5 15:49:48 pixelserv-tls[19594]: handshake failed: unknown cert. client 192.168.2.69:46962 server geo.yahoo.com
Apr  5 15:49:48 pixelserv-tls[19594]: handshake failed: unknown cert. client 192.168.2.69:46963 server geo.yahoo.com
Apr  5 15:49:48 pixelserv-tls[19594]: handshake failed: unknown cert. client 192.168.2.69:46964 server geo.yahoo.com

They show up every time I go to access my email account on my android cell phone. Is yahoo trying to be sneaky about something?

 

[RMerlin]

What's the downside of compiling with that OpenSSL flag?

 

[punkinduster]

I just got a couple thousand of these from pixelserv-tls watching Roku:

handshake failed: unknown CA. client 192.168.2.16:33935 server metrics.brightcove.com

How do you get a Roku box to accept a certificate?

 

[kvic]

I just tested by opening https://mypixelservsip/servstats page from a device without the certificate installed and it's still incrementing slh on every page refresh and not doing anything to uca and uce counters.

People won't see the servstats page over HTTPS if your Pixelserv CA is not imported because handshake will fail and the browser should give you an error page.

 

[JaimeZX]

Now with the updated library... wow!!

pixelserv-tls 2.1.0-rc.3 (compiled: Mar 30 2018 17:10:59) options: 192.168.1.3

uts 0d 23:37 process uptime
log 1 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc 2 number of active service threads
kmx 13 maximum number of service threads
kvg 1.01 average number of requests per service thread
krq 6 max number of requests by one service thread

req 6142 total # of requests (HTTP, HTTPS, success, failure etc)
avg 538 bytes average size of requests
rmx 882 bytes largest size of request(s)
tav 25 ms average processing time (per request)
tmx 75 ms longest processing time (per request)

slh 8 # of accepted HTTPS requests
slm 6 # of rejected HTTPS requests (missing certificate)
sle 0 # of rejected HTTPS requests (certificate available but bad)
slc 1389 # of dropped HTTPS requests (client disconnect without sending any request)
slu 4689 # of dropped HTTPS requests (other TLS handshake errors)

sct 83 cert cache: # of certs in cache
sch 5368 cert cache: # of reuses of cached certs
scm 8 cert cache: # of misses to find a cert in cache
scp 0 cert cache: # of purges to give room for a new cert
sst 0 sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh 68 sess cache: # of reuses of cached TLS sessions
ssm 22 sess cache: # of misses to find a TLS session in cache
ssp 0 sess cache: # of purges to give room for a new TLS session

nfe 36 # of GET requests for server-side scripting
gif 0 # of GET requests for GIF
ico 0 # of GET requests for ICO
txt 4 # of GET requests for Javascripts
jpg 0 # of GET requests for JPG
png 0 # of GET requests for PNG
swf 0 # of GET requests for SWF
sta 2 # of GET requests for HTML stats
stt 0 # of GET requests for plain text stats
ufe 0 # of GET requests /w unknown file extension

opt 0 # of OPTIONS requests
pst 14 # of POST requests
hed 0 # of HEAD requests (HTTP 501 response)
rdr 0 # of GET requests resulted in REDIRECT response
nou 0 # of GET requests /w empty URL
pth 0 # of GET requests /w malformed URL
204 0 # of GET requests (HTTP 204 response)
bad 0 # of unknown HTTP requests (HTTP 501 response)

tmo 3 # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls 1389 # of dropped requests (client disconnect without sending any request)
cly 0 # of dropped requests (client disconnect before response sent)
clt 0 # of dropped requests (reached maximum service threads)
err 0 # of dropped requests (unknown reason)

 

[kvic]

So I whitelisted them in AB-S instead to see if I that would stop the ssl errors. Nope, still the same entries in syslog. Seems I can not identify apps using them. I'm very careful with what I install, well known apps, nothing borderline.... I just wanted to stop the ssl errors, but it appears that ain't gonna happen.

So in the little "task force" we had to study and understand slu during rc.1/rc.2 timeframe, I brought up a few possible solutions and shared my own.

Once people understand the cause of a particular advert/tracker server, one way to suppress it is to assign that server to address 0.0.0.0.

If your adblock script hasn't had this feature for selectively assigning different IP, then a generic approach will be:

• whitelist the host in your adblock script
• manually assign it to 0.0.0.0 in a config file recognised by your adblock script that won't be overwritten to pixelserv ip.

If that still not possible in your adblock script, the fallback approach for ASUSWRT-merlin users will be

• whitelist it in adblock script
• manually add entries to /jffs/configs/hosts.add, e.g.
  • 0.0.0.0 t.appflyer.com
  • 0.0.0.0 x.flyme.com

 

[kvic]

I notice a bunch of these as well as others in my log:

They show up every time I go to access my email account on my android cell phone. Is yahoo trying to be sneaky about something?

Looks like they're. From the tracker server names, I would guess uploading privacy data including your geolocation.

If so, surely they can't let users find out and take evidence. So the app refuses to send the data when server certificate does not match stored fingerprint.

How do you get a Roku box to accept a certificate?

I believe it's possible but I don't know how.

Once people finish with privacy breach investigation, perhaps could lower log LEVEL back to 1.

Another approach is to install syslog-ng (that I advocated a few times in this thread).

Use its filter feature to either send all pixelserv-tls logs to its own file and/or selectively removing messages from being logged altogether. For power users, you need a functional syslog on your router..

 

[kvic]

What's the downside of compiling with that OpenSSL flag?

None for systems built in the past 10 years..

IMO, this feature should not have been added. If OpenSSL feels a bit embarrassing to remove it, they shall have had it set to disable as default..



(@punkinduster, there was a reply to you pending moderator action)

 

[penguin22]

After setting log level to 2, I see similar logs to @punkinduster:

Apr  5 20:02:30 pixelserv-tls[28666]: handshake failed: unknown cert. client 192.168.50.249:58082 server mobile.pipe.aria.microsoft.com
Apr  5 20:12:16 pixelserv-tls[28666]: handshake failed: unknown cert. client 192.168.50.249:58334 server www.googleadservices.com
Apr  5 20:13:04 pixelserv-tls[28666]: handshake failed: unknown cert. client 192.168.50.13:48018 server e.crashlytics.com
Apr  5 20:13:07 pixelserv-tls[28666]: handshake failed: unknown cert. client 192.168.50.13:46065 server control.kochava.com

These all seem to come from various Android or Amazon Fire/FireTV/Echo devices. On the Android phones and Fire tablets, the CA cert has been installed, but it still seems to be non-used by certain services.

Looking through the MAN page @kvic linked, the description for uce would lead a person to be quite concerned, however I think this may be normal in some cases, like the many that are from Microsoft in my above examples, which I believe are related to the Intune Company Portal since synchronizing doesn't occur when on WiFi, but does once dropping (Edit: Might also be Skype for Business, which we also use per this Reddit post.) (Edit 2: Confirmed by Microsoft: 17 Optional: Aria log integration - mobile.pipe.aria.microsoft.com - TCP 443)

uce When a TLS handshake fails due to the client not recognise the
server cert, this counter is incremented by one. Very likely
such cases are of dubious nature. A client may be purposedly
coded to check the server cert against stored fingerprint. If
mismatch, the client refuses to proceed. This is alarming and
often indicates a client attempts to connect to a rogue server.
Users should turn on log LEVEL 2. Catch the client ip and server
name in syslog and proceed with similar investigation steps as
in uca.

uts    0d 08:35    process uptime

log    2    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    22    number of active service threads
kmx    27    maximum number of service threads
kvg    1.77    average number of requests per service thread
krq    75    max number of requests by one service thread
req    7671    total # of requests (HTTP, HTTPS, success, failure etc)
avg    1993 bytes    average size of requests
rmx    47307 bytes    largest size of request(s)
tav    11 ms    average processing time (per request)
tmx    976 ms    longest processing time (per request)
slh    4521    # of accepted HTTPS requests
slm    11    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    429    # of dropped HTTPS requests (client disconnect without sending any request)
slu    1969    # of dropped HTTPS requests (other TLS handshake errors)
uca    39    slu break-down: # of unknown CA reported by clients
uce    1573    slu break-down: # of unknown cert reported by clients

 

[john9527]

None for systems built in the past 10 years..

IMO, this feature should not have been added. If OpenSSL feels a bit embarrassing to remove it, they shall have had it set to disable as default..

Just as an FYI....with the debian patch applied on my fork, I don't see any difference in memory use (running one VPN Server, one VPN Client, NOT running ABSolution/pixelserv)

 

[kvic]

with the debian patch applied on my fork, I don't see any difference in memory use (running one VPN Server, one VPN Client, NOT running ABSolution/pixelserv)

Your little router is mostly idle, not to mention any medium to heavy SSL workload.

 

[Asad Ali]

People won't see the servstats page over HTTPS if your Pixelserv CA is not imported because handshake will fail and the browser should give you an error page.

That was my understanding for it as well but now my tests are showing something else.

I did the tests on two devices:
iPhone with iOS 11.3
iPad with iOS 8

I removed the imported certificate from both devices and then open the statistics page over HTTPS.

On iPhone the browser give me the error but if I force open the page it does open and increment slh on each page refresh.

On iPad the browser doesn't give me any errors and simply open the statistics page and it also keeps incrementing slh on every refresh.

So if my understanding is correct slh should not be incremented in any case if the device don't have the root certificate installed but it's happening on my tests.

 

[penguin22]

I went through most of the Microsoft apps on my Android device, including LinkedIn and was able to greatly reduce the volume of handshake failed messages to mobile.pipe.aria.microsoft.com by disabling as much log gathering as allowed. I'd say that @kvic is right and Pixelserv log level 2 is a great method of finding potentially non-desirable client connectivity.

In this case, if Microsoft used the installed CA cert, I would be equally ok, especially since it appears to be logging rather than functionality related, but I can appreciate they may not permit a non-trusted certificate to be used by their applications as it could be an indication of a compromised device.

 

[Butterfly Bones]

So in the little "task force" we had to study and understand slu during rc.1/rc.2 timeframe, I brought up a few possible solutions and shared my own.

Once people understand the cause of a particular advert/tracker server, one way to suppress it is to assign that server to address 0.0.0.0.

If your adblock script hasn't had this feature for selectively assigning different IP, then a generic approach will be:

• whitelist the host in your adblock script
• manually assign it to 0.0.0.0 in a config file recognised by your adblock script that won't be overwritten to pixelserv ip.

If that still not possible in your adblock script, the fallback approach for ASUSWRT-merlin users will be

• whitelist it in adblock script
• manually add entries to /jffs/configs/hosts.add, e.g.
  • 0.0.0.0 t.appflyer.com
  • 0.0.0.0 x.flyme.com

I remember this discussion earlier, thanks for the bump on this. I need to remember to go back and re-read many posts here as the test /rc releases progress. My memory is not what it used to be.

With the above instructions I have stopped almost all of the "pixelserv-tls[26258]: handshake failed: unknown cert. client 192.168.1.12:38970 server domain_name.com". I'm using the "fallback approach" you mention above. I had one kept popping through and found that I made a spelling error in a long server name. I keep opening phone apps now to see what else I can squash, thanks to your guidance. Then I will do the same with the other two mobile devices.

 

[kvic]

slh should not be incremented in any case if the device don't have the root certificate installed but it's happening on my tests.

Seems to me you're over thinking here.

When you manually override browser warning (and if it actually allow you to do it), then from TLS handshake perspective, no difference to having your Pixelserv CA installed.

iOS 8...I don't have a device to test. Seems to me more like an OS issue though

 

[kvic]

In this case, if Microsoft used the installed CA cert, I would be equally ok, especially since it appears to be logging rather than functionality related, but I can appreciate they may not permit a non-trusted certificate to be used by their applications as it could be an indication of a compromised device.

I think you've brought up a good point on the side of supporting vendors doing fingerprint check in order to protect users from their systems being compromised with 'dubious' or even legitimate CA certs.

I went through both macOS and Windows many times in order to have a user CA cert installed. The security check that has to go through is stringent to a point that people may not want to repeat it casually. But there is no way to avoid a legitimate CA erroneously (or purposely) issue a wrong server certificate.

But I have the suspicion (that can't get rid of) that big (and small) guys are doing so for multiple reasons, the primary is not wanting users to know what actually being sent..

However, in this case, it's good for Microsoft to allow users to turn off the feature. And even better if they allow users to inspect what's actually being sent.