It would be GREAT to have all this documented here or better still in the fork. I think Jack has a helluva time getting all the packages installed properly to compile. Most of these details need skills most of us do not have have. I consider "pixelserv is a "one-of-a-kind" community endeavor as evident in this thread and these AVS forums. Peace.
pixelserv pixelserv - A Better One-pixel Webserver for Adblock
- Thread starter kvic
- Start date
pdc
Regular Contributor
I have a newbie question on the stats. After 20 days, I have around 100K accepted HTTPS requests (slh), but 65K dropped (slu). Of those, the vast majority are "unknown cert reported by clients" (uce).
At the bottom of the stats, there are also 30K "client disconnect before response sent" (cly).
Everything else looks pretty good.
The unknown cert sounds like I didn't install the cert correctly on at least one of my many clients with my pixelserv cert. Is that correct? And if so, is there some way I can figure out which client?
Code:
pixelserv-tls 2.2.1 (compiled: May 22 2019 13:44:50 flags: tfo tls1_3)
uts 20d 14:48 process uptime
log 1 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc 1 number of active service threads
kmx 80 maximum number of service threads
kvg 1.34 average number of requests per service thread
krq 1873 max number of requests by one service thread
req 178915 total # of requests (HTTP, HTTPS, success, failure etc)
avg 1253 bytes average size of requests
rmx 152991 bytes largest size of request(s)
tav 16 ms average processing time (per request)
tmx 5181 ms longest processing time (per request)
slh 104309 # of accepted HTTPS requests
slm 98 # of rejected HTTPS requests (missing certificate)
sle 0 # of rejected HTTPS requests (certificate available but not usable)
slc 8729 # of dropped HTTPS requests (client disconnect without sending any request)
slu 63524 # of dropped HTTPS requests (other TLS handshake errors)
v13 22912 slh/slc break-down: TLS 1.3
v12 90056 slh/slc break-down: TLS 1.2
v10 70 slh/slc break-down: TLS 1.0
zrt 320 slh break-down: TLS 1.3 Early Data aka 0-RTT
uca 2642 slu break-down: # of unknown CA reported by clients
ucb 0 slu break-down: # of bad certificate reported by clients
uce 60222 slu break-down: # of unknown cert reported by clients
ush 597 slu break-down: # of shutdown by clients after ServerHello
(etc...)
cls 8761 # of dropped requests (client disconnect without sending any request)
cly 28510 # of dropped requests (client disconnect before response sent)
clt 0 # of dropped requests (reached maximum service threads)
err 0 # of dropped requests (unknown reason)
dave14305
Part of the Furniture
It's not a question of your own browser cert, but usually the device or app expecting a certain pinned certificate that it doesn't receive back.
You can increase the log level temporarily by browsing http://192.168.1.2/log=2 and then watch the router's system log for pixelserv messages. Then you can reset to normal with http://192.168.1.2/log=1 (be sure to use your actual Pixelserv IP).
Some of my log entries from my iPhone:
Code:
Oct 11 13:21:24 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60144 server e.crashlytics.com
Oct 11 13:21:25 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60150 server e.crashlytics.com
Oct 11 13:21:25 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60151 server settings.crashlytics.com
Oct 11 13:21:27 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60156 server e.crashlytics.com
Oct 11 13:21:27 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60160 server e.crashlytics.com
Oct 11 13:21:31 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60161 server e.crashlytics.com
Oct 11 13:21:31 pixelserv-tls[1043]: handshake failed: shutdown after ServerHello. client 192.168.1.132:60162 server e.crashlytics.comEDIT: @elorimer is more accurate about the uce counts than I am. The extra logging will tell you which IPs report an unknown cert most likely because the CA isn't installed.
Code:
Oct 11 13:37:32 pixelserv-tls[1043]: handshake failed: unknown cert. client 192.168.1.129:60133 server csi.gstatic.com
Last edited:
pdc
Regular Contributor
Cool, I was not aware of the temporary log level trick, that's very handy!
Thanks @elorimer as well, glad to know my pixelserv cert is working as well as it can.
[Edit] Just got a few hits, turns out to also be some of my top-5 blocked domains, so makes sense the numbers are high.
Code:
Oct 11 13:56:10 pixelserv-tls[10800]: handshake failed: unknown cert. client xx.xx.xx.82:37754 server ssl.google-analytics.com
Oct 11 13:57:47 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53977 server p.typekit.net
Oct 11 13:57:51 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53986 server p.typekit.net
Oct 11 13:57:55 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53989 server p.typekit.net
Oct 11 13:57:55 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53990 server p.typekit.net
Oct 11 13:57:56 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53991 server p.typekit.net
Oct 11 13:57:58 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53994 server p.typekit.net
Oct 11 13:57:58 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53995 server p.typekit.net
Oct 11 13:58:00 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:53996 server p.typekit.net
Oct 11 13:59:14 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54014 server p.typekit.net
Oct 11 14:01:14 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54032 server p.typekit.net
Oct 11 14:01:17 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54036 server p.typekit.net
Oct 11 14:01:20 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54039 server p.typekit.net
Oct 11 14:01:20 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54040 server p.typekit.net
Oct 11 14:01:27 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54041 server p.typekit.net
Oct 11 14:01:31 pixelserv-tls[10800]: handshake failed: unknown cert. client xx.xx.xx.82:37774 server ssl.google-analytics.com
Oct 11 14:01:31 pixelserv-tls[10800]: handshake failed: unknown CA. client xx.xx.xx.152:54042 server p.typekit.net
Oct 11 14:01:47 pixelserv-tls[10800]: handshake failed: unknown cert. client xx.xx.xx.81:45661 server ssl.google-analytics.com
Last edited:
dave14305
Part of the Furniture
Dumb question...in which folder do drop the script? Tmp folder or the opt/var/log?
dave14305
Part of the Furniture
I put mine in /jffs/scripts. To run it as-is, you also need to install bash from Entware and supply your own email script. I think it would be cool for one of our scripting gurus to replicate this in the built-in shell.
I seem to recall it also needs dig, for the reverse lookup?
Note that @thelonelycoder does it a different way in Diversion, doesn't use bind-dig.
Note that @thelonelycoder does it a different way in Diversion, doesn't use bind-dig.
Last edited:
jrmwvu04
Very Senior Member
bind-dig as I recall. Been a while though.
Was
Can you explain more on my own "email script"? Is this simple to accomplish?
dave14305
Part of the Furniture
I hacked the original tls-alert.sh script with the following added near the end:
Code:
#${EMAIL_SCRIPT} "pixelserv-tls: Today's TLS handshake errors & new certs" /tmp/tmp.alert
# email settings
SMTP="smtp.gmail.com"
PORT="465"
USERNAME="myemailaddress@gmail.com"
PASSWORD="my gmail app password"
# Mail Enveloppe
FROM_NAME="My Name"
FROM_ADDRESS="myemailaddress@domain.com"
TO_NAME="My Name"
TO_ADDRESS="myemailaddress@domain.com"
echo "From: \"$FROM_NAME\" <$FROM_ADDRESS>" > /tmp/mail.txt
echo "To: \"$TO_NAME\" <$TO_ADDRESS>" >> /tmp/mail.txt
echo "Subject: pixelserv-tls: Today's TLS handshake errors & new certs" >> /tmp/mail.txt
echo "" >> /tmp/mail.txt
cat /tmp/tmp.alert >> /tmp/mail.txt
curl --url smtps://$SMTP:$PORT \
--mail-from "$FROM_ADDRESS" --mail-rcpt "$TO_ADDRESS" \
--upload-file /tmp/mail.txt \
--ssl-reqd \
--user "$USERNAME:$PASSWORD"
rm /tmp/tmp.alert /tmp/mail.txtI've often thought it could be a useful part of Diversion. If one attends to the suppression array, it helps to identify when a device is doing something new that is not associated with browser activity. So for some reason a device is trying to reach a site that Diversion has identified as blocked; not an ad, and not the 'normal' telemetrics, so something worthy of investigating. But maintaining the array is a fair amount of work.
A lot of the Diversion coding concepts could be leveraged:
--the email setup
--the frequency of sending an email
--the maintenance of the suppression array as a separate file, following the same treatment as the whitelist/blacklist maintenance
--checking that the entware packages are installed and updated
And it could be extended:
--a means of automatically adding to the suppression array new combinations generated by the script
--extending existing combinations to new devices
But that would be a lot of work and I'm not sure there are enough paying attention to make it worth it, except that Diversion would draw attention to it.
I did something similar yesterday (no where near as clean a job as yours) as a separate email.sh script passing the two parameters. On my list, though, is to pull all the specs from /opt/share/diversion/.conf/email.conf.
EDIT: Yes, so leave the alert script alone and let Diversion do the heavy lifting (for all four email types) by creating /jffs/scripts/email.sh and make it executable:
Code:
#!/opt/bin/sh
#Parameters passed#
mailsubject=$1
mailbody=$2
# Email settings (mail envelope) #
. /opt/share/diversion/.conf/email.conf
#Build email
echo "From: \"$FRIENDLY_ROUTER_NAME\" <$FROM_ADDRESS>" >/tmp/mail.txt
echo "To: \"$TO_NAME\" <$TO_ADDRESS>" >>/tmp/mail.txt
echo "Subject: $mailsubject " >>/tmp/mail.txt
echo "Date: $(date -R)" >>/tmp/mail.txt
echo >>/tmp/mail.txt
echo " $(cat $mailbody)" >>/tmp/mail.txt
#Send Email
/usr/sbin/curl --url $PROTOCOL://$SMTP:$PORT \
--mail-from "$FROM_ADDRESS" --mail-rcpt "$TO_ADDRESS" \
--upload-file /tmp/mail.txt \
--ssl-reqd \
--user "$USERNAME:$PASSWORD" $SSL_FLAG
#Log results
if [ "$?" = "0" ]; then
logger -t pixelserv-tls "emailed compiled pixelserv breaking stats from $0"
else
logger -t pixelserv-tls "failed to send pixelserv stats from $0"
fi
rm /tmp/mail.txtEDIT: As of Diversion 4.1.7 this no longer works, because the password is now encrypted in a separate file. Change the last line of the curl instruction to
Code:
--user "$USERNAME:$(openssl aes-256-cbc -d -in /opt/share/diversion/.conf/emailpw.enc -pass pass:ditbabot,isoi)
Code:
PASSWORD=$(openssl aes-256-cbc -d -in /opt/share/diversion/.conf/emailpw.enc -pass pass:ditbabot,isoi)
Last edited:
Is there a way to check if the script is running successfully?
dave14305
Part of the Furniture
Check your inbox for the email.
You can run it manually to see what happens. Make sure you adjust this line to point to default syslog.log if you aren’t using syslog-ng as kvic was fond of.
Code:
LOGFILE="/tmp/syslog.log-1 /tmp/syslog.log"
Code:
cru a pixelserv-report "59 23 * * * /opt/bin/bash /jffs/scripts/tls-alert.sh"
Last edited:
thelonelycoder
Part of the Furniture
Though it is there with Entware, as the shebang I use the default Almquist shell instead of Bash, none of my coding uses bash syntax.
#!/opt/bin/sh
Fixed above. The email script doesn't need bash, but the tls-alert script does because of the array declared with the declare statement.
DonnyJohnny
Very Senior Member
Just to share a news for Windows 10 user.
https://docs.microsoft.com/en-us/windows/whats-new/whats-new-windows-10-version-1909
Transport Layer Security (TLS)
An experimental implementation of TLS 1.3 is included in Windows 10, version 1909. TLS 1.3 disabled by default system wide. If you enable TLS 1.3 on a device for testing, then it can also be enabled in Internet Explorer 11.0 and Microsoft Edge by using Internet Options. For beta versions of Microsoft Edge on Chromium, TLS 1.3 is not built on the Windows TLS stack, and is instead configured independently, using the Edge://flags dialog. Also see Microsoft Edge platform status.
https://techcommunity.microsoft.com...nable-TLS-1-3-in-Windows-10-v1909/m-p/1017743
Seems to have some issue for now... let's wait and see.
https://docs.microsoft.com/en-us/windows/whats-new/whats-new-windows-10-version-1909
Transport Layer Security (TLS)
An experimental implementation of TLS 1.3 is included in Windows 10, version 1909. TLS 1.3 disabled by default system wide. If you enable TLS 1.3 on a device for testing, then it can also be enabled in Internet Explorer 11.0 and Microsoft Edge by using Internet Options. For beta versions of Microsoft Edge on Chromium, TLS 1.3 is not built on the Windows TLS stack, and is instead configured independently, using the Edge://flags dialog. Also see Microsoft Edge platform status.
https://techcommunity.microsoft.com...nable-TLS-1-3-in-Windows-10-v1909/m-p/1017743
Seems to have some issue for now... let's wait and see.
Similar threads
Similar threads
| Thread starter | Title | Forum | Replies | Date |
|---|---|---|---|---|
| C | Diversion Pixelserv replacement | Asuswrt-Merlin AddOns | 2 | |
| L | Is Diversion better than NextDNS, PiHole or AdGuard Home? | Asuswrt-Merlin AddOns | 10 |
Similar threads
-
-
Is Diversion better than NextDNS, PiHole or AdGuard Home?
- Started by Logi
- Replies: 10