pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[Butterfly Bones]

troubleshooting now...will report back. standby please!

I used the script and reverted to non-beta, use the line to restart, ran the script a second time, use the line to restart. Finally my time stamp changed from March 29 to March 30. I tried three times before just running the install script and it would keep the same time stamp and stats.

 

[kvic]

I don't use DNScrypt as I consider it an overhead rather than speed boost. Some people use it to bypass ISP logging/filtering on DNS requests. That perhaps can be justified but less of a concern to me personally.

Given that said, anyone running DNSscrypt happen to be unable to resolve "kazoo.ga" or another domain name? If so, any possible explanation..

 

[M@rco]

No issues resolving kazoo.ga here while running DNSCrypt (options: only allowing DNSSEC support, non-logging, non-filtering, automatically selecting fastest server available.)

Results after 8 hours update of your last update @kvic :

pixelserv-tls 2.1.0-rc.3 (compiled: Mar 30 2018 17:10:59) options: 192.168.1.2

uts    0d 08:44    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    3    number of active service threads
kmx    13    maximum number of service threads
kvg    1.04    average number of requests per service thread
krq    17    max number of requests by one service thread
req    3145    total # of requests (HTTP, HTTPS, success, failure etc)
avg    413 bytes    average size of requests
rmx    1425 bytes    largest size of request(s)
tav    1 ms    average processing time (per request)
tmx    40 ms    longest processing time (per request)
slh    0    # of accepted HTTPS requests
slm    4    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    1367    # of dropped HTTPS requests (client disconnect without sending any request)
slu    1673    # of dropped HTTPS requests (other TLS handshake errors)
sct    72    cert cache: # of certs in cache
sch    2974    cert cache: # of reuses of cached certs
scm    13    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    0    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    47    sess cache: # of reuses of cached TLS sessions
ssm    997    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session
nfe    4    # of GET requests for server-side scripting
gif    3    # of GET requests for GIF
ico    5    # of GET requests for ICO
txt    3    # of GET requests for Javascripts
jpg    0    # of GET requests for JPG
png    1    # of GET requests for PNG
swf    0    # of GET requests for SWF
sta    8    # of GET requests for HTML stats
stt    0    # of GET requests for plain text stats
ufe    0    # of GET requests /w unknown file extension
opt    0    # of OPTIONS requests
pst    64    # of POST requests
hed    0    # of HEAD requests (HTTP 501 response)
rdr    0    # of GET requests resulted in REDIRECT response
nou    0    # of GET requests /w empty URL
pth    0    # of GET requests /w malformed URL
204    0    # of GET requests (HTTP 204 response)
bad    0    # of unknown HTTP requests (HTTP 501 response)
tmo    7    # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls    1374    # of dropped requests (client disconnect without sending any request)
cly    0    # of dropped requests (client disconnect before response sent)
clt    0    # of dropped requests (reached maximum service threads)
err    0    # of dropped requests (unknown reason)

 

[DonnyJohnny]

I don't use DNScrypt as I consider it an overhead rather than speed boost. Some people use it to bypass ISP logging/filtering on DNS requests. That perhaps can be justified but less of a concern to me personally.

Given that said, anyone running DNSscrypt happen to be unable to resolve "kazoo.ga" or another domain name? If so, any possible explanation..

I think it would really depend on the dns server. Some dns server have their own filtering... I am using Cloudflare and no problem with the domain above.

 

[punkinduster]

Does this look reasonable? Memory 2.9 percent.

 

[vesalius]

On 3100. Uptime - 13hrs, mem - 2,5%

pixelserv-tls 2.1.0-rc.3 (compiled: Mar 30 2018 17:10:59) options: 10.0.1.3 -l 2
45772 uts, 2 log, 1 kcc, 29 kmx, 3.25 kvg, 336 krq, 7688 req, 2342 avg, 339882 rmx, 8 tav, 798 tmx, 5218 slh, 0 slm, 0 sle, 294 slc, 2017 slu, 100 sct, 2491 sch, 57 scm, 32 scp, 49 sst, 1007 ssh, 103 ssm, 0 ssp, 1470 nfe, 34 gif, 0 ico, 485 txt, 0 jpg, 3 png, 0 swf, 3 sta, 3 stt, 247 ufe, 42 opt, 2732 pst, 0 hed, 314 rdr, 0 nou, 0 pth, 0 204, 0 bad, 41 tmo, 298 cls, 0 cly, 0 clt, 0 err

 

[kvic]

No issues resolving kazoo.ga here while running DNSCrypt (options: only allowing DNSSEC support, non-logging, non-filtering, automatically selecting fastest server available.)

Thanks for the quick check.

I think it would really depend on the dns server. Some dns server have their own filtering... I am using Cloudflare and no problem with the domain above.

I was not aware CF offering public DNS. Let me find time to try. Thanks.



Good to know "kazoo.ga" isn't blocked across the board by DNScrypt servers. Originally I thought *.ga might be blocked by some of these servers but then "www.my.ga" returns IP. So it becomes an interesting problem.

For now, I would settle with some DNScrypt servers misconfigured that give trouble resolving "kazoo.ga" then.

 

[kvic]

Results after 8 hours update of your last update

healthy

Does this look reasonable? Memory 2.9 percent.

healthy

On 3100. Uptime - 13hrs, mem - 2,5%

healthy

Thanks for all the quick feedback. Let me emphasise one point.

The equilibrium point that I found most recently on my RT-56U /w 250MB RAM is between 10 to 13MB.

After that pixelserv-tls will take RAM from OS and give back RAM to OS. Such actions aren't in control by pixelserv-tls solely.

When your system need more RAM, the OS will take back from pixelserv-tls more proactively. So no worry about the situation when you even see higher usage number.

 

[Butterfly Bones]

Memory use is 4.8% - AC86U shows 482 MB used of 512 MB.

pixelserv-tls 2.1.0-rc.3 (compiled: Mar 30 2018 17:11:05 flags: tfo) options: 192.168.1.2
slu better now than before.

uts    0d 11:14    process uptime
log    1    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    30    maximum number of service threads
kvg    4.37    average number of requests per service thread
krq    12351    max number of requests by one service thread
req    18393    total # of requests (HTTP, HTTPS, success, failure etc)
avg    665 bytes    average size of requests
rmx    4573 bytes    largest size of request(s)
tav    4 ms    average processing time (per request)
tmx    1817 ms    longest processing time (per request)
slh    198    # of accepted HTTPS requests
slm    6    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    4332    # of dropped HTTPS requests (client disconnect without sending any request)
slu    638    # of dropped HTTPS requests (other TLS handshake errors)
sct    100    cert cache: # of certs in cache
sch    4977    cert cache: # of reuses of cached certs
scm    81    cert cache: # of misses to find a cert in cache
scp    56    cert cache: # of purges to give room for a new cert
sst    0    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    80    sess cache: # of reuses of cached TLS sessions
ssm    4237    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session
nfe    68    # of GET requests for server-side scripting
gif    9    # of GET requests for GIF
ico    30    # of GET requests for ICO
txt    12476    # of GET requests for Javascripts
jpg    0    # of GET requests for JPG
png    1    # of GET requests for PNG
swf    0    # of GET requests for SWF
sta    30    # of GET requests for HTML stats
stt    2    # of GET requests for plain text stats
ufe    9    # of GET requests /w unknown file extension
opt    1    # of OPTIONS requests
pst    189    # of POST requests
hed    0    # of HEAD requests (HTTP 501 response)
rdr    97    # of GET requests resulted in REDIRECT response
nou    0    # of GET requests /w empty URL
pth    0    # of GET requests /w malformed URL
204    0    # of GET requests (HTTP 204 response)
bad    57    # of unknown HTTP requests (HTTP 501 response)
tmo    275    # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls    4506    # of dropped requests (client disconnect without sending any request)
cly    0    # of dropped requests (client disconnect before response sent)
clt    0    # of dropped requests (reached maximum service threads)
err    0    # of dropped requests (unknown reason)

 

[kvic]

Memory use is 4.8% - AC86U shows 482 MB used of 512 MB.

Doubling RAM shifts up the equilibrium point..? Let's continue to have an eye on this.

 

[DonnyJohnny]

Thanks for the quick check.



I was not aware CF offering public DNS. Let me find time to try. Thanks.



Good to know "kazoo.ga" isn't blocked across the board by DNScrypt servers. Originally I thought *.ga might be blocked by some of these servers but then "www.my.ga" returns IP. So it becomes an interesting problem.

For now, I would settle with some DNScrypt servers misconfigured that give trouble resolving "kazoo.ga" then.

I heard it was very recently only . Cloudflare under 1.1.1.1 DNS over HTTP/2

 

[skeal]

Thanks @kvic for the script to generate certs for the webui! Works great!
EDIT: Great guide found here https://github.com/kvic-z/pixelserv-tls/wiki

 

[kvic]

I heard it was very recently only . Cloudflare under 1.1.1.1 DNS over HTTP/2

Nice. Save me time digging. I did some quick test. From my geolocation, all three DNS servers (1.1.1.1, 8.8.8.8, 9.9.9.9) are near-by in terms of latency.

Network latency: 1.1.1.1 and 8.8.8.8 are on par. The big guys have resource to setup co-lo near you. 9.9.9.9 is worst.

Overall resolving speed: 8.8.8.8 and 9.9.9.9 are on par. 1.1.1.1 is worst. I guess CF still need time to optimise.

 

[kvic]

over HTTP/2

btw, http/2 is the next miracle going to make browsing experience extremely fast!

When its usage gets popular, I hope to add it into pixelserv-tls. I envision it's around v4 timeframe.

Hope I'm still around doing it (with motivation and enthusiasm. lol).

 

[Butterfly Bones]

Nice. Save me time digging. I did some quick test. From my geolocation, all three DNS servers (1.1.1.1, 8.8.8.8, 9.9.9.9) are near-by in terms of latency.

Network latency: 1.1.1.1 and 8.8.8.8 are on par. The big guys have resource to setup co-lo near you. 9.9.9.9 is worst.

Overall resolving speed: 8.8.8.8 and 9.9.9.9 are on par. 1.1.1.1 is worst. I guess CF still need time to optimise.

I just changed DNSCrypt from manual DNS choices to automatic and was surprised to see CloudFlare as well. It is the fastest of 28 DNS servers that are in the automatic list for me on the west coast USA.

Mar 30 18:04:47 dnscrypt-proxy[22125]: Server with the lowest initial latency: cloudflare (rtt: 21ms)
Mar 30 18:04:49 dnscrypt-proxy[22125]: dnscrypt-proxy is ready - live servers: 28

 

[DonnyJohnny]

Nice. Save me time digging. I did some quick test. From my geolocation, all three DNS servers (1.1.1.1, 8.8.8.8, 9.9.9.9) are near-by in terms of latency.

Network latency: 1.1.1.1 and 8.8.8.8 are on par. The big guys have resource to setup co-lo near you. 9.9.9.9 is worst.

Overall resolving speed: 8.8.8.8 and 9.9.9.9 are on par. 1.1.1.1 is worst. I guess CF still need time to optimise.

guess CF still new but heard it doesn't log. So that is a plus compare to Google.
Another this is DNS spoof test in GRC, CF score better than Google and I believe in their DDOS and poisoning resistance.
Quad9 is bad for me in term of latency. CF anycast is in my location at 7-14ms comparable with Google.

 

[Butterfly Bones]

guess CF still new but heard it doesn't log. So that is a plus compare to Google.
Another this is DNS spoof test in GRC, CF score better than Google and I believe in their DDOS and poisoning resistance.
Quad9 is bad for me in term of latency. CF anycast is in my location at 7-14ms comparable with Google.

Very new it turns out.
https://www.wired.com/story/new-encryption-service-adds-privacy-protection-for-web-browsing

Scanning the syslog I saw this in the scan of servers.

Mar 30 14:04:26 dnscrypt-proxy[22125]: [cloudflare] OK (DoH) - rtt: 47ms

DoH is DNS over HTTPS, a very good thing.

 

[kvic]

Very new it turns out.
https://www.wired.com/story/new-encryption-service-adds-privacy-protection-for-web-browsing

DoH is DNS over HTTPS

This is alarming!

I recall reading Google Chrome had been experimenting with resolving DNS over HTTPS long time ago. According to the article it seems this is in the process of being made into a standard. Firefox is onboard as well.

I hope browsers still give users the freedom of picking their DNS servers (including their own running on the router). Otherwise, in practice, browsers bypass user preference, and defeat any DNS-based adblock solutions (as it's getting increasingly popular over the past few years).

I won't be surprised one of the main driving forces for Google is to defeat DNS-based adblock.

 

[pattiri]

I think I've managed to crash rc3. I can ping it but servstat page is inaccessible and there isn't pixelserv in "htop". Latest pixelservs logs are;

Mar 31 09:42:41 Home pixelserv-tls: 172.24.5.2 syndication.twitter.com POST /i/jot HTTP/1.1 secure
Mar 31 09:42:41 Home pixelserv-tls: [dnt=0&tfw_redirect=https%3A%2F%2Fplatform.twitter.com%2Fjot.html&l=%7B%22widget_origin%22%3A%22https%3A%2F%2Ffivethirtyeight.com%2Ffeatures%2Fhow-a-massachusetts-republican-became-one-of-americas-most-popular-politicians%2F%22%2C%22widget_frame%22%3Afalse%2C%22widget_site_screen_name%22%3A%22FiveThirtyEight%22%2C%22widget_site_user_id%22%3A%222303751216%22%2C%22widget_creator_screen_name%22%3A%22perrybaconjr%22%2C%22widget_creator_user_id%22%3A%2220815668%22%2C%22
Mar 31 09:42:43 Home pixelserv-tls: 172.24.5.2 hello.myfonts.net GET /count/29ffaf HTTP/1.1 secure
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 pixel.wp.com GET /g.gif?x_stats-initial-visibility=visible&v=wpcom-no-pv&rand=0.3488582090030202 HTTP/1.1
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 pixel.wp.com GET /g.gif?crypt=UE40eW5QN0p8M2Y%2FRE1BNmNJfGhxNCVxUDExYmtXTGhnRyxzMiZsZ2k2NWxwVnlQUElWXTI5eS5wODglRjBDNGJsNlJnN2RdbWlufE9hdU55UVBqMmEsW2R6ckZbb1EySFpJPywyMk5RdWF4NXZjZHpUWV1BVzR%2BTHVIP2VNWERXX0dMYzJzd3IleC9VWi8sJnU4d1EwQ0dEWXosPzJnJnE5T1dYYUdheFJJaFJjNyU%2FfDYlfD9uRXd1UGN6V3hTW0V4b01LdmhjaEp1ODJMVHl%2BZ193ZHlhSnpxeHNfOSx4R2ZhRGNGLVBsWlRx&v=wpcom-no-pv&rand=0.09848432153652076 HTTP/1.1
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 pixel.wp.com GET /g.gif?blog=64146350&v=wpcom&tz=-4&user_id=0&post=10521&subd=espnfivethirtyeight&host=fivethirtyeight.com&ref=&rand=0.1719868590884338 HTTP/1.1
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 www.googletagservices.com GET /tag/js/gpt.js HTTP/1.1 secure
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 syndication.twitter.com GET /settings HTTP/1.1 secure
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 dpm.demdex.net GET /id?d_visid_ver=1.5.6&d_rtbd=json&d_ver=2&d_verify=1&d_orgid=EE0201AC512D2BE80A490D4C%40AdobeOrg&d_nsid=0&d_cb=s_c_il%5B0%5D._setMarketingCloudFields HTTP/1.1
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 static.chartbeat.com GET /js/chartbeat_mab.js HTTP/1.1 secure
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 tt.onthe.io GET /?k[]=29313:time[page:Main%20Page]&s=1e153215c1c2929dd0cd1756c97f96d8&1522478561710 HTTP/1.1 secure
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 tags.bkrtx.com GET /js/bk-coretag.js?xhr=1 HTTP/1.1 secure
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 static.chartbeat.com GET /js/chartbeat_video.js?xhr=1 HTTP/1.1 secure
Mar 31 09:42:44 Home pixelserv-tls: 172.24.5.2 b.scorecardresearch.com GET /b?c1=2&c2=3000005&ns__t=1522478561986&ns_c=UTF-8&cv=3.1&c8=FiveThirtyEight%20%7C%20Nate%20Silver%E2%80%99s%20FiveThirtyEight%20uses%20statistical%20analysis%20%E2%80%94%20hard%20numbers%20%E2%80%94%20to%20tell%20compelling%20stories%20about%20politics%2C%20sports%2C%20science%2C%20economics%20and%20culture.&c7=http%3A%2F%2Ffivethirtyeight.com%2F&c9= HTTP/1.1
Mar 31 09:42:46 Home pixelserv-tls: 172.24.5.2 espndotcom.tt.omtrdc.net GET /m2/espndotcom/mbox/json?screenHeight=1440&screenWidth=2560&colorDepth=24&browserWidth=2560&browserHeight=1303&browserTimeOffset=180&mboxPage=8d2aeb5248a04064912462d8f23c7cf3&mboxVersion=0.9.2&mboxHost=fivethirtyeight.com&mboxURL=http%3A%2F%2Ffivethirtyeight.com%2F&mboxReferrer=&mboxSession=ae35b82f39bf4c61854699ef17b6ce2a&mboxPC=&mboxTime=1522489363686&mbox=ESPN_NT_Global&mboxCount=1 HTTP/1.1 secure
Mar 31 09:42:48 Home pixelserv-tls: 172.24.5.2 velocecdn.com GET /script/compatibility.js HTTP/1.1 secure
Mar 31 09:42:48 Home pixelserv-tls: 172.24.5.2 www.google-analytics.com GET /analytics.js HTTP/1.1 secure
Mar 31 09:42:49 Home pixelserv-tls: 172.24.5.2 velocecdn.com GET /script/compatibility.js HTTP/1.1 secure
Mar 31 09:42:49 Home pixelserv-tls: 172.24.5.2 www.google-analytics.com GET /analytics.js HTTP/1.1 secure
Mar 31 09:42:50 Home pixelserv-tls: 172.24.5.2 syndication.twitter.com POST /i/jot HTTP/1.1 secure
Mar 31 09:42:50 Home pixelserv-tls: [dnt=0&tfw_redirect=https%3A%2F%2Fplatform.twitter.com%2Fjot.html&l=%7B%22widget_origin%22%3A%22https%3A%2F%2Ffivethirtyeight.com%2F%22%2C%22widget_frame%22%3Afalse%2C%22widget_site_screen_name%22%3A%22FiveThirtyEight%22%2C%22widget_site_user_id%22%3A%222303751216%22%2C%22language%22%3A%22en%22%2C%22message%22%3A%22m%3Awithcount%3A%22%2C%22_category_%22%3A%22tfw_client_event%22%2C%22triggered_on%22%3A1522478567712%2C%22dnt%22%3Afalse%2C%22client_version%22%3A%226ab
Mar 31 09:42:53 Home pixelserv-tls: 172.24.5.2 tt.onthe.io GET /?k[]=29313:time[page:Main%20Page]&s=1e153215c1c2929dd0cd1756c97f96d8&1522478570718 HTTP/1.1 secure
Mar 31 09:43:04 Home pixelserv-tls: 172.24.5.2 tt.onthe.io GET /?k[]=29313:time[page:Main%20Page]&s=1e153215c1c2929dd0cd1756c97f96d8&1522478581724 HTTP/1.1 secure
Mar 31 09:43:04 Home pixelserv-tls: client 172.24.5.2 ssl error:14094412:lib(20):func(148):reason(1042)
Mar 31 09:43:14 Home pixelserv-tls: 172.24.5.2 tt.onthe.io GET /?k[]=29313:time[page:Main%20Page]&s=1e153215c1c2929dd0cd1756c97f96d8&1522478591728 HTTP/1.1 secure
Mar 31 09:43:14 Home pixelserv-tls: 172.24.5.2 w88.espn.com GET /id?d_visid_ver=1.5.6&callback=s_c_il%5B0%5D._setAnalyticsFields&mcorgid=EE0201AC512D2BE80A490D4C%40AdobeOrg&mid=28627361166322715192434296472399301362 HTTP/1.1

I won't be at home for a time but I can check whatever you want later

 

[kvic]

I think I've managed to crash rc3. I can ping it but servstat page is inaccessible and there isn't pixelserv in "htop".

I won't be at home for a time but I can check whatever you want later

Very good!

Looking at the logs, I can't tell what led to the crash. I need some reproducible steps. e.g what websites that you visited right before leading to the crash.

Perhaps you shall try to reproduce the crash when you get time. thanks

edit:

looking closer at the log, I suspect the POST request to syndication.twitter.com might have caused it. Will be difficult for me to guess what trigger this request. However, if you try to recall what actions performed before the crash, shall be quick to reproduce.