pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[Raphie]

5 days and counting....

pixelserv-tls: v2.0.0-rc1 compiled: Nov 26 2017 20:53:17 options: 192.168.1.2
449490 uts, 1 log, 1 kcc, 42 kmx, 1.55 kvg, 220 krq, 23191 req, 953 avg, 28556 rmx, 25 tav, 3194 tmx, 12253 slh, 115 slm, 0 sle, 5432 slc, 350 slu, 2653 nfe, 29 gif, 3 ico, 1396 txt, 0 jpg, 2 png, 0 swf, 13 sta, 7 stt, 59 ufe, 53 opt, 167 pst, 0 hed, 654 rdr, 0 nou, 0 pth, 0 204, 0 bad, 130 tmo, 5509 cls, 8107 cly, 0 clt, 0 err

 

[quant88]

For some reason, pixelserv-tls becomes unresponsive after 2 hours run. Already set - o 10. Will turn on - l 5 on the next restart.

 

[kvic]

For some reason, pixelserv-tls becomes unresponsive after 2 hours run. Already set - o 10. Will turn on - l 5 on the next restart.

When it happens again, can you issue "killall -SIGUSR1 pixelserv-tls" to dump the servstats to syslog. Also try "ping <your pixelserv-tls ip>" and see if any response.

 

[kvic]

Quickly share a piece of finding:

• ver. KK with select_timeout = 1s gives best speed
• v2 is the opposite. 10s yields better speed than 1s.

For v2, perhaps there is an optimal value between 1s and keepalive_timeout. I only tried 1s, 5s, 10s and keepalive_timeout in which 10s gives best speed.

This can be explained by the change in event loop in v2. So you might want to add '-o 10' to your CLI option.

Quite many ppl downloaded v2.0.0-rc1 (and assume you're happily running it). Does any of your v2 get into "stuck" or "hang" state?

 

[jrmwvu04]

Quite many ppl downloaded v2.0.0-rc1 (and assume you're happily running it). Does any of your v2 get into "stuck" or "hang" state?

I haven’t been paying very close attention but I haven’t noticed any problems. That said, I’ve only been running pixelserv with the default timeout options since 2.0.0-rc1 dropped. Prior to v2 I used -o 1.

 

[kvic]

I haven’t been paying very close attention but I haven’t noticed any problems. That said, I’ve only been running pixelserv with the default timeout options since 2.0.0-rc1 dropped. Prior to v2 I used -o 1.

When pixelserv-tls gets into "stuck" or "hang" state, browsing on clients will slow down to a crawl. It's very noticeable. That's also why I'm thinking it might not be a common issue since I don't see many complaints in this thread. lol

I only saw it once during stress test i.e. after 18 million requests in the middle of a KL test build. Didn't pay much attention on the issue back then as I thought home routers eventually disintegrate into such state under heavy workload.

I've been in touch with tom- and protik offline about this issue. @Protik was able to reproduce with '-o 1' a few times and i got some useful traces. Though essentially it points to pixelserv-tls stuck in an abnormal state which very likely caused by bugs in kernels.

Btw while looking at the issue, I found '-o 10' is much better speed for v2. @jrmwvu04, you shall switch to use it. I'll default back to 10s or perhaps another higher & optimal value in a future release.

 

[jrmwvu04]

Btw while looking at the issue, I found '-o 10' is much better speed for v2. @jrmwvu04, you shall switch to use it. I'll default back to 10s or perhaps another higher & optimal value in a future release.

I installed fork 29A9 and made the change to '-o 10' just beforehand. Will let it run for a bit and see how it goes.

 

[kvic]

v2 beats the previous version in speed. Confirmed.

A tabulated summary of "-o 10" vs "-o 1" in v2 and against the previous version.

The same finding shared yesterday but put in a blog post.

 

[quant88]

When it happens again, can you issue "killall -SIGUSR1 pixelserv-tls" to dump the servstats to syslog. Also try "ping <your pixelserv-tls ip>" and see if any response.

I have updated -o 15 on the second try.

Dec 5 21:50:00 pixelserv-tls[8999]: pixelserv-tls: v2.0.0-rc1 compiled: Nov 26 2017 20:53:17 options: 192.168.1.2 -o 15 -l 5
Dec 5 21:50:00 pixelserv-tls[8999]: Listening on :192.168.1.2:80
Dec 5 21:50:00 pixelserv-tls[8999]: Listening on :192.168.1.2:443
Dec 6 19:56:17 pixelserv-tls[8999]: 79578 uts, 5 log, 283 kcc, 283 kmx, 2.03 kvg, 29 krq, 5179 req, 782 avg, 3313 rmx, 59 tav, 11625 tmx, 32 slh, 58 slm, 0 sle, 2930 slc, 324 slu, 91 nfe, 2 gif, 42 ico, 18 txt, 0 jpg, 0 png, 0 swf, 43 sta, 3 stt, 4 ufe, 0 opt, 216 pst, 0 hed, 5 rdr, 0 nou, 0 pth, 0 204, 4 bad, 12 tmo, 3073 cls, 0 cly, 0 clt, 0 err

 

[quant88]

I have updated -o 15 on the second try.

Dec 5 21:50:00 pixelserv-tls[8999]: pixelserv-tls: v2.0.0-rc1 compiled: Nov 26 2017 20:53:17 options: 192.168.1.2 -o 15 -l 5
Dec 5 21:50:00 pixelserv-tls[8999]: Listening on :192.168.1.2:80
Dec 5 21:50:00 pixelserv-tls[8999]: Listening on :192.168.1.2:443
Dec 6 19:56:17 pixelserv-tls[8999]: 79578 uts, 5 log, 283 kcc, 283 kmx, 2.03 kvg, 29 krq, 5179 req, 782 avg, 3313 rmx, 59 tav, 11625 tmx, 32 slh, 58 slm, 0 sle, 2930 slc, 324 slu, 91 nfe, 2 gif, 42 ico, 18 txt, 0 jpg, 0 png, 0 swf, 43 sta, 3 stt, 4 ufe, 0 opt, 216 pst, 0 hed, 5 rdr, 0 nou, 0 pth, 0 204, 4 bad, 12 tmo, 3073 cls, 0 cly, 0 clt, 0 err

Want to add: There was ping response from pixelserv-tls ip though html stats was not responsive.

 

[pattiri]

There was ping response from pixelserv-tls ip though html stats was not responsive.

That's same with my earlier problem. Rebooting router or running "pixelserv-tls <listening ip>" fixed that on my router.

 

[kvic]

@quant88 @Protik @tom-

pls use the install-beta.sh script (or otherwise) to get v2.0.1-rc1. It tries to mitigate the "stuck" issue with a possible cause I could dream of last night.

Changes in v2.0.1-rc1

• mitigate "stuck" issue
• select_timeout default to 10s

 

[Raphie]

Worked perfect via amtm

 

[quant88]

@quant88 @Protik @tom-

pls use the install-beta.sh script (or otherwise) to get v2.0.1-rc1. It tries to mitigate the "stuck" issue with a possible cause I could dream of last night.

Changes in v2.0.1-rc1

• mitigate "stuck" issue
• select_timeout default to 10s

I have tried with new beta but still faced stuck issue. Let me know which other tests you may need. I have installed pixelserv-tls in 2 homes - The other AP (also a Ac88u) is running smoothly with the first beta though. The one seeing the issue is connected with many clients and 1-2 repeaters.

I want to Thank you for the amazing work!

Dec 7 06:59:58 pixelserv-tls[656]: 31059 uts, 1 log, 4 kcc, 30 kmx, 2.09 kvg, 17 krq, 3147 req, 1054 avg, 3315 rmx, 26 tav, 7592 tmx, 41 slh, 53 slm, 1 sle, 1888 slc, 172 slu, 31 nfe, 0 gif, 14 ico, 2 txt, 0 jpg, 0 png, 0 swf, 16 sta, 0 stt, 3 ufe, 0 opt, 156 pst, 0 hed, 0 rdr, 0 nou, 0 pth, 0 204, 0 bad, 7 tmo, 1924 cls, 0 cly, 0 clt, 0 err

 

[baltosml]

When it happens again, can you issue "killall -SIGUSR1 pixelserv-tls" to dump the servstats to syslog. Also try "ping <your pixelserv-tls ip>" and see if any response.

I will save this link... I got stuck this evening, and I could not browse to this link to get the -SIGUSR1 command to post the stats.

Restarting the service immediately restored normal browsing action on my network.

 

[kvic]

I'm repeating my stress test. After 140 million requests and still no luck getting stuck..

I started a conversation with @quant88 @baltosml to get more traces.

 

[Raphie]

I will save this link... I got stuck this evening, and I could not browse to this link to get the -SIGUSR1 command to post the stats.

Restarting the service immediately restored normal browsing action on my network.

Maybe we can add this to amtm ? @thelonelycoder ?

 

[froggy666uk]

Hi kvic,

I'm having the same problem with v2.0.1-rc1 hanging after a few hours of use on an N66u. I decided to compile from source and run on a Raspberry Pi to see if the same problem occurs, in the hope that there might be some more tools available to debug the problem. Sure enough, after a few hours of running it appears to have hung - I can still ping the ip address/telnet on port 443, but attempting to access the servstats page hangs, and opening other websites is slow.

I ran an strace on the process and found the following:

root@raspberrypi:/var/log# strace -p 9869
Process 9869 attached
read(9,


lsof -i^CProcess 9869 detached

...so it appears to be hanging on read.

I ran lsof and netstat to see what files/sockets were open:

root@raspberrypi:/proc/9869# lsof -p 9869
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
pixelserv 9869 nobody cwd DIR 179,2 4096 2 /
pixelserv 9869 nobody rtd DIR 179,2 4096 2 /
pixelserv 9869 nobody txt REG 179,2 39848 1792 /usr/local/bin/pixelserv-tls
pixelserv 9869 nobody mem REG 179,2 46820 1969 /lib/arm-linux-gnueabihf/libnss_files-2.19.so
pixelserv 9869 nobody mem REG 179,2 38612 1977 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so
pixelserv 9869 nobody mem REG 179,2 71628 1966 /lib/arm-linux-gnueabihf/libnsl-2.19.so
pixelserv 9869 nobody mem REG 179,2 30592 1967 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so
pixelserv 9869 nobody mem REG 179,2 9820 1940 /lib/arm-linux-gnueabihf/libdl-2.19.so
pixelserv 9869 nobody mem REG 179,2 1242776 1932 /lib/arm-linux-gnueabihf/libc-2.19.so
pixelserv 9869 nobody mem REG 179,2 1418532 41716 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
pixelserv 9869 nobody mem REG 179,2 300868 41717 /usr/lib/arm-linux-gnueabihf/libssl.so.1.0.0
pixelserv 9869 nobody mem REG 179,2 122308 1992 /lib/arm-linux-gnueabihf/libpthread-2.19.so
pixelserv 9869 nobody mem REG 179,2 18920 10032 /usr/lib/arm-linux-gnueabihf/libarmmem.so
pixelserv 9869 nobody mem REG 179,2 134448 1897 /lib/arm-linux-gnueabihf/ld-2.19.so
pixelserv 9869 nobody 0u CHR 1,3 0t0 1029 /dev/null
pixelserv 9869 nobody 1u CHR 1,3 0t0 1029 /dev/null
pixelserv 9869 nobody 2u CHR 1,3 0t0 1029 /dev/null
pixelserv 9869 nobody 3u unix 0xb91f8c00 0t0 1395555 socket
pixelserv 9869 nobody 4u IPv4 1395558 0t0 TCP *:81 (LISTEN)
pixelserv 9869 nobody 5r FIFO 0,10 0t0 1395560 pipe
pixelserv 9869 nobody 6u IPv4 1395559 0t0 TCP *:https (LISTEN)
pixelserv 9869 nobody 7w FIFO 0,10 0t0 1395560 pipe
pixelserv 9869 nobody 9u IPv4 1466930 0t0 TCP 192.168.1.13:https->android-dad30b14da36932a:58928 (ESTABLISHED)

root@raspberrypi:/proc# netstat -anp | grep 9869
tcp 0 0 0.0.0.0:81 0.0.0.0:* LISTEN 9869/pixelserv-tls
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 9869/pixelserv-tls
tcp 0 0 192.168.1.13:443 192.168.1.207:58928 ESTABLISHED 9869/pixelserv-tls
unix 2 [ ] DGRAM 1395555 9869/pixelserv-tls

...so I'd assume it is hanging attempting read from the following socket:

pixelserv 9869 nobody 9u IPv4 1466930 0t0 TCP 192.168.1.13:https->android-dad30b14da36932a:58928 (ESTABLISHED)

I attached gdb to the pixelserv process and ran the following to kill this socket:

(gdb) p close(9)
$1 = 0
(gdb) quit

...and sure enough, pixelserv started working again!

Hopefully be of use...

Cheers,
Dave.

 

[kvic]

@froggy666uk Thank you! This is helpful.

The output of strace/lsof is similar to what I got from @Protik.

I suspect the issue is that read() performs on file descriptor 9 which is a TCP socket. It's expected to read from a FIFO pipe id 5.

Indeed if that's the case, I haven't figured out how it gets into such a weird state. Do u get a chance to try v2.0.1-rc1 and does it stall at the same place?

 

[froggy666uk]

@froggy666uk Thank you! This is helpful.

The output of strace/lsof is similar to what I got from @Protik.

I suspect the issue is that read() performs on file descriptor 9 which is a TCP socket. It's expected to read from a FIFO pipe id 5.

Indeed if that's the case, I haven't figured out how it gets into such a weird state. Do u get a chance to try v2.0.1-rc1 and does it stall at the same place?

The build I was using on my n66u was v2.0.0-rc1, my raspberry pi build was based on the github master branch. Have your latest v2.0.1-rc1 changes been committed to github master?

Could it be a problem peculiar to android devices? Have you tested using android?

Anything else I could look at on my raspberry pi that might help next time it happens?