pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[penguin22]

@penguin22 I would stop here without digressing into a debate. For me it's a done job. I'm afraid my comments will upset quite many people

I'll make sure not to place any more straws on my camel's back.

 

[punkinduster]

I am happy to say I have successfully installed 2 pixelserv-tls on my router. I also put some ad servers in the hosts.add file that were throwing handshake errors because they either didn't like my certificate or I can't upload the CA to the device. The hosts.add file directed them to the 2nd pixelserv running at Log level 1 so I don't have to see all of the handshake errors in my syslog. Both pixelservs together are only using about 2 percent memory combined. Now I'll run it for a day and post the numbers tomorrow.

Thanks kvic and others for your help.

 

[Makaveli]

So last night I upgraded my firmware from 69_2 to 70_0.

My normal process is to shut down
Skynet
AB-solution
Pixelserv

Then I unmount the USB drive and do the firmware flash.

After the flash is completed I insert back the USB drive restarted those 3 scripts then do a full reboot.

So the interesting thing I noticed after doing that i'm seeing alot lower slc in the same time span.

pixelserv-tls 2.1.0-rc.4 (compiled: Apr 5 2018 21:02:08) options: 192.168.1.3 -l 2

uts    0d 21:06    process uptime
log    2    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    8    maximum number of service threads
kvg    1.26    average number of requests per service thread
krq    15    max number of requests by one service thread
req    769    total # of requests (HTTP, HTTPS, success, failure etc)
avg    630 bytes    average size of requests
rmx    3676 bytes    largest size of request(s)
tav    8 ms    average processing time (per request)
tmx    195 ms    longest processing time (per request)
slh    151    # of accepted HTTPS requests
slm    0    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    443    # of dropped HTTPS requests (client disconnect without sending any request)
slu    125    # of dropped HTTPS requests (other TLS handshake errors)
uca    3    slu break-down: # of unknown CA reported by clients
uce    25    slu break-down: # of unknown cert reported by clients
sct    42    cert cache: # of certs in cache
sch    605    cert cache: # of reuses of cached certs
scm    5    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    4    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    130    sess cache: # of reuses of cached TLS sessions
ssm    103    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

I'm still monitoring the logs and haven't done anything about this yet.


Apr 10 09:06:25 pixelserv-tls[1024]: handshake failed: unknown cert. client 192.168.1.11:37942 server data.flurry.com
Apr 10 09:06:30 pixelserv-tls[1024]: handshake failed: unknown cert. client 192.168.1.11:37958 server www.googleadservices.com
Apr 10 09:20:44 pixelserv-tls[1024]: handshake failed: unknown cert. client 192.168.1.11:38003 server data.flurry.com
Apr 10 09:20:44 pixelserv-tls[1024]: handshake failed: unknown cert. client 192.168.1.11:38005 server data.flurry.com
Apr 10 09:20:44 pixelserv-tls[1024]: handshake failed: unknown cert. client :38005 server data.flurry.com
Apr 10 09:20:44 pixelserv-tls[1024]: handshake failed: unknown cert. client 192.168.1.11:38007 server data.flurry.com
Apr 10 09:25:14 pixelserv-tls[1024]: handshake failed: unknown CA. client 192.168.1.11:38030 server graph.instagram.com

Apr 10 01:56:38 pixelserv-tls[1024]: handshake failed: unknown cert. client 192.168.1.11:37771 server z.moatads.com
Apr 10 01:56:38 pixelserv-tls[1024]: handshake failed: unknown cert. client 192.168.1.11:37772 server config.ioam.de
Apr 10 02:21:43 pixelserv-tls[1024]: handshake failed: unknown cert. client 192.168.1.11:37787 server z.moatads.com
Apr 10 02:53:06 pixelserv-tls[1024]: handshake failed: unknown cert. client 192.168.1.11:37799 server z.moatads.com

 

[JaimeZX]

A connection hanging there forever indicates that clients on your guests network cannot reach 192.168.2.3 (assume that's your pixelserv ip) at all.

I haven't looked into guest networks in detail. Hence, cannot quickly tell how guest/normal network segregation is done.

I assume your guest network can access the DNS server on your router. If so, that will be a good hint how it's provisioned to guest network. Very likely it's a simple iptables rule.

Once confirmed, you could simply duplicate the iptables rule for IP address 192.168.2.3. That I believe shall solve your issue. If still can't please post more details, such as your DNS ip, iptables output...

Yep. So now after regenerating the certificate - I realize this may be the issue too. Nearly all my machines are on the Guest Network. I only have a couple trusted devices on my "LAN." So really, does that mean Pixelserv is not actually that useful for most of these things? Else, how do I set up a rule in the router firewall to allow the guest network (read?) access to .3?

 

[kvic]

After the flash is completed I insert back the USB drive restarted those 3 scripts then do a full reboot.

So the interesting thing I noticed after doing that i'm seeing alot lower slc in the same time span.

Probably just a coincidence...

Nearly all my machines are on the Guest Network. I only have a couple trusted devices on my "LAN." So really, does that mean Pixelserv is not actually that useful for most of these things?

I think most of your router's features as well as add-ons won't apply to guest networks.

My guess is that it shall be playing around with iptables rules. So perhaps you could digging further on that direction.

 

[JaimeZX]

I think most of your router's features as well as add-ons won't apply to guest networks.

Well I assume Skynet still works, and AIProtection, eh? Hopefully they're at the front door, else the guest networks are essentially on the DMZ. I guess on one hand I'm not surprised about not being able to access .3 from Guest network because I don't have/wouldn't want access to the router GUI from a guest network; OTOH AB-S must also be working since if I disable my Adblock browser plugin I still get no ads... So something is going right.

My guess is that it shall be playing around with iptables rules. So perhaps you could digging further on that direction.

*sigh* Yeah, seems like I'll have to learn more about that. Non-trivial...

 

[kvic]

The Km development cycle is near the closing.

I scrolled back for the past posts. It's just too many to go through and I can't do. So some long time and frequent handles I could recall on top of my head @jrmwvu04 @elorimer @Protik @quant88 @Butterfly Bones @SMS786 @Raphie @pattiri as well as those few gents right above this post. Please don't feel left out. All feedback helps and THANK YOU.

In the recent two cycles, we had run into notable issues that were resolved. Not achievable without your constant feedback and testing. In the KL cycle, we had the "stuck" issue. In the latest Km cycle, we had two - the slu issue and the memory issue. Both understood and resolved.

In the last minutes, a gent on GitHub raised a..potential race condition issue that was rectified in the latest code as well. With this refactoring, pixelserv-tls is also more presentable in source code form. The remaining "culprit" that I don't like is socket_handler.c that perhaps will be replaced all together in v3.

In fact, I recently learned the idea and some bit of codes (I suspect) in pixelserv-tls is already used in a closed source China freeware. Well...I take it as a compliment.

Looking back I think Km cycle is much bigger in terms of people's involvement as well as new features. Probably it should have been called v3. But since we've assigned v2.1 and reserved v3 for paradigm shift on our roadmap, let's stick to it. Version number alone means little. Stick to the plan and work out as planned. That's huge.

In the next Kn cycle, lots of things could improve in terms of codes, features, testing as well as user engagement. Don't have a fixed date yet but I need a long break from this forum to get rid of "addiction syndrome."

Many good features could be added to pixelserv-tls. However, as of today, it also does its job and gets it done extremely fast. A companion product perhaps worth a higher priority, so on my spare time I'll think more about that. If decide to do it, initially perhaps will be close source. It'll be a unique piece of software that I haven't seen on this planet yet. lol

I'll upload the final binaries v2.1.0 later today.

 

[SMS786]

The Km development cycle is near the closing.

I scrolled back for the past posts. It's just too many to go through and I can't do. So some long time and frequent handles I could recall on top of my head @jrmwvu04 @elorimer @Protik @quant88 @Butterfly Bones @SMS786 @Raphie @pattiri as well as those few gents right above this post. Please don't feel left out. All feedback helps and THANK YOU.

In the recent two cycles, we had run into notable issues that were resolved. Not achievable without your constant feedback and testing. In the KL cycle, we had the "stuck" issue. In the latest Km cycle, we had two - the slu issue and the memory issue. Both understood and resolved.

In the last minutes, a gent on GitHub raised a..potential race condition issue that was rectified in the latest code as well. With this refactoring, pixelserv-tls is also more presentable in source code form. The remaining "culprit" that I don't like is socket_handler.c that perhaps will be replaced all together in v3.

In fact, I recently learned the idea and some bit of codes (I suspect) in pixelserv-tls is already used in a closed source China freeware. Well...I take it as a compliment.

Looking back I think Km cycle is much bigger in terms of people's involvement as well as new features. Probably it should have been called v3. But since we've assigned v2.1 and reserved v3 for paradigm shift on our roadmap, let's stick to it. Version number alone means little. Stick to the plan and work out as planned. That's huge.

In the next Kn cycle, lots of things could improve in terms of codes, features, testing as well as user engagement. Don't have a fixed date yet but I need a long break from this forum to get rid of "addiction syndrome."

Many good features could be added to pixelserv-tls. However, as of today, it also does its job and gets it done extremely fast. A companion product perhaps worth a higher priority, so on my spare time I'll think more about that. If decide to do it, initially perhaps will be close source. It'll be a unique piece of software that I haven't seen on this planet yet. lol

I'll upload the final binaries v2.1.0 later today.

Congrats @kvic! Knowing firsthand the effort with which you hammered out unique individual bugs in this project, you definitely deserve the break. Looking forward to the final binaries and, perhaps, your new project endevour in the future!

 

[jrmwvu04]

I'll upload the final binaries v2.1.0 later today.

 

[punkinduster]

Here is the 2nd pixelserv catching most of the handshake errors.

 

[punkinduster]

Here is the 1st pixelserv stats.

 

[punkinduster]

The 1st pixelserv is using 1.3 percent memory and the 2nd pixelserv is using just 0.7 percent.

Thanks again kvic for your work.

 

[XIII]

How do we install the final version of 2.1.0?

Do we need to restore the pre-beta version for Entware to update?

 

[penguin22]

How do we install the final version of 2.1.0?

Do we need to restore the pre-beta version for Entware to update?

The update appears now if you check with amtm.

 

[XIII]

The update appears now if you check with amtm.

How do you do that?

(I only see a "update pixelserv-tls beta version")

 

[penguin22]

How do you do that?

(I only see a "update pixelserv-tls beta version")

Yes, when running that it installed the following version:
pixelserv-tls 2.1.0 (compiled: Apr 10 2018 22:22:14

 

[jrmwvu04]

Has anyone tried benchmarking with the 2.1.0 final binary? I get a segfault message when it's finished every time.

 

[Butterfly Bones]

Has anyone tried benchmarking with the 2.1.0 final binary? I get a segfault message when it's finished every time.

Yes, just tried after your messages. @kvic - something to note, you can't leave yet.

/tmp/home/root# pixelserv-tls -B
CERT_PATH: /opt/var/cache/pixelserv
CERT_FILE: _.bing.com
 1. generate cert to disk: 64.024 ms    load from disk: 4.837 ms
 2. generate cert to disk: 68.456 ms    load from disk: 4.792 ms
 3. generate cert to disk: 80.053 ms    load from disk: 4.809 ms
 4. generate cert to disk: 67.300 ms    load from disk: 4.837 ms
 5. generate cert to disk: 41.237 ms    load from disk: 4.900 ms
 6. generate cert to disk: 36.127 ms    load from disk: 4.797 ms
 7. generate cert to disk: 44.236 ms    load from disk: 4.831 ms
 8. generate cert to disk: 74.708 ms    load from disk: 4.819 ms
 9. generate cert to disk: 61.335 ms    load from disk: 4.829 ms
10. generate cert to disk: 57.280 ms    load from disk: 4.851 ms
generate to disk average: 59.476 ms
  load from disk average: 4.830 ms
Segmentation fault

 

[Butterfly Bones]

I just found this in the syslog.

Apr 11 09:23:50 kernel: pgd = ffffffc010ce8000
Apr 11 09:23:50 kernel: [00000007] *pgd=0000000010f5c003, *pud=0000000010f5c003, *pmd=0000000000000000
Apr 11 09:23:50 kernel: CPU: 0 PID: 21715 Comm: pixelserv-tls Tainted: P           O    4.1.27 #2
Apr 11 09:23:50 kernel: Hardware name: Broadcom-v8A (DT)
Apr 11 09:23:50 kernel: task: ffffffc01077f5c0 ti: ffffffc008cf0000 task.ti: ffffffc008cf0000
Apr 11 09:23:50 kernel: PC is at 0x7f8f8e6344
Apr 11 09:23:50 kernel: LR is at 0x7f8f8d6eb4
Apr 11 09:23:50 kernel: pc : [<0000007f8f8e6344>] lr : [<0000007f8f8d6eb4>] pstate: 00000000
Apr 11 09:23:50 kernel: sp : 0000007fe54fd790
Apr 11 09:23:50 kernel: x29: 0000007fe54fd9c0 x28: 0000007f8f8dbd70
Apr 11 09:23:50 kernel: x27: 0000007f8f8dbd90 x26: 0000000000000003
Apr 11 09:23:50 kernel: x25: 0000007f8f98bdd0 x24: 0000007f8f98bda8
Apr 11 09:23:50 kernel: x23: 0000007f8f991000 x22: 0000000000000000
Apr 11 09:23:50 kernel: x21: 0000007f8f98bdd0 x20: 0000007f8f991000
Apr 11 09:23:50 kernel: x19: ffffffffffffffff x18: 40c223483a2964f4
Apr 11 09:23:50 kernel: x17: 0000007f8f83f080 x16: 000000000000003f
Apr 11 09:23:50 kernel: x15: 0000000000007fff x14: 0000000000000000
Apr 11 09:23:51 kernel: x13: 203a656761726576 x12: 0000000100000000
Apr 11 09:23:51 kernel: x11: 0000007fe54fd018 x10: 0000000100000000
Apr 11 09:23:51 kernel: x9 : 0000000000000000 x8 : 0000000000000040
Apr 11 09:23:51 kernel: x7 : 000000000a736d20 x6 : 0000000000000001
Apr 11 09:23:51 kernel: x5 : 0000000000000002 x4 : 0000007f8f8dbaa0
Apr 11 09:23:51 kernel: x3 : 0000000000000000 x2 : 0000007f8f986b08
Apr 11 09:23:51 kernel: x1 : 0000000000000003 x0 : ffffffffffffffff

 

[jrmwvu04]

I just found this in the syslog.

Is that in response to benchmarking specifically or something else? I don't see anything like that in my logs.