pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[penguin22]

Responding to myself, I found that disabling AI Protection, which is arguably useful at best, recovered a decent chunk of memory, bringing the router to 75% in use rather than 95%. With the services running through Entware, I'm comfortable leaving it off as it hasn't done much to date and the other services are better uses of memory.

 

[kvic]

Per htop, Pixelserv-tls is sitting at 3.8% mem.

The memory usage is about 19MB that is very similar to the previous peak using the old libopenssl.

Let's confirm a few things first:
have you installed the new libopenssl from this post?
And restarted pixelserv-tls after updating libopenssl?
what's the output of "opkg list-installed |grep libopenssl"?

 

[penguin22]

The memory usage is about 19MB that is very similar to the previous peak using the old libopenssl.

Let's confirm a few things first:
have you installed the new libopenssl from this post?
And restarted pixelserv-tls after updating libopenssl?
what's the output of "opkg list-installed |grep libopenssl"?

Yes, however I did update to 384.5-alpha2 since then, so it's possible that the version is different than expected.
Output is: libopenssl - 1.0.2n-1b

 

[kvic]

Yes, however I did update to 384.5-alpha2 since then, so it's possible that the version is different than expected.
Output is: libopenssl - 1.0.2n-1b

Okay, this is the reason in bold. You're still using the old libopenssl. That explains the "high" memory usage by pixelserv-tls.

Give the new lib a try. You'll be surprised how little pixelserv-tls ends up using.

 

[penguin22]

Okay, this is the reason in bold. You're still using the old libopenssl. That explains the "high" memory usage by pixelserv-tls.

Give the new lib a try. You'll be surprised how little pixelserv-tls ends up using.

I'm not doing so intentionally:

p22admin@RT-AC86U:/tmp/home/root# opkg list-installed |grep libopenssl
libopenssl - 1.0.2n-1b
p22admin@RT-AC86U:/tmp/home/root# opkg install http://bin.entware.net/aarch64-k3
.10/test/openssl-util_1.0.2n-1c_aarch64-3.10.ipk
Downloading http://bin.entware.net/aarch64-k3.10/test/openssl-util_1.0.2n-1c_aarch64-3.10.ipk
Installing openssl-util (1.0.2n-1c) to root...
Configuring openssl-util.
p22admin@RT-AC86U:/tmp/home/root# opkg list-installed |grep libopenssl
libopenssl - 1.0.2n-1b
p22admin@RT-AC86U:/tmp/home/root# /opt/etc/init.d/S80pixelserv-tls restart
 Shutting down pixelserv-tls...              done.
 Starting pixelserv-tls (AB-Solution)...              done.
p22admin@RT-AC86U:/tmp/home/root# opkg list-installed |grep libopenssl
libopenssl - 1.0.2n-1b

 

[Butterfly Bones]

@penguin22

Perhaps you were right about the Alpha FW. Seems an issue related to filesystem.

In fact, I heard quite a few pecularities regarding the filesystems on RT86...special handling of /opt (where Entware mounts)..."mount -a" (unrelated to this thread) takes no effect..

We need someone with RT86 to step in.

This is what I have as well on the 86U.

/tmp/home/root# opkg list-installed |grep openssl
libopenssl - 1.0.2n-1c
openssl-util - 1.0.2n-1c

That is the one from your post # 1669.

opkg install http://bin.entware.net/aarch64-k3.10/test/openssl-util_1.0.2n-1c_aarch64-3.10.ipk

 

[kvic]

Ah! Perhaps my bad! Should not be openssl-util in that post..

I made a typo in #1669. Should be "libopenssl_1.0.2n-1c_aarch64-3.10.ipk" not openssl-util. I corrected it now.

Very sorry @penguin22. Could you re-update the libopenssl and restart pixelserv-tls again?

edit:

@Butterfly Bones
What happened was that..on your Entware installation, install "openssl-util" triggered the pulling of its dependency which is libopenssl. So you got both installed which is fine.

Somehow on @penguin22's Entware install, the dependency didn't get download and installed.

This in itself is an issue that could be related to various reasons...perhaps filesystem related. But in #1669, I should put libopenssl for ARMv8 64-bit not openssl-util to begin with to be precise and concise.

 

[DonnyJohnny]

Okay, this is the reason in bold. You're still using the old libopenssl. That explains the "high" memory usage by pixelserv-tls.

Give the new lib a try. You'll be surprised how little pixelserv-tls ends up using.

Not sure what is the def between 1.02n and 1.02o in 38.5 alpha 2. Is there a libopenssl for 1.02O?
https://github.com/RMerl/asuswrt-merlin.ng/commit/681b42e309f6e7d58ac2f0f613d8cf0d82c17edb

 

[kvic]

Not sure what is the def between 1.02n and 1.02o in 38.5 alpha 2. Is there a libopenssl for 1.02O?
https://github.com/RMerl/asuswrt-merlin.ng/commit/681b42e309f6e7d58ac2f0f613d8cf0d82c17edb

Thanks for the info. This is unrelated. An good overview of Entware is in this post. Recommended for every Entware user..

For all ARMv8 64-bit users, I recommend you do a check to make sure you have the new lib installed.

$ opkg list-installed |grep libopenssl

libopenssl - 1.0.2n-1c

If you get "1c" then you have the new lib, otherwise, pls re-apply the upgrade to new lib as per updated instruction in #1669.

 

[kvic]

I will appreciate that. I have set up syslog-ng and set up a filter to send pixelserv logging to a separate file. Really handy to leave -l 2 or full time. Anyway through that I’ve discovered that a lot of my uca errors come from an Xbox One that go to a wide variety of common ad servers that I’d prefer not redirect to 0.0.0.0 for all clients. It would, I think, largely defeat the whole point of pixelserv at that point.

We could certainly look into suppression control in a future version v2.2 or perhaps later.

In the meantime, I suggest people consider the secondary instance approach instead of redirecting to 0.0.0.0 if you have "quite a lot" redirections of common advert/tracker servers.

Then the only loss is this portion of statistics in the primary instance as both good and known "bad" connections are concentrated on the secondary instance for these servers you redirected.

HOW-TO and init.d script are available from post #1799.

 

[punkinduster]

What is the command line to get the "/jffs/configs/hosts.add" file re-read after adding additional servers?

I got the 2nd pixelserv running last night but only had a chance to add a few servers to the hosts.add file.

Memory usage is only 1.5 and .6 percent so far.

 

[penguin22]

Thanks for the info. This is unrelated. An good overview of Entware is in this post. Recommended for every Entware user..

For all ARMv8 64-bit users, I recommend you do a check to make sure you have the new lib installed.

$ opkg list-installed |grep libopenssl

libopenssl - 1.0.2n-1c

If you get "1c" then you have the new lib, otherwise, pls re-apply the upgrade to new lib as per updated instruction in #1669.

That did the trick and it updated this time with much lower memory being consumed.

 

[quant88]

Try this :
service restart_dnsmasq

What is the command line to get the "/jffs/configs/hosts.add" file re-read after adding additional servers?

I got the 2nd pixelserv running last night but only had a chance to add a few servers to the hosts.add file.

Memory usage is only 1.5 and .6 percent so far.

 

[penguin22]

As a follow-up to my post above with reg from Pixelserv-tls being much higher with Skynet off; loading Skynet through AMTM had restarted it at that moment when I was making the post.

Apr 10 00:03:25 Skynet: [INFO] Startup Initiated... ( skynetloc=/tmp/mnt/data/skynet )
Apr 10 00:03:46 Skynet: [Complete] 104019 IPs / 1615 Ranges Banned. 0 New IPs / 0 New Ranges Banned. 0 Inbound / 0 Outbound Connections Blocked! [start] [21s]

With Skynet on, req is very low (by about a factor of 10) as I assume Skynet is handling them, but with it off, req shoots up. The net effect is that slu is proportionally higher when Skynet is running since there are overall far fewer req to process. I say that here as if you are running both services (AB-Solution with Pixelserv-tls and Skynet), you may find that the tandem of the two greatly affects the expected results.

 

[kvic]

With Skynet on, req is very low (by about a factor of 10) as I assume Skynet is handling them, but with it off, req shoots up.

Sounds to me between Skynet and ABS, one guy is doing redundant work. You're basically banning yourself i.e. the pixelserv ip.

 

[kvic]

OpenSSL test libraries from Entware.

You'll have to manually install the test library (intructions below).

Then, restart pixelserv-tls through command line (recommended) to take effect:

/opt/etc/init.d/S80pixelserv-tls restart

If you suspect running into any weird issues in other Entware applications, then pls do a reboot. Otherwise, not mandatory.

Installation - ARMv7

opkg install http://bin.entware.net/armv7sf-k2.6/test/libopenssl_1.0.2n-1c_armv7-2.6.ipk

Installation - ARMv8 64-bit

opkg install http://bin.entware.net/aarch64-k3.10/test/libopenssl_1.0.2n-1c_aarch64-3.10.ipk

Revert back to stock Entware - ARMv7 & ARMv8

opkg --force-reinstall install libopenssl

We shall thank @zyxmon (and @ryzhov_al) to provide the test libraries, and listen to feedback from us all.

@ryzhov_al @zyxmon

I believe we can first conclude the testing of the new openssl library. Based on my own tests and other folks feedback in this thread, the result is positive.

I don't see any impact on crypto performance. Medium to high SSL workload improves by >12% in terms of speed, not to mention the tremendous saving in memory.

>12% improvement is based on tav in pixelserv-tls which has a proven record of consistent measurement in the past half year or so.

So thanks to all pixelserv-tls users participating in this new library test. Very good rapport all around on this effort. Hope more Entware users and other people will appreciate the effort we all do on this.

 

[penguin22]

Sounds to me between Skynet and ABS, one guy is doing redundant work. You're basically banning yourself i.e. the pixelserv ip.

Using the + blocklist with ABS should mean that it plays well with Skynet, however I wonder if it is just that Skynet is handling the blocks first, ahead of ABS and thus Pixelserv-tls doesn't need to handle it. The Pixelserv-tls IP is not being blocked by Skynet, so don't think that's the case here, nor that it is necessarily a problem.

 

[kvic]

Using the + blocklist with ABS should mean that it plays well with Skynet, however I wonder if it is just that Skynet is handling the blocks first, ahead of ABS and thus Pixelserv-tls doesn't need to handle it. The Pixelserv-tls IP is not being blocked by Skynet, so don't think that's the case here, nor that it is necessarily a problem.

For ip blocking with iptables + ipset + dns, it's possible that upon DNS lookup, ipset/iptables block it. Hence, packets not reaching pixelserv-tls.

Looks very likely the case. I think that's something for you or ppl having the same observation to figure out. If proven one guy is doing redundant work, it shall be eliminated. it's just waste of CPU cycles...

Separately a reminder to folks running with large host files, i saw some of your dnsmasq consume 30MB to 70MB (?) memory, imo that's over doze. Memory perhaps can be left for better use. And unnecessarily slows down your DNS queries.

 

[penguin22]

For ip blocking with iptables + ipset + dns, it's possible that upon DNS lookup, ipset/iptables block it. Hence, packets not reaching pixelserv-tls.

Looks very likely the case. I think that's something for you or ppl having the same observation to figure out. If proven one guy is doing redundant work, it shall be eliminated. it's just waste of CPU cycles...

Separately a reminder to folks running with large host files, i saw some of your dnsmasq consume 30MB to 70MB (?) memory, imo that's over doze. Memory perhaps can be left for better use. And unnecessarily slows down your DNS queries.

I see dnsmasq sitting steadily at 9.3% mem with a single instance running, so doesn't appear to be an issue here. DNScrypt runs across 10 threads consuming 4.9% per, so 49% memory, but considering the function, that seems expected. Then comes Pixelserv-tls with varies thread based on workload, however very low mem at 1.3% per. I saw it this low previously, so suspect that libopenssl had be overwritten with an older version with either the upgraded alpha firmware or update to one of the applications installed through amtm. I greatly appreciate that you identified how to update it independent of the full openssl library and corrected the post.

I suspect that most on here are in the same boat as me with trying to determine what to use on our router to provide a stable, secure, <insert outcome here> feature set, it takes a little bit of playing around and determining what is needed and how it interacts with other things.

From my perspective, I have the following running, which I believe generally function in the order listed (not necessarily installed in this order):

1. Asuswrt-Merlin fw on RT-AC86U (obvious starting point)
2. OpenVPN (connected to PIA @ SHA256)
   
3. Entware (on primary partition of 32GB USB flash)
   
4. Swap (on separate partition)
5. dnscrypt (Auto w/dnssec & no logging)
   
6. Skynet (appears to filter ahead of AB-Solution w/ Pixelserv-tls (from the other thread, I believe this is expected)
7. AB-Solution w/ Pixelserv-tls & dnsmasq (using AdsBeGone!+, appears to get what isn't filtered by Skynet, thus the change in req volume when Skynet is temporarily disabled)

Considering all that is happening, it's not hard to see why people can get themselves confused or in a pickle with configurations. Tools created to help simplify the process and setup, like amtm, are great values to the community along with the people themselves, yourself most certainly included!

 

[kvic]

@penguin22 I would stop here without digressing into a debate. For me it's a done job. I'm afraid my comments will upset quite many people