pixelserv pixelserv - A Better One-pixel Webserver for Adblock

[sfx2000]

But the new processor in RT-86 is much faster in doing crypto as people can see in the above pixelserv-tls benchmark.

~60ms is about one tenth of my usual 600ms on 1.2GHz RT-AC56

The 4709 along with the HND sdk has impressive performance - crypto sticks up, but there's many system level improvements across the board.

 

[Makaveli]

14 Hours in

Memory usage with updated openssl

pixelserv-tls 2.1.0-rc.4 (compiled: Apr 5 2018 21:02:08) options: 192.168.1.3 -l 2

uts    0d 14:05    process uptime
log    2    critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc    1    number of active service threads
kmx    18    maximum number of service threads
kvg    1.07    average number of requests per service thread
krq    9    max number of requests by one service thread
req    1051    total # of requests (HTTP, HTTPS, success, failure etc)
avg    541 bytes    average size of requests
rmx    6111 bytes    largest size of request(s)
tav    13 ms    average processing time (per request)
tmx    244 ms    longest processing time (per request)
slh    87    # of accepted HTTPS requests
slm    2    # of rejected HTTPS requests (missing certificate)
sle    0    # of rejected HTTPS requests (certificate available but bad)
slc    678    # of dropped HTTPS requests (client disconnect without sending any request)
slu    240    # of dropped HTTPS requests (other TLS handshake errors)
uca    154    slu break-down: # of unknown CA reported by clients
uce    5    slu break-down: # of unknown cert reported by clients
sct    46    cert cache: # of certs in cache
sch    931    cert cache: # of reuses of cached certs
scm    9    cert cache: # of misses to find a cert in cache
scp    0    cert cache: # of purges to give room for a new cert
sst    4    sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh    47    sess cache: # of reuses of cached TLS sessions
ssm    69    sess cache: # of misses to find a TLS session in cache
ssp    0    sess cache: # of purges to give room for a new TLS session

i'm seeing this in the syslog which is coming from my android phone.

Apr 7 05:25:14 pixelserv-tls[14322]: handshake failed: unknown CA. client 192.168.1.11:49660 server graph.instagram.com

*Update* I never actually rebooted my andriod phone after I did the cert import. I rebooted it now got prompted if I want to trust the cert and now viewing the log I don't see any more of these entries for now.

 

[JaimeZX]

Since we brought up Android now... every time I boot my phone I get a warning about having an unsigned certificate installed. Any way to permanently hide that?

 

[kvic]

@JaimeZX or @Makaveli

Could you pls post a screenshot of the cert warning after reboot? I haven't seen one myself on Android N.

 

[kvic]

When i turn pixelserv-tls on in AB-Solution, I'm getting error on my Amazon Android Apps the Oops message, someone know how to fix this?

Sorry, I don't know the exact host(s) to whitelist. You may try @visortgw's suggestion.

On the other hand, everyone is encouraged to learn and use pixelserv-tls to figure out domains to whitelist on troubled websites. Here is a How-To:

Figure out domain(s) to whitelist on a broken website.

 

[sfx2000]

I've been using -O3 for close to four years now on my OpenSSL build, I'd say that's more than "comprehensively tested"...

Here's why -03 can be bad - and this is a good example - gcc <option> cpubench.c && ./a.out

One can optimize things into oblivion - @Voxel and I had a good laugh around this code from the OpenWRT folks...

cpubench.c

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
#include <time.h>
#include <unistd.h>

#define VERSION_STR "v0.6"

// This is the overhead of calling gettimeofday() 2 times
double overhead = -1;

double begin_secs = 0;
double real_get_seconds()
{
    struct timeval tv;
    double total;

    gettimeofday(&tv, NULL);
    total = (double)tv.tv_sec;
    total += ((double)tv.tv_usec)/(double)1000000.0;
   
    //printf("Secs: %d / Usecs: %d / FSecs: %f\n", tv.tv_sec, tv.tv_usec, total);

    return total;
}
double get_seconds()
{
    if (begin_secs == 0)
    {
        // First call
        begin_secs = real_get_seconds();
    }

    return (real_get_seconds() - begin_secs);
}

double run_float_bench()
{
    double secs;
    double begin = get_seconds();
    int i;

    double nb=0;
    for (i=0;i<300000;i++)
    {
        nb *= i;
        int n = i >> 10;
        nb /= n;
    }
   
    double end = get_seconds();
    secs = end-begin;
    printf("Time to run float bench: %.2f[secs]\n", secs);

    return secs;
}

double do_run_memory_bench()
{
    double begin, end;
    int i;
    int* buf;
    int len, index;
    double secs;

          begin = get_seconds();
       
    len = (1<<21)/sizeof(int); // 2Mb
    buf = malloc(len*sizeof(int));

    // Write to memory - sequential
    for (i=0;i<len;i++)
    {
        buf[i] = i;
    }

    // Read memory - sequential
    for (i=0;i<len;i++)
    {
        int a = buf[i];
    }

    // Read memory - random
    for (i=0;i<len;i++)
    {
        index = (i*23)%(len/2);
        int a = buf[index];
    }

    // Write memory - random
    for (i=0;i<len;i++)
    {
        index = (i*23)%(len/2);
        buf[index] = i;
    }
    free(buf);
   
    end = get_seconds();
    secs = (end-begin);

    return secs;
}

double run_memory_bench()
{
    double secs = 0;
    int i;

    for (i=0;i<10;i++)
    {
        secs += do_run_memory_bench();
    }

    printf("Time to run memory bench: %.2f[secs]\n", secs);

    return secs;
}

#define NBD 9009
double run_compute_e()
{
    double begin, end;
    double secs;

    begin = get_seconds();

    int N=NBD, n=N, a[NBD],x;
    while(--n)
    {
        a[n]=1+1/n;
    }

    for(;N>9;)
    {
        for(n=N--;--n; a[n]=x%n, x=10*a[n-1]+x/n)
        {
        }
    }

    end = get_seconds();

    secs = (end-begin);
    printf("Time to run computation of e (%d digits): %.2f[secs]\n", NBD, secs);

    return secs;
}

double run_compute_pi()
{
    double begin, end;
    double secs;
    int i;

    begin = get_seconds();
    //printf("Begin: %f\n", begin);

    for (i=0;i<10;i++)
    {
        int a=10000,b=0,c=8400,d=0,e=0,f[8401],g=0;

        //printf("i:%d\n", i);
        for(;b-c;)
        {
            f[b++]=a/5;
        }

        for(;d=0,g=c*2;c-=14,e=d%a)
        {
            for(b=c;d+=f[b]*a,f[b]=d%--g,d/=g--,--b;d*=b)
            {
            }
        }
        //printf("Mid: %lld\n", (get_seconds()-end));
    }

    end = get_seconds();
    //printf("End: %f\n", end);

    secs = (end-begin);
    printf("Time to run computation of pi (2400 digits, 10 times): %.2f[secs]\n", secs);

    return secs;
}

int main()
{
    printf("This is CPU and memory benchmark for OpenWRT "VERSION_STR". This will then take some time... (typically 30-60 seconds on a 200MHz computer)\n");

    double begin = get_seconds();
    double end = get_seconds();
    overhead = (end-begin)*1000000;
    printf("Overhead for getting time: %.0fus\n", overhead);


    // Nb 1
    double sec_mem = run_memory_bench();

    // Nb 2
    double sec_pi = run_compute_pi();

    // Nb 3
    double sec_e = run_compute_e();

    // Nb 4
    double sec_float = run_float_bench();

    printf("Total time: %.1fs\n", (sec_mem+sec_e+sec_pi+sec_float));

    time_t t = time(0);
    struct tm ti;
          localtime_r(&t, &ti);
    printf("\nYou can copy/paste the following line in the wiki table at: http://wiki.openwrt.org/HardwarePerformance\n");

    printf("|| %04d-%02d-%02d || ''Author'' || %.1fs || %.1fs || %.1fs || %.1fs || " VERSION_STR " || ''OS'' || ''DeviceModel'' || ''CPU model'' || ''CPU Frequency'' || ''LinkToHwPage'' ||\n", (ti.tm_year+1900), (ti.tm_mon+1), ti.tm_mday, sec_mem, sec_pi, sec_e, sec_float);

    return 0;
}

 

[JaimeZX]

@JaimeZX or @Makaveli

Could you pls post a screenshot of the cert warning after reboot? I haven't seen one myself on Android N.

Per your request, sir:

And for reference:

 

[kvic]

every time I boot my phone I get a warning about having an unsigned certificate installed. Any way to permanently hide that?

Okay, after a bit google + personal experience..

The hard way..I believe if people move the user root CA into "system folder" rather "user folder", this warning maybe gone. This, however, may require rooting your device.

The easier way (just my guess). Add a password to your Android device. I suspect that's the only difference between your Android N and mine. I have a password to unlock phone.

Since we're on password to unlock phone...years ago I was so against the hassle to do it and laughed at one colleague doing so. But later I just got used to it and realised she did have wisdom on it... These days with fingerprint sensor, it's even less an hassle with password on.

If none of it helps, then at the moment I don't know the solution.

edit:

I think for some of you using a kind of IP blacklist for both incoming/outgoing traffic, you might have to whitelist 207.244.121.193 (or i.postimg.org aka s17.postimg.org) in order to view @JaimeZX's screenshots.

The black/white ip list on my ER-X has increasingly become a nuisance for little benefit. I'm considering removing it at some point.

 

[JaimeZX]

Yeah, no joy. I already had pattern lock set up. Tried password, same thing. Dunno. Thanks for the thoughts though. It's a minor annoyance, nothing more.

 

[Butterfly Bones]

Per your request, sir:

And for reference:

I do not see this on a Nexus 6 rooted (Magisk) stock or a Pixel 2 XL unrooted stock. I'm guessing it might be a LOS thing not being certified by Google (though they do a great job of keeping old hardware usable and current with releases). When I first installed the cert on my two phones, I got that pop up, but had the option to trust or remove. I choose trust and have never seen that again.

 

[Makaveli]

Okay, after a bit google + personal experience..

The hard way..I believe if people move the user root CA into "system folder" rather "user folder", this warning maybe gone. This, however, may require rooting your device.

The easier way (just my guess). Add a password to your Android device. I suspect that's the only difference between your Android N and mine. I have a password to unlock phone.

Since we're on password to unlock phone...years ago I was so against the hassle to do it and laughed at one colleague doing so. But later I just got used to it and realised she did have wisdom on it... These days with fingerprint sensor, it's even less an hassle with password on.

If none of it helps, then at the moment I don't know the solution.

edit:

I think for some of you using a kind of IP blacklist for both incoming/outgoing traffic, you might have to whitelist 207.244.121.193 (or i.postimg.org aka s17.postimg.org) in order to view @JaimeZX's screenshots.

The black/white ip list on my ER-X has increasingly become a nuisance for little benefit. I'm considering removing it at some point.

I only saw the alert once did another reboot and it never came up again.

My phone has a BES IT policy on it so pin on reboot is on and finger print lock also on Andriod N

Also looks like I spoke too soon seeing this in the log now.

Apr 7 16:34:42 pixelserv-tls[683]: handshake failed: unknown CA. client 192.168.1.11:39281 server graph.instagram.com
Apr 7 16:34:43 pixelserv-tls[683]: handshake failed: unknown cert. client 192.168.1.11:39284 server e.crashlytics.com
Apr 7 16:36:27 pixelserv-tls[683]: handshake failed: unknown cert. client 192.168.1.11:39326 server z.moatads.com
Apr 7 16:36:27 pixelserv-tls[683]: handshake failed: unknown cert. client 192.168.1.11:39327 server config.ioam.de
Apr 7 19:37:28 pixelserv-tls[683]: handshake failed: unknown cert. client 192.168.1.11:43757 server www.googleadservices.com

 

[JaimeZX]

Current security is my #1 android priority.

 

[Butterfly Bones]

@Makaveli most of those I see and added to my lists as well. Keep adding them to the two lists. As @kvic pointed out, they do not accept the cert since they do not want anyone to see what they collect. Shields UP! it is your privacy.

 

[kvic]

My phone has a BES IT policy on it so pin on reboot is on and finger print lock also on Andriod N

Also looks like I spoke too soon seeing this in the log now.

Looks like the BES policy or your phone's customization requires user consent to the user root CA on every reboot. Perhaps there is way to turn it off. Just that I don't know how.

most of those I see and added to my lists as well. Keep adding them to the two lists. As @kvic pointed out, they do not accept the cert since they do not want anyone to see what they collect. Shields UP! it is your privacy.

Thanks for advocating the methodologies. Please continue to advocate the "fallback approach" as well as "two instances of pixelserv-tls in primary/secondary mode" to people in need. Appreciate it.

 

[Makaveli]

Looks like the BES policy or your phone's customization requires user consent to the user root CA on every reboot. Perhaps there is way to turn it off. Just that I don't know how.

i only saw the prompt once i've done 5 reboots of the phone since and it never came back.

I believe it only came up because I never rebooted the first time I did the cert import.

 

[Butterfly Bones]

Thanks for advocating the methodologies. Please continue to advocate the "fallback approach" as well as "two instances of pixelserv-tls in primary/secondary mode" to people in need. Appreciate it.

I plan to try the "two instances" later. I'm needing to get syslog-ng working first (curse this AC86U ) with the updates in pixelserv-tls and now Skynet, my logs need sorting and management because they are getting purged too often. I get it working, but see nothing in the router syslog after a few minutes. Searching another report method that I can view from my Linux desktop.......

 

[JaimeZX]

After 36 hours on the new libraries I'm up to 0.7% in htop. Not sure if the slu# is ridonk or not. Dunno.

uts 1d 21:09 process uptime
log 1 critical (0) error (1) warning (2) notice (3) info (4) debug (5)
kcc 2 number of active service threads
kmx 25 maximum number of service threads
kvg 1.07 average number of requests per service thread
krq 2 max number of requests by one service thread

req 13931 total # of requests (HTTP, HTTPS, success, failure etc)
avg 558 bytes average size of requests
rmx 3144 bytes largest size of request(s)
tav 18 ms average processing time (per request)
tmx 140 ms longest processing time (per request)

slh 40 # of accepted HTTPS requests
slm 30 # of rejected HTTPS requests (missing certificate)
sle 0 # of rejected HTTPS requests (certificate available but bad)
slc 615 # of dropped HTTPS requests (client disconnect without sending any request)
slu 13081 # of dropped HTTPS requests (other TLS handshake errors)

sct 100 cert cache: # of certs in cache
sch 9479 cert cache: # of reuses of cached certs
scm 26 cert cache: # of misses to find a cert in cache
scp 1 cert cache: # of purges to give room for a new cert
sst 0 sess cache: # of cached TLS sessions (for older non-RFC5077 clients)
ssh 171 sess cache: # of reuses of cached TLS sessions
ssm 683 sess cache: # of misses to find a TLS session in cache
ssp 0 sess cache: # of purges to give room for a new TLS session

nfe 64 # of GET requests for server-side scripting
gif 1 # of GET requests for GIF
ico 1 # of GET requests for ICO
txt 22 # of GET requests for Javascripts
jpg 0 # of GET requests for JPG
png 0 # of GET requests for PNG
swf 0 # of GET requests for SWF
sta 2 # of GET requests for HTML stats
stt 0 # of GET requests for plain text stats
ufe 12 # of GET requests /w unknown file extension

opt 0 # of OPTIONS requests
pst 20 # of POST requests
hed 0 # of HEAD requests (HTTP 501 response)
rdr 3 # of GET requests resulted in REDIRECT response
nou 0 # of GET requests /w empty URL
pth 0 # of GET requests /w malformed URL
204 0 # of GET requests (HTTP 204 response)
bad 0 # of unknown HTTP requests (HTTP 501 response)

tmo 29 # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls 644 # of dropped requests (client disconnect without sending any request)
cly 23 # of dropped requests (client disconnect before response sent)
clt 0 # of dropped requests (reached maximum service threads)
err 0 # of dropped requests (unknown reason)

 

[jrmwvu04]

Not sure if the slu# is ridonk or not. Dunno.

Well that’s about 94% of your requests. At only 40 clean serves out of 13k something is amiss.

 

[DonnyJohnny]

Well that’s about 94% of your requests. At only 40 clean serves out of 13k something is amiss.

Try purge your generated cert and regenerate a CA cert and re import.
Maybe due to changes in OpenSSL version or old generated cert using older OpenSSL version. I don’t know but try and error.
Mine was greatly reduce after the steps I tried above.

 

[ScottB]

Hi,

I am running Pixelserv-tls for ad-blocking and followed this guide for how best to set it up on Merlin:

https://github.com/kvic-z/pixelserv-tls/wiki/How-to-best-run-pixelserv-tls-on-Asuswrt-Merlin

It works great on my normal network. However, on my guest WiFi, I can not access 192.168.2.3, even though my device on the guest WiFi is assigned a 192.168.2.X address. The connection just hangs forever. This renders my guest WiFi essentially useless as any web page with an ad never finishes loading and rendering.

Does anyone know how to fix this? Is this something in iptables?

Environment - Asus AC68U running Merlin 380.63_2, subnet: 192.168.2.xxx

Thanks for any help or suggestions.