What's new

[Release] AB-Solution 3 - The Ad Blocking Solution

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Status
Not open for further replies.
I did my own workaround, which could be incorporated as an option in ABS.

  1. dd if=/dev/zero of=./swapfile bs=1k count=256000
  2. mkswap swapfile
  3. swapon swapfile
  4. Run the update from within ab-solution.sh
  5. swapoff -a
My question still remains if there's an option anywhere or a way to turn off autoupdate? As it runs of out memory every time I have to repeat the steps above manually each week.
I was toying with two things yesterday. Adding a swap file as you suggest or monitoring RAM usage when the script is run.
Both are very intrusive and add unnecessary complexity to AB I feel I don't want to handle. AB is complex enough as it is at the moment.
I consider a break down of the way I sort the file, maybe make it a two file sort and then combine.
This is added to the to do list for the coming update.

My question still remains if there's an option anywhere or a way to turn off autoupdate? As it runs of out memory every time I have to repeat the steps above manually each week.
No, updating the hosts files on a weekly basis is an integral part of my ad-blocker and makes it so successful.
You could delete the cron job, but be aware that whenever you start the AB UI this job will be added back silently. Also, if dnsmasq restarts for other reasons it will be re-added.
You best wait for what I come up with for the sort function in the updated update-hosts.add.
 
I'm nearly done with the updates to all of the AB scripts.
I would like this tested outside of my sheltered test environment before the general release.

Changes done:
- AICloud port (443) check with option to directly change it to 9443 (9443 is tested before the option to change comes up)
- In addition to 'entware' and 'entware-ng', 'entware-ng.arm' folder for existing Entware installs is supported
- wget, ping and curl commands simplified and compatible with non-standard versions
- Keeping .ash_history (sh) actually keeps the terminal command history.
- option to reset pixelserv-tls settings. Useful for PS IP change or Entware device has changed

I'm looking for no more than four testers (limit of PM participants is 5) with non-standard installs:
- Merlin forks only (john9527, Xwrt, ..., homebrew)
- With or without pre-existing Entware install
- AiCloud or other packages that may interfere with pixelserv-tls port
- You have a good reason

Let me know if you're interested in this thread, no PM's please.
Quoting myself: The 'nearly done' has just transformed to 'will take a little while' with the things I added to the to do list.
 
I consider a break down of the way I sort the file, maybe make it a two file sort and then combine.
Would it make sense to slice the sort into daily counts then combine?.... could use the log time stamps to do the chopping...
 
I was toying with two things yesterday. Adding a swap file as you suggest or monitoring RAM usage when the script is run.
Both are very intrusive and add unnecessary complexity to AB I feel I don't want to handle. AB is complex enough as it is at the moment.

my ac66u configuration is kind of 'modular', and i happen to have a 'swap management' module.
im posting it here so maybe you can adapt it to your needs.

it first looks for a swap partition and mount it if found.
if not, it looks for a swap image file and mount it if found.
if not, creates the file and mount it.

ramon@ac66u:/tmp/home/root# cat /mnt/EXT3_29G/config/criar-swap.sh

Code:
#!/bin/sh

# ------------------------------------------------------------------------------
# SOURCING AUX FUNCTIONS
# ------------------------------------------------------------------------------

if ! source /mnt/EXT3_29G/config/aux-functions.sh ; then
    SCRIPTNAME=$(basename $0)
    echo "$SCRIPTNAME: Error sourcing /mnt/EXT3_29G/config/aux-functions.sh"
    logger -t "$SCRIPTNAME" "Error sourcing /mnt/EXT3_29G/config/aux-functions.sh"
    exit 1
fi

# ------------------------------------------------------------------------------
# CONFIG
# ------------------------------------------------------------------------------

SWAPSIZE=256
WORKDIR=/mnt/EXT3_29G/config

cd $WORKDIR

# ------------------------------------------------------------------------------
# START
# ------------------------------------------------------------------------------

echoAndLog "Starting script $0"

SWAPDEV=$(fdisk -l | awk '/swap/{print $1}')
SWAPLABEL=$(blkid $SWAPDEV | grep -oP '(?<=LABEL=")\w+(?="\s)')
SWAPUUID=$(blkid $SWAPDEV | grep -oP '(?<=UUID=").*(?="$)')

if [ "$SWAPDEV" ] ; then
    echoAndLog "Found swap partition $SWAPDEV LABEL=$SWAPLABEL UUID=$SWAPUUID"
    echoAndLog "Enabling swap partition"

    swapoff $SWAPDEV 2>/dev/null
    swapon  $SWAPDEV || echoAndLog "Error enabling swap partition"
else
    echoAndLog "No swap partition found. Will try with file"

    if [ -s swapfile.img ] && [ "$(du -m swapfile.img | awk '{print $1}')" -eq 256 ] ; then
        echoAndLog "Found a ${SWAPSIZE}M image file"
    else
        echoAndLog "Creating ${SWAPSIZE}M image file"
        dd if=/dev/zero of=swapfile.img bs=1024 count=${SWAPSIZE}K || echoAndLog "Error creating image file"
    fi

    echoAndLog "Setting image file as swap"
    mkswap swapfile.img || echoAndLog "Error setting image file as swap"

    echoAndLog "Setting swap file permissions"
    chmod 0600 swapfile.img || echoAndLog "Error setting swap file permissions"

    echoAndLog "Enabling swap file"
    swapoff swapfile.img 2>/dev/null
    swapon  swapfile.img || echoAndLog "Error enabling swap file"
fi

echoAndLog "$(free | awk '/Swap/{ print $1, "total", $2, "used", $3, "free", $4 }')"

echoAndLog "Script finished"
 
Would it make sense to slice the sort into daily counts then combine?.... could use the log time stamps to do the chopping...
The thing is the sort is the easy part, that only takes seconds. The sort -u is using a lot of memory to sort out duplicates. And it really only works if you do it on the whole file.
awk or sed would not be faster, I did tests. sort -u is the winner for this every time.
OTOH, running out of memory seems not to happen to many.
I have had it on my RT-AC66U several times. But this is my old and trusted development router. It has seen a lot in it's busy life at my place.
 
my ac66u configuration is kind of 'modular', and i happen to have a 'swap management' module.
im posting it here so maybe you can adapt it to your needs.

it first looks for a swap partition and mount it if found.
if not, it looks for a swap image file and mount it if found.
if not, creates the file and mount it.

ramon@ac66u:/tmp/home/root# cat /mnt/EXT3_29G/config/criar-swap.sh

Code:
#!/bin/sh

# ------------------------------------------------------------------------------
# SOURCING AUX FUNCTIONS
# ------------------------------------------------------------------------------

if ! source /mnt/EXT3_29G/config/aux-functions.sh ; then
    SCRIPTNAME=$(basename $0)
    echo "$SCRIPTNAME: Error sourcing /mnt/EXT3_29G/config/aux-functions.sh"
    logger -t "$SCRIPTNAME" "Error sourcing /mnt/EXT3_29G/config/aux-functions.sh"
    exit 1
fi

# ------------------------------------------------------------------------------
# CONFIG
# ------------------------------------------------------------------------------

SWAPSIZE=256
WORKDIR=/mnt/EXT3_29G/config

cd $WORKDIR

# ------------------------------------------------------------------------------
# START
# ------------------------------------------------------------------------------

echoAndLog "Starting script $0"

SWAPDEV=$(fdisk -l | awk '/swap/{print $1}')
SWAPLABEL=$(blkid $SWAPDEV | grep -oP '(?<=LABEL=")\w+(?="\s)')
SWAPUUID=$(blkid $SWAPDEV | grep -oP '(?<=UUID=").*(?="$)')

if [ "$SWAPDEV" ] ; then
    echoAndLog "Found swap partition $SWAPDEV LABEL=$SWAPLABEL UUID=$SWAPUUID"
    echoAndLog "Enabling swap partition"

    swapoff $SWAPDEV 2>/dev/null
    swapon  $SWAPDEV || echoAndLog "Error enabling swap partition"
else
    echoAndLog "No swap partition found. Will try with file"

    if [ -s swapfile.img ] && [ "$(du -m swapfile.img | awk '{print $1}')" -eq 256 ] ; then
        echoAndLog "Found a ${SWAPSIZE}M image file"
    else
        echoAndLog "Creating ${SWAPSIZE}M image file"
        dd if=/dev/zero of=swapfile.img bs=1024 count=${SWAPSIZE}K || echoAndLog "Error creating image file"
    fi

    echoAndLog "Setting image file as swap"
    mkswap swapfile.img || echoAndLog "Error setting image file as swap"

    echoAndLog "Setting swap file permissions"
    chmod 0600 swapfile.img || echoAndLog "Error setting swap file permissions"

    echoAndLog "Enabling swap file"
    swapoff swapfile.img 2>/dev/null
    swapon  swapfile.img || echoAndLog "Error enabling swap file"
fi

echoAndLog "$(free | awk '/Swap/{ print $1, "total", $2, "used", $3, "free", $4 }')"

echoAndLog "Script finished"
Thanks for the code.
As mentioned, I prefer not to add another potential point of failure to AB to solve a relatively seldom occurrence of a problem.
I'd like to solve it by avoiding the out of memory problem. A smarter way comes up for sure to do what needs to be done.
 
The thing is the sort is the easy part, that only takes seconds. The sort -u is using a lot of memory to sort out duplicates. And it really only works if you do it on the whole file.
awk or sed would not be faster, I did tests. sort -u is the winner for this every time.
OTOH, running out of memory seems not to happen to many.
I have had it on my RT-AC66U several times. But this is my old and trusted development router. It has seen a lot in it's busy life at my place.
is there a good way to see how much memory is getting used?. i monitored the memory usage with netdata ( which grabs data from /proc/meminfo ) and it only peaked about 15Mb more than usual while the sort was running... i guess as lot depends on the size of the dnsmasq log at the end of the week.
 
This project is excellent! Way above average support! Program works as described. Thanks and I will be donating once again. Keep up the great work!
 
is there a good way to see how much memory is getting used?. i monitored the memory usage with netdata ( which grabs data from /proc/meminfo ) and it only peaked about 15Mb more than usual while the sort was running... i guess as lot depends on the size of the dnsmasq log at the end of the week.
free comes with busybox. I use this script to log it to file to have some data when I run tests.
Nothing fancy but it logs the time, used- , available- and percentage of memory.
It either outputs to terminal or file. I use this to keep a record of the models I have.
Code:
log_free(){
echo -e "  time    used  avail      %"
while true; do
    echo -e "$(date '+%H:%M:%S') $(free -m | awk 'NR==2{printf "%s %sMB (%.2f%%)\n", $3,$2,$3*100/$2 }')"
    sleep 1
done
}
#log_free
log_free > /mnt/absolution/free_$(nvram get productid).txt

The mem used depends on how large your hosts file will end up and if you have IPv6 enabled.
With a standard hosts file and no IPv6, the memory usage barely goes up.
But if I enable IPv6 and select the Large hosts file, I can bring it to it's knees if I push it.
 
The mem used depends on how large your hosts file will end up and if you have IPv6 enabled.
With a standard hosts file and no IPv6, the memory usage barely goes up.
But if I enable IPv6 and select the Large hosts file, I can bring it to it's knees if I push it.
makes me wonder how many of the reports of out of memory are from IPv6 users
 
makes me wonder how many of the reports of out of memory are from IPv6 users
With which you brought forth the likely way I'll fix it.
Either separate lists or duplicate the v4 list after sort and append as v6. <-- winner?
 
With which you brought forth the likely way I'll fix it.
Either separate lists or duplicate the v4 list after sort and append as v6. <-- winner?
Highly likely fix i would say....... don't know which would be better. Maybe the first would isolate the code a bit better in case you want to mess with it later.
 
@Beherit are you using IPv6 by any chance?
 
@RMerlin or @john9527
What is a reliable way of knowing that Download Master is installed?
Nvram:
gen_dl_dir (appears to be only present when installed)
apps_mounted_path (has device path when installed)
apps_install_folder (together with apps_mounted_path can be checked for presence)
Any other way I miss to check?

Edit:
apps_state_autorun= (appears to be empty if not installed)
 
Last edited:
Just upgraded to 380.64 and no issues. I know someone reported they lost AB Solution during the upgrade. I had no such issue AB and Pixel serv both survived the flash for me.
 
Just upgraded to 380.64 and no issues. I know someone reported they lost AB Solution during the upgrade. I had no such issue AB and Pixel serv both survived the flash for me.
Yup that was me.... weirdest thing..... the Adblocking folder was completely empty on reboot. But apart from that , nothing touched.... Jffs ok, Entware all ok, all the other folders on the same partition as adblocking ok.
Some kernel fart i think...nothing to do with AB as such
Code:
Dec 17 02:33:26 RT-AC68U-4690 kernel: EXT2-fs (sda2): error: ext2_check_page: bad entry in directory #182897: : rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
Dec 17 02:33:26 RT-AC68U-4690 kernel: EXT2-fs (sda2): error: ext2_readdir: bad page in #182897
 
Yup that was me.... weirdest thing..... the Adblocking folder was completely empty on reboot. But apart from that , nothing touched.... Jffs ok, Entware all ok, all the other folders on the same partition as adblocking ok.
Some kernel fart i think...nothing to do with AB as such
Code:
Dec 17 02:33:26 RT-AC68U-4690 kernel: EXT2-fs (sda2): error: ext2_check_page: bad entry in directory #182897: : rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
Dec 17 02:33:26 RT-AC68U-4690 kernel: EXT2-fs (sda2): error: ext2_readdir: bad page in #182897
As you best know, AB3 silently recreates all jffs files when started or properly exited if they are missing.
Unfortunately, the other way round would be a tad more complicated :rolleyes:.

You mentioned elsewhere that your device is having some difficulties coping with the job it's supposed to do. Time to ditch it?
 
As you best know, AB3 silently recreates all jffs files when started or properly exited if they are missing.
Unfortunately, the other way round would be a tad more complicated :rolleyes:.

You mentioned elsewhere that your device is having some difficulties coping with the job it's supposed to do. Time to ditch it?
Even AB3 can't resurrect itself when the entire contents of the adblocking folder disappear.
Code:
ab-solution-3.sh: line 1: can't create /tmp/mnt/data/adblocking/.config/ab-solution.cfg: nonexistent directory
Once i deleted the now empty adblocking folder, AB3 reinstalled ok without issue.
 
Status
Not open for further replies.

Similar threads

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top