Diversion Diversion - the Router Ad-Blocker

[Dabombber]

It should be possible to compromise and attempt removing the entware-setup.sh parts shouldn't it?

Spoiler

# Remove entware-setup.sh additions
# Usage: entware_clean
entware_clean() {
    # Quote strings suitable for use with sed
    # Usage: sed_quote STRING
    sed_quote() { printf '%s\n' "$1" | sed 's/[]\/$*.^&[]/\\&/g'; }

    if [ -f '/jffs/scripts/services-start' ]; then
        #RC='/opt/etc/init.d/rc.unslung'
        #
        #i=30
        #until [ -x "$RC" ] ; do
        #  i=$(($i-1))
        #  if [ "$i" -lt 1 ] ; then
        #    logger "Could not start Entware"
        #    exit
        #  fi
        #  sleep 1
        #done
        #$RC start
        sed -i "/^$(sed_quote "RC='/opt/etc/init.d/rc.unslung'")$/,/^$(sed_quote "\$RC start")$/d" '/jffs/scripts/services-start'
    fi

    if [ -f '/jffs/scripts/services-stop' ]; then
        #/opt/etc/init.d/rc.unslung stop
        sed -i "/^$(sed_quote '/opt/etc/init.d/rc.unslung stop')/d" '/jffs/scripts/services-stop'
    fi

    if [ -f '/jffs/scripts/post-mount' ]; then
        #if [ -d "$1/entware" ] ; then
        #  ln -nsf $1/entware /tmp/opt
        #fi
        sed -i "/^$(sed_quote "if [ -d \"\$1/entware\" ] ; then")$/,+2d" '/jffs/scripts/post-mount'
    fi

    # Remove empty/broken scripts
    local SCRIPT
    for SCRIPT in services-start services-stop post-mount; do
        [ ! -f "/jffs/scripts/$SCRIPT" ] && continue
        if [ "$(grep -csvE '^[[:space:]]*(#|$)' "/jffs/scripts/$SCRIPT")" = '0' ]; then
            rm -f "/jffs/scripts/$SCRIPT"
        elif ! sh -n "/jffs/scripts/$SCRIPT" 2>/dev/null; then
            echo "Renaming unconvertible script (/jffs/scripts/$SCRIPT -> /jffs/scripts/obsolete_$SCRIPT)"
            mv "/jffs/scripts/$SCRIPT" "/jffs/scripts/obsolete_$SCRIPT"
        fi
    done
}

 

[Ubimo]

During updating the lists I noticed this in syslog:

Feb 26 18:33:47 Diversion: http://winhelp2002.mvps.org/hosts.txt not hosts file, no backup found
Feb 26 18:33:48 Diversion: http://www.malwaredomainlist.com/hostslist/hosts.txt not hosts file, no backup found
Feb 26 18:37:19 Diversion: https://pgl.yoyo.org/adservers/serverlist.php?showintro=0;hostformat=hosts not hosts file, no backup found

I never noticed this before, and the links still work in my browser. Why does diversion say so?

 

[bluepoint]

During updating the lists I noticed this in syslog:

Feb 26 18:33:47 Diversion: http://winhelp2002.mvps.org/hosts.txt not hosts file, no backup found
Feb 26 18:33:48 Diversion: http://www.malwaredomainlist.com/hostslist/hosts.txt not hosts file, no backup found
Feb 26 18:37:19 Diversion: https://pgl.yoyo.org/adservers/serverlist.php?showintro=0;hostformat=hosts not hosts file, no backup found

I never noticed this before, and the links still work in my browser. Why does diversion say so?

Check your "update-blocking-list.log", it'll give you a clue.
Open diversion, sf-->5

 

[SomeWhereOverTheRainBow]

During updating the lists I noticed this in syslog:

Feb 26 18:33:47 Diversion: http://winhelp2002.mvps.org/hosts.txt not hosts file, no backup found
Feb 26 18:33:48 Diversion: http://www.malwaredomainlist.com/hostslist/hosts.txt not hosts file, no backup found
Feb 26 18:37:19 Diversion: https://pgl.yoyo.org/adservers/serverlist.php?showintro=0;hostformat=hosts not hosts file, no backup found

I never noticed this before, and the links still work in my browser. Why does diversion say so?

were those copied over to the file or were they inputted individually in the diversion in the customize list option via ssh terminal? You need to input them individually via the terminal and not manually by copying them into host list file. I noticed this issue when i attempted to copy mine manually to that list as well instead of inputting them individually into the terminal like i should.


if you copied them all to the host list file then diversion assumes you already had a downloaded backup and when it doesn't find it, it gives this error.

 

[Ubimo]

I removed the lines in diversion and re-added them (as suggested) through the menu. I still get these errors:

Feb 27 08:23:01 Diversion: http://winhelp2002.mvps.org/hosts.txt not hosts file, no backup found
Feb 27 08:23:01 Diversion: http://www.malwaredomainlist.com/hostslist/hosts.txt not hosts file, no backup found
Feb 27 08:26:05 Diversion: https://pgl.yoyo.org/adservers/serverlist.php?showintro=0;hostformat=hosts not hosts file, no backup found

 

[thelonelycoder]

I removed the lines in diversion and re-added them (as suggested) through the menu. I still get these errors:

Feb 27 08:23:01 Diversion: http://winhelp2002.mvps.org/hosts.txt not hosts file, no backup found
Feb 27 08:23:01 Diversion: http://www.malwaredomainlist.com/hostslist/hosts.txt not hosts file, no backup found
Feb 27 08:26:05 Diversion: https://pgl.yoyo.org/adservers/serverlist.php?showintro=0;hostformat=hosts not hosts file, no backup found

While evaluating a new feature I found that
- http://winhelp2002.mvps.org/hosts.txt and
- http://www.malwaredomainlist.com/hostslist/hosts.txt
has DOS line endings which is then incorrectly interpreted as a non-valid file and rejected.
This is fixed locally and will be available with the next Diversion update.

https://pgl.yoyo.org/adservers/serverlist.php?showintro=0;hostformat=hosts is not a valid format, use this instead:
https://pgl.yoyo.org/adservers/serv...&showintro=0&mimetype=plaintext&useip=0.0.0.0

 

[thelonelycoder]

Diversion, unfortunately, is extremely sloppy with scripts installed before it - it just deletes them. I mean post-mount and write to the crontab. You can also check the presence of previously installed programs during installation and not touch the records made earlier?

If the known Entware user scripts contain remnants or part of the old-style Entware code, Diversion will remove that file to have a clean slate.
Yours must have had such entries.
See also below.

It should be possible to compromise and attempt removing the entware-setup.sh parts shouldn't it?

Spoiler

# Remove entware-setup.sh additions
# Usage: entware_clean
entware_clean() {
    # Quote strings suitable for use with sed
    # Usage: sed_quote STRING
    sed_quote() { printf '%s\n' "$1" | sed 's/[]\/$*.^&[]/\\&/g'; }

    if [ -f '/jffs/scripts/services-start' ]; then
        #RC='/opt/etc/init.d/rc.unslung'
        #
        #i=30
        #until [ -x "$RC" ] ; do
        #  i=$(($i-1))
        #  if [ "$i" -lt 1 ] ; then
        #    logger "Could not start Entware"
        #    exit
        #  fi
        #  sleep 1
        #done
        #$RC start
       sed -i "/^$(sed_quote "RC='/opt/etc/init.d/rc.unslung'")$/,/^$(sed_quote "\$RC start")$/d" '/jffs/scripts/services-start'
    fi

    if [ -f '/jffs/scripts/services-stop' ]; then
        #/opt/etc/init.d/rc.unslung stop
        sed -i "/^$(sed_quote '/opt/etc/init.d/rc.unslung stop')/d" '/jffs/scripts/services-stop'
    fi

    if [ -f '/jffs/scripts/post-mount' ]; then
        #if [ -d "$1/entware" ] ; then
        #  ln -nsf $1/entware /tmp/opt
        #fi
        sed -i "/^$(sed_quote "if [ -d \"\$1/entware\" ] ; then")$/,+2d" '/jffs/scripts/post-mount'
    fi

    # Remove empty/broken scripts
    local SCRIPT
    for SCRIPT in services-start services-stop post-mount; do
        [ ! -f "/jffs/scripts/$SCRIPT" ] && continue
        if [ "$(grep -csvE '^[[:space:]]*(#|$)' "/jffs/scripts/$SCRIPT")" = '0' ]; then
            rm -f "/jffs/scripts/$SCRIPT"
        elif ! sh -n "/jffs/scripts/$SCRIPT" 2>/dev/null; then
            echo "Renaming unconvertible script (/jffs/scripts/$SCRIPT -> /jffs/scripts/obsolete_$SCRIPT)"
            mv "/jffs/scripts/$SCRIPT" "/jffs/scripts/obsolete_$SCRIPT"
        fi
    done
}

Thanks for the code. I have thought for about a second implementing something like this into Diversion when I rewrote AB-Solution to what became Diversion.
Since I used a much more elaborate jffs code in AB-Solution to cater for almost every (user)configuration, the resulting Entware code in the /jffs/scripts/ folder could be in a number of different formats.

Expecting that after a certain string 2 more lines with the accompanying code follows is not a given. Your code example relies on that. Unfortunately, this would result in remnants or broken user scripts.
I wanted and still do avoid this by simply removing files containing such old-style Entware entries.
My decision back in early 2018 to go that route was obviously the right choice. There are few if any complaints of users saying Entware did not start.

 

[thelonelycoder]

You can lead a horse to water, they say. That aside.. if you know enough, and care, about how your event scripts work, you should surely be able to understand what the installation messages mean pertaining to them. Would everyone understand? Of course not, which brings us to the answer to the last question - support.

At the risk of putting words into someone else’s mouth, the install process for diversion (and entware for that matter) create a known good configuration file without pulling in someone’s potential nonsense that breaks things. This is the best approach for the common user and the easiest for the script writer to “support” (i.e. provide assistance when something goes wrong). It’s implausible to accommodate all possible edits/configurations people could be trying to bring in because it’s impossible to support.

The diversion project to me seeks to be the lowest common denominator for as wide an audience as possible. If you don’t like the way it installs, your choices as I see it would be to either reinstate your personal changes or not install it to begin with.

Well said, thanks.

 

[Dabombber]

Expecting that after a certain string 2 more lines with the accompanying code follows is not a given. Your code example relies on that. Unfortunately, this would result in remnants or broken user scripts.
I wanted and still do avoid this by simply removing files containing such old-style Entware entries.
My decision back in early 2018 to go that route was obviously the right choice. There are few if any complaints of users saying Entware did not start.

It should be pretty safe to assume people are using entware-setup.sh code, any broken scripts would be caught with sh -n. The bigger problem would be the services-start sed statement not having a matching end block which would mean the file gets truncated. This could be fixed with

sed -ni "/^$(sed_quote "RC='/opt/etc/init.d/rc.unslung'")$/{h;bx;};p;b;:x;n;/^$(sed_quote "\$RC start")$/d;H;\${x;p};bx" '/jffs/scripts/services-start'

But if you want a completely fail-safe method you'd want to match the block of code exactly. Should be possible in sed but.. it'd be ugly. The easiest way is to just load the script into a variable and use the built in replace.

Spoiler: services-start example

SCRIPT="$(cat /jffs/scripts/services-start; printf _)"
SCRIPT="${SCRIPT%_}"

REMOVE='
RC='\''/opt/etc/init.d/rc.unslung'\''

i=30
until [ -x "$RC" ] ; do
  i=$(($i-1))
  if [ "$i" -lt 1 ] ; then
    logger "Could not start Entware"
    exit
  fi
  sleep 1
done
$RC start
'

NEWLINE='
'

printf '%s' "${SCRIPT//"$REMOVE"/"$NEWLINE"}" >/jffs/scripts/services-start

 

[thelonelycoder]

It should be pretty safe to assume people are using entware-setup.sh code, any broken scripts would be caught with sh -n. The bigger problem would be the services-start sed statement not having a matching end block which would mean the file gets truncated. This could be fixed with

sed -ni "/^$(sed_quote "RC='/opt/etc/init.d/rc.unslung'")$/{h;bx;};p;b;:x;n;/^$(sed_quote "\$RC start")$/d;H;\${x;p};bx" '/jffs/scripts/services-start'

But if you want a completely fail-safe method you'd want to match the block of code exactly. Should be possible in sed but.. it'd be ugly. The easiest way is to just load the script into a variable and use the built in replace.

Spoiler: services-start example

SCRIPT="$(cat /jffs/scripts/services-start; printf _)"
SCRIPT="${SCRIPT%_}"

REMOVE='
RC='\''/opt/etc/init.d/rc.unslung'\''

i=30
until [ -x "$RC" ] ; do
  i=$(($i-1))
  if [ "$i" -lt 1 ] ; then
    logger "Could not start Entware"
    exit
  fi
  sleep 1
done
$RC start
'

NEWLINE='
'

printf '%s' "${SCRIPT//"$REMOVE"/"$NEWLINE"}" >/jffs/scripts/services-start

Thanks but I'll leave it as it is, for the sanity of me and the enthusiastic and helpful board members minds (thanks guys / gals !).

 

[L&LD]

@Rhialto, you will need to reboot the router.

 

[jrmwvu04]

While evaluating a new feature I found that
- http://winhelp2002.mvps.org/hosts.txt and
- http://www.malwaredomainlist.com/hostslist/hosts.txt
has DOS line endings which is then incorrectly interpreted as a non-valid file and rejected.

It’s been long enough ago that I forget the culprit, but one of the popular lists at the time was including a very unusual character (it looked like a “G” and was blocking a fake Google.com) in one of its blocked domains that was making dnsmasq fail to load similarly. I went to a sed line to strip the non standard stuff, I’m curious about how you handle it.

sed -i 's/[^a-zA-Z0-9#-.]//g' blocklist

 

[maghuro]

Can someone explain me how does primary and secondary lists work?
The secondary one works as a backup if the primary doesn't work?
Or are they used both at same time?

Thanks in advance

 

[L&LD]

@maghuro the first post usually has all the links to the information you may want.

https://diversion.ch/diversion/manual/alternate-blocking-file.html

 

[maghuro]

@maghuro the first post usually has all the links to the information you may want.

https://diversion.ch/diversion/manual/alternate-blocking-file.html

Oh crap, didn't find !!! Thank you...
I'm asking because Large+ list stopped working.... Are everyone aware of this or is it just me?

 

[radurus]

Hello.
I am not a CLI fan but I can manage some things.
I updated the latest fw of Merlin on RT-AC66Ub1.

So I played with amtm interface and features.
I installed Diversion. Everything seemed ok with the a d b l o c k e r but...
The Youtube android app seems not updating the history/recent of the played videos (in the library/history section)
On the PC/laptop browser it is working correctly.
I did not checked the situation of accessing YT from phone browser.

I disabled the a d b l o c k e r via amtm menu and the played videos appeared in the history.
And the YT app works fine now.

I don't know if somebody else observed this issue.
Can somebody check this?

 

[bluepoint]

Oh crap, didn't find !!! Thank you...
I'm asking because Large+ list stopped working.... Are everyone aware of this or is it just me?

Probably because the primary server is down. There are 3 mirror lists you can use temporarily, choose one until the site owner restores the primary server they were updated as of 02/26/2020.
https://www.reddit.com/r/oisd_blocklist/comments/dwxgld/dbloisdnl_internets_1_domain_blocklist/

 

[juched]

Hello.
I am not a CLI fan but I can manage some things.
I updated the latest fw of Merlin on RT-AC66Ub1.

So I played with amtm interface and features.
I installed Diversion. Everything seemed ok with the a d b l o c k e r but...
The Youtube android app seems not updating the history/recent of the played videos (in the library/history section)
On the PC/laptop browser it is working correctly.
I did not checked the situation of accessing YT from phone browser.

I disabled the a d b l o c k e r via amtm menu and the played videos appeared in the history.
And the YT app works fine now.

I don't know if somebody else observed this issue.
Can somebody check this?

From the pie hole page they list some whitelist items to enable YouTube history to work.


https://discourse.pi-hole.net/t/commonly-whitelisted-domains/212

Maybe a good link to add to the original page as a sticky.


There is also a whitelist shared here for these items:

https://github.com/anudeepND/whitelist/blob/master/README.md

 

[Centrifuge]

Whitelisting device-metrics-us.amazon.com and rebooting my echos solved it.

I have a ONeplus 7T, had problems with the Amazon Shopping app, in addition I had to add:

device-metrics-us.amazon.com
aan.amazon.com
fls-na.amazon.com

 

[thelonelycoder]

Question for the next Diversion update:
It will have a new ads only blocking list type option called Minimal in b. This Minimal list only uses http://winhelp2002.mvps.org/hosts.txt to block.

Is it OK to use Minimal as the preset for new installs or should I keep the current preset Standard (https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts)?
Thanks for your replies.