[To all script writers] Suggestion for better environment management

[RMerlin]

Hey folks,

There's a lot of different scripts appearing lately, allowing users to implement advanced firewall-level filtering. This is great

A lot of those require being added to the firewall-start script, and I understand that things can start getting messy, as everyone probably does it his way, and that file could potentially become cluttered.

I would recommend agreeing on some kind of "standard" way of dealing with firewall-start. Here are my recommendations:

First, limit the changes to firewall-start to either a single line, two at most. One (optional) line could be a comment to simply identify the next line.

The second line would be a call to a separate script, which would contain ALL of the changes you want to make at firewall-start launch.

This way, it will make it easier to detect the presence of a given script, and also to handle it. Here's an example.

Let's say your script comes with its own install/uninstall routine. The install routine would insert the following line:

sh /jffs/yourscript/start-superfilter

Then, later on, the user wants to uninstall your script. Your uninstaller can then simply remove that one single line, using sed (you can even use the helper.sh script that I include in the firmware, and which can easily remove/add/replace a line in a given file):

#Uninstaller
pc_delete "sh /jffs/yourscript/start-superfilter" "/jffs/scripts/firewall-start"
rm -rf /jffs/yourscript
service restart_firewall

Similarly, you can easily detect if another script is present, and deal accordingly if you can find that specific line inside firewall-start: warn the user of compatibility issue, insert yourself specifically before that line, etc.... Use something like sed or grep to detect the presence of the other script within firewall-start.


In the end, this will make it easier for the developers, but also for the users (making it easier/simpler to implement an uninstaller).

 

[Jack Yaz]

Oddly enough I implemented this myself for all the scripts I use not half an hour ago!

 

[Adamm]

Added a uninstall and install function as per request. Probably a good idea considering how messy these scripts get from some other scripts. Hopefully scriptwriters using other files also take note

 

[redhat27]

I'll update the wiki to mention that reinstatement block (a snippet that checks if there are already existing ipsets, and re-creates iptable rules if they do) to move to a separate file that is invoked in firewall-start.

 

[Xentrk]

I like the suggestion too. I also implemented the technique to make it easier to manage as the number of scripts started to grow..

 

[spalife]

I would recommend agreeing on some kind of "standard" way of dealing with firewall-start. Here are my recommendations:

First, limit the changes to firewall-start to either a single line, two at most. One (optional) line could be a comment to simply identify the next line.

The second line would be a call to a separate script, which would contain ALL of the changes you want to make at firewall-start launch.

Two lines should be sufficient and should be informative

1st line identifies itself in the script the purpose of its presence in the script
which should be a comment
2nd line call to script

# Application or Script Identifier inserted this line to do this activity
call to script

 

[thelonelycoder]

Thanks @RMerlin for bringing this up.
I fought left and right that script writers not just remove an existing jffs file and write a new one when installing.
The entware-setup.sh shipped with the firmware and other similar installers floating around on the web caused me and the users of AB-Solution a lot of problems in the past.

Since I seem to not be able to convince some script writers that this is a bad idea to do without giving users an option, I had to adapt to to it.
The code resulting from it is a 250 line long series of grep, sed, append or write function called write_jffsfile(), likely never seen before in a shell script as complex as it is.
It works so well that since I implemented it, questions about 'Entware not starting' or similar problems no longer appear in my threads.

I understand that AB-Solution is not directly affected by the many scripts in the Firewall blocking field. But an effort to standardize the automatically generated entries in jffs files would be very welcome and help us all in the future.

However, it would be difficult to adhere to this standard for AB-Solution as it uses several user scripts to perform it's services.
In a future version of AB I _may_ use a one liner in the user script to call a separate file, similar to what I do with dnsmasq.conf.add and what is proposed in this thread.

 

[thelonelycoder]

Two lines should be sufficient and should be informative

1st line identifies itself in the script the purpose of its presence in the script
which should be a comment
2nd line call to script

# Application or Script Identifier inserted this line to do this activity
call to script

I support this suggestion, with a standardized comment line, containing a unique identifier name of the script.
This way, coders could look for that line present and decide the next action to perform.
I already have my comments standardized in all entries that AB writes to jffs, in case anyone ever bothers to check.

 

[Adamm]

I support this suggestion, with a standardized comment line, containing a unique identifier name of the script.
This way, coders could look for that line present and decide the next action to perform.
I already have my comments standardized in all entries that AB writes to jffs, in case anyone ever bothers to check.

Shouldn't the comment really be on the same line if someone thinks its absolutely necessary. The script path in its-self is more or less a dead giveaway in what its related to. Then when uninstalling you can just search for your script path doing something like;

sed -i '\~/jffs/scripts/firewall ~d' /jffs/scripts/firewall-start

 

[thelonelycoder]

Shouldn't the comment really be on the same line if someone thinks its absolutely necessary. The script path in its-self is more or less a dead giveaway in what its related to. Then when uninstalling you can just search for your script path doing something like;

sed -i '\~/jffs/scripts/firewall ~d' /jffs/scripts/firewall-start

Sure, that would simpler simpler to remove with your one-liner.
But it would need a script specific comment. Entries in those files can be the same for scripts made by another maintainer.
I would hate if another un-installer removes code for another script.

 

[Adamm]

Sure, that would simpler simpler to remove with your one-liner.
But it would need a script specific comment. Entries in those files can be the same for scripts made by another maintainer.
I would hate if another un-installer removes code for another script.

I don't see why scriptwriters would be referencing the file path to another users file though right (at least in this file)? Or lets say you comment your execution line, you could search for that string exactly.

My take on the situation is we should be only adding to these files by referencing startup functions in our own scripts and do the real work there.

 

[thelonelycoder]

I don't see why scriptwriters would be referencing the file path to another users file though right (at least in this file)? Or lets say you comment your execution line, you could search for that string exactly.

My take on the situation is we should be only adding to these files by referencing startup functions in our own scripts and do the real work there.

We definitely agree on that. I meant a command that references a file with the same name. Not likely but wanted to add this as a possible conflict.
Over time, AB will be ported completely to Entware and paths will change and be similar to other scripts. Just saying.

 

[spalife]

We definitely agree on that. I meant a command that references a file with the same name. Not likely but wanted to add this as a possible conflict.
Over time, AB will be ported completely to Entware and paths will change and be similar to other scripts. Just saying.

whether its a one-liner or a two-liner
it would be a good start.

Any of the below
Choice 1
# ScriptIdentifier: What it does here. CAUTION do not change
scriptCall

or Choice 2
scriptCall

 

[spalife]

We all are here and having a friendly discussion,
could I also raise the topic of jffs in this gathering ?

I feel that there should be a demarcation of
System Space and User Space.

Someone will surely say "hey jffs is user space"
but is it designed to handle megabytes of writes and
deletes every day and sometimes hours apart ?

It could withstand that load as per the
many posts in this forum but I do have a feeling
that it does decrease its lifespan and will lead
to discarding of a good router instead of a usb disk.

We should encourage users to move to a USB if possible
and reduce wear and tear of their devices.

They should be able to safely revert their devices to
firmware state at their discretion.

Do post on what your thoughts are on jffs here.

 

[spalife]

I don't see why scriptwriters would be referencing the file path to another users file though right (at least in this file)? Or lets say you comment your execution line, you could search for that string exactly.

@thelonelycoder sometimes gets requests for white-listing.

Scenario:
One of the custom scripts is downloading something from
a url(s) and it is getting blocked by ab-solution.
ab-solution will like to know if any file exists from which
it can read the list of urls and white-list them.
A comment in the firewall-start script if that script
is downloading such a file will help him,
Example in firewall-start script:

# ScriptIdentifier: This script downloads urls available in file: XXX or creates file:XXX with urls
scriptCall

Commenting code would help in situations like above

 

[thelonelycoder]

@thelonelycoder sometimes gets requests for white-listing.

Scenario:
One of the custom scripts is downloading something from
a url(s) and it is getting blocked by ab-solution.
ab-solution will like to know if any file exists from which
it can read the list of urls and white-list them.
A comment in the firewall-start script if that script
is downloading such a file will help him,
Example in firewall-start script:

# ScriptIdentifier: This script downloads urls available in file: XXX or creates file:XXX with urls
scriptCall

Commenting code would help in situations like above

The problem with multiple-line entries in jffs is to auto remove them when auto uninstalling. That may be simple for a two liner when you have exact text to sed-away, but with the current AB installation I have entries such as these:

# DO NOT EDIT this part of the file #
# generated by AB-Solution 3.8
RC='/opt/etc/init.d/rc.unslung'

i=30
until [ -x "$RC" ];do
    i=$(($i-1))
    if [ "$i" -lt 1 ];then
        logger "Unable to start Entware"
        exit
    fi
    sleep 1
done
logger "AB-Solution started rc.unslung via $0"
$RC start
# end of DO NOT EDIT #

Add to that that users may have modified such a file manually with their own code.
At the moment, when uninstalling AB, all jffs files created or appended are backed up to the root of jffs and deleted in the scripts/ dir.
It is almost impossible to safely remove multi line entries without possibly remove other code the user may have placed there.

So, as mentioned earlier, in a future version of AB I will implement the one-liner code with comment, running the code in the referenced file, like so:

source /tmp/mnt/absolution/adblocking/scripts/ab_dnsmasq_postconf.sh # added by AB-Solution

 

[thelonelycoder]

We all are here and having a friendly discussion,
could I also raise the topic of jffs in this gathering ?

I feel that there should be a demarcation of
System Space and User Space.

Someone will surely say "hey jffs is user space"
but is it designed to handle megabytes of writes and
deletes every day and sometimes hours apart ?

It could withstand that load as per the
many posts in this forum but I do have a feeling
that it does decrease its lifespan and will lead
to discarding of a good router instead of a usb disk.

We should encourage users to move to a USB if possible
and reduce wear and tear of their devices.

They should be able to safely revert their devices to
firmware state at their discretion.

Do post on what your thoughts are on jffs here.

I fully agree with that but I don't want to add to the fear of many users 'abusing' the jffs partition.
AB rarely changes any of the entries in jffs, all dynamic changes are done to files in either /opt/ or in the /adblocking/ directory on the attached USB device.

The problem with some of the script writers here not even hint at where to install these additional scripts, nor set a default location.

Novice users are often baffled and at a loss where and how they have to do things.
Since this board is not only visited and consulted by unix gurus but also by people simply copy/pasting stuff as is without understanding what it does, a lot of frustration ensues.

I have gone to great lengths to make AB as user friendly as possible for even a complete noob to install.
I do not expect others to follow my way of how I do things in AB, but a bit more explanation text to your #1 post would help and explain what the user has to do.

I repeat again: A lot of users here need guidance for how to do things, you cannot expect them to understand an arcane command if you post a script that is likely to be used by a wide variety of users.

 

[thelonelycoder]

For some reason it went inline...
Perhaps this can help you, it's in my flib_ipBLOCKer.sh.
You can either source or use the function as you deem fit.

remove_file1_from_file2 ()
{
  local file1="${1:-}" file2="${2:-}" tempFile="$2$TEMP_FILE_EXT"
  declare -i beforeCount=0 afterCount=0

  #log "START remove file1 : $file1 from file2: $file2 tempFile: $tempFile"
  [ ! -f "$file1" -o ! -f "$file2" ] && { EXIT_CODE=$EXIT_ERROR; return $EXIT_CODE; }

  EXIT_CODE=$EXIT_NORMAL

  beforeCount=$(wc -l "$file2" 2> /dev/null | awk '{print $1}')

  grep -Fvxf "$file1" "$file2" > "$tempFile" 2> /dev/null      || EXIT_CODE=$EXIT_ERROR
  mv "$tempFile" "$file2"                     > /dev/null 2>&1 || EXIT_CODE=$EXIT_ERROR

  afterCount=$(wc -l "$file2" 2> /dev/null | awk '{print $1}')

  echo "$(($beforeCount-$afterCount))"

  #log "END remove_file1_from_file2"
  return $EXIT_CODE
}

#Usage:
   cnt=$(remove_file1_from_file2 "$file1"   "$file2"
  [ $cnt -gt 0 ] && printf "%-40s %-10s" "Removed from $file2:"   $cnt

Thanks. This appears to require a reference file/code for it to work.
AB is very flexible in what it writes to jffs. For example, the post-mount may already have existing entware code in it that it respects and uses. But the code added can look entirely different when AB installs entware and adds to the file.

This respectful approach is used on every jffs file. I have no way to know how exactly a jffs file looks like on a router, not even when AB installs all of it.

 

[Adamm]

@thelonelycoder sometimes gets requests for white-listing.

Scenario:
One of the custom scripts is downloading something from
a url(s) and it is getting blocked by ab-solution.
ab-solution will like to know if any file exists from which
it can read the list of urls and white-list them.
A comment in the firewall-start script if that script
is downloading such a file will help him,
Example in firewall-start script:

# ScriptIdentifier: This script downloads urls available in file: XXX or creates file:XXX with urls
scriptCall

Commenting code would help in situations like above

I feel like entries here should be nothing more then an execution point, and function descriptions and whatnot should be left for the file its pointing to. Try keep things simple as possible.

sh /path/to/file args # Added By BestScriptEver

And lets say ab-solution for example did want to scan for a potential whitelist, I don't think this is the best place to grab them from or define it. It also gives us less points of failure when removing (and gives users less incentive to make manual edits here)

 

[thelonelycoder]

I feel like entries here should be nothing more then an execution point, and function descriptions and whatnot should be left for the file its pointing to. Try keep things simple as possible.

sh /path/to/file args # Added By BestScriptEver

And lets say ab-solution for example did want to scan for a potential whitelist, I don't think this is the best place to grab them from or define it. It also gives us less points of failure when removing (and gives users less incentive to make manual edits here)

Agreed.
BTW, AB routinely sets the execute permission to all /jffs/scripts/* files and removes DOS line endings if found.