Country blocking script

[Builder71]

... The script has checks that prevent the lists from being loaded if they are already loaded ...

... I believe the script checks to see if the last time it downloaded the files is over 15 days to re-download the country data.

The above seems to interact unexpected, not sure if this is how it should work.

When the script is run for the first time after boot, it will download new /jffs/ipset_lists/country.lst files if older then 15 days.
However every time you run the script again, e.g. cron job, country.lst files will never be updated.
It seems because ipset and/or iptables are already in place and the script exits. (Check to prevent lists from being loaded if they are already loaded.)

In short, when the router stays on for a very long time, without reboot, your country blocking does not get updated.
Can anyone with script skills have a look at this?

 

[octopus]

I can confirm, I have same behave here.

 

[Builder71]

Added below code which works fine on my RT-N66U.
(Add the four lines just above the "# Country blocking by nethashes...." comment line in 'create-ipset-lists.sh'.)

It does makes the existing check that prevent the lists from being loaded if they are already loaded ... kind of obsolete.
However I didn't want to touch the original script because I'm afraid to break something else.

Would be nice if this script gets a bit of polishing.
What I would really like to have is a country block option that only blocks unsolicited incoming packets.
Outgoing packets will be allowed.

For example block the US, but Google DNS, Online games from the kids and so on will keep working.

# Delete old BlockedCountries
logger -t Firewall "Delete old BlockedCountries."
iptables --list INPUT --line-numbers | grep BlockedCountries | cut -d' ' -f1 | while read number; do iptables -D INPUT $number; done
ipset -L | grep BlockedCountries | cut -d' ' -f2 | while read setname; do ipset --destroy $setname; done

# Country blocking by nethashes [Both IPv4 and IPv6 sources]

 

[redhat27]

@octopus and @Builder71, the above issue is fixed in the latest version

 

[Builder71]

Tested a bit and it seems to be working well!
Running a cron job and see how it goes.

One question about the Tor stuff.
Below line a really don't understand.
What's going on here and what is up with the /22 IP range?

$(ipset -q $SWAP BlockedCountries BlockedCountries) && ipset $DELETE BlockedCountries 194.63.140.0/22 2>/dev/null

 

[redhat27]

That just removes the block on the domain for the site torstatus.blutmagie.de where the list of Tor exit nodes is tracked. If you have "ru" in the blocked countries list, you'll need this on the reload. Otherwise its not necessary.
torstatus.blutmagie.de is hosted in Russia (ru) Normally you can access sites in blocked countries (it only prevents inbound connections), but I believe there is some sort of inbound check that happens prior to downloading that file, otherwise it fails to download.

 

[Builder71]

Ahhh, makes sense. Thx.
Needed, because I do have ru in the list.

 

[octopus]

@octopus and @Builder71, the above issue is fixed in the latest version

Thanks for updated script. isn't any faster way to do this? It really sloooow ? @redhat27

for IP in $(cat $IPSET_LISTS_DIR/tor.lst); do
    ipset $ADD TorNodes $IP

 

[redhat27]

isn't any faster way to do this?

Sure there is. Instead of looping through the file, it would be a lot faster to write out a restore file to load all at once. However, since the purpose of this script is to load relatively static ipsets, it would not probably warrant changes to make it faster. I personally do not even run it in cron, just run it once on router boot.

 

[octopus]

Sure there is. Instead of looping through the file, it would be a lot faster to write out a restore file to load all at once. However, since the purpose of this script is to load relatively static ipsets, it would not probably warrant changes to make it faster. I personally do not even run it in cron, just run it once on router boot.

Okey, have you any suggestion to make it faster? I'm using Tornords as standalone program and cron job ~every 7:th day.
On my RT-AC68U it takes for approx 6500 - 7000 ipnumber 1min 20sek and on my RT-16N 5min and 30sek.
Any suggestion to make it faster would be nice!
@redhat27

 

[octopus]

the above issue is fixed in the latest version

I have tested this with a cron job for test nightly (-mtime +1) update but it seems not update. Same date on "tor.lst file" as when installed it.
@redhat27

 

[redhat27]

I have tested this with a cron job for test nightly (-mtime +1) update but it seems not update. Same date on "tor.lst file" as when installed it.
@redhat27

Can you verify the timestamp of the tor.lst file does not update? Is the size of the tor.lst file non-zero?

 

[octopus]

Can you verify the timestamp of the tor.lst file does not update? Is the size of the tor.lst file non-zero?

File time stamp is the same when installed and size is 88KB (6407 ips)
@redhat27

 

[redhat27]

Can you post the output of

ls (minus)l /jffs/ipset_lists/tor.*

Then can you remove the tor.lst and try again?
It appears to be working fine for me.

 

[octopus]

If I run script again after I deleted tor.lst is downloaded again but not replaced in ipset "TorNodes".

root@SMEAGOL:/tmp/home/root# ls (minus) l /jffs/ipset_lists/tor.*
-rw-r--r--    1 root     root         89339 Nov  7 10:40 /jffs/ipset_lists/tor.lst

@redhat27

 

[redhat27]

"TorNodes" ipset will be reloaded if tor.lst is downloaded. There should be a syslog entry once TorNodes is reloaded

 

[octopus]

"TorNodes" ipset will be reloaded if tor.lst is downloaded. There should be a syslog entry once TorNodes is reloaded

I agree If I delete tor.lst and run script its working and tor.lst is replaced in ipset TorNoodes.

root@SMEAGOL:/tmp/home/root# /jffs/TorNodes.sh block
+ IPSET_LISTS_DIR=/jffs/ipset_lists
+ BLOCKLISTS_SAVE_DAYS=1
+ IPTABLES_RULE_TARGET=DROP
+ MATCH_SET=--set
+ CREATE=--create
+ ADD=--add
+ SWAP=--swap
+ TEST=--test
+ DELETE=--del
+ FLUSH=--flush
+ IPHASH=iphash
+ NETHASH=nethash
+ SETNOTFOUND=Unknown set
+ /usr/bin/logger -t Firewall Started TorNodes script
+ find /jffs/ipset_lists/tor.lst -mtime +1 -print
+ [ ! -s /jffs/ipset_lists/tor.lst -o -n  ]
+ curl -s -S -o /jffs/ipset_lists/tor.lst http://torstatus.blutmagie.de/ip_list_all.php/Tor_ip_list_ALL.csv
+ touch /jffs/ipset_lists/tor.lst
+ Action=Add
+ ipset -q --swap TorNodes TorNodes
+
+ ipset --flush TorNodes
+ Action=Reload
+ ipset --swap TorNodes TorNodes
+ grep -q Unknown set
+
+ [ -n Reload ]
+ entryCount=0
+ cat /jffs/ipset_lists/tor.lst
+ ipset --add TorNodes 1.32.49.87
+ [ 0 -eq 0 ]
<<<  SNIP LONG IP NUMBERS  >>>
+ entryCount=6378
+ /usr/bin/logger -t Firewall /jffs/TorNodes.sh: Reloaded TorNodes list (6378 entries)
+ unset Action
+ /usr/sbin/iptables-save
+ grep -q TorNodes
+ /usr/sbin/iptables -t raw -vnL PREROUTING
+ grep TorNodes
+ [ -z     9   540 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           set TorNodes src,dst LOG flags 4 level 4 prefix `[BLOCK-TOR]:'
    9   540 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           set TorNodes src,dst  ]
+ exit 0

If I run with cron job, using every day for testing (-mtime +1) its not working.

root@SMEAGOL:/tmp/home/root# /jffs/TorNodes.sh block
+ IPSET_LISTS_DIR=/jffs/ipset_lists
+ BLOCKLISTS_SAVE_DAYS=1
+ IPTABLES_RULE_TARGET=DROP
+ MATCH_SET=--set
+ CREATE=--create
+ ADD=--add
+ SWAP=--swap
+ TEST=--test
+ DELETE=--del
+ FLUSH=--flush
+ IPHASH=iphash
+ NETHASH=nethash
+ SETNOTFOUND=Unknown set
+ /usr/bin/logger -t Firewall Started TorNodes script
+ find /jffs/ipset_lists/tor.lst -mtime +1 -print
+ [ ! -s /jffs/ipset_lists/tor.lst -o -n  ]
+ ipset --swap TorNodes TorNodes
+ grep -q Unknown set
+
+ [ -n  ]
+ /usr/sbin/iptables-save
+ grep -q TorNodes
+ /usr/sbin/iptables -t raw -vnL PREROUTING
+ grep TorNodes
+ [ -z     9   540 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           set TorNodes src,dst LOG flags 4 level 4 prefix `[BLOCK-TOR]:'
    9   540 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           set TorNodes src,dst  ]
+ exit 0

So my conclusion is, this is not working with -mtime +1

-o -n "$(find $IPSET_LISTS_DIR/tor.lst -mtime +$BLOCKLISTS_SAVE_DAYS -print 2>/dev/null)" ]; then

@redhat27
Is this right "Add" ?

Action="Add"
if [ -n "${Action}" ]; then

 

[redhat27]

@redhat27
Is this right "Add" ?

The $Action options are "Add" and "Reload"

However is the quoted lines in the code?

If you are sure you want to run this every day (I feel that it is too frequent), then you can prefix your run command with:

rm /jffs/ipset_lists/??.lst; rm /jffs/ipset_lists/tor.lst; <script>

It will download the country data and tor data each time then. It defeats the purpose of setting a longer time to keep the downloaded data.

 

[octopus]

The $Action options are "Add" and "Reload"

However is the quoted lines in the code?

If you are sure you want to run this every day (I feel that it is too frequent), then you can prefix your run command with:

rm /jffs/ipset_lists/??.lst; rm /jffs/ipset_lists/tor.lst; <script>

It will download the country data and tor data each time then. It defeats the purpose of setting a longer time to keep the downloaded data.

Thank you for your answer.
I do it that way now to get it downloaded, but feel that -mtime +7 should delete tor.lst after 7 days and download a new set of Tor list.
(just for testing I use 1 day)
It seems it does not work.
@redhat27

 

[redhat27]

I do it that way now to get it downloaded, but feel that -mtime +7 should delete tor.lst after 7 days and download a new set of Tor list.
(just for testing I use 1 day)
It seems it does not work.

It may not work for 1 day, but I've verified that it does work for a value more than one. In your example, if I set BLOCKLISTS_SAVE_DAYS=7
and my .lst files are already a week old, it does re-download. I have tested this.