Yet another malware block script using ipset (v4 and v6)

[redhat27]

FYI to all who is using this script. I've noticed that sometimes the google resolver 8.8.8.8 appears in one of the blocklists. That affects the default resolver configured for the hostip utility.

Please include that in the /jffs/ipset_lists/ya-malware-block.whites file (default location) I've also included that in the default whitelist

 

[Builder71]

I'm using the default ya-malware-block.urls
So this uses firehol_level1, 2, 3 and telemetry_and_scanners.txt

When I try,

admin@RT-N66U:/jffs# MatchIP 8.8.8.8
8.8.8.8 not found in YAMalwareBlockCIDR
8.8.8.8 not found in YAMalwareBlock1IP

the output gives me the impression not all lists from the default ya-malware-block.urls are used.
(Only two output lines.)

Is this as expected?

 

[Builder71]

Looking at it, it seems like "ya-malware-block.urls" is not used.
It always goes for YAMalwareBlock1IP and YAMalwareBlockCIDR, no matter what.
What am I missing?

pastebin

 

[redhat27]

Do not comment out the sources in the .urls file. You may leave level4 commented, but I use all of the lists (none of the lists are commented). Sometimes having level4 enabled may have some false positives. (Like 8.8.8.8 may sometimes be mistakenly included). The sources in these lists are very dynamic in nature and frequently change with time. That is why I'd suggested running it every 6 hours

I would recommend you read post #1 of this thread and let me know if anything is missing or confusing there.

 

[Builder71]

Yes, I know I should not comment out the sources .urls file.
This was only my example to show the file seems to be ignored.
Whatever I do with it, only YAMalwareBlockCIDR and YAMalwareBlock1IP are active.

With a default .urls file (only level4 commented out), this is what the logging shows:

Aug 27 06:00:16 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (55323) and YAMalwareBlockCIDR (6649) in 16 seconds

admin@RT-N66U:/tmp/home/root# iptables -L -t raw
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere            set YAMalwareBlockCIDR src
DROP       all  --  anywhere             anywhere            set YAMalwareBlock1IP src

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
admin@RT-N66U:/tmp/home/root#

I expect also to see something from firehol_level2, 3 and telemetry_and_scanners.txt in the logging.
Or is above OK?

 

[redhat27]

There should be more YAMalwareBlock?IP ipsets, generally. The pastebin output you gave had this:

admin@RT-N66U:/jffs/ipset_lists# cat ya-malware-block.urls
#https://raw.githubusercontent.com/shounak-de/misc-scripts/master/telemetry_and_scanners.txt
#https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset
#https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset
#https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset

And that is why I said that. You can try to un-comment all the sources (even level4) and post your output here. The first line from the terminal run gives summary counts.
See post #336

 

[Builder71]

No problem, I start all over.
First run is with all urls active. (No # sign before any line.)
This seems to work better only no log if telemetry_and_scanners.txt became active?

After that I only put a # sign before the last url.
This is the default situation with level4 not active.

Now you see only two urls become active.
There is something going wrong here.

pastebin

 

[Builder71]

Fixed it!

If I add a space after the # sign in the ya-malware-block.urls file it works.

So it should look like this:

admin@RT-N66U:/jffs/ipset_lists# cat ya-malware-block.urls
https://raw.githubusercontent.com/shounak-de/misc-scripts/master/telemetry_and_scanners.txt
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset
# https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset

If it's #https:// (without space) it doesn't work correctly on my RT-N66U.

 

[VZ3]

Yes, I know I should not comment out the sources .urls file.
This was only my example to show the file seems to be ignored.
Whatever I do with it, only YAMalwareBlockCIDR and YAMalwareBlock1IP are active.

With a default .urls file (only level4 commented out), this is what the logging shows:

Aug 27 06:00:16 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (55323) and YAMalwareBlockCIDR (6649) in 16 seconds

admin@RT-N66U:/tmp/home/root# iptables -L -t raw
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere            set YAMalwareBlockCIDR src
DROP       all  --  anywhere             anywhere            set YAMalwareBlock1IP src

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
admin@RT-N66U:/tmp/home/root#

I expect also to see something from firehol_level2, 3 and telemetry_and_scanners.txt in the logging.
Or is above OK?

That's perfectly normal... This output shows that out of default .urls file with level4 commented out you ONLY have 55323 IP addresses to block this particular time. So, 55323 < 65535 - this way you will have ONLY one YAMalwareBlock1IP active...
Some days, when sources IP count goes more then 65k you will see YAMalwareBlock2IP etc.

PS: you last post with "space in between # and https" does not make sense. Show me output of the next command:

curl -sk #https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset

 

[VZ3]

@redhat27,

btw, there is small bug in your script. When block IP count goes above 65k you are creating YAMalwareBlock2IP etc. BUT, when other days IP count stays below 65k those YAMalwareBlock2IP, ...3IP etc. sets are still active in drop chain and did not get deleted.

I guess this is no biggy.

 

[Builder71]

That's perfectly normal... This output shows that out of default .urls file with level4 commented out you ONLY have 55323 IP addresses to block this particular time. So, 55323 < 65535 - this way you will have ONLY one YAMalwareBlock1IP active...
Some days, when sources IP count goes more then 65k you will see YAMalwareBlock2IP etc.

PS: you last post with "space in between # and https" does not make sense. Show me output of the next command:

curl -sk #https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset

admin@RT-N66U:/jffs/scripts# curl -sk #https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset
curl: no URL specified!
curl: try 'curl --help' for more information
admin@RT-N66U:/jffs/scripts#

Ahh, thanks for explaining, I didn't know that.
When I add the extra space, the ya-malware script always shows level1,2,3 and CIDR are loaded.
Because of this output I assumed this is what it should do.

Without the space I currently see level1,2 and CIDR are loaded.
But from what I understand from your comment is that level3 is also loaded but is not shown in script output.
Correct?

Edit: tested with the MatchIP script and level3 IP's are indeed shown. (As level2 but who cares.)

 

[VZ3]

admin@RT-N66U:/jffs/scripts# curl -sk #https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset
curl: no URL specified!
curl: try 'curl --help' for more information
admin@RT-N66U:/jffs/scripts#

Ahh, thanks for explaining, I didn't know that.
When I add the extra space, the ya-malware script always shows level1,2,3 and CIDR are loaded.
Because of this output I assumed this is what it should do.

Without the space I currently see level1,2 and CIDR are loaded.
But from what I understand from your comment is that level3 is also loaded but is not shown in script output.
Correct?

Edit: tested with the MatchIP script and level3 IP's are indeed shown. (As level2 but who cares.)

Your curl output command with commented out URL as expected - meaning no IP list are downloaded from that URL.

Those level1,2,3 and 4 URLs are sources for YAMB block list which later aggregated into YAMalwareBlock 1, 2, 3 etc. lists. YAMB single set cannot hold more then 65k addresses, so it will "overfill" into next set and so on...

Due to the bug in YAM script, if you run it without router restart, ALL your old/previous "overfill" YAMB sets with grater number WILL stay (will not be cleared up) if on next run you will have less IP addresses in source list. So, that can explain your experience when you see YAMB 1, 2 and 3 sets even if you commented out the source URLs.

And again, source URLs level 1, 2, 3 and 4 has no direct relations (one to one) to YAMB1, 2, 3 etc.

 

[Builder71]

...

And again, source URLs level 1, 2, 3 and 4 has no direct relations (one to one) to YAMB1, 2, 3 etc.

That indeed was my mistake.

Your comment in post #410, is this only "cosmetic" or could it mean we have a possiblity to get false positives?

 

[VZ3]

That indeed was my mistake.

Your comment in post #410, is this only "cosmetic" or could it mean we have a possiblity to get false positives?

When previously some "BAD" IP will become "GOOD" one, maybe due to time or other criteria, your router will still treat it as "BAD" until next "overfill" or router reboot.

As for me, it's like good reputation - one have to treasure it from the very beginning .

 

[Builder71]

Sure thing.

I think redhat27 will fix it when he feels like it.

 

[Builder71]

I added below code block in between "esac" and "startTS" command to the ya-malware-block.sh script.
It seems to work well, but I'm horrible at scripting.
So please shoot at it to make it better.

# Delete existing YAMalwareBlock3IP rule.
number=`iptables -t raw -L --line-numbers | grep YAMalwareBlock3IP | cut -d' ' -f1`
logger -t Firewall "Deleting YAMalwareBlock3IP rule if exist."
echo Deleting YAMalwareBlock3IP rule if exist.
iptables -t raw -D PREROUTING $number > /dev/null 2>&1
# Delete existing YAMalwareBlock2IP rule.
number=`iptables -t raw -L --line-numbers | grep YAMalwareBlock2IP | cut -d' ' -f1`
logger -t Firewall "Deleting YAMalwareBlock2IP rule if exist."
echo Deleting YAMalwareBlock2IP rule if exist.
iptables -t raw -D PREROUTING $number > /dev/null 2>&1

 

[Builder71]

@redhat27 @VZ3
A co-worker helped me making it a one liner.
Works well on my RT-N66U. (ipset v4)

Maybe the only "downside" of deleting the old block rules is that the blockstats counter also resets to zero.
(With 'blockstats' command you can see how well your blocklists are doing.)

# Delete old YAMalwareBlock rules.
logger -t Firewall "Delete old YAMalwareBlock rules."
echo Delete old YAMalwareBlock rules.
iptables -t raw -L --line-numbers | grep YAMalwareBlock | cut -d' ' -f1 | sed '1!G;h;$!d' | while read number; do iptables -t raw -D PREROUTING $number; done

Put above code in between the "esac" and "startTS" command to the ya-malware-block.sh script.

 

[VZ3]

@redhat27 @VZ3
A co-worker helped me making it a one liner.
Works well on my RT-N66U. (ipset v4)

Maybe the only "downside" of deleting the old block rules is that the blockstats counter also resets to zero.
(With 'blockstats' command you can see how well your blocklists are doing.)

# Delete old YAMalwareBlock rules.
logger -t Firewall "Delete old YAMalwareBlock rules."
echo Delete old YAMalwareBlock rules.
iptables -t raw -L --line-numbers | grep YAMalwareBlock | cut -d' ' -f1 | sed '1!G;h;$!d' | while read number; do iptables -t raw -D PREROUTING $number; done

Put above code in between the "esac" and "startTS" command to the ya-malware-block.sh script.

Nice job, you are deleting block set from iptables but you forgeting about actual ipset destroying its unused named set.

 

[Builder71]

Nice job, you are deleting block set from iptables but you forgeting about actual ipset destroying its unused named set.

Thx.

Correct, but removing the iptables rules are enough preventing "false positives".
The ipset named set is rewritten anyway when used again.
But true, it's cleaner to remove that from memory as well. (Although it is never used without iptables rule.)

 

[Builder71]

@VZ3 OK, I could not resist to fix that.

# Delete old YAMalwareBlock rules.
logger -t Firewall "Delete old YAMalwareBlock rules."
echo Delete old YAMalwareBlock rules.
iptables -t raw -L --line-numbers | grep YAMalwareBlock | cut -d' ' -f1 | sed '1!G;h;$!d' | while read number; do iptables -t raw -D PREROUTING $number; done
ipset -L | grep YAMalwareBlock | cut -d' ' -f2 | while read setname; do ipset --destroy $setname; done