Yet another malware block script using ipset (v4 and v6)

[skeal]

Hello
What would cause the sudden loss of the output to log for YAMalwareBlock2IP and YAMalwareBlock3IP the first YAMalwareBlock1IP runs and records 56644 and the CIDR 7424. Any ideas guys??

Edit: I think possibly the update of signatures to ai-protection may have caused this. The update came with the new firmware.

 

[eclp]

Same here with new beta 380.67 ...

 

[skeal]

I have also observed that the count of IP's in the remaining two categories are slowly incrementing downward. Slowly losing ips every report. I run mine on a 4 hour basis not 6 hours. This has started since updating to 380.66_6 and the new ai-protection signature.

 

[Xentrk]

Hello
What would cause the sudden loss of the output to log for YAMalwareBlock2IP and YAMalwareBlock3IP the first YAMalwareBlock1IP runs and records 56644 and the CIDR 7424. Any ideas guys??

Edit: I think possibly the update of signatures to ai-protection may have caused this. The update came with the new firmware.

Good catch @skeal. Perhaps ipset update (ARM) to 6.32 is the culprit. @Adamm has found issues with it.

Here are my results. No stats for YAMalwareBlock2IP after change to 380.67 Beta 1:

380.67 Alpha 3
Jun 24 18:00:11 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65536) YAMalwareBlock2IP (3865) and YAMalwareBlockCIDR (7001) in 11 seconds

380.67 Beta 1
Jun 26 00:00:11 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (52261) and YAMalwareBlockCIDR (7372) in 10 seconds

Ran the script manually and confirmed I am missing two blocking levels:

./ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)...[58547] ~12s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~2s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~1s
>>> Cleaning up... ~0s
./ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (50803) and YAMalwareBlockCIDR (7744) in 15 seconds

I'll post a link to this post in the 380.67 Beta thread.

 

[skeal]

Good catch @skeal. Perhaps ipset update (ARM) to 6.32 is the culprit. @Adamm has found issues with it.

Here are my results. No stats for YAMalwareBlock2IP after change to 380.67 Beta 1:

380.67 Alpha 3
Jun 24 18:00:11 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65536) YAMalwareBlock2IP (3865) and YAMalwareBlockCIDR (7001) in 11 seconds

380.67 Beta 1
Jun 26 00:00:11 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (52261) and YAMalwareBlockCIDR (7372) in 10 seconds

Ran the script manually and confirmed I am missing two blocking levels:

./ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)...[58547] ~12s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~2s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~1s
>>> Cleaning up... ~0s
./ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (50803) and YAMalwareBlockCIDR (7744) in 15 seconds

I'll post a link to this post in the 380.67 Beta thread.

I have the same logs as you do, so I'm not alone with this issue. Please post any fixes for this hiccup.

 

[Adamm]

Hello
What would cause the sudden loss of the output to log for YAMalwareBlock2IP and YAMalwareBlock3IP the first YAMalwareBlock1IP runs and records 56644 and the CIDR 7424. Any ideas guys??

Same here with new beta 380.67 ...

Good catch @skeal. Perhaps ipset update (ARM) to 6.32 is the culprit. @Adamm has found issues with it.

Here are my results. No stats for YAMalwareBlock2IP after change to 380.67 Beta 1:

There is no issue, and the extra IPSets don't indicate "blocking levels". Redhat simply overcomes the maxelem limit in ipset v4.5 by dividing the total number of ipset entries into lists of 65536. The 3 firehol lists include only 61,534 IP/CIDR entries combined as of the time of this post. These lists are basically a compiled version of multiple providers, so the number of entries fluctuates from day to day (firehol2 alone historically can range from 2,000 to 38,000 entries).

That all being said, because there is less than 65536 entries on these lists currently, they are not automatically divided by the script and all fit in 1 IPSet.

 

[Xentrk]

There is no issue, and the extra IPSets don't indicate "blocking levels". Redhat simply overcomes the maxelem limit in ipset v4.5 by dividing the total number of ipset entries into lists of 65536. The 3 firehol lists include only 61,534 IP/CIDR entries combined as of the time of this post. These lists are basically a compiled version of multiple providers, so the number of entries fluctuates from day to day (firehol2 alone historically can range from 2,000 to 38,000 entries).

That all being said, because there is less than 65536 entries on these lists currently, they are not automatically divided by the script and all fit in 1 IPSet.

Thank you @Adamm for the great explanation.

 

[skeal]

Thank you for the excellent explanation of this presentation of information.

 

[Johnathon]

Hi there,

I am a bit of a noob with the scripting. I managed to get it rocking and rolling. However, I can't seem to get level 3 to initialize? Has that been disabled or am I doing something wrong? I am running merlin's beta 1.

Thanks,

J

 

[Xentrk]

Hi there,

I am a bit of a noob with the scripting. I managed to get it rocking and rolling. However, I can't seem to get level 3 to initialize? Has that been disabled or am I doing something wrong? I am running merlin's beta 1.

Thanks,

J

I am experiencing the same thing. The explanation is a few posts above.

https://www.snbforums.com/threads/y...ing-ipset-v4-and-v6.38935/page-17#post-332707

Jun 29 06:00:17 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (61790) YAMalwareBlock2IP (0) and YAMalwareBlockCIDR (8110) in 16 seconds

I reinstalled the code per the instructions on post 1 just to make sure I had not mucked anything up.

Hoping that @redhat27 will chime in soon. I know his day job has been keeping him very busy lately....

 

[Jack Yaz]

I am experiencing the same thing. The explanation is a few posts above.

https://www.snbforums.com/threads/y...ing-ipset-v4-and-v6.38935/page-17#post-332707

Jun 29 06:00:17 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (61790) YAMalwareBlock2IP (0) and YAMalwareBlockCIDR (8110) in 16 seconds

I reinstalled the code per the instructions on post 1 just to make sure I had not mucked anything up.

Hoping that @redhat27 will chime in soon. I know his day job has been keeping him very busy lately....

Looking at firehol 1-3 currently totals 71672 ipset entries. Your log suggests you have 69900 so it's not far off, I'm not sure if whitelist explains the difference

 

[Xentrk]

Looking at firehol 1-3 currently totals 71672 ipset entries. Your log suggests you have 69900 so it's not far off, I'm not sure if whitelist explains the difference

Thanks Jack. That is good to know. I don't have that many entries in whitelist. I looked at the logs and see the number of ipset varies each time the job runs.

 

[Adamm]

Nothing is wrong, again its just duplicates in the 71000 entries being removed before being processed into sets.

 

[redhat27]

Hello everyone.. So sorry for absconding almost a month. Thanks @Adamm for fielding some questions.

Hoping that @redhat27 will chime in soon.

I'll try to answer some the best I can. What @Adamm said is right, the number of sources in the FireHOL lists vary quite a bit with time.

I managed to get it rocking and rolling. However, I can't seem to get level 3 to initialize?

Is it that you do not see the YAMalwareBlock3IP set? That has nothing to do with level 3 of the FireHOL list. These YAMalwareBlock?IP sets are just groups of 64k discrete IP sets that grow or shrink depending on the number of total discrete IPs are there in the combined input list of IPs that are being blocked by the script. You can enable FireHOL Level4 in the

/jffs/ipset_lists/ya-malware-block.urls

file by removing the '#' character in front of the line for https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset (most likely the last line)
That will have more combined IPs.

I am experiencing the same thing.

Jun 29 06:00:17 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (61790) YAMalwareBlock2IP (0) and YAMalwareBlockCIDR (8110) in 16 seconds

I reinstalled the code per the instructions on post 1 just to make sure I had not mucked anything up.

@Xentrk You are right, on no account the script should produce an output to say YAMalwareBlock2IP (0) I will put out a new version soon to take care of the edge cases.

That version worked on Shibby Tomato Firmware 1.28.0000 MIPSR2-140 K26AC USB AIO-64K

Thank you!

Thank you for confirming it! Much appreciated

 

[Xentrk]

Thanks for the clarification @redhat27. I thought there was a one to one relationship with level1 thru 4 and YAMalwareBlock1-4IP. I enabled YAMalwareBlock4IP and now get metrics for all BlockIPs.

 Loaded sets YAMalwareBlock1IP (65536) YAMalwareBlock2IP (65536) YAMalwareBlock3IP (65536) YAMalwareBlock4IP (11101) and YAMalwareBlockCIDR (11839) in 31 seconds

Happy Fire Cracker!

 

[redhat27]

Version 2.4 is up. Tomato version also updated.

This is a minor display fix release, where the counts of the discrete IPs and CIDR ranges are displayed when run from console.

This is a sample run on my (slow) router from the terminal with the default blocking (Level1 through Level3)

admin@RT-AC66R-D700:/tmp/home/root# ya-malware-block.sh
/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)...[131428/122042/9386] ~11s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~4s
>>> Adding data and processing rule for YAMalwareBlock2IP... ~4s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~1s
>>> Cleaning up... ~0s
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (56507) and YAMalwareBlockCIDR (9386) in 20 seconds

Note: the [131428/122042/9386] at the top? It is the [total entries/discrete IPs/CIDR ranges] count after aggregating the sources and eliminating duplicates and applying custom blacklists/whitelists
YAMalwareBlock1IP (65535) YAMalwareBlock2IP (56507): 65535+56507=122042 (discrete IPs)
YAMalwareBlockCIDR (9386) = 9386 (CIDR ranges)
122042+9386=131428 (Total)

And this is with all 4 Levels enabled:

admin@RT-AC66R-D700:/tmp/home/root# ya-malware-block.sh
/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)...[250472/238198/12274] ~23s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~5s
>>> Adding data and processing rule for YAMalwareBlock2IP... ~5s
>>> Adding data and processing rule for YAMalwareBlock3IP... ~5s
>>> Adding data and processing rule for YAMalwareBlock4IP... ~3s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~2s
>>> Cleaning up... ~0s
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (65535) YAMalwareBlock3IP (65535) YAMalwareBlock4IP (41593) and YAMalwareBlockCIDR (12274) in 43 seconds

Similarly:
[250472/238198/12274] means 250472 Total, 238198 discrete IPs and 12274 CIDR ranges
65535+65535+65535+41593=238198 discrete IPs and 12274 CIDR ranges for a total of 250472 source entries.

 

[Csection]

Version 2.4 is up. Tomato version also updated.

This is a minor display fix release, where the counts of the discrete IPs and CIDR ranges are displayed when run from console.

This is a sample run on my (slow) router from the terminal with the default blocking (Level1 through Level3)

admin@RT-AC66R-D700:/tmp/home/root# ya-malware-block.sh
/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)...[131428/122042/9386] ~11s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~4s
>>> Adding data and processing rule for YAMalwareBlock2IP... ~4s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~1s
>>> Cleaning up... ~0s
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (56507) and YAMalwareBlockCIDR (9386) in 20 seconds

Note: the [131428/122042/9386] at the top? It is the [total entries/discrete IPs/CIDR ranges] count after aggregating the sources and eliminating duplicates and applying custom blacklists/whitelists
YAMalwareBlock1IP (65535) YAMalwareBlock2IP (56507): 65535+56507=122042 (discrete IPs)
YAMalwareBlockCIDR (9386) = 9386 (CIDR ranges)
122042+9386=131428 (Total)

And this is with all 4 Levels enabled:

admin@RT-AC66R-D700:/tmp/home/root# ya-malware-block.sh
/jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
>>> Downloading and aggregating malware sources (also processing whitelists)...[250472/238198/12274] ~23s
>>> Adding data and processing rule for YAMalwareBlock1IP... ~5s
>>> Adding data and processing rule for YAMalwareBlock2IP... ~5s
>>> Adding data and processing rule for YAMalwareBlock3IP... ~5s
>>> Adding data and processing rule for YAMalwareBlock4IP... ~3s
>>> Adding data and processing rule for YAMalwareBlockCIDR... ~2s
>>> Cleaning up... ~0s
/jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (65535) YAMalwareBlock2IP (65535) YAMalwareBlock3IP (65535) YAMalwareBlock4IP (41593) and YAMalwareBlockCIDR (12274) in 43 seconds

Similarly:
[250472/238198/12274] means 250472 Total, 238198 discrete IPs and 12274 CIDR ranges
65535+65535+65535+41593=238198 discrete IPs and 12274 CIDR ranges for a total of 250472 source entries.

I ran the 2.4 script from the command line and it appeared to run ok, but it locked my router up.
I had to power off and back on to regain control. 2.3 runs at bootup without any problems.
Please advise!

 

[redhat27]

I ran the 2.4 script from the command line and it appeared to run ok, but it locked my router up.
I had to power off and back on to regain control. 2.3 runs at bootup without any problems.
Please advise!

I can only surmise that the lock up was coincidental. I suggest you try again with the new version.

 

[Csection]

I can only surmise that the lock up was coincidental. I suggest you try again with the new version.

Ok, Will do!
Thanks!

 

[Csection]

Ok, Will do!
Thanks!

I tried it again with the same results.
Router locked up again!
I had to reset it back to the original 2.3 version.