Skynet Is default firewall good enough?

[SomeWhereOverTheRainBow]

just for brain challenge, what do you think about this:

^([0-9]{1,3}\.){3}(?<=[0-9]{1}\.|[0-9]{2}\.|[0-9]{3}\.)[0-9]{1,3}$

View attachment 47861

What do you think about this as a replacement line?

awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++)printf "%s %s\n", RT, FILENAME}' -- * | Filter_PrivateIP > /tmp/skynet/malwarelist.txt

what do you see when you run?

awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++)printf "%s %s\n", RT, FILENAME}' /mnt/sda1/skynet/lists/*

 

[SomeWhereOverTheRainBow]

here is my test

 GNU nano 7.1                                                                                                       mytest
212.193.30.15
212.193.30.15
216.83.46.88
216.83.46.88
705ef00224f3f7b02e29f21eb6e10d02
7ce7c755fc664713a372e9ee635698da
effa0af1a4f1e1ac8023c5c147f9f569
55a46a2415d18093abcd59a0bf33d0a9
dde1f94b7b8dcd720b6952ba9d71763f
a084f7b249471a5f0d53945003b4a7c6
9bef135ad78f1cc980556008af92f385
ffa2e6f6a7a8001f56c352df43af3fe5
0baa1d0cc20d80fa47eeb764292b9e98
d69589f5bd6c3c799be2d2fd2b718af1
b7b0b7eeec44ec80f82a9bf0a99fe471898e0106a2541ba5eb5a48d7ce3a48be
c26339cc6618e4b051cb2aea4e22288e7218b0dee04c2ea1e9dd6bb51feaa950
6bd33a93372e7cc45c5cf3c040991830bee9f2be6959f4b764feb7f3873fc458
15d5605f08420bd6b2ed02d9e08885e442c3f3e0bd4423b2ca7450f593799963
bc31611b03fe427b9c70459b01d95ed5c173a481efc56aabcdfdc0c0808cc347
dd0bd63ada359ba9e0c332af732770c116fdc178d48c4a6dd55e69dc14525340
fb9fdeb110ab64a155a610ecbdbaf7cc780d1c2dd1bb3cc9e544a13a56a992e2
8f5ed5c923256e5ec1bb7c0aa691419f88d0b3c29777f0577f0ba2f2b69fb674
e887881406cf08519db115a8e1dfb4e470a9d4359c918b8d1111aad676ccdb8c
11ad2567eac856e69a0e013936830a52614caa6b3a1e2da4ca8ad08c995b72c0

awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++)printf "%s %s\n", RT, FILENAME}' mytest
212.193.30.15 mytest
216.83.46.88 mytest

 

[SomeWhereOverTheRainBow]

just for brain challenge, what do you think about this:

^([0-9]{1,3}\.){3}(?<=[0-9]{1}\.|[0-9]{2}\.|[0-9]{3}\.)[0-9]{1,3}$

View attachment 47861

With your regex, you have some operators in there that may require grep -P as oppose to grep -E , no ?<=?

 

[commodoro]

@commodoro

Why does this list not show ranges?

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/myfilter.list

I do a little step behind, a first feedback for the 0 ranges banned, skynet script do the following command to calculate the IP's banned and range

https://github.com/Adamm00/IPSet_ASUS/blob/70c58771d7dca7fe6e0842dc3aed22ddb0142c93/firewall.sh#L1292

and seem with this list skynet dosen't match anything with this statement

https://github.com/Adamm00/IPSet_ASUS/blob/70c58771d7dca7fe6e0842dc3aed22ddb0142c93/firewall.sh#L3484

First row with my filter.list, the second row with list you posted

Maybe is related to the point we speaking about

 

[commodoro]

With your regex, you have some operators in there that may require grep -P as oppose to grep -E , no ?<=?

I don't think lookahead needs the -P, --perl-regexp option, but I could be wrong.

I have to do some testing

 

[SomeWhereOverTheRainBow]

I don't think lookahead needs the -P, --perl-regexp option, but I could be wrong.

I have to do some testing

It works only with grep -P, at least that notation. Maybe there might be another notation?

 

[SomeWhereOverTheRainBow]

I do a little step behind, a first feedback for the 0 ranges banned, skynet script do the following command to calculate the IP's banned and range

https://github.com/Adamm00/IPSet_ASUS/blob/70c58771d7dca7fe6e0842dc3aed22ddb0142c93/firewall.sh#L1292

and seem with this list skynet dosen't match anything with this statement

https://github.com/Adamm00/IPSet_ASUS/blob/70c58771d7dca7fe6e0842dc3aed22ddb0142c93/firewall.sh#L3484

First row with my filter.list, the second row with list you posted

View attachment 47863

Maybe is related to the point we speaking about

While I understand the reason for the use of regex with grep, it appears sometimes it breaks when processing huge lists. I am not sure if there is an issue present, or if it is just sheer breakage in certain instances. I like that you are taking the time to review the issue with me. It is something that needs investigating.

 

[JTnola]

While I understand the reason for the use of regex with grep, it appears sometimes it breaks when processing huge lists. I am not sure if there is an issue present, or if it is just sheer breakage in certain instances. I like that you are taking the time to review the issue with me. It is something that needs investigating.

(If ranges aren’t listed, does that mean country bans aren’t working or? Sorry for dumb question.)

 

[SomeWhereOverTheRainBow]

(If ranges aren’t listed, does that mean country bans aren’t working or? Sorry for dumb question.)

@JTnola , there is an issue with skynet parsing certain lists using its regex method. Under normal conditions, there is no issue with the default country blocking, but if you load one of the lists that seems to be "breaking" things, then it will break country blocking and ASN blocking (basically any type of blocking that uses ranges). As long as you don't incorporate one of the broken lists in your skynet filter list, then the rest of your blocking will work fine.

 

[commodoro]

Just an update in troubleshooting, I've created a test script that uses the statements under investigation.
First answer, my regular expression doesn't work with grep -E

and with grep -P it matches nothing

I also try grep a

https://github.com/Adamm00/IPSet_ASUS/blob/70c58771d7dca7fe6e0842dc3aed22ddb0142c93/firewall.sh#L3484

and it works fine, the next step uses a huge list.

P.S. Code of the test.sh

#!/bin/sh

Filter_PrivateIP() {
        grep -vE '^(0\.|10\.|100\.(6[4-9]|[7-9][0-9]|1[0-1][0-9]|12[0-7])\.|127\.|169\.254\.|172\.1[6-9]\.|172\.2[0-9]\.|172\.3[0-1]\.|192\.0\.0\.|192\.0\.2\.|192\.168\.|198\.(1[8-9])\.|198\.51\.100\.|203\.0\.113\.|2(2[4-9]|[3-4][0-9]|5[0
}

awk '{print $1 " " FILENAME}' -- * | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})? E*' | awk '!x[$0]++' | Filter_PrivateIP > /tmp/mnt/sda1/test/original.txt
awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++)printf "%s %s\n", RT, FILENAME}' E* | Filter_PrivateIP > /tmp/mnt/sda1/test/rainbow.txt
#awk '{print $1 " " FILENAME}' -- * | grep -E '^([0-9]{1,3}\.){3}(?<=[0-9]{1}\.|[0-9]{2}\.|[0-9]{3}\.)[0-9]{1,3}$' | awk '!x[$0]++' | Filter_PrivateIP > /tmp/mnt/sda1/test/commodoro_regex_E.txt
#awk '{print $1 " " FILENAME}' -- E* | grep -P '^([0-9]{1,3}\.){3}(?<=[0-9]{1}\.|[0-9]{2}\.|[0-9]{3}\.)[0-9]{1,3}$' | awk '!x[$0]++' | Filter_PrivateIP > /tmp/mnt/sda1/test/commodoro_regex_P.txt

grep -F "/" /tmp/mnt/sda1/test/original.txt | awk '{printf "add Skynet-BlockedRanges %s comment \"BanMalware: %s\"\n", $1, $2 }' >> /tmp/mnt/sda1/test/original_enrich.txt
grep -F "/" /tmp/mnt/sda1/test/rainbow.txt | awk '{printf "add Skynet-BlockedRanges %s comment \"BanMalware: %s\"\n", $1, $2 }' >> /tmp/mnt/sda1/test/rainbow_enrich.txt

 

[Ubimo]

@commodoro
Would you please share your custom filter list link?

 

[commodoro]

@commodoro
Would you please share your custom filter list link?

I use a local path, not a public repo.

I posted here a step to set a local path

Skynet - Is default firewall good enough?

I been using your list for few months. Is this the current list? If I remember correctly it used to be around 170k. The IP list almost doubled? 328944 IPs (+0) -- 9095 Ranges Banned (+0) || 19740 Inbound -- 2098 Outbound Connections Blocked! 481911 IPs try this feed...

www.snbforums.com

and my current custom list is posted here

Skynet - Is default firewall good enough?

Okay so using the first link, i am not sure why there are no ranges since it would be pulling the lists straight from the sources like skynet default does: https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list as you can see, there is not that much difference between...

www.snbforums.com

it's the same as @SomeWhereOverTheRainBow a part of the list that breaks banned ranges

 

[dave14305]

Why doesn’t someone add a set -x near the top of the script and run firewall banmalware to find any errors?

 

[Viktor Jaep]

I use a local path, not a public repo.

I posted here a step to set a local path

Skynet - Is default firewall good enough?

I been using your list for few months. Is this the current list? If I remember correctly it used to be around 170k. The IP list almost doubled? 328944 IPs (+0) -- 9095 Ranges Banned (+0) || 19740 Inbound -- 2098 Outbound Connections Blocked! 481911 IPs try this feed...

www.snbforums.com

and my current custom list is posted here

Skynet - Is default firewall good enough?

Okay so using the first link, i am not sure why there are no ranges since it would be pulling the lists straight from the sources like skynet default does: https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list as you can see, there is not that much difference between...

www.snbforums.com

it's the same as @SomeWhereOverTheRainBow a part of the list that breaks banned ranges

I've been watching this thread unfold, so I thought I'd throw my 2 cents in as well... My custom list (below) also seems to be experiencing a strange issue that I've been noticing this past week... but almost opposite of what you all are experiencing.

https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list

It was running steady around 190,000 blocked IPs... and now it's reduced down to about 16,000. My banned ranges (I believe) were somewhere under 10,000, but are now at 50,000! That's a crazy increase. I mean, it could be from my country bans, or maybe something changed there?

My filter list:

If I switch back to to the standard firewall filter list, it actually increases... which goes against my logic, since my lists were pulling way more blocks than these ever did?

Default filter list:

I downloaded each of the IP block lists in my filter list, and did not find anything out of the ordinary. They all work... they all return results. I'm a bit at a loss myself...

So then I removed that potentially bad list that was poisoning the rest (https://threatview.io/Downloads/Experimental-IOC-Tweets.txt) and things seem back to normal... ranges still seem abnormally high, but ah well.

 

[commodoro]

Why doesn’t someone add a set -x near the top of the script and run firewall banmalware to find any errors?

Because I like to complicate my life unnecessarily

Actually because I didn't remember the debug mode of the shell, unfortunately the use of IDE leads you to forget the basic things

 

[SomeWhereOverTheRainBow]

Just an update in troubleshooting, I've created a test script that uses the statements under investigation.
First answer, my regular expression doesn't work with grep -E

View attachment 47872

and with grep -P it matches nothing

View attachment 47873

I also try grep a

https://github.com/Adamm00/IPSet_ASUS/blob/70c58771d7dca7fe6e0842dc3aed22ddb0142c93/firewall.sh#L3484

View attachment 47874

and it works fine, the next step uses a huge list.

P.S. Code of the test.sh

#!/bin/sh

Filter_PrivateIP() {
        grep -vE '^(0\.|10\.|100\.(6[4-9]|[7-9][0-9]|1[0-1][0-9]|12[0-7])\.|127\.|169\.254\.|172\.1[6-9]\.|172\.2[0-9]\.|172\.3[0-1]\.|192\.0\.0\.|192\.0\.2\.|192\.168\.|198\.(1[8-9])\.|198\.51\.100\.|203\.0\.113\.|2(2[4-9]|[3-4][0-9]|5[0
}

awk '{print $1 " " FILENAME}' -- * | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})? E*' | awk '!x[$0]++' | Filter_PrivateIP > /tmp/mnt/sda1/test/original.txt
awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++)printf "%s %s\n", RT, FILENAME}' E* | Filter_PrivateIP > /tmp/mnt/sda1/test/rainbow.txt
#awk '{print $1 " " FILENAME}' -- * | grep -E '^([0-9]{1,3}\.){3}(?<=[0-9]{1}\.|[0-9]{2}\.|[0-9]{3}\.)[0-9]{1,3}$' | awk '!x[$0]++' | Filter_PrivateIP > /tmp/mnt/sda1/test/commodoro_regex_E.txt
#awk '{print $1 " " FILENAME}' -- E* | grep -P '^([0-9]{1,3}\.){3}(?<=[0-9]{1}\.|[0-9]{2}\.|[0-9]{3}\.)[0-9]{1,3}$' | awk '!x[$0]++' | Filter_PrivateIP > /tmp/mnt/sda1/test/commodoro_regex_P.txt

grep -F "/" /tmp/mnt/sda1/test/original.txt | awk '{printf "add Skynet-BlockedRanges %s comment \"BanMalware: %s\"\n", $1, $2 }' >> /tmp/mnt/sda1/test/original_enrich.txt
grep -F "/" /tmp/mnt/sda1/test/rainbow.txt | awk '{printf "add Skynet-BlockedRanges %s comment \"BanMalware: %s\"\n", $1, $2 }' >> /tmp/mnt/sda1/test/rainbow_enrich.txt

I am giving it all shes got captain! :-
(This is us being double sure of ourselves since awk likes to do funny things sometimes. I can only imagine the process time on this one)

awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++ && RT ~ /(([0-9]{1,3}\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)([[:space:]]|$)/)printf "%s %s\n", RT, FILENAME}' -- * | Filter_PrivateIP > /tmp/skynet/malwarelist.txt

I think she is going to blow!

 

[SomeWhereOverTheRainBow]

Because I like to complicate my life unnecessarily

Actually because I didn't remember the debug mode of the shell, unfortunately the use of IDE leads you to forget the basic things

I think I figured out the culprit. It is the list itself. It has a couple of bad (or non-real) ip addresses in it which may be breaking the loading of ipsets.

Here is my test work using sipcalc.

root@admin:~# curl -fsSL https://threatview.io/Downloads/Experimental-IOC-Tweets.txt | awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++ && RT ~ /(([0-9]{1,3}\.
){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)([[:space:]]|$)/)printf "%s\n", RT}' | while read -r ips; do sipcalc $ips; done | grep -c 'ERR'
2
root@admin:~# curl -fsL --retry 3 --connect-timeout 3 "https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list" | xargs "curl" -fsSL | awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[
0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++ && RT ~ /(([0-9]{1,3}\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)([[:space:]]|$)/)printf "%s\n", RT}' | while read -r ips; do sipcalc $ips; done | grep -c 'ERR'
2

Why doesn’t someone add a set -x near the top of the script and run firewall banmalware to find any errors?

If we had done what you had suggested we would have figured this out a lot sooner.

@commodoro I believe I discovered the culprit in my last test, but set -x would definitely confirm it.

root@admin:~# curl -fsSL https://threatview.io/Downloads/Experimental-IOC-Tweets.txt | awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++ && RT ~ /(([0-9]{1,3}\.
){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)([[:space:]]|$)/)printf "%s\n", RT}' | while read -r ips; do sipcalc $ips | grep ERR && printf "%s\n" "$ips"; done
-[ERR : Unable to retrieve interface information]
451.91.115.161
-[ERR : Unable to retrieve interface information]
447.96.132.96

These are all the good ip addresses on the list.

curl -fsSL https://threatview.io/Downloads/Experimental-IOC-Tweets.txt | awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++ && RT ~ /(([0-9]{1,3}\.
){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)([[:space:]]|$)/)printf "%s\n", RT}' | while read -r ips; do ! sipcalc $ips | grep -q 'ERR' && printf "%s\n" "$ips"; done
89.208.103.122
212.193.30.14
101.43.249.51
209.141.53.178
154.83.17.116
84.32.34.45
3.112.48.183
87.251.64.176
43.142.18.173
108.165.178.43
108.165.178.42
5.57.245.135
201.93.47.22
34.197.227.138
216.238.70.220
157.245.105.72
134.209.104.25
192.144.205.168
198.251.68.79
107.148.130.152
109.172.45.85
81.70.11.25
23.227.196.194
47.92.122.146
42.193.23.91
124.223.182.22
194.87.46.87
67.207.90.203
65.109.1.49
45.56.100.192
45.32.121.12
119.91.148.9
103.142.246.194
54.248.1.227
82.156.177.149
42.81.85.224
125.76.247.137
43.129.158.87
185.225.74.52
156.232.11.5
198.211.9.165
138.124.180.171
195.189.99.65
84.247.51.87
213.252.245.68
194.165.16.90
209.141.52.22
23.234.41.225
107.173.111.16
13.48.54.61
106.75.227.134
54.69.132.184
10.104.128.97
23.94.255.18
69.176.94.39
23.105.215.114
175.178.40.166
23.234.41.226
45.32.157.106
104.248.83.236
185.246.220.26
212.118.39.116
216.127.164.252
157.90.240.174
103.215.81.189
176.124.211.37
206.189.201.57
23.227.203.70
45.145.230.248
20.211.120.220
103.87.240.167
107.174.186.22
103.241.73.58
45.88.221.91
104.207.152.82
124.70.92.91
107.151.203.95
179.60.147.196
109.172.45.38
109.172.45.111
209.141.36.163
43.129.88.120
137.184.10.204
103.215.223.119
185.254.37.182
89.188.222.22
0.0.0.0
177.135.180.180
139.177.146.20
108.163.207.38
8.130.9.56
54.210.2.63
180.184.84.232
123.60.165.221
103.127.124.139
20.239.161.221
5.188.86.194
34.234.209.157
92.255.85.169
81.68.173.143
3.139.62.192
101.35.240.32
192.168.0.104
162.19.155.49
141.98.10.124
199.195.251.23
190.123.44.214
154.26.192.11
88.119.161.139
91.215.85.143
179.43.175.220
47.242.63.91
114.115.135.149
49.232.34.39
38.54.24.164
216.146.25.49
103.227.117.45
47.243.185.202
10.211.55.13
96.43.99.82
192.3.127.174
194.165.16.95
104.243.143.71
84.32.188.75
179.43.187.185
184.72.146.182
45.95.67.211
3.122.234.72
27.122.56.137
137.184.227.180
52.39.206.235
68.183.233.250
18.184.17.94
185.254.37.224
45.61.185.216
45.61.184.196
179.43.154.155
108.166.220.43
192.3.127.76
77.73.131.193
179.43.187.24
3.90.213.150
104.208.73.11
45.61.186.108
92.255.85.150
43.156.232.7
18.183.219.26
5.181.86.249
3.22.116.191
8.219.59.49
3.84.109.117
3.73.0.134
185.174.102.54
124.220.198.212
185.250.148.97
149.28.132.30
109.192.212.70
192.210.162.147
161.117.177.21
104.237.219.36
37.72.168.213
79.137.248.24
54.188.58.32
161.35.17.28
194.165.16.64
38.34.253.57
109.172.45.28
103.20.221.10
210.209.123.100
62.182.85.254
3.121.125.98
68.178.206.43
129.150.60.95
138.2.87.40
212.113.106.118
31.25.10.196
18.176.136.197
217.195.155.140
217.195.155.138
217.195.155.141
217.195.155.142
82.117.252.82
217.195.155.139
82.157.62.138
47.241.255.31
172.81.62.92
213.252.246.35
116.62.168.211
100.42.70.27
43.156.34.251
107.172.208.88
124.223.173.83
124.223.22.86
159.253.120.205
94.102.49.104
143.42.19.99
163.197.211.154
172.245.129.218
43.142.136.237
23.95.67.59
172.67.165.67
172.69.33.115
172.70.210.29
124.221.169.111
172.16.1.100
185.19.212.125
173.254.204.67
124.222.129.148
52.91.134.155
54.157.206.141
172.16.99.33
34.245.162.8
159.223.178.111
47.90.244.75
8.210.74.45
101.43.122.222
45.129.3.134
46.161.40.118
117.52.18.132
37.220.87.31
45.145.231.204
139.224.207.208
167.172.154.189
79.132.128.191
47.106.193.75
110.40.227.251
212.193.30.15
216.83.46.88
190.123.44.207
43.136.168.94
84.32.131.91
45.61.186.121

 

[dave14305]

I haven’t run Skynet in a long time, but testing this scenario, the issue seems to be in the redirection of output. The grep and awk work fine as-is, but redirecting with overwrite to the same destination as the previous grep/awk seems to be the problem.

Remove the redirect and you’ll see the expected add statements on-screen. Redirect to a different filename and it will populate that file. But trying to redirect or tee -a the awk output to “$skynetipset” will not work.

Changing the >> to > will also work, but overwrites all the previous ipset contents.

grep -F "/" /tmp/skynet/malwarelist.txt | awk '{printf "add Skynet-BlockedRanges %s comment \"BanMalware: %s\"\n", $1, $2 }' >> "$skynetipset"

 

[SomeWhereOverTheRainBow]

I haven’t run Skynet in a long time, but testing this scenario, the issue seems to be in the redirection of output. The grep and awk work fine as-is, but redirecting with overwrite to the same destination as the previous grep/awk seems to be the problem.

Remove the redirect and you’ll see the expected add statements on-screen. Redirect to a different filename and it will populate that file. But trying to redirect or tee -a the awk output to “$skynetipset” will not work.

Changing the >> to > will also work, but overwrites all the previous ipset contents.

grep -F "/" /tmp/skynet/malwarelist.txt | awk '{printf "add Skynet-BlockedRanges %s comment \"BanMalware: %s\"\n", $1, $2 }' >> "$skynetipset"

Most likely the issue present is ipsets halts loading with the bad ip address being loaded with the ipset. For both the lists that fail to load the common denominator is they both contain two bad (nonreal) ip addresses. From what I can recall, iptables won't load and will return error if it incounters a problem with an ip address. Unfortunately Skynet does not seem to be capturing this type of failure, so the processing continues. Even skynets small little segment of code allows the fake ip addresses through because they pass the regex test.

 

[dave14305]

Most likely the issue present is ipsets halts loading with the bad ip address being loaded with the ipset. For both the lists that fail to load the common denominator is they both contain two bad (nonreal) ip addresses. From what I can recall, iptables won't load and will return error if it incounters a problem with an ip address. Unfortunately Skynet does not seem to be capturing this type of failure, so the processing continues. Even skynets small little segment of code allows the fake ip addresses through because they pass the regex test.

This part of the process is failing before ipset gets the file as input. This is just writing a text file with the ipset commands. The ranges never make it into the file. Strange, for sure.