What's new

Skynet Is default firewall good enough?

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Okay so using the first link, i am not sure why there are no ranges since it would be pulling the lists straight from the sources like skynet default does:

as you can see, there is not that much difference between the default, versus mine:


in fact I include all the lists that are in skynet default. So it should be banning those "ranges" still.



The question begs to ask why is skynet doing this incorrectly in some instances?

Let us look at what skynet code defines to be a "range"-

This is User added ranges (added by skynet menu option for ranges):


This is ranges blocked by blocking countries:


This is ranges blocked by blocking VIA "ASN" codes


@Adamm- "Why are when users load a custom filter.list, their user defined ranges are not included in the iptable rules; however, when they use the default filter lists their user defined ranges are included in the iptable rules?"

Basically @Ubimo has pointed out when they load a custom filter.list, not much different from the default, there are no longer any ranges showing up as being banned in the statistic ( which implies the skynet-banned ranges rules are not getting loaded when switching to custom filter.list option). However, when they switch back to default filter.list, their banned ranges are once again included in the statistics (which implies skynet-banned ranges are once again loaded properly into the iptable ruleset).

@Ubimo

In the code link below, I see reference to using the custom list with the banmalware option. Possibly different steps are taking if "fast-switch" is enabled in diversion or skynet. Maybe this is causing the skipping of adding ranges when using custom filter lists? But I don't know @Adamm script well enough yet to tell you.


more specifically, this line:



@Ubimo

A temporary solution is to manually restart skynet after switching to custom filter lists and your "blocked ranges" should hopefully be added back. (banned ranges = ranges you have added, any countries you block, and any ASN codes you block, and anything that gets added with a subnet attached like /24 at the end of the ipaddress).

Code:
( firewall ban range 8.8.8.8/24 "Apples" ) This Bans the CIDR Block Specified With The Comment Apples
( firewall ban country "pk cn sa" ) This Bans The Known IPs For The Specified Countries (Accepts Single/Multiple Inputs If Quoted) https://www.ipdeny.com/ipblocks/
( firewall ban asn AS123456 ) This Bans the ASN Specified

@Ubimo

If the custom filter list were loading correctly, here would be the amount of banned IPS ranges contributed.

So if our banned ranges now show zero, does that mean our country bans aren’t working, or?

This is what I now get using your list.

1A9CDAED-ADB5-4CC2-AB63-6D9E9DE5B1CF.jpeg


Banned Countries; af,bd,cn,ge,hk,hu,id,in,iq,ir,kp,no,pk,pl,ro,ru,sa,th,tr,ua

207104 IPs (+0) -- 0 Ranges Banned (+0) || 113 Inbound -- 0 Outbound Connections Blocked!
—————————

I’ve tried resetting to default list (2114 banned ranges show when on default)

Things I’ve tried …
switching to a different custom list
Switching back to your list
Resetting exclusion list
Restarting SkyNet
Force update SkyNet
Re-entering country bans
 
Hi

I have made some checks on list that @SomeWhereOverTheRainBow shared hear

Code:
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

and identified the following list

Code:
https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

that somehow breaks something.

Now my filter list is:

Code:
https://blocklist.greensnow.co/greensnow.txt
https://darklist.de/raw.php
https://feodotracker.abuse.ch/downloads/ipblocklist.txt
https://iplists.firehol.org/files/alienvault_reputation.ipset
https://iplists.firehol.org/files/bds_atif.ipset
https://iplists.firehol.org/files/bi_any_2_30d.ipset
https://iplists.firehol.org/files/blocklist_net_ua.ipset
https://iplists.firehol.org/files/ciarmy.ipset
https://iplists.firehol.org/files/coinbl_hosts_browser.ipset
https://iplists.firehol.org/files/cybercrime.ipset
https://iplists.firehol.org/files/dshield.netset
https://iplists.firehol.org/files/dshield_1d.netset
https://iplists.firehol.org/files/dyndns_ponmocup.ipset
https://iplists.firehol.org/files/et_block.netset
https://iplists.firehol.org/files/et_compromised.ipset
https://iplists.firehol.org/files/firehol_level1.netset
https://iplists.firehol.org/files/firehol_level2.netset
https://iplists.firehol.org/files/firehol_level3.netset
https://iplists.firehol.org/files/greensnow.ipset
https://iplists.firehol.org/files/iblocklist_ciarmy_malicious.netset
https://iplists.firehol.org/files/iblocklist_pedophiles.netset
https://iplists.firehol.org/files/iblocklist_spamhaus_drop.netset
https://iplists.firehol.org/files/malc0de.ipset
https://iplists.firehol.org/files/myip.ipset
https://iplists.firehol.org/files/normshield_high_attack.ipset
https://iplists.firehol.org/files/normshield_high_bruteforce.ipset
https://iplists.firehol.org/files/spamhaus_drop.netset
https://iplists.firehol.org/files/spamhaus_edrop.netset
https://iplists.firehol.org/files/urlvir.ipset
https://lists.blocklist.de/lists/strongips.txt
https://myip.ms/files/blacklist/general/latest_blacklist.txt
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/IPlist.list
https://raw.githubusercontent.com/SecOps-Institute/Tor-IP-Addresses/master/tor-exit-nodes.lst
https://raw.githubusercontent.com/stamparm/ipsum/master/levels/1.txt
https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
https://threatview.io/Downloads/IP-High-Confidence-Feed.txt
https://www.blocklist.de/downloads/export-ips_all.txt
https://www.talosintelligence.com/documents/ip-blacklist

Screenshot 2023-02-10 alle 12.29.08.jpg


I'll try to understand why.

Regards
Commodoro
 
Hi

I have made some checks on list that @SomeWhereOverTheRainBow shared hear

Code:
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

and identified the following list

Code:
https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

that somehow breaks something.

Now my filter list is:

Code:
https://blocklist.greensnow.co/greensnow.txt
https://darklist.de/raw.php
https://feodotracker.abuse.ch/downloads/ipblocklist.txt
https://iplists.firehol.org/files/alienvault_reputation.ipset
https://iplists.firehol.org/files/bds_atif.ipset
https://iplists.firehol.org/files/bi_any_2_30d.ipset
https://iplists.firehol.org/files/blocklist_net_ua.ipset
https://iplists.firehol.org/files/ciarmy.ipset
https://iplists.firehol.org/files/coinbl_hosts_browser.ipset
https://iplists.firehol.org/files/cybercrime.ipset
https://iplists.firehol.org/files/dshield.netset
https://iplists.firehol.org/files/dshield_1d.netset
https://iplists.firehol.org/files/dyndns_ponmocup.ipset
https://iplists.firehol.org/files/et_block.netset
https://iplists.firehol.org/files/et_compromised.ipset
https://iplists.firehol.org/files/firehol_level1.netset
https://iplists.firehol.org/files/firehol_level2.netset
https://iplists.firehol.org/files/firehol_level3.netset
https://iplists.firehol.org/files/greensnow.ipset
https://iplists.firehol.org/files/iblocklist_ciarmy_malicious.netset
https://iplists.firehol.org/files/iblocklist_pedophiles.netset
https://iplists.firehol.org/files/iblocklist_spamhaus_drop.netset
https://iplists.firehol.org/files/malc0de.ipset
https://iplists.firehol.org/files/myip.ipset
https://iplists.firehol.org/files/normshield_high_attack.ipset
https://iplists.firehol.org/files/normshield_high_bruteforce.ipset
https://iplists.firehol.org/files/spamhaus_drop.netset
https://iplists.firehol.org/files/spamhaus_edrop.netset
https://iplists.firehol.org/files/urlvir.ipset
https://lists.blocklist.de/lists/strongips.txt
https://myip.ms/files/blacklist/general/latest_blacklist.txt
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/IPlist.list
https://raw.githubusercontent.com/SecOps-Institute/Tor-IP-Addresses/master/tor-exit-nodes.lst
https://raw.githubusercontent.com/stamparm/ipsum/master/levels/1.txt
https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
https://threatview.io/Downloads/IP-High-Confidence-Feed.txt
https://www.blocklist.de/downloads/export-ips_all.txt
https://www.talosintelligence.com/documents/ip-blacklist

View attachment 47843

I'll try to understand why.

Regards
Commodoro
Commodoro, your list is working now? Do you share it publicly, and if so, would you be willing to do so here?

Thank you to you and to everyone else who has been so responsive & thorough in addressing this...
 
Hi

I have made some checks on list that @SomeWhereOverTheRainBow shared hear

Code:
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

and identified the following list

Code:
https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

that somehow breaks something.

Now my filter list is:

Code:
https://blocklist.greensnow.co/greensnow.txt
https://darklist.de/raw.php
https://feodotracker.abuse.ch/downloads/ipblocklist.txt
https://iplists.firehol.org/files/alienvault_reputation.ipset
https://iplists.firehol.org/files/bds_atif.ipset
https://iplists.firehol.org/files/bi_any_2_30d.ipset
https://iplists.firehol.org/files/blocklist_net_ua.ipset
https://iplists.firehol.org/files/ciarmy.ipset
https://iplists.firehol.org/files/coinbl_hosts_browser.ipset
https://iplists.firehol.org/files/cybercrime.ipset
https://iplists.firehol.org/files/dshield.netset
https://iplists.firehol.org/files/dshield_1d.netset
https://iplists.firehol.org/files/dyndns_ponmocup.ipset
https://iplists.firehol.org/files/et_block.netset
https://iplists.firehol.org/files/et_compromised.ipset
https://iplists.firehol.org/files/firehol_level1.netset
https://iplists.firehol.org/files/firehol_level2.netset
https://iplists.firehol.org/files/firehol_level3.netset
https://iplists.firehol.org/files/greensnow.ipset
https://iplists.firehol.org/files/iblocklist_ciarmy_malicious.netset
https://iplists.firehol.org/files/iblocklist_pedophiles.netset
https://iplists.firehol.org/files/iblocklist_spamhaus_drop.netset
https://iplists.firehol.org/files/malc0de.ipset
https://iplists.firehol.org/files/myip.ipset
https://iplists.firehol.org/files/normshield_high_attack.ipset
https://iplists.firehol.org/files/normshield_high_bruteforce.ipset
https://iplists.firehol.org/files/spamhaus_drop.netset
https://iplists.firehol.org/files/spamhaus_edrop.netset
https://iplists.firehol.org/files/urlvir.ipset
https://lists.blocklist.de/lists/strongips.txt
https://myip.ms/files/blacklist/general/latest_blacklist.txt
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/IPlist.list
https://raw.githubusercontent.com/SecOps-Institute/Tor-IP-Addresses/master/tor-exit-nodes.lst
https://raw.githubusercontent.com/stamparm/ipsum/master/levels/1.txt
https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
https://threatview.io/Downloads/IP-High-Confidence-Feed.txt
https://www.blocklist.de/downloads/export-ips_all.txt
https://www.talosintelligence.com/documents/ip-blacklist

View attachment 47843

I'll try to understand why.

Regards
Commodoro
1. It may be your ,php list as perhaps skynet cannot handle
2. Hint: keep your list clean, no comments or remarks, just the straight urls.
2. I am not sure if you realize or not but:
a. you have a lot of overlapping lists like greensnow from source and greensnow from firehol or https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt is the same as firehol's et-block.netset
b. you have a lot of old lists that you may consider removing. examples include the 2 normsheild lists, the two dshield lists (in fact you get a current copy within https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt).
I recommend to look at the firehol ip list webesite as you see for each of your lists, the last time they were updated. anyway, YMMV
 
1. It may be your ,php list as perhaps skynet cannot handle
2. Hint: keep your list clean, no comments or remarks, just the straight urls.
2. I am not sure if you realize or not but:
a. you have a lot of overlapping lists like greensnow from source and greensnow from firehol or https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt is the same as firehol's et-block.netset
b. you have a lot of old lists that you may consider removing. examples include the 2 normsheild lists, the two dshield lists (in fact you get a current copy within https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt).
I recommend to look at the firehol ip list webesite as you see for each of your lists, the last time they were updated. anyway, YMMV
Skynet removes duplicates immediately on download using a rather quick awk command.
 
Hi

I have made some checks on list that @SomeWhereOverTheRainBow shared hear

Code:
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

and identified the following list

Code:
https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

that somehow breaks something.

Now my filter list is:

Code:
https://blocklist.greensnow.co/greensnow.txt
https://darklist.de/raw.php
https://feodotracker.abuse.ch/downloads/ipblocklist.txt
https://iplists.firehol.org/files/alienvault_reputation.ipset
https://iplists.firehol.org/files/bds_atif.ipset
https://iplists.firehol.org/files/bi_any_2_30d.ipset
https://iplists.firehol.org/files/blocklist_net_ua.ipset
https://iplists.firehol.org/files/ciarmy.ipset
https://iplists.firehol.org/files/coinbl_hosts_browser.ipset
https://iplists.firehol.org/files/cybercrime.ipset
https://iplists.firehol.org/files/dshield.netset
https://iplists.firehol.org/files/dshield_1d.netset
https://iplists.firehol.org/files/dyndns_ponmocup.ipset
https://iplists.firehol.org/files/et_block.netset
https://iplists.firehol.org/files/et_compromised.ipset
https://iplists.firehol.org/files/firehol_level1.netset
https://iplists.firehol.org/files/firehol_level2.netset
https://iplists.firehol.org/files/firehol_level3.netset
https://iplists.firehol.org/files/greensnow.ipset
https://iplists.firehol.org/files/iblocklist_ciarmy_malicious.netset
https://iplists.firehol.org/files/iblocklist_pedophiles.netset
https://iplists.firehol.org/files/iblocklist_spamhaus_drop.netset
https://iplists.firehol.org/files/malc0de.ipset
https://iplists.firehol.org/files/myip.ipset
https://iplists.firehol.org/files/normshield_high_attack.ipset
https://iplists.firehol.org/files/normshield_high_bruteforce.ipset
https://iplists.firehol.org/files/spamhaus_drop.netset
https://iplists.firehol.org/files/spamhaus_edrop.netset
https://iplists.firehol.org/files/urlvir.ipset
https://lists.blocklist.de/lists/strongips.txt
https://myip.ms/files/blacklist/general/latest_blacklist.txt
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/IPlist.list
https://raw.githubusercontent.com/SecOps-Institute/Tor-IP-Addresses/master/tor-exit-nodes.lst
https://raw.githubusercontent.com/stamparm/ipsum/master/levels/1.txt
https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
https://threatview.io/Downloads/IP-High-Confidence-Feed.txt
https://www.blocklist.de/downloads/export-ips_all.txt
https://www.talosintelligence.com/documents/ip-blacklist

View attachment 47843

I'll try to understand why.

Regards
Commodoro
I am able to parse that list just fine using the same commands skynet does .

Code:
curl -fsL --retry 3 --connect-timeout 3 "https://threatview.io/Downloads/Experimental-IOC-Tweets.txt" | awk '{print $0}' | grep -E '^(([0-9]{1,3}\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$' | grep -vE '^(0\.|10\.|100\.(6[4-9]|[7-9][0-9]|1[0-1][0-9]|12[0-7])\.|127\.|169\.254\.|172\.1[6-9]\.|172\.2[0-9]\.|172\.3[0-1]\.|192\.0\.0\.|192\.0\.2\.|192\.168\.|198\.(1[8-9])\.|198\.51\.100\.|203\.0\.113\.|2(2[4-9]|[3-4][0-9]|5[0-5])\.|8\.8\.8\.8|8\.8\.4\.4|1\.1\.1\.1|1\.0\.0\.1)' | awk '!x[$0]++'
89.208.103.122
212.193.30.14
101.43.249.51
451.91.115.161
209.141.53.178
154.83.17.116
447.96.132.96
84.32.34.45
3.112.48.183
87.251.64.176
43.142.18.173
108.165.178.43
108.165.178.42
5.57.245.135
201.93.47.22
34.197.227.138
216.238.70.220
157.245.105.72
134.209.104.25
192.144.205.168
198.251.68.79
107.148.130.152
109.172.45.85
81.70.11.25
23.227.196.194
47.92.122.146
42.193.23.91
124.223.182.22
194.87.46.87
67.207.90.203
65.109.1.49
45.56.100.192
45.32.121.12
119.91.148.9
103.142.246.194
54.248.1.227
82.156.177.149
42.81.85.224
125.76.247.137
43.129.158.87
185.225.74.52
156.232.11.5
198.211.9.165
138.124.180.171
195.189.99.65
84.247.51.87
213.252.245.68
194.165.16.90
209.141.52.22
23.234.41.225
107.173.111.16
13.48.54.61
106.75.227.134
54.69.132.184
23.94.255.18
69.176.94.39
23.105.215.114
175.178.40.166
23.234.41.226
45.32.157.106
104.248.83.236
185.246.220.26
212.118.39.116
216.127.164.252
157.90.240.174
103.215.81.189
176.124.211.37
206.189.201.57
23.227.203.70
45.145.230.248
20.211.120.220
103.87.240.167
107.174.186.22
103.241.73.58
45.88.221.91
104.207.152.82
124.70.92.91
107.151.203.95
179.60.147.196
109.172.45.38
109.172.45.111
209.141.36.163
43.129.88.120
137.184.10.204
103.215.223.119
185.254.37.182
89.188.222.22
177.135.180.180
139.177.146.20
108.163.207.38
8.130.9.56
54.210.2.63
180.184.84.232
123.60.165.221
103.127.124.139
20.239.161.221
5.188.86.194
34.234.209.157
92.255.85.169
81.68.173.143
3.139.62.192
101.35.240.32
162.19.155.49
141.98.10.124
199.195.251.23
190.123.44.214
154.26.192.11
88.119.161.139
91.215.85.143
179.43.175.220
47.242.63.91
114.115.135.149
49.232.34.39
38.54.24.164
216.146.25.49
103.227.117.45
47.243.185.202
96.43.99.82
192.3.127.174
194.165.16.95
104.243.143.71
84.32.188.75
179.43.187.185
184.72.146.182
45.95.67.211
3.122.234.72
27.122.56.137
137.184.227.180
52.39.206.235
68.183.233.250
18.184.17.94
185.254.37.224
45.61.185.216
45.61.184.196
179.43.154.155
108.166.220.43
192.3.127.76
77.73.131.193
179.43.187.24
3.90.213.150
104.208.73.11
45.61.186.108
92.255.85.150
43.156.232.7
18.183.219.26
5.181.86.249
3.22.116.191
8.219.59.49
3.84.109.117
3.73.0.134
185.174.102.54
124.220.198.212
185.250.148.97
149.28.132.30
109.192.212.70
192.210.162.147
161.117.177.21
104.237.219.36
37.72.168.213
79.137.248.24
54.188.58.32
161.35.17.28
194.165.16.64
38.34.253.57
109.172.45.28
103.20.221.10
210.209.123.100
62.182.85.254
3.121.125.98
68.178.206.43
129.150.60.95
138.2.87.40
212.113.106.118
31.25.10.196
18.176.136.197
217.195.155.140
217.195.155.138
217.195.155.141
217.195.155.142
82.117.252.82
217.195.155.139
82.157.62.138
47.241.255.31
172.81.62.92
213.252.246.35
116.62.168.211
100.42.70.27
43.156.34.251
107.172.208.88
124.223.173.83
124.223.22.86
159.253.120.205
94.102.49.104
143.42.19.99
163.197.211.154
172.245.129.218
43.142.136.237
23.95.67.59
172.67.165.67
172.69.33.115
172.70.210.29
124.221.169.111
185.19.212.125
173.254.204.67
124.222.129.148
52.91.134.155
54.157.206.141
34.245.162.8
159.223.178.111
47.90.244.75
8.210.74.45
101.43.122.222
45.129.3.134
46.161.40.118
117.52.18.132
37.220.87.31
45.145.231.204
139.224.207.208
167.172.154.189
79.132.128.191
47.106.193.75
110.40.227.251
212.193.30.15
216.83.46.88
 
Last edited:
Commodoro, your list is working now? Do you share it publicly, and if so, would you be willing to do so here?

Thank you to you and to everyone else who has been so responsive & thorough in addressing this...

Hi @JTnola, yes for me now works, see the image attached.
I use a local list stored in /jffs/configs/filter.list

1. It may be your ,php list as perhaps skynet cannot handle
2. Hint: keep your list clean, no comments or remarks, just the straight urls.
2. I am not sure if you realize or not but:
a. you have a lot of overlapping lists like greensnow from source and greensnow from firehol or https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt is the same as firehol's et-block.netset
b. you have a lot of old lists that you may consider removing. examples include the 2 normsheild lists, the two dshield lists (in fact you get a current copy within https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt).
I recommend to look at the firehol ip list webesite as you see for each of your lists, the last time they were updated. anyway, YMMV

Thanks, I'll keep that in mind

I am able to parse that list just fine using the same commands skynet does .
I didn't mean to question anything, sorry if you felt this way, that's not my intention, just wanted to make a contribution

Regards
Commodoro
 
1. It may be your ,php list as perhaps skynet cannot handle
2. Hint: keep your list clean, no comments or remarks, just the straight urls.
2. I am not sure if you realize or not but:
a. you have a lot of overlapping lists like greensnow from source and greensnow from firehol or https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt is the same as firehol's et-block.netset
b. you have a lot of old lists that you may consider removing. examples include the 2 normsheild lists, the two dshield lists (in fact you get a current copy within https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt).
I recommend to look at the firehol ip list webesite as you see for each of your lists, the last time they were updated. anyway, YMMV
The php list works fine too.

here is part of the end for it.

Code:
93.189.11.246
93.33.202.137
93.42.88.50
93.43.240.145
93.51.176.72
93.61.137.226
93.67.138.66
93.77.135.27
93.84.111.7
94.110.108.120
94.153.212.68
94.153.212.78
94.159.31.10
94.180.247.20
94.180.57.15
94.181.51.252
94.188.177.110
94.191.60.181
94.191.62.195
94.191.71.246
94.30.68.41
94.43.85.6
94.69.226.48
95.111.253.141
95.128.43.164
95.136.11.116
95.140.202.165
95.141.232.2
95.161.97.113
95.216.25.250
95.77.98.196
95.79.31.128
95.79.56.120
95.84.240.168
95.85.15.86
95.85.33.224
95.85.34.53
95.85.38.127
95.85.39.74
95.85.43.241
95.85.9.94
96.1.64.194
96.125.164.115
96.227.85.22
96.30.6.7
96.56.221.138
96.64.11.9
96.69.13.140
96.78.175.36
96.78.175.37
96.79.228.114
96.8.119.39
96.84.149.98
96.93.196.89
97.64.122.66
97.74.81.53
97.74.82.38
97.74.95.243
98.142.141.184
98.143.148.45
98.195.176.219
98.29.18.119
99.149.251.77
99.184.69.177
99.185.76.161
99.191.118.206
99.96.122.99

perhaps a simple word count should suffice.

Code:
curl -fsL --retry 3 --connect-timeout 3 "https://darklist.de/raw.php" | grep -E '^(([0-9]{1,3}\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$' | grep -vE '^(0\.|10\.|100\.(6[4-9]|[7-9][0-9]|1[0-1][0-9]|12[0-7])\.|127\.|169\.254\.|172\.1[6-9]\.|172\.2[0-9]\.|172\.3[0-1]\.|192\.0\.0\.|192\.0\.2\.|192\.168\.|198\.(1[8-9])\.|198\.51\.100\.|203\.0\.113\.|2(2[4-9]|[3-4][0-9]|5[0-5])\.|8\.8\.8\.8|8\.8\.4\.4|1\.1\.1\.1|1\.0\.0\.1)' | awk '!x[$0]++' | wc -l
6037
 
Hi @JTnola, yes for me now works, see the image attached.
I use a local list stored in /jffs/configs/filter.list



Thanks, I'll keep that in mind


I didn't mean to question anything, sorry if you felt this way, that's not my intention, just wanted to make a contribution

Regards
Commodoro
No you are fine, I was mentioning that because I have multiples listed for list resilience ( or availability). I like to be prepared in case one of the services does down. Knowing that @Adamm deduplicates is useful because it allows us to be able to have list resilience. Thus we don't have to worry about the other list being there for the purpose of ensuring our entries are always downloaded.
 
1. It may be your ,php list as perhaps skynet cannot handle
2. Hint: keep your list clean, no comments or remarks, just the straight urls.
2. I am not sure if you realize or not but:
a. you have a lot of overlapping lists like greensnow from source and greensnow from firehol or https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt is the same as firehol's et-block.netset
b. you have a lot of old lists that you may consider removing. examples include the 2 normsheild lists, the two dshield lists (in fact you get a current copy within https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt).
I recommend to look at the firehol ip list webesite as you see for each of your lists, the last time they were updated. anyway, YMMV
As a Matter of fact, here is a quick parse of that entire file using the same method skynet does.

Code:
curl -fsL --retry 3 --connect-timeout 3 "https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list" | awk -F/ '{print $0}' | xargs "curl" -fsSL | grep -E '^(([0-9]{1,3}\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$' | grep -vE '^(0\.|10\.|100\.(6[4-9]|[7-9][0-9]|1[0-1][0-9]|12[0-7])\.|127\.|169\.254\.|172\.1[6-9]\.|172\.2[0-9]\.|172\.3[0-1]\.|192\.0\.0\.|192\.0\.2\.|192\.168\.|198\.(1[8-9])\.|198\.51\.100\.|203\.0\.113\.|2(2[4-9]|[3-4][0-9]|5[0-5])\.|8\.8\.8\.8|8\.8\.4\.4|1\.1\.1\.1|1\.0\.0\.1)' | awk '!x[$0]++' | wc -l

it generates a line count of

Code:
345178
 
In effect the list is downloaded and populated, but then somehow breaks the skynet stats

without https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

View attachment 47844

with https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

View attachment 47845

size of the downloads

View attachment 47846

Regards
Commodoro
Perhaps the entry at the end of the file is the culprit:
Code:
212.193.30.15
216.83.46.88
705ef00224f3f7b02e29f21eb6e10d02
7ce7c755fc664713a372e9ee635698da
effa0af1a4f1e1ac8023c5c147f9f569
55a46a2415d18093abcd59a0bf33d0a9
dde1f94b7b8dcd720b6952ba9d71763f
a084f7b249471a5f0d53945003b4a7c6
9bef135ad78f1cc980556008af92f385
ffa2e6f6a7a8001f56c352df43af3fe5
0baa1d0cc20d80fa47eeb764292b9e98
d69589f5bd6c3c799be2d2fd2b718af1
b7b0b7eeec44ec80f82a9bf0a99fe471898e0106a2541ba5eb5a48d7ce3a48be
c26339cc6618e4b051cb2aea4e22288e7218b0dee04c2ea1e9dd6bb51feaa950
6bd33a93372e7cc45c5cf3c040991830bee9f2be6959f4b764feb7f3873fc458
15d5605f08420bd6b2ed02d9e08885e442c3f3e0bd4423b2ca7450f593799963
bc31611b03fe427b9c70459b01d95ed5c173a481efc56aabcdfdc0c0808cc347
dd0bd63ada359ba9e0c332af732770c116fdc178d48c4a6dd55e69dc14525340
fb9fdeb110ab64a155a610ecbdbaf7cc780d1c2dd1bb3cc9e544a13a56a992e2
8f5ed5c923256e5ec1bb7c0aa691419f88d0b3c29777f0577f0ba2f2b69fb674
e887881406cf08519db115a8e1dfb4e470a9d4359c918b8d1111aad676ccdb8c
11ad2567eac856e69a0e013936830a52614caa6b3a1e2da4ca8ad08c995b72c0
 
Perhaps the entry at the end of the file is the culprit:
Code:
212.193.30.15
216.83.46.88
705ef00224f3f7b02e29f21eb6e10d02
7ce7c755fc664713a372e9ee635698da
effa0af1a4f1e1ac8023c5c147f9f569
55a46a2415d18093abcd59a0bf33d0a9
dde1f94b7b8dcd720b6952ba9d71763f
a084f7b249471a5f0d53945003b4a7c6
9bef135ad78f1cc980556008af92f385
ffa2e6f6a7a8001f56c352df43af3fe5
0baa1d0cc20d80fa47eeb764292b9e98
d69589f5bd6c3c799be2d2fd2b718af1
b7b0b7eeec44ec80f82a9bf0a99fe471898e0106a2541ba5eb5a48d7ce3a48be
c26339cc6618e4b051cb2aea4e22288e7218b0dee04c2ea1e9dd6bb51feaa950
6bd33a93372e7cc45c5cf3c040991830bee9f2be6959f4b764feb7f3873fc458
15d5605f08420bd6b2ed02d9e08885e442c3f3e0bd4423b2ca7450f593799963
bc31611b03fe427b9c70459b01d95ed5c173a481efc56aabcdfdc0c0808cc347
dd0bd63ada359ba9e0c332af732770c116fdc178d48c4a6dd55e69dc14525340
fb9fdeb110ab64a155a610ecbdbaf7cc780d1c2dd1bb3cc9e544a13a56a992e2
8f5ed5c923256e5ec1bb7c0aa691419f88d0b3c29777f0577f0ba2f2b69fb674
e887881406cf08519db115a8e1dfb4e470a9d4359c918b8d1111aad676ccdb8c
11ad2567eac856e69a0e013936830a52614caa6b3a1e2da4ca8ad08c995b72c0
It doesnt parse using skynets regex?

please see - https://www.snbforums.com/threads/is-default-firewall-good-enough.76648/post-822069 .
 
Last edited:
Screenshot 2023-02-10 alle 15.33.40.jpg


also with vim set list

Screenshot 2023-02-10 alle 15.35.02.jpg
 
Yes, this regex also takes into account what is not an IP address, let's see if tonight, family and work permitting, I'd like to try a more robust regex


View attachment 47852



Ok, I'll try it tonight and let you know
check out this one.

Code:
^(([0-9]{1,3}\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)([[:space:]]|$)

1676041991370.png


or even simpler:

Code:
^(([0-9]{1,3}\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$

1676052004061.png


After further testing and playing around-

Old line:
Code:
awk '{print $1 " " FILENAME}' -- * | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})? .*' | awk '!x[$0]++' | Filter_PrivateIP > /tmp/skynet/malwarelist.txt

Improved line:
Code:
awk 'BEGIN{RS="(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)"}{if(RT && !seen[RT]++)printf "%s %s\n", RT, FILENAME}' -- * | Filter_PrivateIP > /tmp/skynet/malwarelist.txt
 
Last edited:
check out this one.

Code:
^(([0-9]{1,3}\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)([[:space:]]|$)

View attachment 47853

or even simpler:

Code:
^(([0-9]{1,3}\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$

View attachment 47860

just for brain challenge, what do you think about this:

Code:
^([0-9]{1,3}\.){3}(?<=[0-9]{1}\.|[0-9]{2}\.|[0-9]{3}\.)[0-9]{1,3}$

Screenshot 2023-02-10 alle 19.42.54.jpg
 

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top