Skynet Is default firewall good enough?

[SomeWhereOverTheRainBow]

This part of the process is failing before ipset gets the file as input. This is just writing a text file with the ipset commands. The ranges never make it into the file. Strange, for sure.

Yea it is definitely alittle mind blowing. My working theory is that since the two failing ip addresses are not ranges, they are getting added to the ipset file before the ranges. The ranges are added at the end of the file after all normal ip addresses. So once the ip addresses are loaded the iptable halts loading the rest of the ipset because the 2 ipset are bad. Hence why none of the ranges ever get loaded because they follow after all single ip addresses in the order in which they are present in the ipset file.

https://github.com/Adamm00/IPSet_ASUS/blob/master/firewall.sh#L3481-L3484

According to the link above, skynet is using a single file to load the ipsets. Regular ip addresses are appended to the file first, then ranges found in the filters are appended second. When iptables breaks loading ipset it is still at the point of loading just the ipaddresses. It never makes it to the point of loading the ranges because the load is breaking due to the bad ip addresses from custom filter lists which explains why loading works normal if we don't use the lists with the bad addresses.

 

[dave14305]

My working theory is that since the two failing ip addresses are not ranges, they are getting added to the ipset file before the ranges. The ranges are added at the end of the file after all normal ip addresses. So once the ip addresses are loaded the iptable halts loading the rest of the ipset because the 2 ipset are bad. Hence why none of the ranges ever get loaded because they follow after all single ip addresses in the order in which they are present in the ipset file.

Are you working from a different codebase at this point? The original Skynet code fails to append the add Skynet-BlockedRanges statements to the file at all. That’s why the count is zero. They aren’t there.

 

[SomeWhereOverTheRainBow]

Are you working from a different codebase at this point? The original Skynet code fails to append the add Skynet-BlockedRanges statements to the file at all. That’s why the count is zero. They aren’t there.

From what I can tell, they are there when I don't load the list with the bad ip addresses, but you could be right as well, there may be an issue with the code being used for ASN and Country blocking. I am not testing the ranges used by those features. Only the ones being loaded by the custom filter list option.

Here is a test:

echo "192.168.1.45/32 localaddress" | grep -F "/" | awk '{printf "add Skynet-BlockedRanges %s comment \"BanMalware: %s\"\n", $1, $2 }'
add Skynet-BlockedRanges 192.168.1.45/32 comment "BanMalware: localaddress"

 

[dave14305]

From what I can tell, they are there when I don't load the list with the bad ip addresses, but you could be right as well, there may be an issue with the code being used for ASN and Country blocking. I am not testing the ranges used by those features. Only the ones being loaded by the custom filter list option.

This is likely a text processing issue, probably related to the total size of the combined lists. The bad IP isn’t a factor unless it also happens to contain some non-printable character that could interfere with the awk output.

 

[JTnola]

From what I can tell, they are there when I don't load the list with the bad ip addresses, but you could be right as well, there may be an issue with the code being used for ASN and Country blocking. I am not testing the ranges used by those features. Only the ones being loaded by the custom filter list option.

Here is a test:

echo "192.168.1.45/32 localaddress" | grep -F "/" | awk '{printf "add Skynet-BlockedRanges %s comment \"BanMalware: %s\"\n", $1, $2 }'
add Skynet-BlockedRanges 192.168.1.45/32 comment "BanMalware: localaddress"

@SomeWhereOverTheRainBow, have you adjusted your list to remove the broken/bad/problem ip addresses? Are we “good” to use your list again?

Thank you everyone for your thoughtful attention to this

 

[Phil Mcavity]

Really interesting thread. Most of which is beyond my capabilities , but a great read

 

[commodoro]

@commodoro I believe I discovered the culprit in my last test, but set -x would definitely confirm it.

I've always done little shell scripting and I'm really rusty with debugging, but I haven't noticed any errors.

What i noticed is that with the list

https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

nothing is inserted in the skynet.ipset file under the installation directory of skynet,

unlike what happens with the other lists, and if I understand correctly it is the file on which the insertion in the iptables chain is based.

This is likely a text processing issue, probably related to the total size of the combined lists. The bad IP isn’t a factor unless it also happens to contain some non-printable character that could interfere with the awk output.

I also think this behavior is not related to an incorrect number range of the IPs, most likely a strange unprintable character

 

[SomeWhereOverTheRainBow]

@SomeWhereOverTheRainBow, have you adjusted your list to remove the broken/bad/problem ip addresses? Are we “good” to use your list again?

Thank you everyone for your thoughtful attention to this

yes it is removed from

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

if you have a chance,

see if I unbroke my list as well.

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/myfilter.list

If I unbroke mine, then myfilter.list is better than the regular filter.list

 

[SomeWhereOverTheRainBow]

I've always done little shell scripting and I'm really rusty with debugging, but I haven't noticed any errors.

What i noticed is that with the list

https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

nothing is inserted in the skynet.ipset file under the installation directory of skynet,

View attachment 47918

unlike what happens with the other lists, and if I understand correctly it is the file on which the insertion in the iptables chain is based.

View attachment 47919


View attachment 47920




I also think this behavior is not related to an incorrect number range of the IPs, most likely a strange unprintable character

Commodoro, If you have time, see if my lists are now workable for you. They seem to be fine for me.

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/myfilter.list

and

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

 

[commodoro]

Commodoro, If you have time, see if my lists are now workable for you. They seem to be fine for me.

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/myfilter.list

and

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

@SomeWhereOverTheRainBow All lists work fine

 

[SomeWhereOverTheRainBow]

With my list, the only thing I changed was the two bad ip addresses, everything else is identical . It still utilizes good entries that were present in the bad list. The other filter list I removed the link to the list that contains the bad ip entries.

 

[dave14305]

I see now. I was misunderstanding the flow of the processing. I didn’t realize skynet.ipset gets overwritten after the banmalware update. I was sorely mistaken.

 

[SomeWhereOverTheRainBow]

I see now. I was misunderstanding the flow of the processing. I didn’t realize skynet.ipset gets overwritten after the banmalware update. I was sorely mistaken.

You are the one that provided the smoking gun when you said to run debugging. We were not thinking in those terms until you mentioned that. We were thinking it was an actual code issue without having ran debug first.

I wasn't sure what was happening 100 percent as well.

https://github.com/Adamm00/IPSet_ASUS/blob/master/firewall.sh#L3480

Is the line that purges the old entries from the list before adding the new ones.

 

[commodoro]

Finally the error is revealed to me, @SomeWhereOverTheRainBow you were right ipset performs a formal check on the IP

Also reading the Ipset documentation

https://ipset.netfilter.org/ipset.man.html

says: "Zero valued IP address cannot be stored in a hash:ip type of set." skynet use that type of ipset.

I did a test by manually entering the IP with a zero value and in fact it goes wrong, among other things this value is present in the list

https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

Thanks @SomeWhereOverTheRainBow an @dave14305 that drive me to the right troubleshooting.

Now a question/curiosity I saw that skynet inserts the networks of the OpenVpn servers present in the router in whitelist, but now that with the 388 version of the firmware there is also wireguard?

 

[chongnt]

Finally the error is revealed to me, @SomeWhereOverTheRainBow you were right ipset performs a formal check on the IP

View attachment 47927

Also reading the Ipset documentation

https://ipset.netfilter.org/ipset.man.html

says: "Zero valued IP address cannot be stored in a hash:ip type of set." skynet use that type of ipset.

I did a test by manually entering the IP with a zero value and in fact it goes wrong, among other things this value is present in the list

https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

View attachment 47928

View attachment 47929

Thanks @SomeWhereOverTheRainBow an @dave14305 that drive me to the right troubleshooting.

Now a question/curiosity I saw that skynet inserts the networks of the OpenVpn servers present in the router in whitelist, but now that with the 388 version of the firmware there is also wireguard?

This list is a bit dodgy with such entries:
451.91.115.161
447.96.132.96
0.0.0.0
10.0.0.8

 

[SomeWhereOverTheRainBow]

Finally the error is revealed to me, @SomeWhereOverTheRainBow you were right ipset performs a formal check on the IP

View attachment 47927

Also reading the Ipset documentation

https://ipset.netfilter.org/ipset.man.html

says: "Zero valued IP address cannot be stored in a hash:ip type of set." skynet use that type of ipset.

I did a test by manually entering the IP with a zero value and in fact it goes wrong, among other things this value is present in the list

https://threatview.io/Downloads/Experimental-IOC-Tweets.txt

View attachment 47928

View attachment 47929

Thanks @SomeWhereOverTheRainBow an @dave14305 that drive me to the right troubleshooting.

Now a question/curiosity I saw that skynet inserts the networks of the OpenVpn servers present in the router in whitelist, but now that with the 388 version of the firmware there is also wireguard?

Are you referring to the upstream wireguard server IP when using wireguard as client on the router?

 

[SomeWhereOverTheRainBow]

This list is a bit dodgy with such entries:
451.91.115.161
447.96.132.96
0.0.0.0
10.0.0.8

Yea I feel the same way about the bad entries. Fortunately skynet recognizes private addresses could appear in any list. It filters accordingly.

https://github.com/Adamm00/IPSet_ASUS/blob/master/firewall.sh#L755-L757

Imagine the heart ache you would experience when one of the much longer lists fails because of such an entry.

Unfortunately, skynet doesn't use an application like
sipcalc (or regexp grep -oE '\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\/(1?[0-9]|2?[0-9]|3?[0-2]))?\b')
to detect other (non-private) fake IP addresses like these:

451.91.115.161
447.96.132.96

which is the cause of the break in the first place.

From a first glance, no one would have suspected those were breaking things.

## Number without bad IPS
root@admin:~# curl -fsSL https://threatview.io/Downloads/Experimental-IOC-Tweets.txt  | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?' | while read -r ips; do [ "$(printf "%s\n" "${PID}" | wc -l)" -ge "$(grep -c processor /proc/cpuinfo)" ] && { wait "$PID"; unset PID; }; { ! sipcalc $ips | grep -q 'ERR' && printf "%s\n" "$ips"; } & PID="${PID} $!"; done | wc -l
334
root@admin:~# curl -fsSL https://threatview.io/Downloads/Experimental-IOC-Tweets.txt  | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?' | while read -r ips; do [ "$(printf "%s\n" "${PID}" | wc -l)" -ge "$(grep -c processor /proc/cpuinfo)" ] && { wait "$PID"; unset PID; }; { printf "%s\n" "$ips" | grep -qoE '\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\/(1?[0-9]|2?[0-9]|3?[0-2]))?\b' && printf "%s\n" "$ips"; } & PID="${PID} $!"; done | wc -l
334

## Number of bad IPS
root@admin:~# curl -fsSL https://threatview.io/Downloads/Experimental-IOC-Tweets.txt  | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?' | while read -r ips; do [ "$(printf "%s\n" "${PID}" | wc -l)" -ge "$(grep -c processor /proc/cpuinfo)" ] && { wait "$PID"; unset PID; }; { sipcalc $ips | grep -q 'ERR' && printf "%s\n" "$ips"; } & PID="${PID} $!"; done | wc -l
2
root@admin:~# curl -fsSL https://threatview.io/Downloads/Experimental-IOC-Tweets.txt  | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?' | while read -r ips; do [ "$(printf "%s\n" "${PID}" | wc -l)" -ge "$(grep -c processor /proc/cpuinfo)" ] && { wait "$PID"; unset PID; }; { ! printf "%s\n" "$ips" | grep -qoE '\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\/(1?[0-9]|2?[0-9]|3?[0-2]))?\b' && printf "%s\n" "$ips"; } & PID="${PID} $!"; done | wc -l
2

but the more complex the regex, the longer the time to process lists.

The most accurate one I have been able to devise without look a heads.
^(((0?|[1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-5][0-5])\.){3}(0?|[1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-5][0-5])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$
and
^(((25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])\.){3}(25[0-5]|(2[0-4]|1[0-9]|[1-9]|)[0-9])(\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$

Other ones will typically match Addresses like:

1.01.0.01
or
09.09.09.09

 

[JTnola]

Stats (click to expand/collapse)

IPs Banned Ranges Banned Inbound Blocks Outbound Blocks 496221 63924 2034 0

@SomeWhereOverTheRainBow that’s using your “My List”

 

[chongnt]

Recently I noticed one of the github.io address is blocked. When I look for the list, it appears this list has not been updated for over 500 days.
Can we still trust those list that are not updated for so long?

FireHOL IP Lists | IP Blacklists | IP Reputation Feeds

350+ IP blacklists, IP blocklists and IP Reputation feeds, about Cybercrime, Fraud, Botnets, Μalware, Virus, Abuse, Attacks, Open Proxies, Anonymizers. See their changes and updates, size over time, retention policy, geographic coverage, comparisons and overlaps.

iplists.firehol.org

github.io.              4000    IN      A       185.199.111.153

coinbl_hosts_browser.ipset:185.199.111.153

 

[SomeWhereOverTheRainBow]

Recently I noticed one of the github.io address is blocked. When I look for the list, it appears this list has not been updated for over 500 days.
Can we still trust those list that are not updated for so long?

FireHOL IP Lists | IP Blacklists | IP Reputation Feeds

350+ IP blacklists, IP blocklists and IP Reputation feeds, about Cybercrime, Fraud, Botnets, Μalware, Virus, Abuse, Attacks, Open Proxies, Anonymizers. See their changes and updates, size over time, retention policy, geographic coverage, comparisons and overlaps.

iplists.firehol.org

github.io.              4000    IN      A       185.199.111.153

coinbl_hosts_browser.ipset:185.199.111.153

Two questions I would ask myself in this type of situation are "do you believe there are other legitimate blocks coming from the list?" and "does any of your other list also include those entries present in this list?" Also, keep in mind the nature and purpose of this list you are referencing.

If these are IP addresses resolved from DNS hostnames, I suspect those IP addresses would not change that much depending on how often new DNS hostnames get added to that list. It may turn out that those DNS hostnames may not change very often.

You have to first ask yourself "Is that DNS hostname list still being maintain?" I suspect the answer is yes, but it doesn't change that often.