Skynet Skynet - Router Firewall & Security Enhancements

[Bamsefar]

I guess I am to old then, since re-install from scratch has proven to be the best solution in all the time I worked with Linux and Windows.

 

[Adamm]

I guess I am to old then, since re-install from scratch has proven to be the best solution in all the time I worked with Linux and Windows.

In most cases I’d agree with you, Skynet just stores so little unnessesry data (and refreshes stale data) which makes the process unnessesary if you wish to keep the same blacklist entries.

 

[Bamsefar]

Well that assumes that the backup of the installation is 100% correct and that no errors are part of that specific backup that one might restore. Keep in mind, every week or so you already reload all information from all sites (URL/Countries and so on) - so what is the difference?

 

[Adamm]

Keep in mind, every week or so you already reload all information from all sites (URL/Countries and so on) - so what is the difference?

Here’s an example scenario. Someone imports a custom list with 10,000 entries.

With your suggestion, Skynet would have to scan the blacklist for these entries then generate 10,000 individual commands for you to manually enter.

With the current implementation we do all this with 1 command.

I know which method I’d pick

 

[Bamsefar]

So the fact is, you don't save which lists are imported. Just what's inside each list.

 

[Bamsefar]

We will most likely never agree - so let's agree that we do not agree - I know what I like, you prefer it differently.

I know from my experience that rebuild, from scratch to full blown with the commands / script needed is superior to any backup, since you will need to validate the backup on a regular base (never mind if the backup in it self contains errors and all that). The install commands and scripts can be saved very easily, and emailed. Most likly most people do no enter 10.000 entries via the AMTM menu driven system, nor with 10.000 commands. They will provide a URL or a file location where a file is located maybe with 100.000 entries.

You like backups - I prefer the old hard cold long way (to reproduce problems).

 

[wbartels]

Hello Addam,

On the first page of this Skynet post and on the Github site I read:

This script will always be open source and free to use, but if you want to support future development you can do so by Donating With PayPal.

But I don't see any license, is there any?

I'm working on a proof of concept based on Skynet and want to respect the license.
If you like the idea it can be used to back port.
I'm not skilled enough to implement it in the full package.
But with the base functions like Load_IPTables, Unload_LogIPTables, etc... I think I will get it working.

The idea is to make use off curl --time-cond (If-Modified-Since header) and only download an ipset if it has changed.
Most websites including https://iplists.firehol.org support this feature.
Then it will only add or update (add temp set, swap and destroy temp set) required ipset's.
With the ipset swap feature Skynet will keep working without any downtime!
This way you can for example check every 30 minutes and only download and update when needed.

I have the first download step ready and working on the rest.
I hope you like the idea.

Willem

 

[Adamm]

Hello Addam,

On the first page of this Skynet post and on the Github site I read:

But I don't see any license, is there any?

I'm working on a proof of concept based on Skynet and want to respect the license.
If you like the idea it can be used to back port.
I'm not skilled enough to implement it in the full package.
But with the base functions like Load_IPTables, Unload_LogIPTables, etc... I think I will get it working.

The idea is to make use off curl --time-cond (If-Modified-Since header) and only download an ipset if it has changed.
Most websites including https://iplists.firehol.org support this feature.
Then it will only add or update (add temp set, swap and destroy temp set) required ipset's.
With the ipset swap feature Skynet will keep working without any downtime!
This way you can for example check every 30 minutes and only download and update when needed.

I have the first download step ready and working on the rest.
I hope you like the idea.

Willem

I never listed an official licence, but I kind of like the sounds of this one

https://dbad-license.org/

As for modifications, feel free to submit a pull request / POC and we can work from there. Do note though that during the list update process we also do various other things like refreshing whitelists etc. The owner of firehol also requested people update their lists no sooner then 12 hour intervals so keep that in mind.

 

[wbartels]

I never listed an official licence, but I kind of like the sounds of this one

https://dbad-license.org/

As for modifications, feel free to submit a pull request / POC and we can work from there. Do note though that during the list update process we also do various other things like refreshing whitelists etc. The owner of firehol also requested people update their lists no sooner then 12 hour intervals so keep that in mind.

That is a good one
I will come back if I have a working version.

The owner of firehol also requested people update their lists no sooner then 12 hour intervals so keep that in mind.

I will keep that in mind. The overhead with a "If-Modified-Since" request is very low.
When a request is done with "If-Modified-Since" and there has nothing changes the server only send a "304 Not Modified" header without any content.

 

[Adamm]

That is a good one
I will come back if I have a working version.


I will keep that in mind. The overhead with a "If-Modified-Since" request is very low.
When a request is done with "If-Modified-Since" and there has nothing changes the server only send a "304 Not Modified" header without any content.

So I had a little free time today and implemented your (-z, --time-cond <time> Transfer based on a time condition) suggestion locally. In the process I happened to stumble across an extremely handy feature added to curl in September (-Z, --parallel Perform transfers in parallel) which has netted some great performance gains along with simplifying a hacky method we used to replicate this previously missing functionality.

Watch this space over the next few days

 

[wbartels]

So I had a little free time today and implemented your (-z, --time-cond <time> Transfer based on a time condition) suggestion locally. In the process I happened to stumble across an extremely handy feature added to curl in September (-Z, --parallel Perform transfers in parallel) which has netted some great performance gains along with simplifying a hacky method we used to replicate this previously missing functionality.

Watch this space over the next few days

WOW! That is very cool
I'm looking forward

 

[wbartels]

Here is the code I came up with so far.
The download part is working, but the rest is just a brainstorm!

Please be gentle with me, I'm new with Shell programing
And I can't really overview the full code of Skynet.

Load_IPSet () {
    md5=$1
    name=$2
    if ! ipset -L -n "Skynet-Master" >/dev/null 2>&1; then
        :
        # create Skynet-Master list:set size 64
    fi

    if ipset -L -n "$md5" >/dev/null 2>&1; then
        # SWAP SET, HASHSIZE NEED TO BE CALCULATED FROM IPSET:
        # create $md5-temp hash:net family inet hashsize 2048 comment
        # add $md5-temp 66.70.180.127 comment "Banalware: $name"
        # ...
        # ipset swap $md5 $md5-temp
        # ipset destroy $md5-temp
        logger -st Skynet "[i] Swap  00:00:00 $md5 $name"
    else
        # ADD SET, HASHSIZE NEED TO BE CALCULATED FROM IPSET:
        # create $md5 hash:net family inet hashsize 2048 comment
        # add $md5 66.70.180.127 comment "Banalware: $name"
        # ...
        # add Skynet-Master $md5
        logger -st Skynet "[i] Add   00:00:00 $md5 $name"
    fi
}


File_IDLE () {
    file=$1
    sec=$(($(date +%s) - $(date +%s -r $file)))
    printf '%02d:%02d:%02d' $(($sec/3600)) $(($sec%3600/60)) $(($sec%60))
}


download () {
    url=$1
    md5="$(echo -n "$url" | md5sum | awk '{print $1}')"
    name="${url##*/}"
    file="$cache$md5"
    temp="/tmp/wall/curl.tmp"
    if [ ! -f "$file" ] && http_code="$(curl $url -sfL --retry 3 --output "$temp" --write-out %{http_code})" && [ "$http_code" = "200" ]; then
        mv "$temp" "$file" # download to temp and move afterward to prevent a half downloaded file
        Load_IPSet "$md5" "$name"
    elif [ ! -f "$file" ]; then
        logger -st Skynet "[*] Download failed $http_code: $url"
    elif http_code="$(curl -sfL --retry 3 $url --output "$temp" --time-cond "$cache$md5" --write-out %{http_code})" && [ $http_code = "200" ]; then
        mv "$temp" "$file" # download to temp and move afterward to prevent a half downloaded file
        Load_IPSet "$md5" "$name"
    elif [ "$http_code" = "304" ] && ! ipset -L -n "$md5" >/dev/null 2>&1; then
        Load_IPSet "$md5" "$name"
    elif [ "$http_code" = "304" ]; then
        logger -st Skynet "[i] Fresh $(File_IDLE $file) $md5 $name"
    else
        logger -st Skynet "[*] Download failed $http_code: $url"
    fi
}


download 'https://iplists.firehol.org/files/bds_atif.ipset'
download 'https://iplists.firehol.org/files/blocklist_de.ipset'
download 'https://iplists.firehol.org/files/cleantalk_1d.ipset'

 

[Adamm]

I've pushed v7.0.0

Add New Logo
Fix Spacing Issues
Add ASN Whitelisting
Improve Comment Based Whitelist Removal
Cache Malware Blacklists Locally ($sknetloc/lists)
Only Download Malware List If Newer Timestamp (thanks @wbartels for the suggestion)
Replace Parellel Downloads With Native curl Functionality (-Z)
Significicantly Increase Reliability Of Malware List Download/Processing
Significiantly Improve "stats search malware" Lookup Time

Also for anyone keeping track, this is our 1000th Github commit, so quite fitting we celebrate the milestone with a major version change (v6 came out on Mar 19, 2018)

 

[skeal]

Congratulations @Adamm, nice work!

 

[skeal]

Very fast update process!!!

 

[skeal]

My logo doesn't look anything like yours. Mine is rendered like this in putty.

 

[Adamm]

My logo doesn't look anything like yours. Mine is rendered like this in putty.View attachment 20363

Looks like putty doesn't use UTF-8 by default, this setting should fix it.

https://thegreyblog.blogspot.com/2009/08/configuring-putty-to-use-utf-8.html

 

[WuTang LAN]

Great work @Adamm

 

[skeal]

Looks like putty doesn't use UTF-8 by default, this setting should fix it.

https://thegreyblog.blogspot.com/2009/08/configuring-putty-to-use-utf-8.html

My putty was already set to UTF-8, still the same issue sir.

 

[Adamm]

My putty was already set to UTF-8, still the same issue sir.

Weird, seems like a Mac only issue. Try out some of the suggestions in this thread;

https://serverfault.com/questions/475925/how-to-fix-putty-showing-garbled-characters