Skynet Skynet - Router Firewall & Security Enhancements

[Adamm]

Updated yesterday. getting this in logs over and over again..

an 19 17:09:10 Skynet: [%] Startup Initiated... ( skynetloc=/tmp/mnt/DATA1/skynet )
Jan 19 17:19:06 rc_service: udhcpc 20830:notify_rc start_firewall
Jan 19 17:19:06 dhcp_client: bound 76.71.182.30 via 10.11.18.65 during 600 seconds.
Jan 19 17:19:07 miniupnpd[19801]: shutting down MiniUPnPd
Jan 19 17:19:07 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
Jan 19 17:19:07 custom_script: Running /jffs/scripts/firewall-start (args: ppp0)
Jan 19 17:19:07 miniupnpd[20850]: HTTP listening on port 37849
Jan 19 17:19:07 miniupnpd[20850]: Listening for NAT-PMP/PCP traffic on port 5351
Jan 19 17:19:08 Skynet: [%] Startup Initiated... ( skynetloc=/tmp/mnt/DATA1/skynet )
Jan 19 17:19:33 Skynet: [#] 143412 IPs (+0) -- 1352 Ranges Banned (+0) || 0 Inbound -- 0 Outbound Connections Blocked! [start] [25s]
Jan 19 17:19:58 kernel: [BLOCKED - INBOUND] IN=ppp0 OUT= MAC= SRC=185.176.27.34 DST=70.55.54.122 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=58074 PROTO=TCP SPT=50038 DPT=4381 SEQ=851953085 ACK=0 WINDOW=1024 RES=0x00 SYN URGP=0
Jan 19 17:20:02 Diversion: updated ads counter: 218,503 total, 3,316 this week, 3,100 new since last count, from /opt/bin/diversion
Jan 19 17:20:18 kernel: [BLOCKED - INBOUND] IN=ppp0 OUT= MAC= SRC=172.107.96.221 DST=70.55.54.122 LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=11739 PROTO=TCP SPT=49565 DPT=1433 SEQ=3777568312 ACK=0 WINDOW=1024 RES=0x00 SYN URGP=0

Keeps
Jan 19 17:19:08 Skynet: [%] Startup Initiated... ( skynetloc=/tmp/mnt/DATA1/skynet )
over and over again..

Skynet is working fine, something is just restarting the firewall service (and Skynet in the process).

Jan 19 17:19:06 rc_service: udhcpc 20830:notify_rc start_firewall

 

[cousinit99]

Just wanted to say thanks, @Adamm ! This script is slick. I now have, dare I say, a formidable firewall that feeds directly from major threat intelligence sources on a regular basis. For a sub-$300 appliance, that's practically unheard of. Major props to you, sir!

 

[consorts]

sorry if this was asked, but i used search and could not find an answer, so here goes...
months ago i tried skynet and it made a mess of things (i'll spare you the gory details)
since then i've only used diversion(small) and stubby. tonight i added skynet once again,
coordinated it with diversion, rebooted and got the 18/18 test pass, and all seems well
EXCEPT now my router's CPU load averages >60% instead of <6%
https://i.imgur.com/F0BPZDc.jpg (default install, i didn't in/out black/whitelist modify)
so what's going on here? is there some way to reduce the stress on my router now that
skynet is running? keep in mind my network's doing practically nothing at the moment,
and it's been running flawlessly this past Month (no reboots needed or 5g radio hangs)
i'm using a plain usb2 flash, would buying a faster one put less stress on the router CPU?

RAM shows only 38% in use ( 194, 318, 512 )
Jan 20 21:29:02 avahi-daemon[29302]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!

 

[Blackbox]

Hey guys,

I am having some trouble adding IP lists to Skynet. When importing an IP list (5) and entering the web URL, it immediately bringgs me back to the amtm menu and the IP counter did not change.
Is there a way to show the blocked IPs/Whitelist/Blacklist etc in Skynet (the content of the lists) to double check if they got properly added?

Also, when trying to add my own lists of IPs is there a way to directly copy them into the IP list that Skynet uses? I cannot find the "Skynet" folder when connecting on a Mac (Cmd+K) to the USB Drive on the Router (with loginname@RouterIP).
Is there a way to directly access the folder/lists that Skynet uses from a Mac?
(I do find the lists that Diversion uses, but there simply is no folder named "Skynet")

Thanks a lot!

 

[Adamm]

sorry if this was asked, but i used search and could not find an answer, so here goes...
months ago i tried skynet and it made a mess of things (i'll spare you the gory details)
since then i've only used diversion(small) and stubby. tonight i added skynet once again,
coordinated it with diversion, rebooted and got the 18/18 test pass, and all seems well
EXCEPT now my router's CPU load averages >60% instead of <6%
https://i.imgur.com/F0BPZDc.jpg (default install, i didn't in/out black/whitelist modify)
so what's going on here? is there some way to reduce the stress on my router now that
skynet is running? keep in mind my network's doing practically nothing at the moment,
and it's been running flawlessly this past Month (no reboots needed or 5g radio hangs)
i'm using a plain usb2 flash, would buying a faster one put less stress on the router CPU?

RAM shows only 38% in use ( 194, 318, 512 )
Jan 20 21:29:02 avahi-daemon[29302]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!

The CPU graph by its-self doesn't mean much, it could be any process using your CPU. Skynet at most has a spike in CPU usage for 20s when the banmalware command is run, after that the performance impact is impossible to measure. I suggest using a tool like htop from entware to see which process is actually causing issues (fyi the avahi error is unrelated)

I am having some trouble adding IP lists to Skynet. When importing an IP list (5) and entering the web URL, it immediately bringgs me back to the amtm menu and the IP counter did not change.

Scroll up in your terminal to see the reason Skynet didn't import the list, most likely it was formatted incorrectly.

Also, when trying to add my own lists of IPs is there a way to directly copy them into the IP list that Skynet uses? I cannot find the "Skynet" folder when connecting on a Mac (Cmd+K) to the USB Drive on the Router (with loginname@RouterIP).

If you are referring to adding your own list to banmalware, that will require you to host your own custom filter list in the same format as the default one

 

[consorts]

Skynet at most has a spike in CPU usage for 20s when the banmalware command is run, after that the performance impact is impossible to measure. I suggest using a tool like htop from entware to see which process is actually causing issues

thanks for reassuring me that skynet alone should not be contributing this much utilization, and i can appreciate the troubleshooting approach of running the linux form of task utilization display in order to better illustrate what is taking up so much of the cpu's time. unfortunately i'm not familiar with linux and use putty merely to implement install/uninstall commands as specified by the app developer. so overnight i uninstalled skynet, diversion, stubby, then reinstalled diversion and stubby and after their required reboots - i am happy to report that my router is back to it's former <6% cpu utilization pattern. (i must compliment merlin's current stability on being able to withstand all these changes without requiring it be reinstalled as well)

the symptomatology of my obvious conflict with skynet was not merely the cpu meter going so active when the lan itself was doing nothing, i also notice my wan bandwidth was decreased from 100mbps to 85mbps and latency increased from 2ms to 4ms for hops between my wired 1gig pc and the isp's mnt (first hop pon beyond my ont), and as expected the router cpu temperature reading noticeably increased. these observations persisted for 18 hours while skynet was still installed, and despite various equipment reboot cycles, so this was not a one time thing.

i'm sure it would be interesting to diagnose this and narrow down what is causing it, but it's pretty obvious it has something having to do with skynet's install and or operation procedures or coexistence with these two other popular apps on my popular router, so i will bide my time waiting for others more knowledgeable here to notice a similar issue, then attempt whatever solutions they find of merit. till then i suppose i'll have to be satisfied with the level of ad blocking and firewall protection already afforded to me by aiprotection and diversion(small file).

 

[Skeptical.me]

I just updated to 6.6.6. and just before I did, I noticed the following

IPTables Rules                      | [Failed]

Is this a significant problem?

 

[Adamm]

thanks for reassuring me that skynet alone should not be contributing this much utilization, and i can appreciate the troubleshooting approach of running the linux form of task utilization display in order to better illustrate what is taking up so much of the cpu's time.

If you post the snippet here of the log we can identify pretty fast what is causing CPU usage (assuming the issue is still apparent)

I just updated to 6.6.6. and just before I did, I noticed the following

If you still see that after updating (and when there's no lock file), then yes it is an issue as something is removing Skynet's IPTables rules.

 

[Skeptical.me]

If you still see that after updating (and when there's no lock file), then yes it is an issue as something is removing Skynet's IPTables rules.

Do yu think a reinstallation would be the best way to fix the issue? (that is since I'm not a Command Line wiz )

 

[Adamm]

Do yu think a reinstallation would be the best way to fix the issue? (that is since I'm not a Command Line wiz )

That would be the quickest way as Skynet would re-do everything, but we should be able to track it down via other means.

 

[Skeptical.me]

That would be the quickest way as Skynet would re-do everything, but we should be able to track it down via other means.

Okay, if you've got time to direct me I'm all ears. I have added a number of things like blocking country IP ranges, and specific servers etc, so fixing it would be preferable, but only if yu have time.

 

[dave14305]

thanks for reassuring me that skynet alone should not be contributing this much utilization, and i can appreciate the troubleshooting approach of running the linux form of task utilization display in order to better illustrate what is taking up so much of the cpu's time. unfortunately i'm not familiar with linux and use putty merely to implement install/uninstall commands as specified by the app developer. so overnight i uninstalled skynet, diversion, stubby, then reinstalled diversion and stubby and after their required reboots - i am happy to report that my router is back to it's former <6% cpu utilization pattern. (i must compliment merlin's current stability on being able to withstand all these changes without requiring it be reinstalled as well)

the symptomatology of my obvious conflict with skynet was not merely the cpu meter going so active when the lan itself was doing nothing, i also notice my wan bandwidth was decreased from 100mbps to 85mbps and latency increased from 2ms to 4ms for hops between my wired 1gig pc and the isp's mnt (first hop pon beyond my ont), and as expected the router cpu temperature reading noticeably increased. these observations persisted for 18 hours while skynet was still installed, and despite various equipment reboot cycles, so this was not a one time thing.

i'm sure it would be interesting to diagnose this and narrow down what is causing it, but it's pretty obvious it has something having to do with skynet's install and or operation procedures or coexistence with these two other popular apps on my popular router, so i will bide my time waiting for others more knowledgeable here to notice a similar issue, then attempt whatever solutions they find of merit. till then i suppose i'll have to be satisfied with the level of ad blocking and firewall protection already afforded to me by aiprotection and diversion(small file).

Not too long ago someone reported CPU spikes and it was related to Diversion's attempt to sort the .ash_history file, which may have been corrupt. You would really need to run "top" in ssh (or htop from entware) to see what's going on during the CPU spikes.

 

[Adamm]

Okay, if you've got time to direct me I'm all ears. I have added a number of things like blocking country IP ranges, and specific servers etc, so fixing it would be preferable, but only if yu have time.

Restart Skynet and wait about 60s for it to startup;

sh /jffs/scripts/firewall restart

Then check debug info;

sh /jffs/scripts/firewall debug info

Assuming its just the same error again, post the output of;

sh /jffs/scripts/firewall save

And anything relevant in syslog

 

[Skeptical.me]

Restart Skynet and wait about 60s for it to startup;

sh /jffs/scripts/firewall restart

Then check debug info;

sh /jffs/scripts/firewall debug info

Assuming its just the same error again, post the output of;

sh /jffs/scripts/firewall save

And anything relevant in syslog

I ran those commands and I have no idea why, but its working again. No error is showing in Skynet. ¯\_(ツ)_/¯

 

[consorts]

Not too long ago someone reported CPU spikes and it was related to Diversion's attempt to sort the .ash_history file, which may have been corrupt.

you mean clvk07 ?

https://www.snbforums.com/threads/diversion-the-router-ad-blocker.48538/page-89#post-457475

everything with diversion was fine till i added skynet, so i don't know if this is relevant, but thanks

 

[Maverickcdn]

Is there a way to keep Skynet from using syslog?

I have my syslog uploading to a syslog server through the GUI and it is now flooded with the logs of Skynet, I know Skynet self manages the syslog file on the router itself but is there a way to have it save this temp data somewhere else, or anyone know a way to keep from sending them to my syslog server? nearly 4000+ posts and searching for log I never saw the answer I was looking for...

Thanks for the great tool btw @Adamm!

EDIT:
Thought I had the answer using filtering on my server, but the QNAP syslog server is pretty poor and can't change the 'then' option, defaulted to send to the log which is pretty dumb imho.

EDIT 2: And later in the evening....

Realized QNAPs server is perty to look at, but zero customization (basic syslog not syslog-ng), will look into getting a rsyslog server running and I can do my own filtering

 

[cmkelley]

@Adamm - so I was thinking (which is a bad sign) about my 'sed' edits to your firewall script to help skynet play nice with syslog-ng. The question occurred to me - by inhibiting skynet from deleting lines from syslog.log, am I messing up skynet's statistics anyways? Is it depending on those lines being deleted so it only gets the "new" lines? Diff file attached for context of what I'm changing.

Maybe I should change my seds to point to /opt/var/log/messages instead of only removing /tmp/syslog.log and log-1? Either that or figure out how to have syslog-ng send them to the correct file and obviate all the seds.

 

[Blackbox]

The CPU graph by its-self doesn't mean much, it could be any process using your CPU. Skynet at most has a spike in CPU usage for 20s when the banmalware command is run, after that the performance impact is impossible to measure. I suggest using a tool like htop from entware to see which process is actually causing issues (fyi the avahi error is unrelated)

Scroll up in your terminal to see the reason Skynet didn't import the list, most likely it was formatted incorrectly.

If you are referring to adding your own list to banmalware, that will require you to host your own custom filter list in the same format as the default one

Thanks a lot for your help Adamm.

Scroll up in your terminal to see the reason Skynet didn't import the list, most likely it was formatted incorrectly.

There are actually no messages when I scroll up. I basically add an IP list and suddenly Skynet brings me back to the amtm menu, not the Skynet main menu. Just for ca 0.5 secs there appears "This Function Extracts All IPs and adds them to the Blacklist", which is way to short to read, just via screenshot in the right moment.
So it is not clear to me if the list has been added. Is there a function to list which lists (URLs) have been added already, like in Diversion to have an overview?
These are the IP lists I am trying to add (mainly to block Win10 Telemetry IPs) Would these work or are they in the wrong format?
https://www.encrypt-the-planet.com/downloads/Microsoft-ip-blocklist.p2p
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/p2p/extra.txt
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/p2p/spy.txt

If you are referring to adding your own list to banmalware, that will require you to host your own custom filter list in the same format as the default one

I rather was looking where I can find the Skynet folder on the USB drive when connecting to it on the Router?
I just cannot see any folder with saved lists/IPs from Skynet. Could you please point me in the right direction where to find these as Skynet is supposed to create a folder with all its content in there but I simply cannot find it.
My thinking was if I want to add a very long own list with IP ranges to copy them into one of the lists that Skynet uses and save it in a folder on the USB drive.
(For example Diversion has all the Blacklists, Whitelists in the folder share/diversion/list and I can directly add hundreds of domains just by copy paste and after that just process the list in Diversion.)

Or how would I do that?

 

[Adamm]

@Adamm - so I was thinking (which is a bad sign) about my 'sed' edits to your firewall script to help skynet play nice with syslog-ng. The question occurred to me - by inhibiting skynet from deleting lines from syslog.log, am I messing up skynet's statistics anyways? Is it depending on those lines being deleted so it only gets the "new" lines? Diff file attached for context of what I'm changing.

Maybe I should change my seds to point to /opt/var/log/messages instead of only removing /tmp/syslog.log and log-1? Either that or figure out how to have syslog-ng send them to the correct file and obviate all the seds.

I'll look into adding a special configuration option to specify a custom syslog/syslog-1 location (Skynet only edits syslog to collect its own logs essentially). In my head that should resolve the issue or atleast give you a possible way of dealing with it rather then editing the script every time it updates.

 

[Adamm]

These are the IP lists I am trying to add (mainly to block Win10 Telemetry IPs) Would these work or are they in the wrong format?

Skynet is working correctly, I guess you missed the error;

Router Model; RT-AX88U
Skynet Version; v6.6.6 (13/01/2019) (7b9be528df4c0df260c44f6357aec36b)
iptables v1.4.15 - (ppp0 @ 192.168.1.1)
ipset v6.32, protocol version: 6
FW Version; 384.9_alpha2-g665419b4b (Jan 14 2019) (4.1.51)
Install Dir; /tmp/mnt/Elements/skynet (1022.1G / 1.7T Space Available)
SWAP File; /tmp/mnt/Elements/myswap.swp (512.0M)
Boot Args; /jffs/scripts/firewall start skynetloc=/tmp/mnt/Elements/skynet

143805 IPs (+0) -- 1601 Ranges Banned (+0) || 358 Inbound -- 0 Outbound Connections Blocked!

Select Menu Option:
[1]  --> Unban
[2]  --> Ban
[3]  --> Banmalware
[4]  --> Whitelist
[5]  --> Import IP List
[6]  --> Deport IP List
[7]  --> Save
[8]  --> Restart Skynet
[9]  --> Temporarily Disable Skynet
[10] --> Update Skynet
[11] --> Settings
[12] --> Debug Options
[13] --> Stats
[14] --> Install Skynet
[15] --> Uninstall

[r]  --> Reload Menu
[e]  --> Exit Menu

[1-15]: 5

Select Where To Import List:
[1]  --> Blacklist
[2]  --> Whitelist

[1-2]: 1

Input URL/Local File To Import:

[File]: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/p2p/spy.txt

[$] /opt/bin/firewall import blacklist https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/p2p/spy.txt


=============================================================================================================


[i] This Function Extracts All IPs And Adds Them ALL To Blacklist
[i] Remote Custom List Detected: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/p2p/spy.txt
[*] 404 Error Detected - Stopping Import

These lists are all formatted incorrectly so they fail to import, you need to have 1 IP per line, use one of the default lists as an example;

https://iplists.firehol.org/files/alienvault_reputation.ipset

I believe the telemetry lists you are referring to are formatted correctly elsewhere on the repo in question;

https://github.com/crazy-max/WindowsSpyBlocker/tree/master/data/firewall

I rather was looking where I can find the Skynet folder on the USB drive when connecting to it on the Router?
I just cannot see any folder with saved lists/IPs from Skynet. Could you please point me in the right direction where to find these as Skynet is supposed to create a folder with all its content in there but I simply cannot find it.
My thinking was if I want to add a very long own list with IP ranges to copy them into one of the lists that Skynet uses and save it in a folder on the USB drive.
(For example Diversion has all the Blacklists, Whitelists in the folder share/diversion/list and I can directly add hundreds of domains just by copy paste and after that just process the list in Diversion.)

The functionality is implemented in another way, we don't store raw data locally. Skynet uses the IPSet comment extension to identify all its entries and you can view them using the respective commands. You can either import them as mentioned above in a one time hit, or you can implement your own "banmalware filter" so they are auto renewed using the default filter list as a reference to create your own.

I understand the documentation could be improved (it's a one man show), but all the things you listed are possible if you read through the possible commands and menu items listed here.