Skynet Skynet keeps failing to start

[jorgsmash]

Hey guys. I just noticed that Skynet seems to be failing to start. The tab doesn't show up in the GUI, and logging in via SSH and opening the skynet script, it shows a status of failed. I've rebooted the router about 10 times in the past few days (just moved, setting up the network), and I've updated skynet and router firmware.

Any ideas?

All help is appreciated, thanks.

 

[jjones7791]

Just a suggestion and not sure how much this will help, but if you running your scripts and swap on a flash drive, move to a SSD.

I made that mistake on my initial setup, and it caused all sorts of unexpected and odd behavior with Diversion, but I'm sure it would impact Skynet as well. Example, first pixelserv wouldn't start, even though Diversion was running fine, then Diversion just disappeared altogether, but that was when my flash drive completely died.

 

[Viktor Jaep]

Hey guys. I just noticed that Skynet seems to be failing to start. The tab doesn't show up in the GUI, and logging in via SSH and opening the skynet script, it shows a status of failed. I've rebooted the router about 10 times in the past few days (just moved, setting up the network), and I've updated skynet and router firmware.

View attachment 49989


View attachment 49990

Any ideas?

All help is appreciated, thanks.

Have you tried uninstalling / reboot / reinstall?

 

[jorgsmash]

Just a suggestion and not sure how much this will help, but if you running your scripts and swap on a flash drive, move to a SSD.

I made that mistake on my initial setup, and it caused all sorts of unexpected and odd behavior with Diversion, but I'm sure it would impact Skynet as well. Example, first pixelserv wouldn't start, even though Diversion was running fine, then Diversion just disappeared altogether, but that was when my flash drive completely died.

I have it running on a spinning HDD. But it's been running that way for a few years. And the drive is still good.

Have you tried uninstalling / reboot / reinstall?

I haven't because I don't know exactly what all I have configured and don't want to lose settings. Can you help me figure out what IPtables rules I have? I think I had to do a special rule a long time ago for something related to VPN clients.

 

[Viktor Jaep]

I haven't because I don't know exactly what all I have configured and don't want to lose settings. Can you help me figure out what IPtables rules I have? I think I had to do a special rule a long time ago for something related to VPN clients.

Uninstalling Skynet would only remove the rules that it was taking care of at the time. If you are using your own custom rules, they would need to get added each time your router reboots, so you must have a script out there adding those in each time?

 

[jorgsmash]

Uninstalling Skynet would only remove the rules that it was taking care of at the time. If you are using your own custom rules, they would need to get added each time your router reboots, so you must have a script out there adding those in each time?

I do believe they are being added by a script. It was a while ago, probably a year or two that I did this, and I don't fool with IPtables very often, so I wouldn't remember where specifically the script runs. I know there is an IPtables "save" script that can be installed on Linux flavors like Ubuntu that I've used before, but I doubt that is supported here.

 

[Viktor Jaep]

I do believe they are being added by a script. It was a while ago, probably a year or two that I did this, and I don't fool with IPtables very often, so I wouldn't remember where specifically the script runs. I know there is an IPtables "save" script that can be installed on Linux flavors like Ubuntu that I've used before, but I doubt that is supported here.

You might want to take a look under /jffs/scripts/firewall-start?

 

[jorgsmash]

You might want to take a look under /jffs/scripts/firewall-start?

Ah. Yes there is some stuff here

Looks like I made a comment for where I got the settings from https://www.snbforums.com/threads/openvpn-server-and-client-question.38378/page-2 - Post 39

I'll have to go back and read up on that to figure out what issue I was trying to solve. I no longer use flexqos, or diversion so I guess I could comment those out.

I had an issue with YazFi which I need to create a post for.

So uninstalling and reinstalling skynet would not harm these rules?

 

[SomeWhereOverTheRainBow]

Hey guys. I just noticed that Skynet seems to be failing to start. The tab doesn't show up in the GUI, and logging in via SSH and opening the skynet script, it shows a status of failed. I've rebooted the router about 10 times in the past few days (just moved, setting up the network), and I've updated skynet and router firmware.

View attachment 49989


View attachment 49990

Any ideas?

All help is appreciated, thanks.

It might be a result of you trying to use too big of a blocklist. Or your AI protect is crashing.

I have an active pull request open that fixes a subnet bug with skynet ipset as well.

But it looks like what @Viktor Jaep suggests fixed it.

 

[Viktor Jaep]

Ah. Yes there is some stuff here

View attachment 49998

Looks like I made a comment for where I got the settings from https://www.snbforums.com/threads/openvpn-server-and-client-question.38378/page-2 - Post 39

I'll have to go back and read up on that to figure out what issue I was trying to solve. I no longer use flexqos, or diversion so I guess I could comment those out.

I had an issue with YazFi which I need to create a post for.

So uninstalling and reinstalling skynet would not harm these rules?

Exactly... it won't harm your rules... these will stay in place even after uninstalling/reinstalling Skynet.

 

[jorgsmash]

It might be a result of you trying to use too big of a blocklist. Or your AI protect is crashing.

I have an active pull request open that fixes a subnet bug with skynet ipset as well.

But it looks like what @Viktor Jaep suggests fixed it.

Did you have this issue too? I'm confused what you mean by "it looks like what @Viktor Jaep suggests fixed it." because I haven't done it yet. But I will give it a shot. Any help on taking a look at my blocklist? I don't think I did a ton of configuring of skynet, I just installed and let it ride.

What does skynet actually do if I have no open ports on my WAN? I see blocked connections from public IPs in the system log, but what are they trying to connect to? I don't have any ports open besides the OpenVPN udp port which (to my knowledge) doesn't respond as open unless you have the correct certificate during authentication. Or at least that is what I have seen with running port scans and vuln scans against my IP from a host not connected to the router.

 

[Viktor Jaep]

Did you have this issue too? I'm confused what you mean by "it looks like what @Viktor Jaep suggests fixed it." because I haven't done it yet. But I will give it a shot. Any help on taking a look at my blocklist? I don't think I did a ton of configuring of skynet, I just installed and let it ride.

I think he meant "that would probably fix it"...

What does skynet actually do if I have no open ports on my WAN? I see blocked connections from public IPs in the system log, but what are they trying to connect to? I don't have any ports open besides the OpenVPN udp port which (to my knowledge) doesn't respond as open unless you have the correct certificate during authentication. Or at least that is what I have seen with running port scans and vuln scans against my IP from a host not connected to the router.

Skynet will present you with a log of blocked connections that would match whatever you're currently blocking in its blocklist. Since you have no open ports, your firewall would have been blocking these anyways, but this just gives you some nice visibility, and a nice web interface to boot. The nice thing about Skynet is that it also allows for country blocking as well as outbound blocking. So if you're blocking North Korea, and you get infected with malware internally that's trying to reach out to its command & control servers, Skynet would block that traffic in its tracks. Also, there's a plethora of great blocklists out there, and @SomeWhereOverTheRainBow maintains a great one -- highly recommended!

 

[jorgsmash]

I think he meant "that would probably fix it"...


Skynet will present you with a log of blocked connections that would match whatever you're currently blocking in its blocklist. Since you have no open ports, your firewall would have been blocking these anyways, but this just gives you some nice visibility, and a nice web interface to boot. The nice thing about Skynet is that it also allows for country blocking as well as outbound blocking. So if you're blocking North Korea, and you get infected with malware internally that's trying to reach out to its command & control servers, Skynet would block that traffic in its tracks. Also, there's a plethora of great blocklists out there, and @SomeWhereOverTheRainBow maintains a great one -- highly recommended!

Awesome! I would definitely be interested in tweaking all this. I'll do some googling. I'll report back after I uninstall and reboot, reinstall, reboot.

 

[SomeWhereOverTheRainBow]

Did you have this issue too? I'm confused what you mean by "it looks like what @Viktor Jaep suggests fixed it." because I haven't done it yet. But I will give it a shot. Any help on taking a look at my blocklist? I don't think I did a ton of configuring of skynet, I just installed and let it ride.

What does skynet actually do if I have no open ports on my WAN? I see blocked connections from public IPs in the system log, but what are they trying to connect to? I don't have any ports open besides the OpenVPN udp port which (to my knowledge) doesn't respond as open unless you have the correct certificate during authentication. Or at least that is what I have seen with running port scans and vuln scans against my IP from a host not connected to the router.

I ran into a similar issue with skynet, the first issue presented itself when networks with subnets /32 were filling up the skynet block ranges list, because skynet had no logic to filter out IP addresses with /32. The second time I experienced it was with a Failing AI Protect bug present on a newer model router. The third time I experience it was with a Massive single IP blocklist (i.e. -no ranges).

 

[SomeWhereOverTheRainBow]

Awesome! I would definitely be interested in tweaking all this. I'll do some googling. I'll report back after I uninstall and reboot, reinstall, reboot.

here is with my current blocklist

 

[SomeWhereOverTheRainBow]

@Viktor Jaep You were right, I do maintain an awesome list

here is the analysis using iprange

326293 printed CIDRs, break down by prefix:
        - prefix /3 counts 1 entries
        - prefix /8 counts 3 entries
        - prefix /9 counts 1 entries
        - prefix /10 counts 16 entries
        - prefix /11 counts 31 entries
        - prefix /12 counts 76 entries
        - prefix /13 counts 122 entries
        - prefix /14 counts 196 entries
        - prefix /15 counts 322 entries
        - prefix /16 counts 929 entries
        - prefix /17 counts 629 entries
        - prefix /18 counts 949 entries
        - prefix /19 counts 1508 entries
        - prefix /20 counts 2028 entries
        - prefix /21 counts 2481 entries
        - prefix /22 counts 5771 entries
        - prefix /23 counts 5314 entries
        - prefix /24 counts 13500 entries
        - prefix /25 counts 424 entries
        - prefix /26 counts 560 entries
        - prefix /27 counts 583 entries
        - prefix /28 counts 798 entries
        - prefix /29 counts 1285 entries
        - prefix /30 counts 2631 entries
        - prefix /31 counts 9815 entries
        - prefix /32 counts 276320 entries

totals: 326293 lines read, 302935 distinct IP ranges found, 26 CIDR prefixes, 326293 CIDRs printed, 1100260370 unique IPs
completed in 13.67130 seconds (read 0.22549 + think 0.22359 + speak 13.22222)

 

[SomeWhereOverTheRainBow]

lets say I reduce it by 100,000 entries using iprange

326293 printed CIDRs, break down by prefix:
        - prefix /3 counts 1 entries
        - prefix /8 counts 3 entries
        - prefix /9 counts 1 entries
        - prefix /10 counts 16 entries
        - prefix /11 counts 31 entries
        - prefix /12 counts 76 entries
        - prefix /13 counts 122 entries
        - prefix /14 counts 196 entries
        - prefix /15 counts 322 entries
        - prefix /16 counts 929 entries
        - prefix /17 counts 629 entries
        - prefix /18 counts 949 entries
        - prefix /19 counts 1508 entries
        - prefix /20 counts 2028 entries
        - prefix /21 counts 2481 entries
        - prefix /22 counts 5771 entries
        - prefix /23 counts 5314 entries
        - prefix /24 counts 13500 entries
        - prefix /25 counts 424 entries
        - prefix /26 counts 560 entries
        - prefix /27 counts 583 entries
        - prefix /28 counts 798 entries
        - prefix /29 counts 1285 entries
        - prefix /30 counts 2631 entries
        - prefix /31 counts 9815 entries
        - prefix /32 counts 276320 entries

totals: 326293 lines read, 302935 distinct IP ranges found, 26 CIDR prefixes, 326293 CIDRs printed, 1100260370 unique IPs
completed in 12.08756 seconds (read 0.22889 + think 0.21960 + speak 11.63906)

and here is if I reduce it by 1 million

742955 printed CIDRs, break down by prefix:
        - prefix /16 counts 15653 entries
        - prefix /24 counts 289012 entries
        - prefix /32 counts 438290 entries

totals: 326293 lines read, 302935 distinct IP ranges found, 3 CIDR prefixes, 742955 CIDRs printed, 1100260370 unique IPs
completed in 28.41651 seconds (read 0.22530 + think 0.23088 + speak 27.96033)

the last list would probably be the most "optimized" for IPset hash:ip in-regards to memory (RAM) consumption because I have reduced the differences in prefix lengths to its lowest possible outcome.

For hash:net, the first list would be the most optimized.

As a side note 1100260370 unique IPs could imply that my list blocks slightly more than approximately 1/4th about 25.62% the worlds IP addresses (on all open incoming ports) and (all outbound connections). @Tech9 .

 

[SomeWhereOverTheRainBow]

Hey guys. I just noticed that Skynet seems to be failing to start. The tab doesn't show up in the GUI, and logging in via SSH and opening the skynet script, it shows a status of failed. I've rebooted the router about 10 times in the past few days (just moved, setting up the network), and I've updated skynet and router firmware.

View attachment 49989


View attachment 49990

Any ideas?

All help is appreciated, thanks.

enable inbound firewall rules.

 

[jorgsmash]

enable inbound firewall rules.

View attachment 50002

I don't have any inbound firewall rules configure in the GUI.

lets say I reduce it by 100,000 entries using iprange

326293 printed CIDRs, break down by prefix:
        - prefix /3 counts 1 entries
        - prefix /8 counts 3 entries
        - prefix /9 counts 1 entries
        - prefix /10 counts 16 entries
        - prefix /11 counts 31 entries
        - prefix /12 counts 76 entries
        - prefix /13 counts 122 entries
        - prefix /14 counts 196 entries
        - prefix /15 counts 322 entries
        - prefix /16 counts 929 entries
        - prefix /17 counts 629 entries
        - prefix /18 counts 949 entries
        - prefix /19 counts 1508 entries
        - prefix /20 counts 2028 entries
        - prefix /21 counts 2481 entries
        - prefix /22 counts 5771 entries
        - prefix /23 counts 5314 entries
        - prefix /24 counts 13500 entries
        - prefix /25 counts 424 entries
        - prefix /26 counts 560 entries
        - prefix /27 counts 583 entries
        - prefix /28 counts 798 entries
        - prefix /29 counts 1285 entries
        - prefix /30 counts 2631 entries
        - prefix /31 counts 9815 entries
        - prefix /32 counts 276320 entries

totals: 326293 lines read, 302935 distinct IP ranges found, 26 CIDR prefixes, 326293 CIDRs printed, 1100260370 unique IPs
completed in 12.08756 seconds (read 0.22889 + think 0.21960 + speak 11.63906)

and here is if I reduce it by 1 million

742955 printed CIDRs, break down by prefix:
        - prefix /16 counts 15653 entries
        - prefix /24 counts 289012 entries
        - prefix /32 counts 438290 entries

totals: 326293 lines read, 302935 distinct IP ranges found, 3 CIDR prefixes, 742955 CIDRs printed, 1100260370 unique IPs
completed in 28.41651 seconds (read 0.22530 + think 0.23088 + speak 27.96033)

the last list would probably be the most "optimized" for IPset hash:ip in-regards to memory (RAM) consumption because I have reduced the differences in prefix lengths to its lowest possible outcome.

For hash:net, the first list would be the most optimized.

As a side note 1100260370 unique IPs could imply that my list blocks slightly more than approximately 1/4th about 25.62% the worlds IP addresses (on all open incoming ports) and (all outbound connections). @Tech9 .

How can I go about utilizing your optimized lists? I have posted before (couple years ago) about my AX88U's memory usage. I've been rocking like 50-75mb free memory since I installed WRT Merlin and various scripts. Memory utilization is always at like 95%. I was told it was fine, and that unused RAM was a waste, but running at 95% doesn't leave much headroom. I'd like to figure out why I have so little RAM or just go get a new router. The AX88U was supposed to be future proof but with only 1 GB of RAM and using 95% of it, seems like I need more.

Almost forgot, would I like to remove the swap file?

Would You Like To Remove Skynet Generated Swap File?
[1] --> Yes
[2] --> No

 

[dave14305]

Run the debug command to gather more info:

firewall debug info extended