Aegis Aegis 1.7.x

[HELLO_wORLD]

Due to new forum rules on threads older than six months, here is a fresh new one, good until August 2021.

Previous thread

Aegis
A firewall blocklist script for Netgear R7800, R9000 and Orbi Routers with Voxel firmware.
Should work with some other models as well.

What is it?
It is a script that allows to block a list of IP adresses or ranges for inbound and outbound traffic.
The main purpose is blocking dangerous adresses known for spam, hacking, malware, etc...
The blocklist is automatically generated from known sources (this is editable) and you can add your own IP adresses/ranges as well.

Instructions and installation
https://github.com/bolemo/aegis/blob/stable/README.md


Aegis is totally free, and I am not asking for anything if you are using it.
If you wish to make a donation for my work, and are able to, here is a PayPal link:
https://paypal.me/bolemoDonation
If you wish to make a donation, please consider first @Voxel who is doing an incredible amount of work for our routers, and @kamoj who is making an amazing add-on.

 

[HELLO_wORLD]

1.7.0

It is a major upgrade.

Changes are:

• Since iprange is now inside the @Voxel firmware, I removed the portions of code used to work without it, and made aegis relying more on it, making aegis lighter.
• aegis has now the ability to have custom blacklists and whitelists specific to WAN or VPN.
• the upraising, logging and status portions of the code have been heavily changed to adapt for that change.
• optimizations, bug fixes.
• web companion has been updated to work with new status and log code, and the lists editor has been changed to allow to view/create or edit default lists (sources, custom global blacklist, custom global whitelist, custom WAN black and white lists, VPN custom black and white lists).

Install procedure
1.7 being a major upgrade from 1.6, it is strongly advised to upgrade this way from terminal:

aegis unset
aegis upgrade
aegis up

In case you have problems with 1.7, it is possible to revert to 1.6.12; from terminal:

aegis unset
aegis upgrade -repo=1.6
aegis up

 

[blueliner]

1.7.0

It is a major upgrade.

Changes are:

• Since iprange is now inside the @Voxel firmware, I removed the portions of code used to work without it, and made aegis relying more on it, making aegis lighter.
• aegis has now the ability to have custom blacklists and whitelists specific to WAN or VPN.
• the upraising, logging and status portions of the code have been heavily changed to adapt for that change.
• optimizations, bug fixes.
• web companion has been updated to work with new status and log code, and the lists editor has been changed to allow to view/create or edit default lists (sources, custom global blacklist, custom global whitelist, custom WAN black and white lists, VPN custom black and white lists).

Install procedure
1.7 being a major upgrade from 1.6, it is strongly advised to upgrade this way from terminal:

aegis unset
aegis upgrade
aegis up

In case you have problems with 1.7, it is possible to revert to 1.6.12; from terminal:

aegis unset
aegis upgrade -repo=1.6
aegis up

Hello,

I followed the upgrade instructions and Aegis upgraded from 1.6.12 to 1.7.0 OK. However, I had the following error message when I executed "aegis up":

It didn't seem to be a problem for Aegis as the Status page showed everything was OK. But I tried a restart and reinstall anyway and had the same result. I did have two lines of commented (## ...) text at the top of both my global whitelist and global blacklist. I removed the commented lines and the error (?) message disappeared on the next restart.

I like the expanded functionality of 1.7.0 and so far it is running well- thank you!

BL

 

[HELLO_wORLD]

Hello,

I followed the upgrade instructions and Aegis upgraded from 1.6.12 to 1.7.0 OK. However, I had the following error message when I executed "aegis up":

View attachment 31401

It didn't seem to be a problem for Aegis as the Status page showed everything was OK. But I tried a restart and reinstall anyway and had the same result. I did have two lines of commented (## ...) text at the top of both my global whitelist and global blacklist. I removed the commented lines and the error (?) message disappeared on the next restart.

I like the expanded functionality of 1.7.0 and so far it is running well- thank you!

BL

Yes, since iprange is mandatory, I release on it to strip comments, instead of my own code.
However, you should not have any errors with comments. I tried several commented files (and just did with ## ... ##), and never had a complaint like you did... could it be that there were strange characters in your list around the comments?

Anyway, glad it works, I think aegis is getting a nice maturity now.

 

[blueliner]

Yes, since iprange is mandatory, I release on it to strip comments, instead of my own code.
However, you should not have any errors with comments. I tried several commented files (and just did with ## ... ##), and never had a complaint like you did... could it be that there were strange characters in your list around the comments?

Anyway, glad it works, I think aegis is getting a nice maturity now.

Hello,

Reading the Aegis "error" message, I thought the same thing about strange characters around the comments. I couldn't see anything unusual except that I had ## both at the beginning and end of the comment in Line #1 of the blacklist and whitelist. I had the same two lines of comments in the blacklist and whitelist ever since I've used Aegis...maybe they were not the problem but it worked after removing them, so I am happy! This is what I had:

## Custom blacklist of IPs for Aegis ##
## bl - Added blacklisted IPs and netsets below. One per line

## Custom whitelist of IPs for Aegis ##
## bl - Added whitelisted IPs and netsets below. One per line

Best wishes,
BL

 

[HELLO_wORLD]

Hello,

Reading the Aegis "error" message, I thought the same thing about strange characters around the comments. I couldn't see anything unusual except that I had ## both at the beginning and end of the comment in Line #1 of the blacklist and whitelist. I had the same two lines of comments in the blacklist and whitelist ever since I've used Aegis...maybe they were not the problem but it worked after removing them, so I am happy! This is what I had:

## Custom blacklist of IPs for Aegis ##
## bl - Added blacklisted IPs and netsets below. One per line

## Custom whitelist of IPs for Aegis ##
## bl - Added whitelisted IPs and netsets below. One per line

Best wishes,
BL

Just added your blacklist comment lines (copy/paste) to my custom blacklist. Refreshed aegis, no error.

Strange.

Thankfully, all ok for you

 

[NetBytes]

Looking at the output above and where the single quote lines are located, is it possible there is an
invalid carriage return at the end of your comment lines?
The difference i mean is between \r\n and just \n - like u edited the file in windows notepad?

 

[blueliner]

Looking at the output above and where the single quote lines are located, is it possible there is an
invalid carriage return at the end of your comment lines?
The difference i mean is between \r\n and just \n - like u edited the file in windows notepad?

That's a good point... all my devices are Linux and I normally use Mousepad. I hadn't modified the lists since 1.6.9 but I must admit I don't pay enough attention to formatting things such as carriage returns, etc, and don't even know much about them. I pasted the comments back in from a saved copy and restarted Aegis. All was OK. Maybe it had nothing to do with the list comments at all, even though its the only thing I changed?

Here is a pic from a saved copy of the blacklist (I had typed in the comments for my previous post.:

Either way I will be looking at formatting more carefully in the future as your suggestion makes sense to me. Thank you!

BL

 

[mith_y2k]

I tried manually updating and stopped, maybe it's not super important, but it looks like something is wrong with sort. See print from my terminal:

root@RBR50:~# aegis upgrade

Upgrading:
- version installed: 1.6.12
/usr/bin/sort: invalid option -- Vn...
BusyBox v1.24.1 (2020-12-23 11:30:05 UTC) multi-call binary.

Usage: sort [-nru] [FILE]...

Sort lines of text

    -n    Sort numbers
    -r    Reverse sort order
    -u    Suppress duplicate lines
- this version is higher than than the one available: 1.7.0!       
? do you want to downgrade from 1.6.12 to 1.7.0 anyway (y/n)?

 

[HELLO_wORLD]

I tried manually updating and stopped, maybe it's not super important, but it looks like something is wrong with sort. See print from my terminal:

root@RBR50:~# aegis upgrade

Upgrading:
- version installed: 1.6.12
/usr/bin/sort: invalid option -- Vn...
BusyBox v1.24.1 (2020-12-23 11:30:05 UTC) multi-call binary.

Usage: sort [-nru] [FILE]...

Sort lines of text

    -n    Sort numbers
    -r    Reverse sort order
    -u    Suppress duplicate lines
- this version is higher than than the one available: 1.7.0!      
? do you want to downgrade from 1.6.12 to 1.7.0 anyway (y/n)?

I see.
You can answer yes here.

The problem is minor (related to version detection and not aegis core functions).
It appears that the sort present in Orbi differs, and does not accept options like -V.
I will need to either do without sort or see if @Voxel can include the same sort into Orbi as other models.

 

[HELLO_wORLD]

1.7.1

Changed version comparison algorithm.
Now using a little more complicated one, but still short, that should work fine on Orbi as well; it uses sort too but with -n, not -V
Also, in case of future betas, it detects (for example) 1.1.1b4 as lower than 1.1.1 as it should be since 1.1.1 would be the release coming after the beta.

upgrade won’t show in web GUI for now
Upgrade from terminal will show as 1.7.0 (deliberately), but saying yes will upgrade to 1.7.1
This is to prevent web GUI upgrade, that is defective (inferno loop).

 

[blueliner]

Hello,

As info, my router running Aegis is an R9000 with Voxel 1.0.4.48.1HF and the Kamoj 5.4b22 Add-on. I noticed the post about Aegis 1.7.1, and sure enough the Web Companion showed an upgrade from 1.7.0 to 1.7.1 was available. I selected to upgrade and things seemed to proceed until I got this pop-up on my browser:

Checking the Status tab, I saw this:

At that point nothing more would happen. Logging into the router and doing Aegis info showed nothing and Aegis would not respond to other commands either. There was the normal dialog about installing Aegis to the (correct) drive before the upgrade stopped. I looked in the bolemo directory on that drive (a usb hdd), and all the normal folders/files appeared to be there and were updated with "new" timestamps. I rebooted the router but Aegis still was not available.

I decided to do a manual install. It proceeded normally and looked like it was successful. However, the Aegis menu option in the Kamoj Add-on was not available and the Aegis web page would not open. Aegis info showed all ok. A reboot did not fix the problem, so I did a second reinstall...everything worked after that. I've attached screenshots of the console commands in case I may have done something incorrectly...

Anyway, a little drama on the upgrade from 1.7.0 to 1.7.1 but in the end all is well! All of the 1.6x versions of Aegis upgraded without any issues, so I thought I would pass this along.

Thank you,
BL

 

[HELLO_wORLD]

There is definitely something broken with upgrade from web GUI...
Working on it.

meanwhile, please AVOID upgrading from web until it is fixed.

 

[mith_y2k]

Quick update that I upgraded from CLI to 1.7.1, refreshed directives and started aegis and all looks good. Small glitch that I noticed:

root@RBR50:~# aegis upgrade

Upgrading:
- version installed: 1.6.12
/usr/bin/sort: invalid option -- Vn...
BusyBox v1.24.1 (2020-12-23 11:30:05 UTC) multi-call binary.

Usage: sort [-nru] [FILE]...

Sort lines of text

    -n    Sort numbers
    -r    Reverse sort order
    -u    Suppress duplicate lines
- this version is higher than than the one available: 1.7.0!       
? do you want to downgrade from 1.6.12 to 1.7.0 anyway (y/n)? y
- Downloading:
/tmp/aegis.dl                    100%[==========================================================>]  69.43K  --.-KB/s    in 0.01s   
- Script installed to /opt/bolemo/scripts/aegis
- downloading Web Companion:
/tmp/aegis.dl                    100%[==========================================================>]  24.06K  --.-KB/s    in 0.002s 
- Web Companion htm file installed to /opt/bolemo/www/aegis.htm
/tmp/aegis.dl                    100%[==========================================================>]  19.97K  --.-KB/s    in 0.002s 
- Web Companion cgi file installed to /opt/bolemo/www/cgi-bin/aegis_web.cgi
root@RBR50:~# aegis upgrade
Upgrading:
- version installed: 1.7.1
- this version is higher than than the one available: 1.7.0!       
? do you want to downgrade from 1.7.1 to 1.7.0 anyway (y/n)? n

See how the first time I tried to upgrade it told me the new version was 1.7.0 and if I wanted to downgrade. I thought that maybe you wanted to force upgrade to 1.7.0 and then upgrade again to 1.7.1, but when I tried the second upgrade I was already on 1.7.1. TBH it looks like a text string issue as the upgrade went well, so super minor.

Thank you as usual for the work!

 

[mith_y2k]

One more thing, it looks like logMaxLen is back to 300 by default which causes the Error 502 on Orbi. I'm going to manually change it back to 100 that seemed to work fine in the past in my local copy.

 

[HELLO_wORLD]

One more thing, it looks like logMaxLen is back to 300 by default which causes the Error 502 on Orbi. I'm going to manually change it back to 100 that seemed to work fine in the past in my local copy.

1.7.1 showing as 1.7.0 on server is intentional, to prevent upgrade from web GUI.
There is a nasty bug where it just loops into upgrading again and again... only solution is from terminal killall aegis
I am looking into this, but meanwhile, I prevent web upgrade.

Next upgrade to 1.7.2 when ready will have to be made from terminal.

I did not touch the logMaxLen code at all since I put the orbi recognition code to limit to 150 (when Orbi). Not sure why it did not work for you anymore... Is it 1.7.1 only or 1.7.0 was the same?

 

[HELLO_wORLD]

Ok, problem solved.

First, thank you @mith_y2k for your reports, it was helpful to detect a very tricky nasty weird bug.

When aegis web is upgraded, it calls the downloaded file for some post-install process. For some unknown and unexplained reason, it started to misbehave and it would put the http server in an infernal loop, upgrading aegis over and over and over...
The very weird part is that none of the post install code or code involved in this was changed in a while, and it happens only when called from http server.
Anyway, when aegis web is downloaded, it is first going in a temporary file, and calling the post install in the temporary file does not trigger this frenzy.

Maybe this bug has been there for a while, most likely from before 1.7. It worked, and something changed (could be outside of aegis) creating this weird behavior. Maybe this explains some issues some of you had in the past; I am thinking of @R. Gerrits who had problems with too many processes running, maybe it was already this infernal loop, or maybe not.

Anyway this nasty thing is fixed, and I discovered a (minor) bug that was introduced since Orbi support: for lighttpd to not throw error, any called cgi file has to output something, it freaks if there is no output. I therefore had introduced the output of a blank space at the end of every calls to aegis cgi. This broke the JavaScript that was reloading the page after an upgrade (reloading new html). This was not a real issue, but could lead to problems having the old version loaded into the web browser.
So that is fixed as well.

Now, @mith_y2k , the code in aegis web for the log has this:

[ "$(cat /module_name)" = "RBR50" ] && MAX=150 || MAX=300

and

elif [ $ARG -gt $MAX ]; then LEN=$MAX

So whatever logMaxLen is in html file, the cgi part caps it to 150 for Orbi before outputting the log, so same as if you would have changed
logMaxLen to 150 in html file.
If you still have that issue with 1.7.2, it means that 150 is too high (but was ok before), and I will reduce it to 100

Ok, 1.7.2 on its way.
Upgrade MUST NOT BE MADE FROM WEB any version prior to 1.7.2 has the upgrade from web bug. Once 1.7.2 is installed, then future web upgrades will be ok.

 

[HELLO_wORLD]

1.7.2

Fix MAJOR BUG with web upgrade that became defective (weird http cgi server behavior).
Fix minor bug where html page would not automatically reload itself after an upgrade (to load upgraded version in web browser).

Because of the bug, and to prevent any problem, the upgrade procedure MUST BE DONE FROM terminal. Once upgraded to 1.7.2, upgrade from web will be ok.

Exceptionnal:

aegis unset
aegis upgrade -repo=stable
aegis up -v

Please, note that future updates won’t show if you do not upgrade to 1.7.2 this way first (or reinstall aegis from install procedure explained in 1.7.2 and up readme).

 

[blueliner]

Hello,

Upgrade as instructed worked OK. 1.7.2 up and running. Thanks!

BL

 

[Warlord1981]

Upgraded from 1.6.12 to 1.7.2 via terminal and all is well! Many thanks!

EDIT: I see a slighlty increased RAM Usage. Is there a way we can clear ram cache?

EDIT2: Just noticed that on LOGS traffic "from" to "to" appear with big space between them. Also, clicking on an IP does not open an IP Lookup page. (thought this was added as feature to 1.7.0 and later)