Kamoj Kamoj Add-on Beta testing

[R. Gerrits]

But today it would again crash with an out of memory error, almost immediately after starting it.

Anyways, I'll now try I simple hack and put the AdGuard Home data folder on my USB. See if that fixes it.
tbc...

With the data folder on USB, it does stay up longer.
But already 2x today it OOMed.

After the first time, I added a cronjob to dump the memory usage (using free) every 15 minutes.

When I checked it after the second time, it had this:

Fri Feb 19 17:00:01 GMT 2021
              total         used         free       shared      buffers
  Mem:       483524       367944       115580            0        41764
 Swap:            0            0            0
Total:       483524       367944       115580
Fri Feb 19 17:15:01 GMT 2021
              total         used         free       shared      buffers
  Mem:       483524       277484       206040            0        38636
 Swap:            0            0            0
Total:       483524       277484       206040

115 MB free before the crash and 206 MB free after the crash...
guess I need to dump "free" more often to really see the oom event.

 

[R. Gerrits]

getting closer....
just almost caught a crash in the act:

2021/02/19 21:24:03 [info] Saving filter 1590435211 contents to: /tmp/addons/adguard_home/data/filters/1590435211.txt
2021/02/19 21:24:03 [info] Updated filter #1.  Rules: 39182 -> 39140
2021/02/19 21:24:03 [info] Updated filter #1590433495.  Rules: 876360 -> 833009
2021/02/19 21:24:03 [info] Updated filter #1590435211.  Rules: 2764 -> 2765
2021-02-19 22:29:08 [ADGUARD] adguard_home.sh 18644: 823480.79:Start called
2021-02-19 22:29:10 [ADGUARD] adguard_home.sh 18644: 823482.59:Information: Time is OK
2021-02-19 22:29:10 [ADGUARD] adguard_home.sh 18644: 823482.74:Adguard Home started. Accessable GUI: http://192.168.1.1:8080
2021/02/19 21:29:10 [info] AdGuard Home, version v0.105.1, channel release, arch linux arm v7
2021/02/19 21:29:10 [info] Initializing auth module: /tmp/addons/adguard_home/data/sessions.db

(some of the timestamps are in utc, hence the hour difference)
and the memory usage:

Fri Feb 19 22:23:01 GMT 2021
              total         used         free       shared      buffers
  Mem:       483524       391484        92040            0        53852
Swap:            0            0            0
Total:       483524       391484        92040
Fri Feb 19 22:24:01 GMT 2021
              total         used         free       shared      buffers
  Mem:       483524       421640        61884            0        53928
Swap:            0            0            0
Total:       483524       421640        61884
Fri Feb 19 22:25:01 GMT 2021
              total         used         free       shared      buffers
  Mem:       483524       255540       227984            0        16308
Swap:            0            0            0
Total:       483524       255540       227984

So it seems that during the update of some blocklists, the memory quickly fills up.
perhaps I should pick a smaller blocklist...


EDIT: unselected all blocklists and only kept the abp.oisd.nl one -> Pressed "Check for Updates" and boom, it crashed.
Then I rebooted the router, tried again to refresh -> this time it did successfully refresh...
So it seems the full oisd list is pushing it to the limit.

I also noticed AdGuard tries the refresh every 12 hours -> this explains also my previous logs showing supervision restarting it also every 12 hours.

EDIT2:
AdGuard Home made it through the night without crashes, but its really pushing the limits:

              total         used         free       shared      buffers
  Mem:       483524       460448        23076            0        83068
Swap:            0            0            0
Total:       483524       460448        23076

And I saw also moments where fee memory was less than 16 MB during the night.

I looked at procps-ng-ps -> the two biggest consumers seem to be AdGuard Home and Netdata.
I stopped Netdata -> free memory jumped to 113 MB.

So i'll disable netdata, and hope that this will finally give me a stable situation.

 

[Droidrat]

I had a very similar experience with a simultaneous upgrade to FW V1.0.2.82.1SF (Voxel) and Kamoj Add-on:5.4b20. Lost access to my /opt (for Entware). I had to recreate the Entware drive using a new usb stick as the the existing usb is locked “read only” and cannot be reformatted. Also, no partition tools (tried several) can remove the ext4 partition. This usb does not have a physical lock.

Thank you. It is helpful to have validation. I deleted the partition on my USB thumb drive; reformatted in ext4; and reinstalled entware. From now on, I think I will remove the USBs any time I install new firmware.

 

[Droidrat]

After creating a new entware USB, and rebooting (without any USBs installed), I noticed that my MTD System Disk Usage is maxed out at 100% (see attached image). I've never seen it this way before. The last screenshot I took (in April 2020) showed it at 19.6% with 9.758 in use.

Any ideas as to what caused this, and how to fix it?

 

[R. Gerrits]

Did you perhaps backup some files on the router itself?

Can you do du -h /overlay and see how big that is?
(on my R7800, it is 11.9 M)

If it does have more, then look at the files and see what you can cleanup. Perhaps entware installed into /overlay/opt ??
(i.e. USB was not mounted correctly when you installed it) ?

 

[Droidrat]

Did you perhaps backup some files on the router itself?

Can you do du -h /overlay and see how big that is?
(on my R7800, it is 11.9 M)

If it does have more, then look at the files and see what you can cleanup. Perhaps entware installed into /overlay/opt ??
(i.e. USB was not mounted correctly when you installed it) ?

I did not back up any files to the router (at least not knowingly).

Yes, my overlay output shows a bit more than yours, 699.5M. ;-)

I have attached it. I see the there is an /entware directory, so I deleted it. I also deleted the file "entware-cortex-a15-3x-initial-generic.tar.gz", and the massive file "swap." As you suggested, they were all created during a busted installation of entware.

After deleting all of this, I reran "du" and the usage is a more reasonable 13.5M. This is still more than yours. Please see the attached output (Output of du overlay2.txt). I would appreciate guidance as to what else to delete.

 

[R. Gerrits]

After deleting all of this, I reran "du" and the usage is a more reasonable 13.5M. This is still more than yours. Please see the attached output (Output of du overlay2.txt). I would appreciate guidance as to what else to delete.

At some point I deleted these folders:
/overlay/opt/rcagent
/overlay/opt/remote
/overlay/opt/leafp2p
/overlay/opt/xagent

They belong to some netgear stufff that I'm not using.
Can't recall anymore what. (might be ReadyCloud related).

Anyways, if you are not sure, just leave them there.
13.5 MB usage is perfectly fine. And I think it is even compressed before it is stored.

 

[sppmaster]

After beta 21 when I reboot the router the DNSCrypt 2 won't start automatically anymore and all LAN clients don't have Internet connection. The router itself report that the connection is OK.
@kamoj can you adopt the script to properly take info about the new @Voxel firmware snapshot versions. I've installed the snapshot and it says there is a new version.

 

[kamoj]

Changes in kamoj-addon beta version 5.4b22
--------------------------------------------------
- Router Information: System Version Information: Display "Latest Voxel SnapShot" in green,
if newer than current version. (@sppmaster)
- DNSPrivacy/Ad-Blocking: Adblocking: Added "--no-check-certificate" for wget
- DNSPrivacy/Ad-Blocking: DNSCrypt2 : Added fix for dnscrypt2 crashing at start if given
blacklist-file does not exist. (@sppmaster)
(Dnscrypt2 was though later restarted when blacklist downloaded was ok.)
- DNSPrivacy/Ad-Blocking: DNSCrypt2 : Prepared for usage of USB-device to store lists.
- Minor fixes

N.B: If you use DNSCrypt ad-blocking and experience problems, report the problem and revert to 5.4b20, please.

 

[n1llam1]

Using two R7800 devices with Voxel 1.0.2.82.2SF, one in router mode and one in AP mode

Router mode device has Kamoj Add-on V5.4b20
AP mode device has Kamoj Add-on V5.4b21

Both V5.4b20 and V5.4b21 display only the following 2 lines when clicking on System Information->AP Stats->Show button

apstats: No application recognized options. Using defaults: AP level, non-recursive.
Use -h for help

It does not appear to be capturing / displaying the full apstats info.

Also, on both versions, clicking on System Information->WiFi Info->Show button does not display any info.

Upgraded device in AP mode to V5.4b22. Observed issue with WiFi Info has been resolved and information is now displayed for both bands:
###2.4G###
###5G###

The AP Stats Show button still displays only the 2 lines as previously described.

 

[blueliner]

Changes in kamoj-addon beta version 5.4b22
--------------------------------------------------
- Router Information: System Version Information: Display "Latest Voxel SnapShot" in green,
if newer than current version. (@sppmaster)
- DNSPrivacy/Ad-Blocking: Adblocking: Added "--no-check-certificate" for wget
- DNSPrivacy/Ad-Blocking: DNSCrypt2 : Added fix for dnscrypt2 crashing at start if given
blacklist-file does not exist. (@sppmaster)
(Dnscrypt2 was though later restarted when blacklist downloaded was ok.)
- DNSPrivacy/Ad-Blocking: DNSCrypt2 : Prepared for usage of USB-device to store lists.
- Minor fixes

N.B: If you use DNSCrypt ad-blocking and experience problems, report the problem and revert to 5.4b20, please.

Hello,

DNSCrypt is running fine as near as I can tell on 5.4b22. I should also mention that I was able to stop the DNSSEC error I had on 5.4b20 (?) by starting with a fresh dnscrypt-proxy-2.toml configuration. I couldn't find any errors and set the new copy up the same way I always do...but I must have had an error or file corruption? Anyway everything with DNSCrypt is working now.

I have still gotten "DNS Status ERRORS" in the Router Information page at times even though it says "OK" in the Supervisor log. Right now they both say "OK" but that wasn't the case the previous time I checked. I still wonder if it has something to do with timing or otherwise affected by the VPN...as sometimes turning off the VPN seems to correct the issue but other times it doesn't????

BL

 

[sppmaster]

Changes in kamoj-addon beta version 5.4b22
--------------------------------------------------
- Router Information: System Version Information: Display "Latest Voxel SnapShot" in green,
if newer than current version. (@sppmaster)
- DNSPrivacy/Ad-Blocking: Adblocking: Added "--no-check-certificate" for wget
- DNSPrivacy/Ad-Blocking: DNSCrypt2 : Added fix for dnscrypt2 crashing at start if given
blacklist-file does not exist. (@sppmaster)
(Dnscrypt2 was though later restarted when blacklist downloaded was ok.)
- DNSPrivacy/Ad-Blocking: DNSCrypt2 : Prepared for usage of USB-device to store lists.
- Minor fixes

N.B: If you use DNSCrypt ad-blocking and experience problems, report the problem and revert to 5.4b20, please.

The reported issues seem resolved to me in b22.

 

[kamoj]

DNSCrypt is extremely picky about the toml-file.
Overall I think it is not fault tolerant, still a bit immature...
One example was this bug I corrected in the add-on b22.
I don't think it's ok for dnscrypt to just crash because there is no blaocklist!

So, Warning when editing the toml-file; be extremely cautious since just a blank or tab character in wrong place crashes dnscrypt completely.

Hello,

DNSCrypt is running fine as near as I can tell on 5.4b22. I should also mention that I was able to stop the DNSSEC error I had on 5.4b20 (?) by starting with a fresh dnscrypt-proxy-2.toml configuration. I couldn't find any errors and set the new copy up the same way I always do...but I must have had an error or file corruption? Anyway everything with DNSCrypt is working now.

I have still gotten "DNS Status ERRORS" in the Router Information page at times even though it says "OK" in the Supervisor log. Right now they both say "OK" but that wasn't the case the previous time I checked. I still wonder if it has something to do with timing or otherwise affected by the VPN...as sometimes turning off the VPN seems to correct the issue but other times it doesn't????

BL

 

[VinnyJ]

Where are the new beta test builds going to be distributed, once the current hosting service shuts down in about 5 days?

 

[blueliner]

Hello,

I never seem to show a good DNS Status on the Router Information page (yet pings are often OK). I decided to look into it and now I am confused.

I telnetted into the router and ran nslookup and curl on what I had entered into kamoj_ping_www (I changed it earlier because of the DNS Status "ERROR"s). I got a failure with the first address in my list so I deleted that from Kamoj_ping_www and did nvram commit. The Router Information page now showed a good DNS Status - success! Then when the Router Information page refreshed, the DNS Status showed an error again (as shown in the pic below).

View attachment 30919

Yet the Supervisor log (attached) showed all pings and curl tests successful - not one failure. I cleared the cache on my browser - same thing. I used a different browser - same thing. I rebooted the router and the fist view of the DNS Status was good and at that time the VPN had not completed its start-up. The next page refresh showed the VPN was up and now the DNS Status showed "ERROR". My VPN does apply malware filtering, so I decided to try an OpenVPN profile that does not filter. Same thing, the Add-on Supervisor log shows all ping and curl successful but the router information page shows DNS Status: ERROR. Resetting to other values, including the default in kamoj_ping_www makes no difference. I have also increased kamoj_restart_ping_timeout to a value one number higher than what it takes to get a successful "curl --max-time () https://ifconfig.io/ip" command.

No matter what, the Router Information page shows DNS Status: ERROR yet the supervisor log shows 100% success. I assume I am misunderstanding how this works or have done something incorrect? It doesn't seem to affect my router's operation but like I said, I am confused (as usual!) by this. Thoughts please...

Thanks,
BL

Hello,

As a follow-up to my post on the DNS Status ERROR on the Router Information Page... whether the VPN is on or off, I get consistent OK's on the Supervisor log:

However, the VPN must be off for me to get a DNS Status OK on the Router Information page.

As soon as I turn the VPN back on, the DNS Status goes to ERROR:
'

The results are consistent but don't seem to affect the router operation. DNSCrypt is running well and I have seen no other issues besides the very rare need for Supervisor to restart the VPN (working good). RAM use seems to spike higher than it had in the past after a reboot, or starting DNSCrypt, etc.; but it returns to normal soon thereafter. CPU use seems to be normal as well.

Best Wishes,
BL

 

[kamoj]

Thank you again for your report!

Your internet and ping times are bad, and it might be that when using VPN the 3 sec. ping used in Router Information times out?

For next version I'll revert to use the value of "nvram get kamoj_restart_ping_timeout".
Until then you can try this patch:

sed -i 's/timeout 3/timeout 6/g' /usr/bin/addon_info_update.sh

Hello,

As a follow-up to my post on the DNS Status ERROR on the Router Information Page... whether the VPN is on or off, I get consistent OK's on the Supervisor log:

View attachment 31285

However, the VPN must be off for me to get a DNS Status OK on the Router Information page.

View attachment 31284

As soon as I turn the VPN back on, the DNS Status goes to ERROR:
'
View attachment 31286

The results are consistent but don't seem to affect the router operation. DNSCrypt is running well and I have seen no other issues besides the very rare need for Supervisor to restart the VPN (working good). RAM use seems to spike higher than it had in the past after a reboot, or starting DNSCrypt, etc.; but it returns to normal soon thereafter. CPU use seems to be normal as well.

Best Wishes,
BL

 

[blueliner]

Thank you again for your report!

Your internet and ping times are bad, and it might be that when using VPN the 3 sec. ping used in Router Information times out?

For next version I'll revert to use the value of "nvram get kamoj_restart_ping_timeout".
Until then you can try this patch:

sed -i 's/timeout 3/timeout 6/g' /usr/bin/addon_info_update.sh

That took care of it. For the past 4 hours I have seen no DNS Errors on the Router Information page!

Thank you Kamoj.

BL

 

[manup85]

Hello All, i have flashed the new beta b22 and have lost my 5ghz band.

i have backup and erase everything but i still see this issue. if i disable smarth connect i loose wifi completely
thank you to have a look

 

[sppmaster]

I had problems with wi-fi during a high CPU usage (reported it earlier but I was unable to pinpoint the exact reason for it). Can you check what process causes the high CPU load on your router.

 

[kamoj]

Seems like you are out-of-memory, right?
Then anything can happen. Linux OS may reboot the system at any time to regain memory e.g.
Run top2 and ps2 commands, and see who is the hog. (I expect AdGuard, as you can read about here in this thread).
As a beta-tester the community hope you will examine this and find the culprit, and hopefully solution.

Hello All, i have flashed the new beta b22 and have lost my 5ghz band.

View attachment 31399

i have backup and erase everything but i still see this issue. if i disable smarth connect i loose wifi completely
thank you to have a look