Scribe scribe - syslog-ng and logrotate installer

[juched]

Sharing my findings, experimenting with removing settings and changing settings. one thing I found:

if this is my file list:
blankmsg diversion openvpn skynet spdmerlin syslogng unbound wlceventd

I get memory usage of ~78M


if I add "ntpd" memory goes up to 148M.

if I add "logrotate" memory goes up to 275M.

if I remove "ntpd" memory goes down to 274M.


So, what is special about ntpd and logrotate? Don't have time to dig right now.

 

[cmkelley]

Sharing my findings, experimenting with removing settings and changing settings. one thing I found:

if this is my file list:
blankmsg diversion openvpn skynet spdmerlin syslogng unbound wlceventd

I get memory usage of ~78M


if I add "ntpd" memory goes up to 148M.

if I add "logrotate" memory goes up to 275M.

if I remove "ntpd" memory goes down to 274M.


So, what is special about ntpd and logrotate? Don't have time to dig right now.

Memory "usage" in htop or top doesn't tell you anything of value. Removing ntpd doesn't necessarily mean the memory it was using is marked "free" - it's probably marked "used, but able to be freed immediately" which means it's included in the "used" memory. Unix memory management isn't binary, it isn't "free" or "in use", there's a couple levels of gray area in between.

That said, ntpd is a daemon, and so it remains in memory. logroate is a program that only runs once a day, in the middle of the night. I'd guess, completely out of my nether-regions, that the big jump you saw when installing logrotate was the memory manager saying "ah-ha, the user ran opkg twice in a short period of time, so I'm going to keep it in memory and mark it "used, but able to be freed immediately" so I know where it is." And removing ntpd required running opkg (that was already in memory), but again it didn't mark the memory it used as "free", even after opkg exited.

No single value for "used" or "free" memory is useful, these are nebulous concepts in unix-land. I have a feeling that the way memory is reported as "used", made a lot more sense when computers and storage systems were hugely slower than they are now.

 

[juched]

Memory "usage" in htop or top doesn't tell you anything of value. Removing ntpd doesn't necessarily mean the memory it was using is marked "free" - it's probably marked "used, but able to be freed immediately" which means it's included in the "used" memory. Unix memory management isn't binary, it isn't "free" or "in use", there's a couple levels of gray area in between.

That said, ntpd is a daemon, and so it remains in memory. logroate is a program that only runs once a day, in the middle of the night. I'd guess, completely out of my nether-regions, that the big jump you saw when installing logrotate was the memory manager saying "ah-ha, the user ran opkg twice in a short period of time, so I'm going to keep it in memory and mark it "used, but able to be freed immediately" so I know where it is." And removing ntpd required running opkg (that was already in memory), but again it didn't mark the memory it used as "free", even after opkg exited.

No single value for "used" or "free" memory is useful, these are nebulous concepts in unix-land. I have a feeling that the way memory is reported as "used", made a lot more sense when computers and storage systems were hugely slower than they are now.

All of my memory usage numbers was just the usage shown in top of syslog-ng. Not free or total memory used of the system. Just used by syslog-ng.

I am not concerned about the reported memory usage of syslog-ng. But based on the questions and posts in this thread I would report what I saw.

To be honest once syslog-ng loads the memory reported used by it in top never changes. So this isn’t a leak and doesn’t cause any issues that I can see.

Free memory is wasted memory.... I agree!

 

[cmkelley]

All of my memory usage numbers was just the usage shown in top of syslog-ng. Not free or total memory used of the system. Just used by syslog-ng.

I am not concerned about the reported memory usage of syslog-ng. But based on the questions and posts in this thread I would report what I saw.

To be honest once syslog-ng loads the memory reported used by it in top never changes. So this isn’t a leak and doesn’t cause any issues that I can see.

Free memory is wasted memory.... I agree!

My bad, I didn't get that from your reply.

 

[Butterfly Bones]

There is a syslog-ng update in today's bunch of Entware updates. I do not have time to fuss if it breaks anything, so I will hold off on updates, and a heads up.

 

[SomeWhereOverTheRainBow]

There is a syslog-ng update in today's bunch of Entware updates. I do not have time to fuss if it breaks anything, so I will hold off on updates, and a heads up.

Yea I am curious if any one has already tested it, or experienced any breakage from it. that is why i am holding off as well.

 

[Dan Ljungberg]

No problems here, just had to restart syslog-ng in Scribe (rs)
Just use "scribe status" at the command line or the "s" option on the scribe menu

 

[Butterfly Bones]

I ran the update today with no issues as stated by others. I did restart syslog-ng (rs) as stated by @Dan Ljungberg and then scribe status (s) that stated that syslog-ng conf was out of sync, and corrected it and synced up just fine, no issues. Humming along now.

 

[cmkelley]

I ran the update today with no issues as stated by others. I did restart syslog-ng (rs) as stated by @Dan Ljungberg and syslog-ng stated that syslog-ng conf was out of sync, so I ran (rs) again and it synced up just fine, no issues. Humming along now.

So, that's poor placement of the update message on my part, and actually just running "scribe status" or "s' from the menu will fix it and restart syslog-ng.

# scribe status
                             _
                          _ ( )
        ___    ___  _ __ (_)| |_      __
      /',__) /'___)( '__)| || '_`\  /'__`\
      \__, \( (___ | |   | || |_) )(  ___/
      (____/`\____)(_)   (_)(_,__/'`\____)
      syslog-ng and logrotate installation
      v2.4_2 (master)  Coded by cynicastic


       checking syslog-ng daemon ... alive.

  checking system for necessary scribe hooks ...

           checking S01syslog-ng ... updated.
          checking service-event ... present.
             checking post-mount ... present.
                checking unmount ... present.
     checking logrotate cron job ... present.
        checking directory links ... present.

  checking syslog-ng configuration ...

    syslog-ng.conf version check ... out of sync! (3.25)
  Checking syslog-ng...              alive.
  Shutting down syslog-ng...              done.
  Starting syslog-ng...              done.
  Restarting uiScribe ...            done.
          syslog-ng.conf version ... updated! (3.26)
     syslog-ng.conf syntax check ... okay!

           scribe installed version: v2.4_2 (master)
              scribe GitHub version: v2.4_2 (master)
                     scribe is up to date!

After the "Restarting uiScribe" line you can see it updates the syslog-ng.conf version.

 

[Butterfly Bones]

So, that's poor placement of the update message on my part, and actually just running "scribe status" or "s' from the menu will fix it and restart syslog-ng.

# scribe status
                             _
                          _ ( )
        ___    ___  _ __ (_)| |_      __
      /',__) /'___)( '__)| || '_`\  /'__`\
      \__, \( (___ | |   | || |_) )(  ___/
      (____/`\____)(_)   (_)(_,__/'`\____)
      syslog-ng and logrotate installation
      v2.4_2 (master)  Coded by cynicastic


       checking syslog-ng daemon ... alive.

  checking system for necessary scribe hooks ...

           checking S01syslog-ng ... updated.
          checking service-event ... present.
             checking post-mount ... present.
                checking unmount ... present.
     checking logrotate cron job ... present.
        checking directory links ... present.

  checking syslog-ng configuration ...

    syslog-ng.conf version check ... out of sync! (3.25)
  Checking syslog-ng...              alive.
  Shutting down syslog-ng...              done.
  Starting syslog-ng...              done.
  Restarting uiScribe ...            done.
          syslog-ng.conf version ... updated! (3.26)
     syslog-ng.conf syntax check ... okay!

           scribe installed version: v2.4_2 (master)
              scribe GitHub version: v2.4_2 (master)
                     scribe is up to date!

After the "Restarting uiScribe" line you can see it updates the syslog-ng.conf version.

Now that you state that I did run (rs) first then (s) and that is what fixed it. I did this a few hours ago and just posted, so I missed how I actually did it until I see your message. Your message placement is fine.

EDIT - also edited earlier post

 

[cmkelley]

There is a syslog-ng update in today's bunch of Entware updates. I do not have time to fuss if it breaks anything, so I will hold off on updates, and a heads up.

The only thing of interest in the release notes is

config version: Made the config version check of the configuration more liberal by accepting version numbers
that had no changes relative to the current version. This means that if you are running 3.26 and the last
semantic change in the configuration was 3.22, then anything between 3.22 and 3.26 (inclusive) is accepted

which now makes updating the config file version number automatically a double edge sword. But I think the risk of them breaking basic functionality with the config file is low enough to just keep doing it.

 

[cmkelley]

For anyone reading who hasn't updated their Entware yet, you don't need to restart syslog-ng through scribe. Just use "scribe status" at the command line or the "s" option on the scribe menu and it will update the config file and figure out on its own that it needs to restart syslog-ng.

 

[elorimer]

Where did you get the release notes? I don't see it on syslog-ng.com and I didn't find it easily on github.

I'm thinking the new group-by function might have some utility in dealing with the startup process.

 

[cmkelley]

Where did you get the release notes? I don't see it on syslog-ng.com and I didn't find it easily on github.

I'm thinking the new group-by function might have some utility in dealing with the startup process.

I googled "syslog-ng 3.26 release notes" and got https://github.com/syslog-ng/syslog-ng/releases as the first hit.

 

[elorimer]

Hah! Interesting that it came through entware before the documentation made it onto syslog-ng.

 

[markusd1984]

My logs stopped working and won't restart as it seems, trying to figure out what's going on. Any thoughts?

FYI - I think my log file sizes still grow too much despite log rotation

 

[elorimer]

1. Your logs aren't rotating often enough. Your last rotation of the messages and syslog-ng log was March 8.
2. You might start by updating scribe and entware.
3. Your syslog-ng.conf is throwing errors. As scribe suggests, do the -Fevd start at the CLI to pinpoint where.

 

[markusd1984]

1. Your logs aren't rotating often enough. Your last rotation of the messages and syslog-ng log was March 8.
2. You might start by updating scribe and entware.
3. Your syslog-ng.conf is throwing errors. As scribe suggests, do the -Fevd start at the CLI to pinpoint where.

Thanks for the quick reply, tried to re-install it, but can't install entware packages for scribe. I think because tmp swap space shows 0 space free, at least during skynet update i got the below error (everything else updated fine).

/dev/sdb1 seems to be full

admin@RT-AC86U-9620:/jffs# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
ubi:rootfs_ubifs         79016     65244     13772  83% /
devtmpfs                220096         0    220096   0% /dev
tmpfs                   220208       600    219608   0% /var
tmpfs                   220208      4060    216148   2% /tmp/mnt
mtd:bootfs                4480      3356      1124  75% /bootfs
tmpfs                   220208      4060    216148   2% /tmp/mnt
mtd:data                  8192       584      7608   7% /data
tmpfs                   220208      4060    216148   2% /tmp
/dev/mtdblock8           49152     15212     33940  31% /jffs
/dev/mtdblock8           49152     15212     33940  31% /www/Main_LogStatus_Content.asp
tmpfs                   220208      4060    216148   2% /www/require/modules/menuTree.js
/dev/sdb1               954792    937644         0 100% /tmp/mnt/swap

 

[elorimer]

Bad USB stick?

 

[cmkelley]

My logs stopped working and won't restart as it seems, trying to figure out what's going on. Any thoughts?

Something in Entware didn't update properly - you have "error while loading shared libraries: /opt/lib/libd1.so.2: unsupported version 4097 of Verneed record" in 2 places above.

FYI - I think my log file sizes still grow too much despite log rotation

Something is very wrong, your messages file has a May 5 2018 date, which is the default firmware date. So your clock may not be set correctly. That would also be why nothing appears to have been written to any log file since April 2nd at 16:04.

syslog-ng not working is a symptom of a much larger problem. The first problem to tackle is ensuring your clock is set correctly.