Scribe scribe - syslog-ng and logrotate installer

[joe scian]

I am sorry but how do i use this script. The menu system does not display any logs, my web guis does not show any logs. What is the point of this if I can not as a stupid user even find any ways to look at logs?

Even a stupid user surely can read the first post?
In addition to syslog-ng and logrotate, scribe also installs syslog-ng and logrotate configuration files to send syslog-ng's own logs, logrotate logs, WLCEVENTD logs, and (hopefully most) crash logs to their own separate files. All log files will be in /opt/var/log.

You should install WinSCP as an easy way to see the logs in that directory.

 

[akb]

Even a stupid user surely can read the first post?
In addition to syslog-ng and logrotate, scribe also installs syslog-ng and logrotate configuration files to send syslog-ng's own logs, logrotate logs, WLCEVENTD logs, and (hopefully most) crash logs to their own separate files. All log files will be in /opt/var/log.

Yes, but
1.) why dont i still see logs in the webgui, like nothing happend.
2.)why cant the menu have options on which logs you want to read?

 

[akb]

I feel like a better approach would be to tag the log line with the source so that when we use a remote log server with the built-in feature to separate the logs
[WLCEVENTD] eth1 mac connect/disccont
[Skynet] ips blocked
[Diversion] stats updated
etc...

 

[Butterfly Bones]

I feel like a better approach would be to tag the log line with the source so that when we use a remote log server with the built-in feature to separate the logs
[WLCEVENTD] eth1 mac connect/disccont
[Skynet] ips blocked
[Diversion] stats updated
etc...

Like this?

This is how I have a AC86u/syslog-ng/Skynet search. I can turn line wrap off and get this.

 

[cmkelley]

I am sorry but how do i use this script. The menu system does not display any logs, my web guis does not show any logs. What is the point of this if I can not as a stupid user even find any ways to look at logs?

The idea of scribe is not that of a log viewer, it is an installation program for the syslog-ng and logrotate programs (available via Entware) that allow breaking the single system log file into multiple dedicated logs for later processing or viewing, and in the case of logrotate, rotating the logs when they get larger than some desired value for ease of searching and keeping some amount of history.

The firmware system logging daemon (syslogd) is very limited in what it can do, installing syslog-ng allows people to do more with their logs. Some people use external services such a loggy to help them sort through the logs, some might use syslog-ng to push the router logs to a centralized log server, or perhaps use the router as a centralized log server. syslog-ng is incredibly powerful, but has a very steep learning curve. What I supply with scribe is just the basics of getting syslog-ng (& logrotate) installed and working, which is unfortunately not as simple as just installing it from entware. For instance, scribe takes care of ensuring that when syslogd or klogd are restarted by the system (which is fortunately predictable), they are terminated because by their nature they conflict with syslog-ng.

The webgui should still show the main system log, but many messages will be removed from it. To view all the logs, you need to ssh into your router, then "cd /opt/var/log" All of the files in that directory should be log files that are most conveniently viewed with the firmware "less" utility (e.g. "less messages"). Assuming you didn't copy any of the scripts from /opt/share/syslog-ng/examples, you may only have a couple logs there, such as crash (all the dcd crashes) or syslog-ng.log. Your main log will be called "messages", but that should be repeated in the webgui.

If you are looking for a log viewer, I'm sorry but this isn't it. This is simply a script to install and maintain a more sophisticated logging daemon than the one that is included in the router firmware.

 

[cmkelley]

I feel like a better approach would be to tag the log line with the source so that when we use a remote log server with the built-in feature to separate the logs
[WLCEVENTD] eth1 mac connect/disccont
[Skynet] ips blocked
[Diversion] stats updated
etc...

Well, that's not really the point of syslog-ng, and I wouldn't really have a clue how to make it do that, although I suspect it could using a template of some sort. If you're already using a remote log server, syslog-ng can push your logs to that server, although I couldn't advise on the best way to accomplish what you need.

Although it's more than a bit cryptic, the documentation for syslog-ng is at: https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition/3.20 ... note that if you download the pdf version of the adminstrator's guide, it's on the order of 900 pages!

 

[joe scian]

The idea of scribe is not that of a log viewer, it is an installation program for the Entware syslog-ng and logrotate programs that allow breaking the single system log file into multiple dedicated logs for later processing or viewing, and in the case of logrotate, rotating the logs when they get larger than some desired value for ease of searching and keeping some amount of history.

The firmware system logging daemon (syslogd) is very limited in what it can do, installing syslog-ng allows people to do more with their logs. Some people use external services such a loggy to help them sort through the logs, some might use syslog-ng to push the router logs to a centralized log server, or perhaps use the router as a centralized log server. syslog-ng is incredibly powerful, but has a very steep learning curve. What I supply with scribe is just the basics of getting syslog-ng (& logrotate) installed and working, which is unfortunately not as simple as just installing it from entware. For instance, scribe takes care of ensuring that when syslogd or klogd are restarted by the system (which is fortunately predictable), they are terminated because by their nature they conflict with syslog-ng.

The webgui should still show the main system log, but many messages will be removed from it. To view all the logs, you need to ssh into your router, then "cd /opt/var/log" All of the files in that directory should be log files that are most conveniently viewed with the firmware "less" utility (e.g. "less messages"). Assuming you didn't copy any of the scripts from /opt/share/syslog-ng/examples, you may only have a couple logs there, such as crash (all the dcd crashes) or syslog-ng.log. Your main log will be called "messages", but that should be repeated in the webgui.

If you are looking for a log viewer, I'm sorry but this isn't it. This is simply a script to install and maintain a more sophisticated logging daemon than the one that is included in the router firmware.

Hi cmkelley - you stated above :-

Your main log will be called "messages", but that should be repeated in the webgui.
I dont see the messages log replicated in the webgui - Its certainly there in /opt/var/log. Should we really see it in the GUI too ?

 

[cmkelley]

Hi cmkelly - you stated above :-

Your main log will be called "messages", but that should be repeated in the webgui.
I dont see the messages log replicated in the webgui - Its certainly there in /opt/var/log. Should we really see it in the GUI too ?

It definitely should be visible in the System Log - General Log tab. There is not a separate tab or anything for it, the /tmp/syslog.log file (which is what is displayed on that page) is symlinked to messages on installation and the symlink is re-created when the router is re-booted.

 

[joe scian]

It definitely should be visible in the System Log - General Log tab. There is not a separate tab or anything for it, the /tmp/syslog.log file (which is what is displayed on that page) is symlinked t
o messages on installation and the symlink is re-created when the router is re-booted.

Hmmmm- well definitely nothing in my GUI system log - general log tab from a fresh install of Scribe v 2.0 on an RT-AC5300 and I dont have /tmp/syslog.log file

 

[cmkelley]

Hmmmm- well definitely nothing in my GUI system log - general log tab from a fresh install of Scribe v 2.0 on an RT-AC5300

Huh. I'm at a loss as to why that could be. Sounds like maybe the symlink isn't getting created but I don't understand why it wouldn't be ... unless the AC5300 uses a file other than /tmp/syslog.log for its system log?

I need to get to bed, but if you'd like, you can use the "d" option on the menu to create the debug data and PM the file to me, and I'll take a look when I have a chance, which might not be until tomorrow night Los Angeles time - I have an all-day customer meeting tomorrow.

@RMerlin, sorry to be a pain, but can you confirm all of the routers you support look at /tmp/syslog.log when viewing the system log in the webgui System Log - General Log tab?

 

[thelonelycoder]

IMPORTANT: If you use Diversion, you must update Diversion to v1.4.0 (or later) for new installations.

That would be Diversion 4.1.x or later.

 

[joe scian]

Huh. I'm at a loss as to why that could be. Sounds like maybe the symlink isn't getting created but I don't understand why it wouldn't be ... unless the AC5300 uses a file other than /tmp/syslog.log for its system log?

I need to get to bed, but if you'd like, you can use the "d" option on the menu to create the debug data and PM the file to me, and I'll take a look when I have a chance, which might not be until tomorrow night Los Angeles time - I have an all-day customer meeting tomorrow.

@RMerlin, sorry to be a pain, but can you confirm all of the routers you support look at /tmp/syslog.log when viewing the system log in the webgui System Log - General Log tab?

tmp/syslog.log was definitely there as well as tmp/syslog.log-1 before installing scribe . Before installing scribe I also have /jffs/syslog.log and /jffs/syslog.log-1. As soon as scribe is installed tmp/syslog.log and tmp/syslog.log-1 are no longer there. However /jffs/syslog.log and /jffs/syslog.log-1 still remain. However these files are not being updated - they remain as a snapshot of syslog.log at the time that scribe was installed.

 

[joe scian]

HI cmkelley
Here is the debug output

Please select an option: d

 gathering debugging information ...ls: /tmp/syslog*: No such file or directory
 taring the output ... done.

 Debug output stored in /opt/tmp/scribe_debug.log, please review this file
 to ensure you understand what information is being disclosed.

 Tarball of debug output is /opt/tmp/scribe_debug.log.tar.gz

 Press [Enter] to continue:
### Scribe Version: v2.0_0 (master)
###  Local Scribe md5: 982df81757563620d47e9978cf21287b
### GitHub Version: v2.0_0 (master)
### GitHub Scribe md5: 982df81757563620d47e9978cf21287b
### Router: RT-AC5300 (armv7l)
### Firmware Version: ASUSWRT-Merlin 384.12

### check running log processes:
 8125 redacted  8448 S    {syslog-ng} supervising syslog-ng
 8126 redacted 15320 S    syslog-ng
12131 nobody    1368 S    dnsmasq --log-async
12132 redacted  1332 S    dnsmasq --log-async
21392 redacted  4492 S    grep log

### check crontab:
5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#

### directory check:
drwxrwxrwx    2 redacted root             0 Jun  4 18:45 /jffs/syslog.log
drwxrwxrwx    2 redacted root             0 Jun  4 18:45 /jffs/syslog.log-1

### top output:
Mem: 277752K used, 237432K free, 2056K shrd, 2792K buff, 26840K cached
CPU:  0.0% usr  4.5% sys  0.0% nic 95.4% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.12 0.12 0.32 1/138 21405
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 8126  8125 redacted S    15320  2.9   1  0.0 syslog-ng
  361     1 redacted S    12512  2.4   0  0.0 networkmap --bootwait
 8125     1 redacted S     8448  1.6   0  0.0 {syslog-ng} supervising syslog-ng
 4623     1 redacted S     7856  1.5   0  0.0 minidlna -f /etc/minidlna.conf -r
 4641  4640 redacted S N   7856  1.5   0  0.0 minidlna -f /etc/minidlna.conf -r
 4640  4623 redacted S     7856  1.5   1  0.0 minidlna -f /etc/minidlna.conf -r

### *log references in top:
 8126  8125 redacted S    15320  2.9   1  0.0 syslog-ng
 8125     1 redacted S     8448  1.6   0  0.0 {syslog-ng} supervising syslog-ng
21408 21171 redacted S     4492  0.8   0  0.0 grep log
12131     1 nobody   S     1368  0.2   0  0.0 dnsmasq --log-async
12132 12131 redacted S     1332  0.2   0  0.0 dnsmasq --log-async

### init.d directory:
-rwxr-xr-x    1 redacted root           250 Jun  4 18:45 S01syslog-ng
-rwxrwxrwx    1 redacted root           731 Jun  4 18:42 S77ntpd
-rwxr-xr-x    1 redacted root          1524 Jun  4 18:31 S80pixelserv-tls
-rw-r--r--    1 redacted root          2822 May 25 02:49 rc.func
-rw-r--r--    1 redacted root          2602 Jun  4 18:30 rc.func.div
-rw-r--r--    1 redacted root           999 Jun  4 18:45 rc.func.syslog-ng
-rwxr-xr-x    1 redacted root           966 May 25 02:49 rc.unslung

### contents of S01syslog-ng
#!/bin/sh

ENABLED=yes
PROCS=syslog-ng
ARGS=""
PREARGS=""
DESC=$PROCS
PATH=/opt/sbin:/opt/bin:/opt/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

. /opt/etc/init.d/rc.func.syslog-ng # added by scribe
/opt/tmp/scribe_debug.log

 

[thelonelycoder]

amtm now also supports scribe

 

[elorimer]

@joe scian: The webgui displays in the system log window whatever is in /tmp/syslog.log is. Scribe turns that into a symlink to /opt/var/log messages when /opt/etc/init.d/S01syslog-ng is run by calling /opt/etc/init.d/rc.func.syslog-ng. rc.func.syslog-ng is where the work is done and something in that process is messed up for you.

1. S01syslog-ng should end with this line:

. /opt/etc/init.d/rc.func

and I don't see that in your debug.
2. Check that rc.func.syslog-ng says this:

#!/bin/sh
#tof
kill_logger (){
    # kill any/all running klogd and/or syslogd
    [ -n "$( pidof klogd )" ] && killall klogd
    [ -n "$( pidof syslogd )" ] && killall syslogd
    # webGUI System Log = /tmp/syslog.log
    if [ ! -L "/tmp/syslog.log" ]
    then
        cat /tmp/syslog.log >> /opt/var/log/messages
        rm -f /tmp/syslog.log /tmp/syslog.log-1
        ln -s /opt/var/log/messages /tmp/syslog.log
        echo "### Top of Log File ###" >> /tmp/syslog.log-1
    fi
    # make /jffs/syslog.log and log-1 directories if not already
    # prevents system log saver from writing to jffs
    if [ ! -d "/jffs/syslog.log" ] || [ ! -d "/jffs/syslog.log-1" ]
    then
        rm -rf /jffs/syslog.log /jffs/syslog.log-1
        mkdir /jffs/syslog.log /jffs/syslog.log-1
    fi
}
# export timezone if not already set
[ -z "$TZ" ] && export TZ=$( cat /etc/TZ )
PRECMD="kill_logger"
# enabling the below can be useful when having problems,
# but fills up the logfile fast
#ARGS="-v"
#eof

From what I can tell, the fact that you have files, not directories, named /jffs/syslog.log suggests that kill_logger is not being run. But it doesn't look like klogd or syslogd are running, which suggests it is.

 

[cmkelley]

That would be Diversion 4.1.x or later.

Fixed. Thanks!

 

[akb]

I knew something was up, if the webgui was always there i would have been fine. I just thought it was odd that the webgui logs were gone and forced me to log into the router to parse logs.

 

[joe scian]

@joe scian: The webgui displays in the system log window whatever is in /tmp/syslog.log is. Scribe turns that into a symlink to /opt/var/log messages when /opt/etc/init.d/S01syslog-ng is run by calling /opt/etc/init.d/rc.func.syslog-ng. rc.func.syslog-ng is where the work is done and something in that process is messed up for you.

1. S01syslog-ng should end with this line:

. /opt/etc/init.d/rc.func

and I don't see that in your debug.
2. Check that rc.func.syslog-ng says this:

#!/bin/sh
#tof
kill_logger (){
    # kill any/all running klogd and/or syslogd
    [ -n "$( pidof klogd )" ] && killall klogd
    [ -n "$( pidof syslogd )" ] && killall syslogd
    # webGUI System Log = /tmp/syslog.log
    if [ ! -L "/tmp/syslog.log" ]
    then
        cat /tmp/syslog.log >> /opt/var/log/messages
        rm -f /tmp/syslog.log /tmp/syslog.log-1
        ln -s /opt/var/log/messages /tmp/syslog.log
        echo "### Top of Log File ###" >> /tmp/syslog.log-1
    fi
    # make /jffs/syslog.log and log-1 directories if not already
    # prevents system log saver from writing to jffs
    if [ ! -d "/jffs/syslog.log" ] || [ ! -d "/jffs/syslog.log-1" ]
    then
        rm -rf /jffs/syslog.log /jffs/syslog.log-1
        mkdir /jffs/syslog.log /jffs/syslog.log-1
    fi
}
# export timezone if not already set
[ -z "$TZ" ] && export TZ=$( cat /etc/TZ )
PRECMD="kill_logger"
# enabling the below can be useful when having problems,
# but fills up the logfile fast
#ARGS="-v"
#eof

From what I can tell, the fact that you have files, not directories, named /jffs/syslog.log suggests that kill_logger is not being run. But it doesn't look like klogd or syslogd are running, which suggests it is.

Hi Elorimer and cmkelley
My rc.func.syslog-ng below exactly like yours and yes the last line of S01syslog-ng does end correctly.
@elorimer "From what I can tell, the fact that you have files, not directories, named /jffs/syslog.log suggests that kill_logger is not being run. But it doesn't look like klogd or syslogd are running, which suggests it is."

Actually Im sorry they are directories - not files - neither have anything in them but that must have happened overnight. They were files as at yesterday with the contents as per my post above - ie a snapshot of syslog as and when scribe gets installed.

As at this morning there are no /tmp/syslog.log or /tmp/syslog.log-1 files and nothing appears in webgui. I must say that even when I had this going about a month ago when I was corresponding with you and butterflybones regarding setting up loggly - when i had everything working with loggly there was never any WEBGUI messages. I thought that was intended operation at that time and i didnt report it.

#!/bin/sh
#tof

kill_logger (){
    # kill any/all running klogd and/or syslogd
    [ -n "$( pidof klogd )" ] && killall klogd
    [ -n "$( pidof syslogd )" ] && killall syslogd

    # webGUI System Log = /tmp/syslog.log
    if [ ! -L "/tmp/syslog.log" ]
    then
        cat /tmp/syslog.log >> /opt/var/log/messages
        rm -f /tmp/syslog.log /tmp/syslog.log-1
        ln -s /opt/var/log/messages /tmp/syslog.log
        echo "### Top of Log File ###" >> /tmp/syslog.log-1
    fi

    # make /jffs/syslog.log and log-1 directories if not already
    # prevents system log saver from writing to jffs
    if [ ! -d "/jffs/syslog.log" ] || [ ! -d "/jffs/syslog.log-1" ]
    then
        rm -rf /jffs/syslog.log /jffs/syslog.log-1
        mkdir /jffs/syslog.log /jffs/syslog.log-1
    fi
}

# export timezone if not already set
[ -z "$TZ" ] && export TZ=$( cat /etc/TZ )

PRECMD="kill_logger"
# enabling the below can be useful when having problems,
# but fills up the logfile fast
#ARGS="-v"

#eof

#!/bin/sh

ENABLED=yes
PROCS=syslog-ng
ARGS=""
PREARGS=""
DESC=$PROCS
PATH=/opt/sbin:/opt/bin:/opt/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

. /opt/etc/init.d/rc.func.syslog-ng # added by scribe

. /opt/etc/init.d/rc.func

 

[elorimer]

Such a puzzle! No /tmp/syslog.log in the directory at all. If you make the symlink in a terminal what happens:

ln -s /opt/var/log/messages /tmp/syslog.log

 

[elorimer]

they are directories - not files - neither have anything in them

This is a kludge to fool the firmware. By making these names directories, the firmware doesn't try to make jffs copies of the log. The fact that nothing is in the directories is good.