Scribe scribe - syslog-ng and logrotate installer

[Butterfly Bones]

err, not quite, you missed a "};" (extra LFs added for clarity)

# log ethernet change to /opt/var/log/ethernet.log only

destination d_ethernet {
    file("/opt/var/log/ethernet.log");
};

filter f_ethernet {
    message("eth1") or
    message("eth2") or
    message("eth3") or
    message("eth4") or
    message("eth5") or
    message("eth6") or
    message("eth7") or
    message("eth8") or
    message("br0:");
};

filter f_kernel  { program("kernel") ; };
};

log {
    source(src);
    source(kernel);
    filter(f_ethernet);
    filter(f_kernel);
    destination(d_ethernet);
    flags(final);
};
#eof

Putting the filter statements on separate lines is entirely stylistic.

Thank you again. I just seem to have some block to understand the proper syntax on scripts. I keep hoping if I stay immersed in this, it will eventually get through the clouds the keep me from understanding. That is why I ask first on scripts. Everything else I just charge in and make a mess, then figure out how to clean it up.

EDIT - corrected the above filter list (ethernet filter) to work correctly

 

[Butterfly Bones]

That change and rename back to ethernet breaks syslog-ng.

     Checking syslog-ng daemon... dead.
 syslog-ng NOT running!  Starting syslog-ng ...
 Starting syslog-ng...              done.

I reverted to the z_ethernet filter file and restarted. Now it is alive again. ¯\_(ツ)_/¯

 

[elorimer]

? Don't you have an extra "};" now in the filter f_kernel?

filter f_kernel  { program("kernel") ; };
};

 

[Butterfly Bones]

? Don't you have an extra "};" now in the filter f_kernel?

Which one? 1 or 2 in the code?

# log ethernet change to /opt/var/log/ethernet.log only
destination d_ethernet {
    file("/opt/var/log/ethernet.log");
};
filter f_ethernet {
    message("eth1") or
    message("eth2") or
    message("eth3") or
    message("eth4") or
    message("eth5") or
    message("eth6") or
    message("eth7") or
    message("eth8") or
    message("br0:");
};
filter f_kernel  { program("kernel") ; };  <<=== 1
};  <<== 2
log {
    source(src);
    source(kernel);
    filter(f_ethernet);
    filter(f_kernel);
    destination(d_ethernet);
    flags(final);
};
#eof

 

[elorimer]

One or the other, but not both. You are defining a filter within the {}. syslog-ng sees its configuration has a mismatched } and that's why it doesn't start.

PS. Almost all the installation issues we were wrestling with a few days ago was syslog-ng having this kind of problem. In my case, be cause I had double-defined the messages log.
@Martineau's suggestion to look in syslog-ng's own logs was the place I should have focused.

 

[Butterfly Bones]

One or the other, but not both. You are defining a filter within the {}. syslog-ng sees its configuration has a mismatched } and that's why it doesn't start.

PS. Almost all the installation issues we were wrestling with a few days ago was syslog-ng having this kind of problem. In my case, be cause I had double-defined the messages log.
@Martineau's suggestion to look in syslog-ng's own logs was the place I should have focused.

That was it! Now it works. I have corrected my post above with all the filters to correct and update the ethernet filter. Thank you!

I just looked at the /opt/var/log/syslog-ng.log and see nothing there that looks like an error. It is a little hard to read. (EDIT - reformatted to make it easier to read)

/tmp/home/root# cat /opt/var/log/syslog-ng.log
Apr  7 03:11:12 RT-AC86U-4608 syslog-ng[4219]: 
Log statistics; 
processed='destination(d_syslogng)=0', 
processed='center(received)=2642', 
processed='destination(d_pixelserv)=1230', 
processed='destination(d_vpnfailover)=108', 
processed='destination(d_openvpn)=81', 
processed='center(queued)=2642', queued='global(scratch_buffers_count)=2929167695872', 
processed='source(src)=2637', 
processed='destination(d_wlceventd)=0', 
processed='destination(messages)=29', 
processed='destination(d_ethernet)=22', 
processed='global(payload_reallocs)=1602', 
processed='global(sdata_updates)=0', 
processed='destination(d_skynet)=779', 
processed='destination(d_crash)=357', 
processed='destination(d_chkwan)=36', 
processed='src.internal(src#0)=1', stamp='src.internal(src#0)=1554610272', 
processed='source(kernel)=5', 
processed='global(msg_clones)=0', queued='global(scratch_buffers_bytes)=0', 
processed='global(internal_queue_length)=0'

Is there another log somewhere?

 

[elorimer]

@cmkelley I just updated to 0.7 and got this:

 scribe successfully updated.
/opt/bin/scribe: line 536: g: not found
/opt/bin/scribe: line 537: syntax error: unexpected ";;"

EDIT: nvm. I forced a reinstallation and the problem didn't occur.

 

[elorimer]

Now, for giggles, here is how to send the logs to loggly (unencrypted though).
1. Sign up for the trial account at www.loggly.com. After 14 days it will convert to a free account.
2. Follow the instructions for logs from syslog-ng, basically copy their config file and paste it into /opt/etc/syslog-ng.d as [zero]loggly. You need this to be the first file processed, because using the flags(final) will mess you up. You need messages to be processed here first, and then passed along, so DON"T USE THE FLAG.
3. Log into loggly and get your sending token. Copy it, and paste it into the TOKEN place in the config file. Set the TAG to something that identifies your router.
4. If you are running diversion, the loggly website is likely blocked. Copy the website from the config file, and whitelist it in diversion.
5. Loggly's config file gave me an error if I used their source defintion, so I deleted it and used source(src) instead in the log definition.
6. restart scribe.

If you want to stop sending the logs, just move the config file out of the directory and restart scribe. EDIT: Or rename it with a starting "."

 

[elorimer]

Is there another log somewhere?

In a terminal, enter

syslog-ng -d

You will get a fairly verbose set of messages scrolled out on the terminal. If your terminal doesn't scroll, time for Xshell.

syslog-ng sends its start up errors to stderr; if there is a config error it aborts, and its messages never go to that file.

 

[Butterfly Bones]

In a terminal, enter

syslog-ng -d

You will get a fairly verbose set of messages scrolled out on the terminal. If your terminal doesn't scroll, time for Xshell.

syslog-ng sends its start up errors to stderr; if there is a config error it aborts, and its messages never go to that file.

I cannot find Xshell for my Linux box, but x-term scrolls infinitely, so I am good there.

That "syslog-ng -d" works great! Just copy the file into /opt/etc/syslog-ng.d and run that in a term to get result. You don't even need to restart syslog-ng to get the result, like below.

I've been trying to get Loggly going for ages, now, thanks to your instructions, I am so close. There is an error that is beyond me (script dunce). All syntax seems ok, with my limited understanding, but I need some help again. I have changed the token and tags from what I will use, but the problems (^) are in the same place.

Error parsing config, syntax error, unexpected $undefined, expecting ')' in /opt/etc/syslog-ng.d/0loggly:
2       source s_loggly {
3          system();    # Check which OS & collect system logs
4          internal();    # Collect syslog-ng logs
5       };
6    
7-----> template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [noiowjerojnoin09u2930489jiafdjfioj9250@41058 tag=\"TAG\xzd39p"] $MSG\n");
7----->                                                                                                                                                          ^
8          template_escape(no);
9       };
10  
11      destination d_loggly {
12          tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
Included from /opt/etc/syslog-ng.conf:
10      #   source net function to get udp log messages from local network commented out
11      #   destination log_server function added (commented out) to illustrate sending udp log messages to server on local network
12  
13      @version: 3.19
14      @include "/opt/etc/syslog-ng.d/" # Put any customization files in this directory
15---->
15----> ^
16      options {
17          chain_hostnames(no); # Enable or disable the chained hostname format.
18          create_dirs(yes);
19          keep_hostname(yes); # Enable or disable hostname rewriting.
20          log_fifo_size(256); # The number of messages that the output queue can store.

 

[cmkelley]

@cmkelley I just updated to 0.7 and got this:

 scribe successfully updated.
/opt/bin/scribe: line 536: g: not found
/opt/bin/scribe: line 537: syntax error: unexpected ";;"

EDIT: nvm. I forced a reinstallation and the problem didn't occur.

Right, the syntax errors were in the script you were upgrading from.

 

[elorimer]

I'm not quite following your code. But do a few things to change it like this:

### Syslog-ng Logging Directives for Loggly.com ###
template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [PasteYourTokenRightHere@41058 tag=\"PutWhateverTagYouWantRightHere\" ] $MSG\n");
    template_escape(no);
};
destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));  #<--make sure this is whitelisted
};
log { 
    source(src); 
    destination(d_loggly); 
};
### END Syslog-ng Logging Directives for Loggly.com ###

 

[skeal]

Is putty ok for a terminal?

 

[cmkelley]

It's what I use ...

 

[Butterfly Bones]

Well this is exciting, like I needed more!
Here is a better Loggly Dashboard

 

[cmkelley]

v0.8_0 is up now

• Adjusted the logic for copying example files, /opt/etc/*.d/ files, and conf files.
• Changed the name of the alternate syslog-ng.conf file from -small to -no_scl. The "small" conf file is actually larger than the package file due to large # of comments
• Tried a bit harder to avoid overwriting files user have modified ... still needs a bit more work there I think.
• Got rid of a bash-ism
• Some cosmetic and code flow changes

 

[skeal]

Ok can I now turn on skynet and not see the dropped traffic but still benefit from the stat collecting and if so, can you lead a guy along the way. I want to try this on my AX88U.

 

[Butterfly Bones]

v0.8_0 is up now

• Adjusted the logic for copying example files, /opt/etc/*.d/ files, and conf files.
• Changed the name of the alternate syslog-ng.conf file from -small to -no_scl. The "small" conf file is actually larger than the package file due to large # of comments
• Tried a bit harder to avoid overwriting files user have modified ... still needs a bit more work there I think.
• Got rid of a bash-ism
• Some cosmetic and code flow changes

Gotta love a new code perfectionist!

Upgraded and this is the output, assuming like @elorimer posted earlier, this error from old code ?

scribe installed version: v0.7_0 
            scribe GitHub version: v0.8_0 
 New version available!
 Do you wish to upgrade [y|n] y
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 18461  100 18461    0     0  50578      0 --:--:-- --:--:-- --:--:-- 77242
 scribe successfully updated.
/opt/bin/scribe: line 555: cru l: not found
/opt/bin/scribe: line 556: syntax error: unexpected ")"

 

[Butterfly Bones]

Ok can I now turn on skynet and not see the dropped traffic but still benefit from the stat collecting and if so, ca you lead a guy along the way. I want to try this on my AX88U.

Why of course. All my thrashing in the deep end for the last two days has warmed the water.

 

[cmkelley]

Gotta love a new code perfectionist!

Upgraded and this is the output, assuming like @elorimer posted earlier, this error from old code ?

scribe installed version: v0.7_0
            scribe GitHub version: v0.8_0
 New version available!
 Do you wish to upgrade [y|n] y
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 18461  100 18461    0     0  50578      0 --:--:-- --:--:-- --:--:-- 77242
 scribe successfully updated.
/opt/bin/scribe: line 555: cru l: not found
/opt/bin/scribe: line 556: syntax error: unexpected ")"

I'm starting to think it's because I'm overwriting the script while it's running and it's doing funky things?