Scribe scribe - syslog-ng and logrotate installer

[Butterfly Bones]

@Butterfly Bones, now that you have free time, perhaps you would describe what you've done with loggly? I haven't dug into it myself but I know you spent some time customizing your charts and I'd like to go to school on what you did.

Well now that my free trial has expired, it is not as nice and useful now for a home user of syslog-ng only. I would actually pay a small monthly to use it, but $79 USD monthly and they only bill annually, so that is not worth it to me. This is the only page I can access now. I *think* it can be useful if I spend the time and learn how to use the search options there. I just spent too much time jousting with the syslog-ng windmills recently.

You need to sign up for an account, that is free 14 days with full access.
https://www.loggly.com/signup/

I started here.
https://www.loggly.com/docs/logging-setup/

Get your customer token and save it for the next step.
https://www.loggly.com/docs/customer-token-authentication-token/

This page for syslog-ng setup to get a loggly filter (name it 0loggly so it gets read first, that is numeral zero) and all others get sent to Loggly. It goes in /opt/entware/etc/syslog-ng,d with the other filter files.
https://www.loggly.com/docs/syslog-ng-manual-configuration/

This is my 0loggly filter (replace YOUR_CUSTOMER_TOKEN_HERE and YOUR_TAG). I remember checking and this is from the one you posted earlier in this thread.

### Syslog-ng Logging Directives for Loggly.com ###
template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [YOUR_CUSTOMER_TOKEN_HERE0@41058 tag=\"YOUR_TAG\" ] $MSG\n");
    template_escape(no);
};
destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log {
    source(src);
    destination(d_loggly);
};
### END Syslog-ng Logging Directives for Loggly.com ###

Restart scribe and then log into your Loggly account. I used the Dashboard (it has preconfigured ones) and Charts along the top banner, now I only have Search without subscribing.

It is pretty simple, unless you get tripped up by all the scripting symbols like I do.
That is all I remember doing, if anyone does this and has questions, let me know.

 

[elorimer]

I think the free tier continues but you are limited to a certain amount of data. That's one of the reasons my loggly config inverts the skynet and pixelserv filters, so I don't send that stuff.

 

[Butterfly Bones]

I think the free tier continues but you are limited to a certain amount of data. That's one of the reasons my loggly config inverts the skynet and pixelserv filters, so I don't send that stuff.

Yes that screen shot above of Search is the only screen with free tier, here is the free account details.

Subscription Plan Summary
Plan: Lite
Daily volume limit: 200MB
Log retention: 7 days
Price per month: $0.00

That max I have sent is only 7 MB, by their accounting, with full Skynet debug and invalid packet logging and Pixelserv at log level 4 so I cannot imagine getting close to 200MB. From what I can find if one exceeds 200 MB it is just dropped. I need to learn to use the loggly search to see if it gives me a better option to watch Skynet logging.

 

[joe scian]

@Butterfly Bones, I have the sense you are fighting this too hard. If you let scribe do its thing, you should end up in the right place.

1. Uninstall scribe. Check to see that the /opt/etc/init.d/S01syslog-ng and rc.func.syslog-ng files are gone. Check to see the /opt/etc/logrotate.d and /syslog-ng.d directories are gone. Delete the logs in /opt/var/log EXCEPT for the Dnsmasq logs. Killall syslog-ng.
2. Now, copy the curl instruction from post 1 and install scribe. This is to insure you start with the corrected 0.9.1.
3. run "pidof syslog-ng". You should get two numbers. Run "pidof klogd" and "pidof syslogd"; neither should return a number.
4. In firewall settings, make sure that the custom location is now /opt/var/log/syslog-o.
5. From examples, copy the skynet and openvpn configs back to /opt/etc/syslog-ng.d. Make sure they do not have a kernel source. I don't think you need to bother with logrotate until you've got the rest working.
6. In s01syslog-ng, see that the only source statement that isn't commented out is the src {system(); internal();} statement.
7. Scribe restart. Repeat #3.

Here is the sequence of how the skynet log statements work (I think):

1. Iptables uses the kernel logger to generate its BLOCKED statements. Those go to the /proc/kmsg file. Syslog-ng picks those up using the system() function, adds the timestamp and the host, and then they fall to the skynet config, where they are sent to syslog-o and further processing of that message stops. Other kernel or system log messages are picked up, don't hit one of the separate configs, and get written to /opt/var/log/messages. That message file is symlinked to /tmp/syslog, which appears in the webgui. If skynet is pointed to /tmp/syslog for any reason, this breaks, and almost everything about skynet stats breaks, because what happens next doesn't happen.
2. Hourly, cron runs the /jffs/scripts/firewall save command. That runs the Purge_Logs function in skynet, which takes all the Blocked messages and copies them to the /skynet/skynet.log file and the events.log file. Then it deletes all of those messages from the syslog-0 file. Then, it runs stats, because anytime a command is run, stats is run. Stats reads the skynet.log file, and then prints that hourly stats line to the system logger using the function Print_Log. That message gets picked up by the system() function, and then is sent by syslog-ng to the skynet-0 file. (I confess I'm having a hard time understanding exactly how and what Purge_Logs is doing here, because of the tildes and the !d stuff, and how Print_Log gets called). If you've changed the skynet log location, Purge_Logs will run on the file it reads from the symlink to messages, which doesn't have any skynet messages, and then destroy the symlink, which means the webui doesn't ever see messages again, even though syslog-ng is still running. It also means no lines are written to the skynet.log file and events.log file, and no stats are there to be generated. That will continue until you do scribe restart, which will through a bit of cleverness by @cmkelley, notice that the symlink is gone and reform it. But at the hour it will get broken again, because it doesn't change the skynet setting.

I am not getting an hourly stat message either, but I think that is because one isn't being generated by skynet, not anything to do with this. I'm not sure why.

Interesting - I am getting an hourly stats message in my skynet-0.log file

 

[Butterfly Bones]

Interesting - I am getting an hourly stats message in my skynet-0.log file

I think it is the $#@%& AC86U, it has been a bear to get some things going, and has a known shut down only when one issues a reboot, via script or webGUI. It was a PITA with syslog-ng and ntpd by kvic (I gave up) and Stubby had some issues. Diversion has some niggles that thelonelycoder had to solve. Danged AC86U, love it and hate it at the same time.

 

[elorimer]

Interesting - I am getting an hourly stats message in my skynet-0.log file

Yes, I think you are supposed to. The hourly save instruction strips everything else out, runs the stats, pumps that log message out through the system logger, it goes into syslog-ng that strips it out to skynet-o.log.

 

[joe scian]

Noticed this entry twice in syslog-ng.log file at 00:05 - I dont think its an issue since all working fine.

pr 18 00:05:01 RT-AC5300 syslog-ng[1694]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_s

 

[Butterfly Bones]

Yes, I think you are supposed to. The hourly save instruction strips everything else out, runs the stats, pumps that log message out through the system logger, it goes into syslog-ng that strips it out to skynet-o.log.

.. except on my AC86U....

 

[Butterfly Bones]

Of course the AC86U is going to make me eat my words!

Finally the [SAVE] entries are being generated by Skynet. I can not see them in the Skynet search script, but they are in skynet-0.log. They go by so fast, I found them with grep and then figured out how to search for them in Loggly, so win-win.

 

[cmkelley]

.. except on my AC86U....

FWIW, the hourly report stripping is not consistent on my 86U either. Not sure how that's possible, the codebase for all that should be common across all models.

 

[cmkelley]

Here are my observations of installing and configuring Scribe 0.9_1 yesterday after about 24 hours without it. I used the temporary disable Skynet before install. Install was smooth, I had downloaded the new filter files from GitHub first and used those to update my own to match the new scheme. First thing I noticed is that I had no webGUI syslog entries at all, none over a 5 hour span. Here is the result of "cat /opt/var/log/messages" ( deleted all entries prior to install).

Apr 16 14:13:26 kernel: klogd: exiting
Apr 16 14:13:26 syslogd exiting
Apr 16 14:27:18 RT-AC86U-4608 rc_service[23687]: service 23687:notify_rc restart_firewall
Apr 16 14:27:18 RT-AC86U-4608 custom_script[1]: Running /jffs/scripts/service-event (args: restart firewall) - max timeout = 120s
Apr 16 14:27:18 RT-AC86U-4608 nat[1]: apply nat rules (/tmp/nat_rules_eth0_eth0)
Apr 16 14:27:18 RT-AC86U-4608 custom_script[1]: Running /jffs/scripts/nat-start
Apr 16 14:27:18 RT-AC86U-4608 (install_stubby.sh)[23782]: 23777 Starting Script Execution (checkipv6)
Apr 16 14:27:18 RT-AC86U-4608 custom_script[1]: Running /jffs/scripts/firewall-start (args: eth0)
Apr 16 14:32:33 RT-AC86U-4608 kernel:
Apr 16 14:32:33 RT-AC86U-4608 kernel:
Apr 16 14:48:55 RT-AC86U-4608 kernel:
Apr 16 14:48:56 RT-AC86U-4608 kernel:
Apr 16 15:07:31 RT-AC86U-4608 kernel:
Apr 16 15:07:31 RT-AC86U-4608 kernel:
Apr 16 15:26:53 RT-AC86U-4608 kernel:
Apr 16 15:26:53 RT-AC86U-4608 kernel:
Apr 16 15:43:00 RT-AC86U-4608 kernel:
Apr 16 15:43:00 RT-AC86U-4608 kernel:
Apr 16 16:05:20 RT-AC86U-4608 kernel:
Apr 16 16:05:21 RT-AC86U-4608 kernel:
Apr 16 16:10:33 RT-AC86U-4608 kernel:
Apr 16 16:10:33 RT-AC86U-4608 kernel:
Apr 16 16:15:49 RT-AC86U-4608 kernel:
Apr 16 16:15:49 RT-AC86U-4608 kernel:
Apr 16 16:18:26 RT-AC86U-4608 kernel:
Apr 16 16:18:26 RT-AC86U-4608 kernel:
Apr 16 16:42:13 RT-AC86U-4608 kernel:
Apr 16 16:42:13 RT-AC86U-4608 kernel:
Apr 16 17:03:18 RT-AC86U-4608 kernel:
Apr 16 17:03:18 RT-AC86U-4608 kernel:
Apr 16 17:31:28 RT-AC86U-4608 kernel:
Apr 16 17:31:28 RT-AC86U-4608 kernel:
Apr 16 17:37:39 RT-AC86U-4608 kernel:
Apr 16 17:37:39 RT-AC86U-4608 kernel:
Apr 16 18:03:01 RT-AC86U-4608 kernel:
Apr 16 18:03:01 RT-AC86U-4608 kernel:
Apr 16 18:25:12 RT-AC86U-4608 kernel:
Apr 16 18:25:12 RT-AC86U-4608 kernel:
Apr 16 18:47:55 RT-AC86U-4608 kernel:
Apr 16 18:47:55 RT-AC86U-4608 kernel:
Apr 16 19:03:53 RT-AC86U-4608 kernel:
Apr 16 19:03:53 RT-AC86U-4608 kernel:
Apr 16 19:22:00 RT-AC86U-4608 kernel:
Apr 16 19:22:00 RT-AC86U-4608 kernel:
Apr 16 19:26:34 RT-AC86U-4608 kernel:
Apr 16 19:26:34 RT-AC86U-4608 kernel:
Apr 16 19:27:25 RT-AC86U-4608 kernel:
Apr 16 19:27:25 RT-AC86U-4608 kernel:
Apr 16 19:50:32 RT-AC86U-4608 kernel:
Apr 16 19:50:32 RT-AC86U-4608 kernel:

Is this because of this in v.0.9_0 changelog that I see nothing after "kernel:"?

• related, the source kernel directive has been removed from all syslog-ng files.

The main issue for me is Skynet. As stated I temporarily halted it, in settings and checked the syslog location setting and changed it back from "/opt/var/log/skynet-0.log" to "/tmp/syslog". As I stated earlier, this was an attempt to leave Skynet logs alone and only scrape most other syslog entries with syslog-ng. Bottom line it did not work. Skynet logging of skynet.log and events.log in "/mnt/<USB_name>/skynet/" stopped. Here is what I saw in the events.log, all stopped after install of Scribe.

Apr 16 00:00:03 Skynet: [#] 139383 IPs (+0) -- 1710 Ranges Banned (+0) || 1772 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 01:00:03 Skynet: [#] 139383 IPs (+0) -- 1710 Ranges Banned (+0) || 1890 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 02:00:03 Skynet: [#] 139383 IPs (+0) -- 1710 Ranges Banned (+0) || 2019 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 03:00:03 Skynet: [#] 139383 IPs (+0) -- 1710 Ranges Banned (+0) || 2163 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 03:25:22 Skynet: [#] 147960 IPs (+8577) -- 1764 Ranges Banned (+54) || 2216 Inbound -- 0 Outbound Connections Blocked! [banmalware] [21s]
Apr 16 04:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 2274 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 05:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 2376 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 06:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 2537 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 07:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 2651 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 08:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 2768 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 09:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 2908 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 10:00:38 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 3013 Inbound -- 0 Outbound Connections Blocked! [stats] [2s]
Apr 16 11:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 3129 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 12:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 3252 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 13:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 3375 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 14:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 3498 Inbound -- 0 Outbound Connections Blocked! [save] [3s]

Like joe scian I got the errors running logrotate manually, I know now from the above post by cmkelley that it is understandable so I will not post the captures of those results.

Bottom line for me is the Skynet logging issue, and as I understand that does not occur for others? I know in the past there were many problems getting the syslog-ng by kvic (and ntpd by kvic as well), on the RT-AC86U with the HND chipset. I uninstalled scribe last night to get Skynet logging back.

Apr 16 13:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 3375 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 14:00:03 Skynet: [#] 147960 IPs (+0) -- 1764 Ranges Banned (+0) || 3498 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
===> uninstall scribe and reboot
Apr 16 20:00:05 Skynet: [#] 149020 IPs (+0) -- 1831 Ranges Banned (+0) || 1 Inbound -- 0 Outbound Connections Blocked! [start] [25s]
Apr 16 21:00:03 Skynet: [#] 149020 IPs (+0) -- 1831 Ranges Banned (+0) || 16 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 22:00:04 Skynet: [#] 149020 IPs (+0) -- 1831 Ranges Banned (+0) || 136 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 16 23:00:03 Skynet: [#] 149020 IPs (+0) -- 1831 Ranges Banned (+0) || 261 Inbound -- 0 Outbound Connections Blocked! [save] [2s]
Apr 17 00:00:03 Skynet: [#] 149020 IPs (+0) -- 1831 Ranges Banned (+0) || 358 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 17 01:00:03 Skynet: [#] 149020 IPs (+0) -- 1831 Ranges Banned (+0) || 475 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 17 02:00:03 Skynet: [#] 149020 IPs (+0) -- 1831 Ranges Banned (+0) || 579 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 17 02:25:22 Skynet: [#] 150849 IPs (+1829) -- 1791 Ranges Banned (-40) || 613 Inbound -- 0 Outbound Connections Blocked! [banmalware] [22s]

This is not a primary issue for Scribe development and testing. I'm trying to figure out a method to keep at least minimal Skynet logging and use Scribe because I really want to use it for log segregation and to continue to contribute to testing. If someone has suggestions, they would be welcome. Don't let solving my issue get in the way of more important Scribe troubleshooting and progress. Thanks!

I'm getting tons of blank entries as well. I've got no time in the next couple days to chase it down.

It may be related to removing the kernel directive, however if you 'scribe config' and scroll through, the src directive expands to:

# syslog-ng gets messages from the system and syslog-ng (internal)
source src {

#Start Block source generator system
channel {
    source {
channel {
    source { unix-dgram("/dev/log" so_rcvbuf(8192) flags(syslog-protocol));
 };
    rewrite { set("${.unix.pid}" value("PID") condition("${.unix.pid}" ne "")); };
};
file("/dev/kmsg" program-override("kernel") flags(kernel) format(linux-kmsg) keep-timestamp(no));

    }; # source
channel {
  channel {
    parser {

#Start Block parser generator app-parser

channel {
    junction {
    };
}
#End Block parser generator app-parser
;

#Start Block parser generator app-parser

channel {
    junction {
    };
}
#End Block parser generator app-parser
;
    };
    flags(final);
  };
  channel { flags(final); };
};
}; # channel

#End Block source generator system
;
    internal();
};

So it expands to include /dev/kmsg rather than /proc/kmsg, but I don't know if /dev/kmsg is necessarily 'wrong'. Even after googling a bit I'm not clear about the difference.

 

[tomsk]

Noticed this entry twice in syslog-ng.log file at 00:05 - I dont think its an issue since all working fine.

pr 18 00:05:01 RT-AC5300 syslog-ng[1694]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_s

Yes this is normal. If you read from a file and it gets recreated, syslog-ng is smart enough to remember and carries on where it left off. For example i grab the dnsmasq logs to send to loggly, but when Diversion rotates the logs, syslog-ng handles it without me having to do anything with it.

Apr 18 05:20:12 RT-AC68U-4690 Diversion: rotated dnsmasq log files, from /opt/share/diversion/file/rotate-logs.div
Apr 18 05:20:22 RT-AC68U-4690 syslog-ng[18070]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_sd_curpos(/tmp/mnt/Entware/entware/var/log/dnsmasq.log)', stored_inode='13 $

From the admin guide

The syslog-ng application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When syslog-ng is restarted, it records the position of the last sent log message in the /opt/syslog-ng/var/syslog-ng.persist file, and continues to send messages from this position after the restart./QUOTE]

 

[joe scian]

Well now that my free trial has expired, it is not as nice and useful now for a home user of syslog-ng only. I would actually pay a small monthly to use it, but $79 USD monthly and they only bill annually, so that is not worth it to me. This is the only page I can access now. I *think* it can be useful if I spend the time and learn how to use the search options there. I just spent too much time jousting with the syslog-ng windmills recently.

You need to sign up for an account, that is free 14 days with full access.
https://www.loggly.com/signup/

I started here.
https://www.loggly.com/docs/logging-setup/

Get your customer token and save it for the next step.
https://www.loggly.com/docs/customer-token-authentication-token/

This page for syslog-ng setup to get a loggly filter (name it 0loggly so it gets read first, that is numeral zero) and all others get sent to Loggly. It goes in /opt/entware/etc/syslog-ng,d with the other filter files.
https://www.loggly.com/docs/syslog-ng-manual-configuration/

This is my 0loggly filter (replace YOUR_CUSTOMER_TOKEN_HERE and YOUR_TAG). I remember checking and this is from the one you posted earlier in this thread.

### Syslog-ng Logging Directives for Loggly.com ###
template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [YOUR_CUSTOMER_TOKEN_HERE0@41058 tag=\"YOUR_TAG\" ] $MSG\n");
    template_escape(no);
};
destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log {
    source(src);
    destination(d_loggly);
};
### END Syslog-ng Logging Directives for Loggly.com ###

Restart scribe and then log into your Loggly account. I used the Dashboard (it has preconfigured ones) and Charts along the top banner, now I only have Search without subscribing.

It is pretty simple, unless you get tripped up by all the scripting symbols like I do.
That is all I remember doing, if anyone does this and has questions, let me know.

Hi again - trying to get loggly to work -
did you also update the syslog-ng.conf file asper below ? And is it correct to remove this from the the 0loggly filter

source s_loggly {
    system();    # Check which OS & collect system logs
    internal();    # Collect syslog-ng logs
};

Does this get appended to syslog-ng.conf?

### Syslog-ng Logging Directives for Loggly.com ###
source s_loggly {
    system();    # Check which OS & collect system logs
    internal();    # Collect syslog-ng logs
};

template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [TOKEN@41058 tag=\"TAG\" ] $MSG\n");
    template_escape(no);
};

destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};

log {
    source(s_loggly);
    destination(d_loggly);
};
### END Syslog-ng Logging Directives for Loggly.com ###

 

[elorimer]

I changed my loggly configuration to just use src, and not to send skynet or pixelserv messages:

filter f_loggly1 { not program("pixelserv-tls"); };
filter f_loggly2 { not program("Skynet"); };

destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};

log {
    source(src);
    filter(f_loggly1);
    filter(f_loggly2);
    destination(d_loggly);

I left out my LogglyFormat template from what I pasted, but remember you need to fill in the TOKEN and TAG stuff. Also, I made my loggly configuration its own file under syslog-ng.d, and renamed it to 0loggly so it would operate first. If you put it in syslog-ng.conf it has to be before the messages log statement.

 

[elorimer]

So it expands to include /dev/kmsg rather than /proc/kmsg, but I don't know if /dev/kmsg is necessarily 'wrong'. Even after googling a bit I'm not clear about the difference.

My googling turned up this (from around 3 years ago, so I don't know if it applies to this kernel:

The kernel generates output to an in-memory buffer.
 Application softwares can access this in two ways. 
The logging subsystem usually accesses it as a pseudo-FIFO named /proc/kmsg. 
This source of log information cannot usefully be shared amongst log readers, because it is read-once. 
If multiple processes share it, they each get only a part of the kernel log data stream. 
It is also read-only.

The other way to access it is the newer /dev/kmsg character device. 
This is a read-write interface that is shareable amongst multiple client processes.
 If multiple processes share it, they all read the same complete data stream, unaffected by one another. 
If they open it for write access, they can also inject messages into the kernel's log stream, as if they were generated by the kernel.

/proc/kmsg and /dev/kmsg provide log data in a non-RFC-5424 form.

 

[tomsk]

My googling turned up this (from around 3 years ago, so I don't know if it applies to this kernel:

The kernel generates output to an in-memory buffer.
 Application softwares can access this in two ways.
The logging subsystem usually accesses it as a pseudo-FIFO named /proc/kmsg.
This source of log information cannot usefully be shared amongst log readers, because it is read-once.
If multiple processes share it, they each get only a part of the kernel log data stream.
It is also read-only.

The other way to access it is the newer /dev/kmsg character device.
This is a read-write interface that is shareable amongst multiple client processes.
 If multiple processes share it, they all read the same complete data stream, unaffected by one another.
If they open it for write access, they can also inject messages into the kernel's log stream, as if they were generated by the kernel.

/proc/kmsg and /dev/kmsg provide log data in a non-RFC-5424 form.

looks like we should be using /dev/kmsg rather than /proc/kmsg as a kernel source then ?

 

[elorimer]

I'm using the system() function as my source, and before we went down the road of killing klogd, I was getting garbling. If system() is reading from dev/kmsg this kernel doesn't seem to be handling /dev/kmsg any different.

EDIT: On my 56U, when I do scribe config, my source expands to read /proc/kmsg. So my experience is not going to be representative. Setting up a file read on /dev/kmsg might be a different way to go, although if we leave klogd running we might end up with duplicates. Also, syslog-ng -Fevd includes

[2019-04-18T12:42:12.214112] system(): Enabling Linux kernel log device; device='/proc/kmsg', format='(null)'

 

[tomsk]

I'm using the system() function as my source, and before we went down the road of killing klogd, I was getting garbling. If system() is reading from dev/kmsg this kernel doesn't seem to be handling /dev/kmsg any different.

my understanding is that klogd reads from /proc/kmsg and not /dev/kmsg

Kernel logging:

The kernel logs messages to an in-memory buffer. There are three ways to access them.

1. dmesg - usually runs on demand
2. /proc/kmsg - read only + read once mechanism
3. /dev/kmsg - character device interface, read + write, can be read multiple times

User space logging:

User space normally use libc library function called syslog which in turn logs the user space messages to UNIX domain socket /dev/log. There can be multiple such sockets as defined and commonly understood between user space components and syslogd

klogd:

A daemon that reads the messages from either /proc/kmsg or calls sys_syslogd. It provides kernel log data streams to another daemon called syslogd

syslogd:

syslogd daemon listens on a number of UNIX domain sockets. It receives messages from klogd. It connects to an AF_LOCAL data gram socket called /dev/log to read them. It can read from other UNIX domain sockets as commonly agreed upon with user space applications. syslogd can write the messages to any log file or UDP ports for remote hosts
/QUOTE]

 

[elorimer]

So just to beat on this a bit, my 86U and 56U are both on kernel 2.6.36, and @cmkelley has a 86U which I guess is on 4.1. So that might be why his system() expands differently than mine does.

 

[Butterfly Bones]

Hi again - trying to get loggly to work -
did you also update the syslog-ng.conf file asper below ? And is it correct to remove this from the the 0loggly filter

source s_loggly {
    system();    # Check which OS & collect system logs
    internal();    # Collect syslog-ng logs
};

That looks like the one from the Loggly site. I looked at it, but then used the one @elorimer posted. Here it is named 0loggly - place it in syslog.d directory:

### Syslog-ng Logging Directives for Loggly.com ###
template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [PLACE_YOUR_TOKEN_HERE@41058 tag=\"YOUR_TAG\" ] $MSG\n");
    template_escape(no);
};
destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log { 
    source(src); 
    destination(d_loggly); 
};
### END Syslog-ng Logging Directives for Loggly.com ###

Does this get appended to syslog-ng.conf?

### Syslog-ng Logging Directives for Loggly.com ###
source s_loggly {
    system();    # Check which OS & collect system logs
    internal();    # Collect syslog-ng logs
};

template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [TOKEN@41058 tag=\"TAG\" ] $MSG\n");
    template_escape(no);
};

destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};

log {
    source(s_loggly);
    destination(d_loggly);
};
### END Syslog-ng Logging Directives for Loggly.com ###

When I run scribe config or the syslog-ng command "syslog-ng -Fevd" I see it, yes.

Have you succeeded with Loggly? I was out early this morning, and just now reading messages.