Scribe Feature to support AiMesh node(s) log?

[unknownz]

Not sure if this has been mentioned before but was wondering if this is something possible,

1. main node with Scribe which acts as the central log collector
2. mesh node(s) push the log to the main node using the default log daemon

Right now checking the mesh node(s) log requires logging into the node itself through a shell access which is a hassle; a central managed log would be easier

 

[Jack Yaz]

Assuming syslog-ng supports a remote host as a destination, I think this is probably possible. It may be as simple as installing scribe and manually replacing the log filters to send everything to tue "main" collector.

 

[JGrana]

I just did this remote logging this morning.
As @Jack Yaz said, use scribe, @cmkelley has 90% of it setup already.

You will need to install Entware on your AiMesh nodes and main Router (recommend using amtm). Then install scribe on all the devices. The router will be the syslog server - the nodes will be clients.


On your main router (usually 192.168.1.1) you would edit /opt/etc/syslog-ng.conf and uncomment these three lines (shown here already uncommented):

source net {
udp(ip(192.168.1.1) port(514));
};

and also uncomment this line in the log definition a few lines below the above:

log {
source(src);
source(net); # uncomment this and "source net" function above to get udp log messages from local network
destination(messages);
# destination(log_server); # uncomment this and "destination log_server" function above to send udp log messages to local network
};

For each mesh node, you will need to create a conf file that resides in the /opt/etc/syslog-ng.d directory.

Here is a simple one that I am using for my node:

# MeshBasement - log all AI Mesh (Basement) logs to /opt/var/log/MeshBasement.log

destination d_MeshBasement {
    file("/opt/var/log/MeshBasement.log");
};

log {
    source(net);
    destination(d_MeshBasement);
#    flags(final);
};

Be sure to create a log file in /opt/var/log with the name you defined in destination - in this case MeshBasement.log. Also make SURE to have permissions for addmin only:

chmod go-rwx /opt/var/log/MeshBasement.log (or whatever you named the file)

I would then recommend you run scribe from the shell and select "s" (check status). This will check your conf files.
Assuming ok, select rl to have scribe re-load the conf files.

The clients (nodes are much simpler). ssh into the Mesh node and again make sure scribe is installed.

First, edit the client/nodes /opt/etc/syslog-ng.log file and uncomment the lines @cmkelley has as log_server. Here is what the lower part of syslog-ng.conf looks like after uncomment:

# if you only want to pass network messages through some syslog-ng filters, uncomment the source line below
# then add "soource(net);" to the log statement in any filter you want to pass network messages through
#source net { udp(ip(192.168.x.y) port(514)); };

# set the filename for the default log file - anything not filtered out will end up here
destination messages { file("/opt/var/log/messages"); };

# to send log messages to the local network, uncomment the destination line below
# then add "destination(log_server);" to the log statement in any filter you want to pass network messages through
destination log_server { udp("192.168.1.1" port(515)); };

log {
    source(src);
#    source(net); # uncomment this and "source net" function above to get udp log messages from local network
    destination(messages);
   log messages to local network destination(log_server); # uncomment this and "destination log_server" function above to send udp
};

Last, copy the file /opt/share/syslog-ng/examples/A00remote to /opt/etc/syslog-ng.d.

Edit the copy in /opt/share/syslog-ng.d and make sure to comment the line "destination log_server { udp("192.168.x.y" port(514)); };" - we have already defined the log_server in /opt/etc/syslog-ng.conf

Leave the rest as is.

This will result in all system log messages to be stored both locally on the node and also sent to the Router at 192.168.1.1 running syslog-ng.

BTW, also make sure to setup the logrotate file that corresponds to the nodes definition file.
For example, in my case the definition file is BasementMesh and the log file (defined in it) is BasementMesh.log.

So, I have a file called BasementMesh in /opt/etc/logrotate.d. Here is what it has:

/opt/var/log/BasementMesh.log {
    postrotate
        /usr/bin/killall -HUP syslog-ng
    endscript
}

Again, run scribe, check status and make sure no errors - the reload (rl) the new conf files.

As a reminder - if you have installed uiScribe (and if you haven't - do so - highly recommended) you should run uiScribe on the Router and allow your new node to be shown for display (option 1 - customize list of log displays and make sure your node is enabled in the list)

 

[CaptnDanLKW]

FWIW, I did this some time back but did not install entware on the remote node - instead, on the node I did a
nvram set log_ipaddr=(router_running_syslog-ng)
nvram commit
restarted syslogd
This does NOT survive reboots. I suppose I could have merlinized it by enabling jffs, enabled startup scripts and put the commands in one of the scripts but never bothered.

The hardest part is the syslog-ng.conf part, then deciding what and where and how you want the events logged (all from remote in one file vs. each event source combined in the scribe filter buckets, etc. I got myself wrapped around the axle, screwed everything up so I started fresh and never got around to remote logging again.

 

[unknownz]

FWIW, I did this some time back but did not install entware on the remote node - instead, on the node I did a
nvram set log_ipaddr=(router_running_syslog-ng)
nvram commit
restarted syslogd
This does NOT survive reboots. I suppose I could have merlinized it by enabling jffs, enabled startup scripts and put the commands in one of the scripts but never bothered.

The hardest part is the syslog-ng.conf part, then deciding what and where and how you want the events logged (all from remote in one file vs. each event source combined in the scribe filter buckets, etc. I got myself wrapped around the axle, screwed everything up so I started fresh and never got around to remote logging again.

thanks all for the input and feedback

the approach by @CaptnDanLKW is something im looking at i.e. using the default syslogd daemon; reason being it's more lightweight and removes the need of having a usb stick in the mesh node(s)

 

[roboots21]

Hmmm .. I have been interested in seeing the logs on my AiMesh nodes and this may just be the ticket. That being said, this would be a great addition to the scribe!

 

[elorimer]

FWIW, I did this some time back but did not install entware on the remote node - instead, on the node I did a
nvram set log_ipaddr=(router_running_syslog-ng)
nvram commit
restarted syslogd
This does NOT survive reboots. I suppose I could have merlinized it by enabling jffs, enabled startup scripts and put the commands in one of the scripts but never bothered.

Interesting. I just noticed that the webgui equivalent of this is not in 386.1. It used to be in on the Administration|system page, didn't it?

destination log_server { udp("192.168.1.1" port(515)); };

This should be port 514, shouldn't it?

Also, not sure whether syslog-ng will do this for you anyway, but I use a template to collect my logs from my NAS and backup NAS, with all the messages going into one file but labeled by the sender:

destination d_TrueNAS {
    file("/opt/var/log/truenas-main.log" template("${R_DATE} ${HOST} ${MSGHDR}${MESSAGE}\n"));
};

So you could do something like that so that the nodes all go to one file but are labeled by node.

On your main router (usually 192.168.1.1) you would edit /opt/etc/syslog-ng.conf and uncomment these three lines (shown here already uncommented):
source net {
udp(ip(192.168.1.1) port(514));
};

My net source to accept from anyplace on the lan is:

source net {
    udp(ip(0.0.0.0) port(514));
};

Which is the default, binding to all addresses and seems to collect my NAS messages. Not sure about this now since I see this is now obsoleted and should be replaced with the network() drivers.

 

[Maverickcdn]

Interesting. I just noticed that the webgui equivalent of this is not in 386.1. It used to be in on the Administration|system page, didn't it?

Is this not available on the log tab on nodes?

 

[elorimer]

Is this not available on the log tab on nodes?

I just have my main router, no nodes. On 386.1 with scribe and uiScribe, this option doesn't appear. I guess that makes sense, since syslogd isn't running.

Assuming this is a uiScribe feature and not a 386.1 feature (I can't check this), then this could work for sending the node logs to syslog-ng on the main router if it is configured to catch them.

 

[Digimite]

Is this not available on the log tab on nodes?
View attachment 30322

Node access via gui is not directly supported although there are ways around it. From what I've read, making changes while using these workarounds is not recommended. That being said, setting nvram variables, using scripts and/or altering config files could accomplish the same thing.

 

[Digimite]

Currently receiving this message when scribe is reloading after I change the config to receive messages from the node ...

Error binding socket; addr='AF_INET(192.168.0.XX:514)', error='Cannot assign requested address (99)'
Error initializing message pipeline; plugin_name='udp', location='/opt/etc/syslog-ng.conf:36:14'
Error initializing new configuration, reverting to old config;
Configuration reload finished;

I had set log_ipaddr on the node to the router and the log_port to 514 but I guess I'm missing something else...

As well, I'm only using the default syslog on the node.

 

[elorimer]

The syslog-ng.conf scribe uses includes network sources that have been obsoleted. Dunno why, but try changing the source net to read like this:

source net {
network(
transport(udp)
);
};

And then see what happens. (This defaults to port 514).

 

[Digimite]

The syslog-ng.conf scribe uses includes network sources that have been obsoleted. Dunno why, but try changing the source net to read like this:

source net {
network(
transport(udp)
);
};

And then see what happens. (This defaults to port 514).

Will give that a try and let you know how it goes. Thanks!

 

[Digimite]

The syslog-ng.conf scribe uses includes network sources that have been obsoleted. Dunno why, but try changing the source net to read like this:

source net {
network(
transport(udp)
);
};

And then see what happens. (This defaults to port 514).

Works nicely thanks! Now to have it last through a reboot of the node and my OCD for log messages will be fulfilled ...

 

[aex.perez]

Having this would cover a blind spot. Both ways presented here, I've not been successful with. got to sit and focus one weekend to see what I'm doing wrong. Adding Scribe on each AIMESH node, the modification of SYSLOG-NG and a log (or logs based on filtered events) for the node(s) either based on defined name, AIMESH location lable, or default host name of router, would be a big plus. I would take it a little further and mess with the filters to get better message classification/visibility and help facilitate correlation of events. Then on to being able to select/preset which of the logs are expended or collapsed when opening the page, and the expanded ones to be prioritized at the top of the page when opened, would be the icing on the proverbial cake. One can dream...

 

[JGrana]

Interesting. I just noticed that the webgui equivalent of this is not in 386.1. It used to be in on the Administration|system page, didn't it?

This should be port 514, shouldn't it?

Also, not sure whether syslog-ng will do this for you anyway, but I use a template to collect my logs from my NAS and backup NAS, with all the messages going into one file but labeled by the sender:

destination d_TrueNAS {
    file("/opt/var/log/truenas-main.log" template("${R_DATE} ${HOST} ${MSGHDR}${MESSAGE}\n"));
};

So you could do something like that so that the nodes all go to one file but are labeled by node.

Yes, 514 is the default. You can use a range of port numbers. Being lazy (and learning syslog_ng syntax) I have 2 ports on the router assigned (514 and 515) one for each node pointing to a unique log file for each node. One node sends to 514 the other to 515.
I get separate windows (and downloads if I want) on uiScribe.

Your recommendation is more elegant! I just might try that method. Thanks!

 

[elorimer]

Adding Scribe on each AIMESH node, the modification of SYSLOG-NG and a log (or logs based on filtered events) for the node(s) either based on defined name, AIMESH location lable, or default host name of router, would be a big plus.

I don't have nodes so this is just a thought experiment. As I think about it, it seems to me the most straightforward approach is to have the node use the existing remote log feature to send log messages to port 514 on the main router. Unless there is some other reason for having entware on the node, I don't see a value in adding syslog-ng or scribe on the node. I haven't found, for example, a need to send via tcp rather than udp in this situation.

Then, on the main router, you can use syslog-ng to slice and dice the messages however you want. You can filter each node into separate log files, or particular messages across all nodes into one file, or repeat messages into several log files, etc. The one thing to be mindful of is to substitute the network() driver for the udp() driver.

Then with uiScribe you can choose which of these to show in the webgui.

 

[Digimite]

I don't have nodes so this is just a thought experiment. As I think about it, it seems to me the most straightforward approach is to have the node use the existing remote log feature to send log messages to port 514 on the main router. Unless there is some other reason for having entware on the node, I don't see a value in adding syslog-ng or scribe on the node. I haven't found, for example, a need to send via tcp rather than udp in this situation.

Then, on the main router, you can use syslog-ng to slice and dice the messages however you want. You can filter each node into separate log files, or particular messages across all nodes into one file, or repeat messages into several log files, etc. The one thing to be mindful of is to substitute the network() driver for the udp() driver.

Then with uiScribe you can choose which of these to show in the webgui.

I was successful in sending log messages back to the router from the node but had to kill and restart the syslog daemon with the -R option (remote log server) using the router IP. I didn't have to install scribe or entware. I haven't tried any filtering through syslog-ng yet and still have to figure out if there's a more elegant way of getting this to work on the node that will survive a reboot. It's a work in progress right now.

 

[Digimite]

So I enabled custom scripts on the Aimesh node, created the services-start file, changed its permissions to execute and called another executable script I had created that contains the following...

PID=`ps | grep "syslogd" | grep -v "grep" | awk '{print $1}'`
`kill "$PID"`
/sbin/syslogd -m 0 -S -l 8 -R 192.168.xx.xx

Probably not very elegant, but it's been years since I've done any coding at all. It did survive the reboot and perform as intended, so that's good. Maybe a start for someone who would like to take it and operationalize it a bit more.

 

[aex.perez]

I went in the other direction. Got scribe on router and both nodes. Have nodes forwarding to router (A00remote). Syslog-ng filters putting all of it where it suppossed to go. Debating creating my own filters based on the hostname of the AImiesh nodes (logrotate as well), the part two. This way I can view interactions between all three based on the filters, and then for each AIMesh node individually as well.

Odd thing though, as I copied the filters (cp -p "filter" /opt/etc/syslog-ng-d and logrotate-d from the exampls location) Skynet and spdMerlin stopped logging, had to reinstall them to get it to startup logging again. As if the previous results acessed from the gui didn't exist. All good though, all working, last step was changing the router/AImesh nodes host names to a model-location designation to better tell them apart.