Scribe Feature to support AiMesh node(s) log?

[Digimite]

I went in the other direction. Got scribe on router and both nodes. Have nodes forwarding to router (A00remote). Syslog-ng filters putting all of it where it suppossed to go. Debating creating my own filters based on the hostname of the AImiesh nodes (logrotate as well), the part two. This way I can view interactions between all three based on the filters, and then for each AIMesh node individually as well.

Odd thing though, as I copied the filters (cp -p "filter" /opt/etc/syslog-ng-d and logrotate-d from the exampls location) Skynet and spdMerlin stopped logging, had to reinstall them to get it to startup logging again. As if the previous results acessed from the gui didn't exist. All good though, all working, last step was changing the router/AImesh nodes host names to a model-location designation to better tell them apart.

That’s great! Did you have to put entware on the nodes as well? My solution was definitely a quick fix only to get something up and running. Yours could possibly be rolled completely into scribe as an install on node option with a couple of standard filters to be displayed in uiScribe on the router. Something to think about...

 

[roboots21]

So I enabled custom scripts on the Aimesh node, created the services-start file, changed its permissions to execute and called another executable script I had created that contains the following...

PID=`ps | grep "syslogd" | grep -v "grep" | awk '{print $1}'`
`kill "$PID"`
/sbin/syslogd -m 0 -S -l 8 -R 192.168.xx.xx

Probably not very elegant, but it's been years since I've done any coding at all. It did survive the reboot and perform as intended, so that's good. Maybe a start for someone who would like to take it and operationalize it a bit more.

I was able to get this working using your method, albeit manual terminal commands. What is the best way to enable custom scripts on the node? Only way I know of is to use nvram variables to enable remote access via the GUI, but I don't want to change anything while in there risking messing with the configuration of the node. Is there an nvram variable that enables custom scripts to run?

 

[Digimite]

I was able to get this working using your method, albeit manual terminal commands. What is the best way to enable custom scripts on the node? Only way I know of is to use nvram variables to enable remote access via the GUI, but I don't want to change anything while in there risking messing with the configuration of the node. Is there an nvram variable that enables custom scripts to run?

I used this from another post I found on the site and it seemed to work. You have to execute it on the node.

nvram set jffs2_scripts="1"
nvram commit
reboot

 

[aex.perez]

That’s great! Did you have to put entware on the nodes as well? My solution was definitely a quick fix only to get something up and running. Yours could possibly be rolled completely into scribe as an install on node option with a couple of standard filters to be displayed in uiScribe on the router. Something to think about...

Yes, but its impact and footprint are minimal, compared to the convience and the capabilities it opens up. in addition to scribe, added scMerlin as well not because I needed it it just makes it I can restart spefic functions without restarting the nodes. Plus I had a few low profile Samsung 32GB USB 3.1 sticks that match the color of the router so you can't really tell they are there.

To change the host names on the nodes;
nvram set lan_hostname="hostname"
nvram commit
reboot

 

[aex.perez]

Still playing around with log levels and whether to include the feeds before all the filtering as well (by device / 3 more logs, might be too busy). In any case here's the first pass, have scribe, entware, scMerlin and LCD scheduler running on on the two AIMESH nodes, forwarding everything to the router. Scribe on the router setup with additional filers to better focus on specific areas for any interaction between the nodes and router. Not any real impact on the router in terms of CPU or RAM that I've been able to measure. The AC5300's are using a bit more RAM, but as nodes not really doing much to use the RAM they have anyway.

While implementing this, I did run into a few problems with a few scripts on the router in terms of losing logs, having to reinstall, turning logging off than of and so on. Eliminated the GUI for vpnmgr somehow and not even a reinstall brought it back. This was with 386.1 and now running 386.1_2, router and nodes. Post reboot, haven't tried to reinstall vpnmgr but will try again soon.

The expanded roamast log below, you can see entries for AX88U-MAIN (Router) and both AC5300's in the bedrooms, hence the LCD scheduler - While you can turn off the LCD's in AIMESH 2.0 it doesn't carry the setting between a reboot, LCD scheduler provides consistency.

 

[aex.perez]

Still playing around with log levels and whether to include the feeds before all the filtering as well (by device / 3 more logs, might be too busy). In any case here's the first pass, have scribe, entware, scMerlin and LCD scheduler running on on the two AIMESH nodes, forwarding everything to the router. Scribe on the router setup with additional filers to better focus on specific areas for any interaction between the nodes and router. Not any real impact on the router in terms of CPU or RAM that I've been able to measure. The AC5300's are using a bit more RAM, but as nodes not really doing much to use the RAM they have anyway.

While implementing this, I did run into a few problems with a few scripts on the router in terms of losing logs, having to reinstall, turning logging off than of and so on. Eliminated the GUI for vpnmgr somehow and not even a reinstall brought it back. This was with 386.1 and now running 386.1_2, router and nodes. Post reboot, haven't tried to reinstall vpnmgr but will try again soon.

The expanded roamast log below, you can see entries for AX88U-MAIN (Router) and both AC5300's in the bedrooms, hence the LCD scheduler - While you can turn off the LCD's in AIMESH 2.0 it doesn't carry the setting between a reboot, LCD scheduler provides consistency.

View attachment 30800

Quick update, just finished the install of "vpnmgr" after the upgrade/reboot to 386.1_2 and the GUI came back. Thinking back, what might have happened as I was trying to get logging back for some of the addons, where the WebUI page got loaded for vpnmgr may have accidently been overwritten during one of the reinstalls or by my adding of an additional script during all of this. During the reinstall saw this pop up, "Mounted vpnmgr WebUI page as user6.asp" and I distinctly remember it as "user5.asp".

Wonder if there might be a way to list which addons have added a WebUI extension and which user#.asp it is, from the /www directory grabbing the title element of the header, <title>connmon</title> as an example. Maybe as a enhancement to scMerlin?

All good so far...
Let it run a few more days and I'll post up the few changes I made to syslog-ng to make this happen...

 

[Martineau]

Wonder if there might be a way to list which addons have added a WebUI extension and which user#.asp it is, from the /www directory

grep -E "user.*\.asp" /tmp/menuTree.js; cat /www/user/user*.title;df | grep menuTree

 

[aex.perez]

grep -E "user.*\.asp" /tmp/menuTree.js; cat /www/user/user*.title;df | grep menuTree

That does it

Master@AX88U-MAIN:/www# grep -E "user.*\.asp" /tmp/menuTree.js; cat /www/user/user*.title;df | grep menuTree
{url: "user6.asp", tabName: "vpnmgr"},
{url: "user5.asp", tabName: "Skynet"},
{url: "user1.asp", tabName: "scMerlin"},
{url: "user3.asp", tabName: "ntpMerlin"},
{url: "user4.asp", tabName: "spdMerlin"},
{url: "user2.asp", tabName: "connmon"},
scMerlin
connmon
ntpMerlin
spdMerlin
vpnmgr
tmpfs 451792 1516 450276 0% /www/require/modules/menuTree.js

Now I just need to grab the date/time attributes to see when one was changed, updated... From the looks of it, this sort of aligns to the issues I had as 4, 5 are the ones I had to reinstall/play with to get to log again.

 

[Jack Yaz]

I'll add something to scMerlin. Webui mounting is being improved by my scripts but until all other WebUI scripts are updated to take consideration of ensuring exclusive access to a particular file before attempting the tab mount there will still be a risk of failure

 

[Martineau]

Webui mounting is being improved by my scripts but all other WebUI scripts are updated to take consideration of ensuring exclusive access to a particular file before attempting the tab mount there will still be a risk of failure

I believe you meant to say

"Webui mounting is being improved by my scripts but UNTIL all other WebUI scripts are updated to take consideration of ensuring exclusive access to a particular file before attempting the tab mount there will still be a risk of failure"

i.e. any devs that still use your original flawed WEBUI management code in their published scripts need to adopt your corrected methodology.

 

[JGrana]

Tough crowd this evening ;-)

Thank goodness software is “soft”…

 

[pdc]

I spent some time working on all the ideas from this thread, especially the approach by @Digimite, and wanted to condense everything into one post.

Here's my setup:

• Main router: full Merlin + entware + scribe + uiScribe
• AiMesh node: Merlin + scripts enabled, no entware/scribe needed

I took this approach for simplicity, after all, the only change needed on the client is to have syslogd broadcast the logs.

Client​

1. Install Merlin
2. ssh to client (you can find the IP from AiMesh info on the main router web UI)
3. Enable scripts: nvram set jffs2_scripts="1"
4. (Optional) Change hostname: nvram set lan_hostname="hostname"
5. nvram commit
6. Create services-start (see below)
7. chmod +x /jffs/scripts/services-start
8. reboot

After the reboot, log in and use ps w | grep syslogd to confirm the syslogd now has the -R flag.

/jffs/scripts/services-start

#!/bin/sh

PID=`ps | grep "syslogd" | grep -v "grep" | awk '{print $1}'`
`kill "$PID"`

# System logging utility
#
#        -n              Run in foreground
#        -R HOST[:PORT]  Log to HOST:PORT (default PORT:514)
#        -L              Log locally and via network (default is network only if -R)
#        -C[size_kb]     Log to shared mem buffer (use logread to read it)
#        -K              Log to kernel printk buffer (use dmesg to read it)
#        -O FILE         Log to FILE (default:/var/log/messages, stdout if -)
#        -s SIZE         Max size (KB) before rotation (default:200KB, 0=off)
#        -b N            N rotated logs to keep (default:1, max=99, 0=purge)
#        -l N            Log only messages more urgent than prio N (1-8)
#        -S              Smaller output
#        -H NAME         Use NAME as hostname
#        -D              Drop duplicates
#        -f FILE         Use FILE as config (default:/etc/syslog.conf)

# default: /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 6

/sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 6 -L -R <router_IP>

Router​

This assumes syslog-ng/scribe is already installed and configured the way you want it. My approach with scribe was to integrate the AiMesh logs with files I already split out.

Make changes to /opt/etc/syslog-ng.conf as follows, adding lines starting with '+', DO NOT INCLUDE THE PLUS SIGN IN THE FILE :

# if you only want to pass network messages through some syslog-ng filters, uncomment the source line below
# then add "soource(net);" to the log statement in any filter you want to pass network messages through
#source net { udp(ip(192.168.x.y) port(514)); };
+source net { network( transport(udp) ); };

[...]

log {
     source(src);
+    source(net); # uncomment this and "source net" function above to get udp log messages from local network
     destination(messages);
#    destination(log_server); # uncomment this and "destination log_server" function above to send udp log messages to local network
};

The above will include remote messages intermixed with router messages in the main syslog file.

To include remotes in other files, for each file you want to include remote syslog messages, add source(net); to the log statement. In my case, only ethernet and wlceventd were modified.

These files are in /opt/etc/syslog-ng.d, for example ethernet:

log {
     source(src);
+    source(net);
     filter(f_kernel);
     filter(f_ethernet);
     destination(d_ethernet);
     flags(final);
};

After all the changes, run scribe, check status (s) and make sure there are no errors, then reload (rl) the new config files.

That's it!

 

[CaptnDanLKW]

Good job @pdc !

One caveat and a question.

Caveat is that, per Asus guidance (and my experience) if we ever have random issues with AI Mesh, such as after an upgrade, the best practice is to remove the node from the UI and re-add. That's because the graceful removal guarantees a factory reset.

So keep in mind you would need to reconfigure the client.


My question is how to configure syslog-ng to log remote messages into a separate file, such as...
/opt/var/log/messages(NODEhostname)

and for any sub filtering (if I chose), like
/opt/var/log/ethernet(NODEhostname)


and to have uiScribe Log display these new sections.

Thoughts?

 

[Digimite]

Good job @pdc !

One caveat and a question.

Caveat is that, per Asus guidance (and my experience) if we ever have random issues with AI Mesh, such as after an upgrade, the best practice is to remove the node from the UI and re-add. That's because the graceful removal guarantees a factory reset.

So keep in mind you would need to reconfigure the client.


My question is how to configure syslog-ng to log remote messages into a separate file, such as...
/opt/var/log/messages(NODEhostname)

and for any sub filtering (if I chose), like
/opt/var/log/ethernet(NODEhostname)


and to have uiScribe Log display these new sections.

Thoughts?

Great job @pdc and good comments @CaptnDanLKW.

Offhand, I think a script could be written that could handle the setup on the node, I.e check for the status of the jffs to see if it is enabled to run scripts and if not, enable it and reboot, otherwise setup the services-start script to kill the syslogd process and restart it with remote-logging enabled. You could probably operationalize it further and query for input on the remote logging host ip, etc. There are certainly scripting experts on this forum who would be better to get advice from in setting this up than myself.

On the question of creating a separate logfile for entries coming from the remote node(s), following the examples of the different log files already being created by including .conf files in the syslog-ng.d directory should allow the creation of these log files the way you would like and with the filters you want. Should being operative word... You would have to dig into uiScribe a little more to have them displayed and rotated properly, etc.

 

[pdc]

Caveat is that, per Asus guidance (and my experience) if we ever have random issues with AI Mesh, such as after an upgrade, the best practice is to remove the node from the UI and re-add. That's because the graceful removal guarantees a factory reset.

So keep in mind you would need to reconfigure the client.

Good point, I may be referring back to this thread

For a node-specific setup, full credit goes to @JGrana (post #3)

To log content to a node-specific file, try something like the files below. Also, comment out the "source(net);" line in /opt/etc/syslog-ng.conf (that I had you add) so the messages don't end up in your main log file.

Don't forget a rule for log rotate, as @Digimite mentioned. The format looks pretty straightforward, probably start with the one for "messages" and update to match your new remote node log file. Okay, so I created an example for that too for completeness.

In /opt/etc/syslog-ng.d/meshnode

# MeshNode - log all AiMesh logs to /opt/var/log/MeshNode.log

destination d_MeshNode {
    file("/opt/var/log/MeshNode.log");
};

log {
    source(net);
    destination(d_MeshNode);
};

In /opt/etc/logrotate.d/meshnode:

/opt/var/log/MeshNode.log{
    minsize 1024k
    daily
    rotate 9
    postrotate
        /usr/bin/killall -HUP syslog-ng
    endscript
}

 

[CaptnDanLKW]

Thanks again @pdc
I've never been good at coding or scripting from scratch but I'm quite adept at modifying them once I see the matrix.
I got the view I was looking for.

Had to use alphabetizing logic to make the other logs appear at the bottom and in the order I wanted.

One issue is that editing the logs to see in the web 'General Configuration' of uiScribe doesn't work, gets confused and I cant enable some logs. If I use the command line, it sticks.

/opt/etc/syslog.d/zmeshSystem

# MeshNode - log all AiMesh logs to /opt/var/log/zmeshSystem.log

destination d_zmeshSystem {
    file("/opt/var/log/zmeshSystem.log");
};

log {
    source(net);
    destination(d_zmeshSystem);
};

and /opt/etc/syslog.d/zmeshwlceventd

# put wlceventd Assoc/ReAssoc/Disassoc messages into /opt/var/log/zmeshwlceventd.log

destination d_zmeshwlceventd {
    file("/opt/var/log/zmeshwlceventd.log");
};

filter f_zmeshwlceventd {
    ( ( program("WLCEVENTD") or
    program("wlceventd") ) and
    ( message("ssoc") or
    message("uth") ) ) or
    ( program("syslog") and
    message("wlceventd") );
};

log {
    source(net);
    filter(f_zmeshwlceventd);
    destination(d_zmeshwlceventd);
    flags(final);
};

 

[CaptnDanLKW]

Well, actually all is not well with that above config (there's also an additional /opt/etc/syslog.d/zmeshethernet filter I didn't bother pasting in).

Thinking about the logic of how the filters work, I don't see a way to essentially nest filters of filters, which is what I want. As it stands now, the filter for 'all' logs from the node go into their own file and my additional filters duplicate them as they are put into their own files.

 

[pdc]

Well, actually all is not well with that above config (there's also an additional /opt/etc/syslog.d/zmeshethernet filter I didn't bother pasting in).

Thinking about the logic of how the filters work, I don't see a way to essentially nest filters of filters, which is what I want. As it stands now, the filter for 'all' logs from the node go into their own file and my additional filters duplicate them as they are put into their own files.

You are so close! It turns out since you want the general zmeshSystem.log to be the default, you need to name it so it is last in alphabetical order. That's because the @include in syslog.conf loads in alphabetical order. Normally that doesn't matter, but in your case the zmeshSystem filter is grabbing and logging everything, and then your other filters run. Since the more specific filters have flags(final) in them, when they are triggered they will stop processing any other filters. So if you rename your zmeshSystem.log to zzmeshSystem.log (two 'z') then it should work the way you want.

 

[MvW]

I'm confused. I read through the thread twice and it's unclear to me which method requires entware and scribe and installed on the node and which one does not? I just want one logfile (/opt/var/log/messages from the (single) node I have) to be accessible on the main router through uiScribe in a separate logfile. Can someone please lay out the steps I need to take, preferably as lightweight as possible. I can always try the entware/Scribe way at a later time, if I feel up to it and see a need for it.

 

[CaptnDanLKW]

I'm confused. I read through the thread twice and it's unclear to me which method requires entware and scribe and installed on the node and which one does not? I just want one logfile (/opt/var/log/messages from the (single) node I have) to be accessible on the main router through uiScribe in a separate logfile. Can someone please lay out the steps I need to take, preferably as lightweight as possible. I can always try the entware/Scribe way at a later time, if I feel up to it and see a need for it.

None require entware & scribe on the node. The default syslogd built into AsusWRT, including all forks (Merlins, Johns, etc) all support sending to a remote node.

The ongoing discussion has more to do with how we (I) want to have the logs separated and represented as their own section in uiScribe.

@pdc post https://www.snbforums.com/threads/feature-to-support-aimesh-node-s-log.69868/post-680304 #32 in this thread is perfect for you and don't read further.

All your remote nodes will send their messaged to your primary router, your primary router, running syslog-ng and configured as he describes will gobble them up and drop them into the same /opt/var/log/messages. Your log will look like this

Good Luck