[DEVEL] Asuswrt-Merlin Remote Log Server (Syslog & Syslog-ng Synergy) to Centralize Primary Router & AiMesh Node System Logs

[garycnew]

ATTENTION: It has come to my attention that there is an established Asuswrt-Merlin Add-on in AMTM called scribe and a user interface for it called uiscribe that takes the native Asuswrt-Merlin Syslog implementation to the next level, which is the recommended approach over this tutorial.

The following is an Asuswrt-Merlin Remote Log Server (Syslog & Syslog-ng Synergy) to Centralize Primary Router & AiMesh Node System Logs Tutorial gleaned from Existing Posts in this Forum (RE: References), but taking a slightly different approach by making use of the existing Syslog process, Remote Logging to Syslog-ng, maintaining the existing Syslog (/tmp/syslog.log) location, and using iptables to Drop Duplicate Logging of Primary Router.

Requirements/Assumptions:

1. An Asuswrt-Merlin Compatible Router (i.e., Asus RT-AC66U)
2. Asuswrt-Merlin Compatible Firmware (i.e., 384.19)
3. Formated JFFS Partition and Enabled JFFS Custom Scripts and Configs
4. Capable of Modifying NVRAM Settings
5. Capable of Editing the following User Scripts:
/jffs/configs/firewall-start

### Primary Router: Update/Install Entware Syslog-ng ###

# ssh admin@192.168.0.1
# opkg update
# opkg install syslog-ng
Installing syslog-ng (3.32.1-1) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/syslog-ng_3.32.1-1_armv7-2.6.ipk
Installing libiconv-full (1.16-1) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/libiconv-full_1.16-1_armv7-2.6.ipk
Installing libintl-full (0.21-2) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/libintl-full_0.21-2_armv7-2.6.ipk
Installing libattr (2.5.1-3) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/libattr_2.5.1-3_armv7-2.6.ipk
Installing glib2 (2.68.1-3) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/glib2_2.68.1-3_armv7-2.6.ipk
Installing libdbi (0.9.0-5) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/libdbi_0.9.0-5_armv7-2.6.ipk
Installing libjson-c (0.15-2) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/libjson-c_0.15-2_armv7-2.6.ipk
Installing libcurl (7.77.0-1) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/libcurl_7.77.0-1_armv7-2.6.ipk
Configuring libiconv-full.
Configuring libintl-full.
Configuring libattr.
Configuring glib2.
Configuring libdbi.
Configuring libjson-c.
Configuring libcurl.
Configuring syslog-ng.

### (Optional) Remove Syslog-ng & Dependencies ###

# opkg remove syslog-ng glib2 libiconv-full libintl-full libattr libdbi libjson-c libcurl
# opkg remove logrotate libpopt

### Primary Router: Edit syslog-ng.conf to Change file() Location & Uncomment source() to Open Port 514 ###

# vi /opt/etc/syslog-ng.conf
…
        #file("/opt/var/log/messages");                           
        file("/tmp/syslog.log");
…
    # uncomment this line to open port 514 to receive messages
    source(s_network);

### Primary Router: Start Syslog-ng ###

# /opt/etc/init.d/S01syslog-ng start
Starting syslog-ng...              done.

### Primary Router: Verify Syslog-ng is Listening on the Any Address (0.0.0.0) and UDP Port 514 ###

# netstat -anp | grep syslog-ng
tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN      2064/syslog-ng
tcp        0      0 0.0.0.0:6514            0.0.0.0:*               LISTEN      2064/syslog-ng
tcp        0      0 0.0.0.0:601             0.0.0.0:*               LISTEN      2064/syslog-ng
udp        0      0 0.0.0.0:514             0.0.0.0:*                           2064/syslog-ng
udp        0      0 127.0.0.1:514           0.0.0.0:*                           2064/syslog-ng

### Primary Router: Verify Syslog's Pre-WebUI Run-Time Configuration ###

# ps w | grep -i syslog
32650 admin     1416 S    /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 6

### Primary Router: In the Asuswrt-Merlin WebUI under Advance Settings > System Log > General Log for Remote Log Server enter 192.168.0.1 (Private Gateway Address) and for Port enter 514 (Syslog-ng UDP Port) then Click "Apply" ###

### Primary Router: Verify Syslog's Post-WebUI Run-Time Configuration ###

# ps w | grep -i syslog
  110 admin     1424 S    /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 6 -R 192.168.0.1:514 -L -H gnutech-wap01-CA04B43-C
2062 admin     8540 S    {syslog-ng} supervising syslog-ng
2064 admin    22936 S    syslog-ng

### Primary Router: Add iptables Rule to Drop Duplicate Logging of Primary Router to Syslog-ng over UDP Port 514

# iptables -I INPUT -i lo -p udp -s 192.168.0.1 --dport 514 -j DROP

### Primary Router: Create/Edit firewall-start Script to Persist Drop Duplicate Logging of Primary Router to Syslog-ng ###

# touch /jffs/scripts/firewall-start
# chmod 755 /jffs/scripts/firewall-start

# vi /jffs/configs/firewall-start
#!/bin/sh

sleep 10  # During the boot process firewall-start may run multiple times so this is required

# Drop Duplicate Logging of Primary Router to Syslog-ng over UDP Port 514
# iptables -I INPUT -i lo -p udp -s 192.168.0.1 --dport 514 -j DROP

### Primary Router: Section for Potential Implementation of Syslog Filters ###

### Primary Router: Section for Potential Implementation of Logrotate ###

### AiMesh Node: Verify NVRAM log_ Settings ###

# ssh admin@192.168.0.11
# nvram show | grep -i "^log_"
log_ipaddr=192.168.0.1
log_port=514
log_size=256
log_level=7
log_path=/jffs

### AiMesh Node: Change NVRAM log_level from 7=Debug to 6=Notice to be less Verbose ###

# nvram set log_level=6

### AiMesh Node: Verify NVRAM lan_hostname Setting ###

# nvram show | grep -i lan_hostname
lan_hostname=RT-AC66U_B1-3C73

### AiMesh Node: Change NVRAM lan_hostname to something more readable in the Syslogs ###

nvram set lan_hostname=Office-3C73

### AiMesh Node: Commit NVRAM Settings & Reboot ###

# nvram commit
# reboot

### AiMesh Node: Verify Syslog's Post-NVRAM Change Run-Time Configuration ###

# ssh admin@192.168.0.11
# ps w | grep -i syslog
5286 admin     1424 S    /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 6 -R 192.168.0.11:514 -L -H Office-3C73-CA04B43-R

### (Optional) Primary Router: Restart Syslog & Syslog-ng ###

# ssh admin@192.168.0.1
# service restart_logger
Done.
# /opt/etc/init.d/S01syslog-ng reconfigure
Sending SIGHUP to syslog-ng...

### Primary Router: View the Centralized Primary Router & AiMesh Node System Logs from the Command-Line or Asuswrt-Merlin WebUI ###

# ssh admin@192.168.0.1
# tail -f /tmp/syslog.log
Aug 21 23:20:44 syslog: WLCEVENTD wlceventd_proc_event(500): eth2: Auth F8:38:80:A9:F1:A9, status: Successful (0)
Aug 21 23:20:44 syslog: WLCEVENTD wlceventd_proc_event(529): eth2: Assoc F8:38:80:A9:F1:A9, status: Successful (0)
Aug 21 23:22:03 gnutech-wap01 rc_service: service 22892:notify_rc restart_logger
Aug 21 23:22:04 kernel: klogd started: BusyBox v1.25.1 (2020-08-14 15:17:43 EDT)
Aug 22 05:22:15 gnutech-wap01 syslog-ng[14682]: Accepting connections; addr='AF_INET(0.0.0.0:514)'
Aug 22 05:22:15 gnutech-wap01 syslog-ng[14682]: You have a TLS enabled source without a X.509 keypair. Make sure you have tls(key-file() and cert-file()) options, TLS handshake to this source will fail; location='/opt/etc/syslog-ng.conf:33:2'
Aug 22 05:22:15 gnutech-wap01 syslog-ng[14682]: Accepting connections; addr='AF_INET(0.0.0.0:6514)'
Aug 22 05:22:15 gnutech-wap01 syslog-ng[14682]: Accepting connections; addr='AF_INET(0.0.0.0:601)'
Aug 22 05:22:15 gnutech-wap01 syslog-ng[14682]: Configuration reload request received, reloading configuration;
Aug 22 05:22:15 gnutech-wap01 syslog-ng[14682]: Configuration reload finished;
Aug 21 23:22:56 dropbear[23028]: Pubkey auth succeeded for 'admin' with key sha1!! 12:2f:2c:eb:8d:9a:89:c7:ed:7d:08:e8:cf:ab:94:75:db:b7:76:be from 192.168.0.232:63523
Aug 21 23:23:09 Living_Room-C293-CA04B43-R dropbear[2965]: Pubkey auth succeeded for 'admin' with key sha1!! 12:2f:2c:eb:8d:9a:89:c7:ed:7d:08:e8:cf:ab:94:75:db:b7:76:be from 192.168.0.232:63524
Aug 21 23:23:14 Data_Center-D448-CA04B43-R dropbear[22487]: Pubkey auth succeeded for 'admin' with key sha1!! 12:2f:2c:eb:8d:9a:89:c7:ed:7d:08:e8:cf:ab:94:75:db:b7:76:be from 192.168.0.232:63525
Aug 21 23:23:17 Office-3C73-CA04B43-R dropbear[22037]: Child connection from 192.168.0.232:63526
Aug 21 23:23:18 Office-3C73-CA04B43-R dropbear[22037]: Pubkey auth succeeded for 'admin' with key sha1!! 12:2f:2c:eb:8d:9a:89:c7:ed:7d:08:e8:cf:ab:94:75:db:b7:76:be from 192.168.0.232:63526
Aug 21 23:23:21 Garage-AE61-CA04B43-R dropbear[14223]: Child connection from 192.168.0.232:63527
Aug 21 23:23:22 Garage-AE61-CA04B43-R dropbear[14223]: Pubkey auth succeeded for 'admin' with key sha1!! 12:2f:2c:eb:8d:9a:89:c7:ed:7d:08:e8:cf:ab:94:75:db:b7:76:be from 192.168.0.232:63527
Aug 21 23:23:25 Wiring_Closet-5610-CA04B43-R dropbear[6917]: Child connection from 192.168.0.232:63528
Aug 21 23:23:25 Wiring_Closet-5610-CA04B43-R dropbear[6917]: Pubkey auth succeeded for 'admin' with key sha1!! 12:2f:2c:eb:8d:9a:89:c7:ed:7d:08:e8:cf:ab:94:75:db:b7:76:be from 192.168.0.232:63528

Congratulations! You have a successfully working Asuswrt-Merlin Remote Log Server (Syslog & Syslog-ng Synergy) to Centralize Primary Router & AiMesh Node System Logs.

A BIG "Thank You" to those who Pioneered this Solution (RE: References).

Referernces:

https://www.snbforums.com/threads/configuring-syslog-ng-with-merlin-firmware.35095/

 

[Jack Yaz]

firewall-start would be better than nat-start

Also did you come across the scribe and uiscribe addons in your research?

 

[garycnew]

firewall-start would be better than nat-start

Also did you come across the scribe and uiscribe addons in your research?

@Jack Yaz

I am unbiased to where the DROP filter lives as long as it gets the job done. I am curious to why you feel firewall-start would be a better location.

As for the scribe and uiscribe add-ons, they didn't come up in my research. It seems there's more ways to skin a Merlin.

Respectfully,


Gary

 

[Jack Yaz]

@Jack Yaz

I am unbiased to where the DROP filter lives as long as it gets the job done. I am curious to why you feel firewall-start would be a better location.

As for the scribe and uiscribe add-ons, they didn't come up in my research. It seems there's more ways to skin a Merlin.

Respectfully,


Gary

Either script should be fine but nat-start is typically for nat table rules, and firewall start for filter etc.
You should also consider adding iptables -D prior to -I to prevent duplicate rules being created, if the scripts run for whatever reason without the old rules being cleared.

scribe installs and configures syslog-ng. It does some nice things to stop the routers own logging. It's primarily aimed at separate log files per message type, but since its syslog-ng you can still add your configuration on top. https://www.snbforums.com/threads/scribe-syslog-ng-and-logrotate-installer.55853/

Uiscribe extends the system log page to display each of the new log files

 

[garycnew]

Either script should be fine but nat-start is typically for nat table rules, and firewall start for filter etc.
You should also consider adding iptables -D prior to -I to prevent duplicate rules being created, if the scripts run for whatever reason without the old rules being cleared.

@Jack Yaz

I'll update the tutorial to use the firewall-start script.

I haven't ever had issues with duplicating the exact same iptables rule. I just tried to manually add the iptables rule twice and it wouldn't let me, so I don't think duplicating the iptables rule is a big concern.

I appreciate the feedback and alternative scribe solution.

Respectfully,

Gary

 

[elorimer]

I remember that old thread! I didn't realize it was five years ago folks backed this bulldozer into position.

I think you will find that @cmkelley took logging to a whole new level with scribe's implementation of syslog-ng, amtm familiarized it for the addon audience and @Jack Yaz GUI-ized the results. That is all really stable and a solid building block. It works to combine receipt of logs from other devices, or sending logs to other places.

Scribe avoids running two logging solutions at once, and some of the improvements cover your placeholders. Duplicates you can handle in filters rather than IPTables.

There is another thread on logging from mesh nodes. My suggestion would be for you to retool this for scribe and move it to the addons subforum.

PS: Also, lookup how @cmkelley and @Adamm made syslog-ng work with the firewall script.

 

[garycnew]

I remember that old thread! I didn't realize it was five years ago folks backed this bulldozer into position.

I think you will find that @cmkelley took logging to a whole new level with scribe's implementation of syslog-ng, amtm familiarized it for the addon audience and @Jack Yaz GUI-ized the results. That is all really stable and a solid building block. It works to combine receipt of logs from other devices, or sending logs to other places.

Scribe avoids running two logging solutions at once, and some of the improvements cover your placeholders. Duplicates you can handle in filters rather than IPTables.

There is another thread on logging from mesh nodes. My suggestion would be for you to retool this for scribe and move it to the addons subforum.

@elorimer

It seems I'm the last one to the scribe/uiscribe party. Not sure why they didn't come up in my Syslog research. I'm sure I've seen them in AMTM before. I suppose I just didn't realize it would cover my AiMesh Node Syslog Centralization need. I'll have to check them out and incorporate the AiMesh Node System Log Centralization as you suggest. I like your idea to use Syslog filters opposed to iptables to deal with duplicate logs.

I'll likely leave this tutorial as is and create something different with scribe/uiscribe.

Much Appreciated.


Gary

 

[garycnew]

@Jack Yaz, et al:

Please confirm whether scribe & uiscribe make use of the Remote Log Server functionality of Syslog? It is needed to centralized AiMesh Node Syslogs on the Primary Router.

I was looking at the uiscribe screenshots and Remote Log Server appears to be removed. I love everything else about uiscribe, though.

Much Appreciated.


Gary

 

[SomeWhereOverTheRainBow]

@Jack Yaz, et al:

Please confirm whether scribe & uiscribe make use of the Remote Log Server functionality of Syslog? It is needed to centralized AiMesh Node Syslogs on the Primary Router.

I was looking at the uiscribe screenshots and Remote Log Server appears to be removed. I love everything else about uiscribe, though.

Much Appreciated.


Gary

If I remember correctly , the gui remote log server function got removed because it related to using the routers native syslog remote server feature.(i.e. scribe turns off native system log.) Syslog-ng still allows for one to use a remote server with in the main.conf.

 

[garycnew]

If I remember correctly , the gui remote log server function got removed because it related to using the routers native syslog remote server feature.(i.e. scribe turns off native system log.) Syslog-ng still allows for one to use a remote server with in the main.conf.

@SomeWhereOverTheRainBow

That makes sense as to why uiscribe would remove the Remote Log Server option, if scribe turns off the native syslog daemon.

As long as I'm able to configure syslog-ng to listen on udp/514 (as I did in this tutorial) and manually configure the AiMesh Nodes' nvram set log_ipaddr=192.168.0.1 to remote send logs to syslog-ng via the AiMesh Nodes' native syslog daemon–it should be doable with the added advantage of uiscribe.

Thanks for the quick response.

Much Appreciated.


Gary

 

[Jack Yaz]

@SomeWhereOverTheRainBow

That makes sense as to why uiscribe would remove the Remote Log Server option, if scribe turns off the native syslog daemon.

As long as I'm able to configure syslog-ng to listen on udp/514 (as I did in this tutorial) and manually configure the AiMesh Nodes' nvram set log_ipaddr=192.168.0.1 to remote send logs to syslog-ng via the AiMesh Nodes' native syslog daemon–it should be doable with the added advantage of uiscribe.

Thanks for the quick response.

Much Appreciated.


Gary

what @SomeWhereOverTheRainBow said is correct, the UI option was for the router's syslog to send remote. fesabily i could add it back and instead drop the configuration into a syslog-ng.conf file but that could prove tricky depending on how users have customised scribe/syslog-ng beyond the "out of the box" experience.

 

[elorimer]

As long as I'm able to configure syslog-ng to listen on udp/514 (as I did in this tutorial) and manually configure the AiMesh Nodes' nvram set log_ipaddr=192.168.0.1 to remote send logs to syslog-ng via the AiMesh Nodes' native syslog daemon–it should be doable with the added advantage of uiscribe

Here's the thing. You can use the node's native logging to send messages to the main router at port 514. Then you can use syslog-ng on the main router to listen on that port for messages and process them.

What.you.do.not.want.to.do is use the syslog-ng.conf that entware installs. You.do.not.want.to.use the rc script either. The fact that it works in this limited instance is because the updated network driver just happens to default to what you are trying to do. Scribe installs its own version and you should use that.

I see the update to the OP. Scribe does not "automate" many of the steps in your tutorial. Scribe does a lot of different things.

I don't see the point in reinventing this wheel. But if you want to do it, you should dig out the original scribe thread and read it from beginning to end. All 70 pages. In the meantime, I think you should take "Solution" and "Tutorial" out of the title, until you've got the logrotates and the hangups sorted.

Sorry if this seems harsh, but your resource is the middle thread of at least three (here's my annual shoutout to @kvic!) in which @kvic, @Butterfly Bones, @tomsk laid down the original path and then @cmkelley weaponized it.

For anyone else looking at this thread, go to AMTM and install scribe. Then open the network driver.

 

[dave14305]

I don't see the point in reinventing this wheel. But if you want to do it, you should dig out the original scribe thread and read it from beginning to end. All 70 pages. In the meantime, I think you should take "Solution" and "Tutorial" out of the title, until you've got the logrotates and the hangups sorted.

Sorry if this seems harsh, but your resource is the middle thread of at least three (here's my annual shoutout to @kvic!) in which @kvic, @Butterfly Bones, @tomsk laid down the original path and then @cmkelley weaponized it.

For anyone else looking at this thread, go to AMTM and install scribe. Then open the network driver.

Why dampen the enthusiasm of someone who wants to do new things and share what he learns? The old guard of developers has unfortunately gone quiet. Many scripts are now stagnant in their development (Skynet, Diversion, Scribe, Pixelserv, et al), at least from an end-user perspective. Have contingencies in case they become abandoned.

 

[elorimer]

That's fair; even more fair if we are talking about new things. Skynet, scribe and pixelserv are quiet also because there isn't much at the moment that needs to be done. Diversion is the same while @thelonelycoder works on a full gui version. But I don't think we need anyone to pop up and say, "hey, I've just figured out a way to do adblocking using dnsmasq", at least not before reading through the 5,000 posts on the subject.

 

[thelonelycoder]

Have contingencies in case they become abandoned.

FYI, I am not abandoning my projects. But if anyone wants to have a go at their own adblocker script, I'd be delighted to see with what they come up as alternatives to the solutions Diversion offers. Unbound also offers adblocking.

 

[dave14305]

FYI, I am not abandoning my projects. But if anyone wants to have a go at their own adblocker script, I'd be delighted to see with what they come up as alternatives to the solutions Diversion offers. Unbound also offers adblocking.

I tried to give mine up for adoption but no one wanted to take it.

 

[SomeWhereOverTheRainBow]

FYI, I am not abandoning my projects. But if anyone wants to have a go at their own adblocker script, I'd be delighted to see with what they come up as alternatives to the solutions Diversion offers. Unbound also offers adblocking.

@thelonelycoder I have to say, along with @Jack Yaz , you are one of the most dedicated to your projects technical support. It barely takes a whisper of your name for you to be here helping out. I hope your development has been eventful lately as we patiently wait for your next masterful breakthrough.

 

[SomeWhereOverTheRainBow]

I tried to give mine up for adoption but no one wanted to take it.

You have become one of the experts around here. Not everyone can fill the same shoes you have come to fill.

 

[elorimer]

It barely takes a whisper of your name for you to be here helping out.

I knew what I was doing with that whisper. I just wish it would work so well for @kvic, but I fear that is not in his control.

 

[SomeWhereOverTheRainBow]

I welcome @garycnew to keep writing guides, even if it is not for everyone. A crafty developer (such as @Jack Yaz )on here may see something they want to incorporate. It is the lack of innovation that will keep things stale.