Scribe scribe 3.x_y - syslog-ng and logrotate installer

[Digimite]

The syslog-ng.conf file that scribe installs does have the @include “/opt/etc/syslog-ng.d/“ line near the top.

#############################################################################
# syslog-ng.conf customized for scribe on Asuswrt-Merlin firmware
# compare to /opt/share/syslog-ng/examples/syslog-ng.conf-opkg for differences from Entware distribution
#
# syslog-ng documentation: https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition
#
# Release notes: https://github.com/syslog-ng/syslog-ng/releases

@version: 4.2
#@include "scl.conf" # uncomment this line to for additional functionality, see syslog-ng documentation
@include "/opt/etc/syslog-ng.d/" # Put any customization files in this directory

That said, yes, it would make sense that using a config file with the @include at the bottom would result in duplicated messages.

Thanks for the clarification cmkelley. I’m not sure how the include got to the bottom of the config file, but I know that I have had some issues with scribe recently where it stopped working for reasons unknown. There were a couple of times where I checked the config file and the only thing present was line after line of “4.2” so there was something strange going on. If it happens again, I’ll touch base and let you know…

 

[zwitterion]

Flash drives vary in their durability depending on brand but as far as I can tell, the Samsung Fit Plus USB 64GB, 128GB, etc should be able to do the job without issue. Operating temps need to be below 140 Fahrenheit or 60 Celsius. Mine operates around 110 F. USB warranty covers 5 years, which shows their confidence in product reliability. -Although I couldn't verify if warranty is voided by R/W life cycles, their warranty page only shows their standard line of permanent storage warranty is voided if you exceed the W/R amount, the external drives however are covered by how many years. If it fails before 5 years, I will see if they will replace it. I figured that since the capacity is such a low amount that this particular USB would suffice for the job. If anyone has this configuration I'd be interested in your mileage if over 2 years.

If the USB does have the capabilities for the job, an important test would be to stress it to see if it does stay within operating temps, otherwise the heat will certainly cause W/R failures

 

[SomeWhereOverTheRainBow]

Flash drives vary in their durability depending on brand but as far as I can tell, the Samsung Fit Plus USB 64GB, 128GB, etc should be able to do the job without issue. Operating temps need to be below 140 Fahrenheit or 60 Celsius. Mine operates around 110 F. USB warranty covers 5 years, which shows their confidence in product reliability. -Although I couldn't verify if warranty is voided by R/W life cycles, their warranty page only shows their standard line of permanent storage warranty is voided if you exceed the W/R amount, the external drives however are covered by how many years. If it fails before 5 years, I will see if they will replace it. I figured that since the capacity is such a low amount that this particular USB would suffice for the job. If anyone has this configuration I'd be interested in your mileage if over 2 years.

If the USB does have the capabilities for the job, an important test would be to stress it to see if it does stay within operating temps, otherwise the heat will certainly cause W/R failures

I typically have gotten the best results with these two flash drive:

DataTraveler Max - USB 3.2 Gen 2 USB-C, USB-A Series Flash Drives - Kingston Technology

Kingston’s DataTraveler® Max series are two high performance USB 3.2 Gen 2 flash drives: one Type-C and one Type-A. With capacities up to 1TB, they’re ideal for transferring and storing large digital files such as HD photos, 4K/8K videos, music and more.

www.kingston.com

https://www.corsair.com/eu/en/p/data-storage/cmfvygtx3c-1tb/flash-voyager-gtx-usb-3-1-1tb-premium-flash-drive-cmfvygtx3c-1tb

But I have gotten good results with this one as well:

BAR Plus USB 3.1 Flash Drive 256GB Champagne Silver | Samsung US

www.samsung.com

The Samsung Fit Plus is pretty good too.


But I currently use a Samsung SSD.

 

[zwitterion]

I typically have gotten the best results with these two flash drive:

DataTraveler Max - USB 3.2 Gen 2 USB-C, USB-A Series Flash Drives - Kingston Technology

Kingston’s DataTraveler® Max series are two high performance USB 3.2 Gen 2 flash drives: one Type-C and one Type-A. With capacities up to 1TB, they’re ideal for transferring and storing large digital files such as HD photos, 4K/8K videos, music and more.

www.kingston.com

https://www.corsair.com/eu/en/p/data-storage/cmfvygtx3c-1tb/flash-voyager-gtx-usb-3-1-1tb-premium-flash-drive-cmfvygtx3c-1tb

But I have gotten good results with this one as well:

BAR Plus USB 3.1 Flash Drive 256GB Champagne Silver | Samsung US

www.samsung.com

The Samsung Fit Plus is pretty good too.


But I currently use a Samsung SSD.

I actually wasn't aware corsair made storage drives! I love their RAM but I guess I never bothered to look. Thanks! My hesitancy with external SSD is simply having a limp extremity protruding from my router, although I do realize as a long term solution that would be best. Hmmm, maybe if I had a 3-D printer I could whip up a permanent enclosure that would stealthily* mount to router... I'll put it on the endless list of future projects

 

[SomeWhereOverTheRainBow]

Hmmm, maybe if I had a 3-D printer I could whip up a permanent enclosure that would stealthily* mount to router... I'll put it on the endless list of future projects

I didn't need a 3D printer or enclosure for mine, it came with everything i needed and fits snuggly next to the router.
https://www.amazon.com/dp/B0874XN4D8/?tag=snbforums-20
The only buyers remorse I have is that I wish mine was $69 US dollars when I bought it.

 

[JAX1337]

is it possible to push scribe logs to graylog , if yes how to do it the right way ?

 

[elorimer]

It's in the graylog documentation: https://go2docs.graylog.org/5-1/getting_in_log_data/ingest_syslog.html?Highlight=syslog-ng

Also, if you search here back a few years, you will see folks building a destination for loggly.

 

[JAX1337]

It's in the graylog documentation: https://go2docs.graylog.org/5-1/getting_in_log_data/ingest_syslog.html?Highlight=syslog-ng

Also, if you search here back a few years, you will see folks building a destination for loggly.

i read the graylog config , i wanted to know if their is anyway to forward scribe logs as is to graylog rather than using syslog-ng

 

[elorimer]

Graylog appears to ingest files but not directly: https://go2docs.graylog.org/5-1/get...ogs|Log Sources|Additional Log Sources|_____6

Scribe sets up syslog-ng to parse messages into log files, but you can manually set up other destinations. If you only want to send messages to graylog that meet specific filters, isn't it easier to just add a graylog destination to each logging statement?

(As an aside, I didn't see exactly how to define a graylog destination; there must be something similar to a loggly token that is needed. But I stopped my loggly collection around the time of the solarwinds data breach.)

 

[JAX1337]

Graylog appears to ingest files but not directly: https://go2docs.graylog.org/5-1/getting_in_log_data/ingest_from_files.html?tocpath=Getting in Logs|Log Sources|Additional Log Sources|_____6

Scribe sets up syslog-ng to parse messages into log files, but you can manually set up other destinations. If you only want to send messages to graylog that meet specific filters, isn't it easier to just add a graylog destination to each logging statement?

(As an aside, I didn't see exactly how to define a graylog destination; there must be something similar to a loggly token that is needed. But I stopped my loggly collection around the time of the solarwinds data breach.)

its was easy to forward logs for graylog to injest but the default logs are are too scrambled for easy extraction , it would have been great if scribe had an option to integrate a remote log server like graylog

 

[elorimer]

To be honest, I've kind of lost the thread of what you want to accomplish-- "easy extraction" and "scrambled" don't give me a guide. You'll have to attract the interest of the two other folks here who use graylog.

 

[joe scian]

have a situation where syslog-ng.conf file got corrupted after a reboot with multiple lines of "4.2" contained in the file. Restored from a backup conf file. Syslog-ng started fine and all looks ok. Except now my skynet filters and unbound filters are not working. I forced updated scribe and did a log rotate and rebooted router. I get duplicate entries of unbound replies in system messages and also in the unbound log . ALL the skynet blocked entries appear in system messages only but not in skynet log. This used to work fine before the corrupt syslog-ng.conf was found this morning. Any ideas ? PS updated all the filters as well.

 

[maghuro]

have a situation where syslog-ng.conf file got corrupted after a reboot with multiple lines of "4.2" contained in the file. Restored from a backup conf file. Syslog-ng started fine and all looks ok. Except now my skynet filters and unbound filters are not working. I forced updated scribe and did a log rotate and rebooted router. I get duplicate entries of unbound replies in system messages and also in the unbound log . ALL the skynet blocked entries appear in system messages only but not in skynet log. This used to work fine before the corrupt syslog-ng.conf was found this morning. Any ideas ? PS updated all the filters as well.

Happened to me a couple of times.

Tried to debug, no success.

 

[joe scian]

Corruption occurred again after 2 hours - this is crazy - /opt/etc/syslog-ng.conf is now a file that contains 60 or 70 "4.2" entries. The system messages have halted as have every other log file. Do the experts elorimer and cmkelley have any idea whats going on please? I used restore configuration files in scribe utilities to get back to a working state.
Also had to move syslog-ng.conf file @include “/opt/etc/syslog-ng.d/“ line near the top. it was at the bottom so causing duplicate entries i was having.
So current situation is still that the skynet filter not working. Hopefully I wont have corruption problem again.

Please select an option: s

checking syslog-ng daemon ... dead.
the system logger (syslogd) ... is running.

Type scribe restart at shell prompt or select rs
from scribe main menu to start syslog-ng.
syslog.log default location ... /tmp/syslog.log
... & agrees with config file ... okay!

checking system for necessary scribe hooks ...

checking S01syslog-ng ... present.
checking post-mount ... present.
checking unmount ... present.

checking syslog-ng configuration ...

syslog-ng.conf version check ... out of sync! ()
*** Updating syslog-ng.conf and restarting syslog-ng ***
Checking syslog-ng... dead.
sed: bad option in substitution expression
Starting syslog-ng... done.
Restarting uiScribe ...Error parsing config, syntax error, unexpected LL_FLOAT, expecting end of file in /opt/etc/syslog-ng.conf:1:1-1:4:
1-----> 4.2
1-----> ^^^
2 4.2
3 4.2
4 4.2
5 4.2
6 4.2

 

[joe scian]

I sorted out the problem with the above post. It seems the corruption was still in the syslog conf file after restoring configurations. I copied syslog-ng.conf from another router at an alternate installation and all OK now - sknet filter once again is working.

 

[SomeWhereOverTheRainBow]

has any one here seen these errors :- https://www.snbforums.com/threads/i...over-the-place-after-restarting-scribe.87949/

 

[Ripshod]

has any one here seen these errors :- https://www.snbforums.com/threads/i...over-the-place-after-restarting-scribe.87949/

Nope, but I'm not running Skynet.

 

[SomeWhereOverTheRainBow]

Nope, but I'm not running Skynet.

well if you run into this issue with scribe in the future (w/o skynet), the only solution I found was to uninstall it. I see @joe scian had the same issue. I will keep scribe uninstalled in the future for now until this "concern" is no longer a "concern".

 

[Ripshod]

Decided to take one for the team. Installed skynet, and through several restarts of skynet and scribe I have seen zero errors. Entware is up to date.

 

[SomeWhereOverTheRainBow]

Decided to take one for the team. Installed skynet, and after several restarts of skynet and scribe I have zero errors. Entware is up to date.

For me it took months for this error to show up. If you are willing to ride it out that long, then maybe the issue could be resolved.