Permanently moving traffic.db and syslog.log over to a USB Drive?

[dandle]

Yes, everything in the code box in post #6 should be cut'n'pasted into /jffs/scripts/syslog_move.sh


There should be no need to change the source???


To make my syslog move script 'portable' I documented/included a base init-start script as a comment header within the syslog_move.sh script.

..but to make it even easier I have expanded the comments as code, so simply cut'n'paste everything in this code box to /jffs/scripts/init-start (assuming of course that you don't already have an existing init-start script! )

#!/bin/sh

/usr/bin/logger -s -t "($(basename $0))" $$ "Martineau $MYROUTER BOOT in progress... [$@]"

MYROUTER=$(nvram get computer_name)
HARDWARE_MODEL=$(nvram get productid)
BUILDNO=$(nvram get buildno)
EXTENDNO=$(nvram get extendno)


# NOTE: Can't use Flash drive /tmp/mnt/$MYROUTER/ 'cos it hasn't been mounted yet :-(
# 'flock' is probably a better solution rather than the 'echo' ;-)
echo $$"-"`date` > /tmp/BOOTINPROGRESS

# Should be sufficient to cover physical BOOT process?
logger -st "($(basename $0))" $$ "Paused for 3 mins....."
sleep 180

# call custom scripts/commands e.g. cifs.sh / cru etc. here...

# Move Syslog to USB Flash drive
/jffs/scripts/syslog-move.sh

rm /tmp/BOOTINPROGRESS

/usr/bin/logger -st "($(basename $0))" $$ "Martineau" $MYROUTER "BOOT Completed Firmware build" $BUILDNO $EXTENDNO "[$@]"

exit 0

Thanks. Looks easy enough. Carried out all instructions. One small issue (or big). I don't have the syslog.log file anymore within the /Jffs directory

I understand, if I reboot the router it should rebuild itself? I'm not seeing any output from within the GUI when logged into the router.

This is what I see by the way when I run syslog_move.sh

(syslog_move.sh): 21418 SYSLOG Housekeeping Running...... Tue Jan 24 21:47:42 GMT 2017 []
mv: can't rename '/tmp/mnt/sda1/Syslog/syslog.log': No such file or directory
cp: can't stat '/tmp/syslog.log': No such file or directory
(syslog_move.sh): 21418 Creating '/tmp/mnt/sda1/Syslog/syslog.log-20170124-214743-ZZ 'persistent BOOT log 1
rm: can't remove '/tmp/syslog.log': No such file or directory
(syslog_move.sh): 21418 Syslog moved to USB drive '/tmp/mnt/sda1/Syslog/syslog.log'
Syslog move OK. Script: syslog_move.sh
(syslog_move.sh): 21418 SYSLOG Housekeeping Complete...... Tue Jan 24 21:47:48 GMT 2017 for '/tmp/mnt/sda1/Syslog/syslog.log-20170124-214743'
(syslog_move.sh): 21418 SYSLOG Housekeeping scanning: /tmp/mnt/sda1/Syslog/syslog.log-20170124-214743-ZZ
syslog_move.sh: line 97: can't create /tmp/mnt/RT-AC68U-/Syslog/BOOT_Errors.txt: nonexistent directory
grep: /tmp/mnt/sda1/Syslog/syslog.log-20170124-214743-ZZ: No such file or directory
[: bad number
syslog_move.sh: line 105: can't create /tmp/mnt/RT-AC68U-/Syslog/BOOT_Errors.txt: nonexistent directory
(syslog_move.sh): 21418 SYSLOG Housekeeping custom script action scanning: '/tmp/mnt/RT-AC68U-/Syslog/MyCustomActions.txt'
syslog_move.sh: line 108: can't create /tmp/mnt/RT-AC68U-/Syslog/MyCustomActions.txt: nonexistent directory

 

[Martineau]

Thanks. Looks easy enough. Carried out all instructions. One small issue (or big). I don't have the syslog.log file anymore within the /Jffs directory

I understand, if I reboot the router it should rebuild itself?

Have you manually run /jffs/scripts/syslog_move.sh

cd /jffs/scripts
./syslog_move.sh

and checked if you can see the syslog in the router GUI, and also using WinSCP to check that the syslog file is now on the USB drive?

Also you may wish to run ./init-start to check it also runs as expected, then reboot to check out both scripts in anger so to speak.

Yes, the syslog will be created as normal during the reboot, then 3 mins later it should be moved to the USB drive.

 

[Martineau]

Thanks. Looks easy enough. Carried out all instructions. One small issue (or big). I don't have the syslog.log file anymore within the /Jffs directory

I understand, if I reboot the router it should rebuild itself? I'm not seeing any output from within the GUI when logged into the router.

This is what I see by the way when I run syslog_move.sh

(syslog_move.sh): 21418 SYSLOG Housekeeping Running...... Tue Jan 24 21:47:42 GMT 2017 []
mv: can't rename '/tmp/mnt/sda1/Syslog/syslog.log': No such file or directory
cp: can't stat '/tmp/syslog.log': No such file or directory
(syslog_move.sh): 21418 Creating '/tmp/mnt/sda1/Syslog/syslog.log-20170124-214743-ZZ 'persistent BOOT log 1
rm: can't remove '/tmp/syslog.log': No such file or directory
(syslog_move.sh): 21418 Syslog moved to USB drive '/tmp/mnt/sda1/Syslog/syslog.log'
Syslog move OK. Script: syslog_move.sh
(syslog_move.sh): 21418 SYSLOG Housekeeping Complete...... Tue Jan 24 21:47:48 GMT 2017 for '/tmp/mnt/sda1/Syslog/syslog.log-20170124-214743'
(syslog_move.sh): 21418 SYSLOG Housekeeping scanning: /tmp/mnt/sda1/Syslog/syslog.log-20170124-214743-ZZ
syslog_move.sh: line 97: can't create /tmp/mnt/RT-AC68U-0998/Syslog/BOOT_Errors.txt: nonexistent directory
grep: /tmp/mnt/sda1/Syslog/syslog.log-20170124-214743-ZZ: No such file or directory
[: bad number
syslog_move.sh: line 105: can't create /tmp/mnt/RT-AC68U-0998/Syslog/BOOT_Errors.txt: nonexistent directory
(syslog_move.sh): 21418 SYSLOG Housekeeping custom script action scanning: '/tmp/mnt/RT-AC68U-0998/Syslog/MyCustomActions.txt'
syslog_move.sh: line 108: can't create /tmp/mnt/RT-AC68U-0998/Syslog/MyCustomActions.txt: nonexistent directory

The variable $MYROUTER needs to reflect the name of the drive... so it looks like you didn't change it, but only some of the lines where you hard-coded 'sda1'

Did you create the /tmp/mnt/sda1/Syslog folder on the drive?

 

[dandle]

Right ok. I think I may have got it to work, sort of.

So the $MYROUTER variable. You're right. I didn't change it for each I came across. Only for the ones you listed earlier. Do I need to change each one to sda1? Either way I've done that now. I haven't yet run init-start. To be honest, I haven't created the script for that yet as I wanted to test this one out first.

I created a new Syslog folder in /tmp/mnt/sda1. I noticed straight away I saw these files become populated inside.

syslog.log
MyCustomActions.txt
BOOT_Errors.txt

So I guess it worked. I got the GUI output back too however it seems every second I'm seeing this message

topology change detected, propagating"

 

[dandle]

Update: Ok, after a reboot of the router, the GUI output is now showing the syslog correctly and it's copied over correctly to the USB and has synced up nicely

I assume it's worked then! One question. Although the syslog has moved over to the USB drive, I still see it within the /JFFS partition using up 2KB of space. Is that normal? Should that still be there?

Thanks for the help!

 

[Martineau]

Update: Ok, after a reboot of the router, the GUI output is now showing the syslog correctly and it's copied over correctly to the USB and has synced up nicely

I assume it's worked then! One question. Although the syslog has moved over to the USB drive, I still see it within the /JFFS partition using up 2KB of space. Is that normal? Should that still be there?

Thanks for the help!

As @ColinTaylor stated, normally when syslog is in /tmp, when it reaches its threshold size (basically it is restricted by the flash memory) it dumps a copy to /jffs and also when you reboot.

Hopefully now it's on USB, it should never reach a threshold as I restarted syslog with 'unlimited' size, and better yet, if the router crashes, as the syslog is on the USB, hopefully you no longer lose the 'useful' syslog leading up to the crash!

Now you can call yourself a scripter!

 

[dandle]

Feels good

So basically just so I understand correctly. So currently any activity from the syslog will of course be written to the sda1 drive but not jffs correct? Only if the router is rebooted or the threshold size is reached? What would the threshold size be? Does that mean if the syslog file gets too big for the USB drive itself? Sorry if that sounds dumb.

So now syslog is on the USB drive, any way to move over traffic.db

 

[dandle]

Sorry for the constant questions

But it seems I've come across the message I mentioned earlier in my post when I thought I had got rid of it by rebooting.

kern.info kernel: br0: topology change detected, propagating

What does this mean? The syslog seems to update to this every 1 second.

 

[ColinTaylor]

kern.info kernel: br0: topology change detected, propagating

What does this mean? The syslog seems to update to this every 1 second.

You get those messages if the network interfaces are restarted for some reason. The messages that follow tell you which interface it refers to. You should only see a few of them though. Have you bee making lots of changes to the wireless settings?

https://www.snbforums.com/threads/wireless-restart.8464/

 

[dandle]

Hey @ColinTaylor.

Sorry for the late reply. Hmm, no I haven't really been making any changes to the wireless settings.

I've had a look at the link you posted and I believe what I'm seeing is probably a little different. I should have added in my earlier post the full extract of what I see.

(vlan1) received tcn bpdu
kernel: br0: topology change detected, propagating

I believe this may be swtiching related? I reset my router yesterday to factory defaults and started fresh. Later that night I started noticing these messages again. A quick reboot of the router halted it but it has started to appear again. I did some reading and it was recommended to disable STP from within settings which I have done. This rebooted the router and will now have to wait to see whether this has solved this issue.

I currently have 3 switches on the network. All 3 are not linked to each other. 1 switch is linked to the other 2.

Will this solve it or is there another way?

 

[Shonk]

compile a build your self

JFFS2LOG=n

job done

 

[ColinTaylor]

should have added in my earlier post the full extract of what I see.

(vlan1) received tcn bpdu
kernel: br0: topology change detected, propagating

I believe this may be swtiching related?

You're still missing the first part of the first line which tells you the port number.


Yes those are messages are generated when STP is reporting a change to your network. vlan1 is the router's LAN ports. Turning off STP on the router should hide those messages. Presumably the messages are coming from your switch(es) so you might want to investigate them to see why they think the network topology has changed.

 

[Diamond67]

This obsession people have with thinking they are wearing out the flash is wrong and has been debunked here many times (including by Merlin himself).

Speaking of wearing out (obsession): I think I'm going to test the scripts discussed in this thread and use the usb stick which contains my Entware and AB Solution 3. However, when I installed them, I used a usb stick which I had earlier formatted EXT4 with Ubuntu. Only afterwards I read that with usb sticks recommended file system would be EXT2 (because of journaling -> more writes -> wearing out).

Should I reformat my usb stick to EXT2 and start from scratch before installing more shi... stuff into it?

 

[rdy2]

@Martineau

I am working on deploying your syslog-move.sh. Thank you for making the code available. linux I am at the point of testing it by hand. It has successfully switched to USB. Looking at the code, the logged lines and the execution trace on the terminal, it seems there are some informative logger calls in between killall syslogd and the subsequent start. Was that intentional? My skill level is too low to know for sure what happens to the messages, but I suspect they get lost.

Edit: I can take care of moving a few lines around...

 

[Martineau]

@Martineau

Looking at the code, the logged lines and the execution trace on the terminal, it seems there are some informative logger calls in between killall syslogd and the subsequent start. Was that intentional? My skill level is too low to know for sure what happens to the messages, but I suspect they get lost.

Yes you are correct, the logger statements ONLY go to the console during the switch, clearly useful if the script is called manually, but the messages would indeed be lost if the script was called by a cron job, but the 'lost' messages are not significantly important!

There have been several changes to the original script, i.e. cosmetic message changes in colour etc. but also I do now spoof Syslog messages using 'echo' rather than 'logger' for instances when syslogd is down.
NOTE: There is also a change to the script when creating the edited BOOT log as Asus changed the base default Syslog message tagging date from Aug to Feb....
e.g. LINE 144

        # Remove all duplicated \'shutdown\' lines - they are always incomplete anyway?
        logger -st "($(basename $0))" $$ "Editing \'"$SYSLOG-$NOW-BOOT.raw"\' -> \'"$SYSLOG-$NOW-BOOT.txt"\'
        RC=$(sed -n \'/^Aug/,$p\' $SYSLOG-$NOW"-BOOT.raw" > $SYSLOG-$NOW"-BOOT.txt")  # i.e. upto \'Aug  1 01:00:15 syslogd started: BusyBox v1.25.1\'

has been modified to

        # Remove all duplicated 'shutdown' lines - they are always incomplete anyway?
        echo -en $cBCYA
        echo `date "+%b   %d %T"` $MYROUTER "Spoof.logger Trimming" $SYSLOG-$NOW-BOOT.raw "to create" $SYSLOG-$NOW-BOOT.txt >>$SYSLOG-$NOW-BOOT.raw
        # v382.xx> no longer uses Aug
        FIRMWARE=$(echo $(nvram get buildno) | awk 'BEGIN { FS = "." } {printf("%03d%02d",$1,$2)}')
        if [ $FIRMWARE -ge 38201 ];then
            #Feb  5 14:46:56 RT-AC68U kern.info kernel: usbcore: deregistering interface driver usblp
            #Feb 14 00:00:16 syslogd started: BusyBox v1.25.1
            #Feb 14 00:00:16 kernel: klogd started: BusyBox v1.25.1 (2018-02-04 12:53:06 EST)
            #Feb 14 00:00:16 kernel: Linux version 2.6.36.4brcmarm (merlin@ubuntu-dev) (gcc version 4.5.3 (Buildroot 2012.02) ) #1 SMP PREEMPT Sun Feb 4 13:00:47 EST 2018
            RC=$(sed -n '/^Feb 14 00:00:16 kernel/,$p' $SYSLOG-$NOW"-BOOT.raw" > $SYSLOG-$NOW"-BOOT.txt")   # i.e. upto 'Feb 14 00:00:16 syslogd started: BusyBox v1.25.1'
        else
            RC=$(sed -n '/^Aug/,$p' $SYSLOG-$NOW"-BOOT.raw" > $SYSLOG-$NOW"-BOOT.txt")  # i.e. upto 'Aug  1 01:00:15 syslogd started: BusyBox v1.25.1'
        fi

P.S. I didn't think anyone still used the script - hence I haven't bothered to update the original link, as I believe there is now a possible conflict with another script which breaks the functionality.

 

[rdy2]

Thank you. I appreciate the fast response and the updated code. (I pulled your ANSIColours from another thread)

P.S. I didn't think anyone still used the script - hence I haven't bothered to update the original link, as I believe there is now a possible conflict with another script which breaks the functionality.

What do you mean? The architected solution may not work in latest levels ? Have you noticed something with 384.4 ? I am running 384.3.

 

[dony71]

i have 2 /tmp/sda1 when do df
this line below will mess up DEV_MOUNT
DEV_MOUNT=`df | grep -F "$SDx" | awk '{print $6}'`
how to fix if only wants to pick 1st /tmp/sda1 ?

 

[Dabombber]

The pastebin link on the previous page is meant for embedding, try using the original, or even easier, the raw.

 

[Martineau]

i have 2 /tmp/sda1 when do df

this line below will mess up DEV_MOUNT
DEV_MOUNT=`df | grep -F "$SDx" | awk '{print $6}'`

how to fix if only wants to pick 1st /tmp/sda1 ?

Change

DEV_MOUNT=`df | grep -F "$SDx" | awk '{print $6}'`

to

DEV_MOUNT=$(df | grep -m 1 -F "$SDx" | awk '{print $6}')

 

[Turgut Kalfaoglu]

It seems that the parameters have changed and/or I am not using the GNU versions of the logger and syslogd?


# ./move-syslog
(move-syslog): 26604 Syslog '/tmp/syslog.log' is in RAM
(move-syslog): 26604 Syslog Housekeeping starting..... []
(move-syslog): 26604 Creating '/tmp/mnt/2TBEXTERNAL/Syslog/syslog.log-20200424-091757.txt'
cp: can't create '/tmp/mnt/2TBEXTERNAL/Syslog/syslog.log-20200424-091757.txt': No such file or directory
(move-syslog): 26604 Syslog moved to USB drive '/tmp/mnt/2TBEXTERNAL/Syslog/syslog.log'
syslogd: invalid option -- 's'
BusyBox v1.31.1 () multi-call binary.

Usage: syslogd [OPTIONS]

System logging utility
(this version of syslogd ignores /etc/syslog.conf)

-n Run in foreground
-O FILE Log to FILE (default: /var/log/messages, stdout if -)
-l N Log only messages more urgent than prio N (1-8)
-S Smaller output
-t Strip client-generated timestamps
logger: invalid option -- 'O'
BusyBox v1.31.1 () multi-call binary.

Usage: logger [OPTIONS] [MESSAGE]

Write MESSAGE (or stdin) to syslog

-s Log to stderr as well as the system log
-t TAG Log using the specified tag (defaults to user name)
-p PRIO Priority (numeric or facility.level pair)