GX device logging to ram (tmpfs) | log2ram

Seems that this idea was not yet discussed in more details in the concept of Venus OS, but I have had bad experience with embedded systems doing a lot of I/O to general purpose flash storage like it is the microSD card in Raspberry Pi and similar. I do not know what type is the memory in the generic GX devices (Cerbo, Venus, etc.), so this is targeted mainly toward raspberry pi implementations of Venus OS.

This was originally raised by me within Victron community here, but seems that there is no interest.

In this thread, I will share my implementation of log2ram for Venus OS. log2ram was officially developed for a systemd service, so it needs some modification.
Of course, it is not perfect, but I believe, can be easily modified to the venus services system. I am using the classical SysV Init.

… to be continued in the next posts.

1 Like

Here is a step-by-step procedure that can be automated with a script if someone is willing to do it. It can also be implemented it as a package (similar to Kevin Windrem packages).
I will paste file content in-line. If possible, will attach the files to an archive.

1) Downloading:

mkdir -p /data/my_files
cd /data/my_files
curl -L https://github.com/azlux/log2ram/archive/master.tar.gz | tar zxf -
cd log2ram-master

2) Install/copy and create the needed files to a systems dir in /data:

First creating log2ram system directory in /data so we can have all needed files to be easily copied to the needed places even after upgrade.

mkdir -p /data/log2ram_system
cp log2ram /data/log2ram_system/log2ram
chmod 755 /data/log2ram_system/log2ram
cp log2ram.conf /data/log2ram_system/log2ram.conf
chmod 644 /data/log2ram_system/log2ram.conf
cp log2ram.logrotate /data/log2ram_system/log2ram.logrotate
chmod 644 /data/log2ram_system/log2ram.logrotate

Creating the init script (SysV Init). This will provide proper start and sync on reboot and gracefull shutdown as well.

cat > /data/log2ram_system/log2ram-init.sh <<EOF
#!/bin/sh
#
# log2ram SysV Init script
# Developed by kivanov for VenusOS
#
### BEGIN INIT INFO
# Provides:          log2ram
# Required-Start:       
# Required-Stop:        
# Default-Start:     S
# Default-Stop:      0 6
# Short-Description: Provides ramdrive for system logging
### END INIT INFO

# Init start
START=06
PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
LOG2RAM_SCRIPT=/usr/local/bin/log2ram
CONFIG=/etc/log2ram.conf

# Check if all is fine with the needed files
[ -f $CONFIG ] || exit 1
[ -x $LOG2RAM_SCRIPT ] || exit 1

# Source function library.
if [ -f /etc/init.d/functions ]; then
      . /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ]; then
      . /etc/rc.d/init.d/functions
fi


case "$1" in
  start)
    echo -n "Starting log2ram: "
    #touch /data/log2ram.started.$(date +"%Y-%m-%d_%T")
    ${LOG2RAM_SCRIPT} start
    RETVAL=$?
    if [ $RETVAL -eq 0 ] ; then
        echo "OK"
    else
        echo "FAIL"
    fi
    ;;
  stop)
    echo -n "Stopping log2ram: "
    #touch /data/log2ram.stopped.$(date +"%Y-%m-%d_%T")
    ${LOG2RAM_SCRIPT} stop
    RETVAL=$?
    if [ $RETVAL -eq 0 ] ; then
        echo "OK"
    else
        echo "FAIL"
    fi
    ;;
  sync)
    echo -n "This operation is generally provided by cron."
    while true; do
    read -p "Continue (y/n)?" choice
    case ${choice} in
        [Yy]* ) break;;
        [Nn]* ) exit 1;;
        * ) echo "Please answer yes or no.";;
    esac
	done
		
    echo -n "Force log2ram write to disk on-the-fly from the cli: "
    #touch /data/log2ram.synched.$(date +"%Y-%m-%d_%T")
    ${LOG2RAM_SCRIPT} write
    RETVAL=$?
    if [ $RETVAL -eq 0 ] ; then
        echo "OK"
    else
        echo "FAIL"
    fi
    ;;
  status)
    cat /proc/mounts | grep -w log2ram > /dev/null && { echo -ne "log2ram is running\n"; };
    cat /proc/mounts | grep -w log2ram > /dev/null || { echo -ne "log2ram is NOT running\n"; };
    exit $?
    ;;
  restart)
    $0 stop && sleep 1 && $0 start
    ;;
  *)
    echo "Usage: /etc/init.d/$(basename $0) {start|stop|sync|status|restart}"
    exit 1
esac

exit 0
EOF

chmod 755 /data/log2ram_system/log2ram-init.sh

Creaging cron script which can be copied to cron daily/weekly/ etc folder or set manually in cron to be execuded for particular periods of time in order to sync the log ramdisk to the flash memory

cat > /data/log2ram_system/log2ram-cron.sh <<EOF
#!/bin/sh
#
# cron.daily/log2ram -- daily write/sync ramlog to disk
#
#

LOG2RAM_SCRIPT=/usr/local/bin/log2ram
CONFIG=/etc/log2ram.conf

# Check if all is fine with the needed files
[ -f $CONFIG ] || exit 1
[ -x $LOG2RAM_SCRIPT ] || exit 1
cat /proc/mounts | grep -w log2ram > /dev/null || { echo -ne "log2ram is NOT running\n"; exit 1; }

exec ${LOG2RAM_SCRIPT} write

EOF
chmod 755 /data/log2ram_system/log2ram-cron.sh

Finally fix ownership (if some of the copied filed are not owned by root)

chown -R root:root /data/log2ram_system

3) Configure log2ram setting:

Set the bellow parameters in /data/log2ram_system/log2ram.conf (editing the file using nano for example)

SIZE=100M
USE_RSYNC=true
MAIL=false
PATH_DISK="/data/log"
ZL2R=false

It Highly recommended to install rsync if it has not been installed yet:

opkg update
opkg install rsync

IMPORTANT: Install findmnt - needed to log2ram core tool:

opkg install util-linux-findmnt

4) Installing to system:

cd /data/log2ram_system
mkdir -p /usr/local/bin/
cp log2ram /usr/local/bin/log2ram
cp log2ram.conf /etc/log2ram.conf
cp log2ram-init.sh /etc/init.d/log2ram-init.sh
# we create daily cron for syncing
cp log2ram-cron.sh /etc/cron.daily/log2ram-cron.sh
[ -d /etc/logrotate.d ] && cp /data/log2ram_system/log2ram.logrotate /etc/logrotate.d/log2ram

Remove a previous log2ram version (if present for some reason):

rm -rf /var/log.hdd
rm -rf /data/log.hdd
# Make sure we start clean
rm -rf /var/hdd.log
rm -rf /data/hdd.log

5) Setting up init and check

update-rc.d log2ram-init.sh start 06 S . stop 80 0 6 . 

If we want to disable:

update-rc.d -f log2ram-init.sh remove

Checking correct symlinks are installed:

ls -alh /etc/rcS.d/    # symlink should be present
ls -alh /etc/rc3.d/    # symlink should NOT be present
ls -alh /etc/rc6.d/    # symlink should be present

6) reboot

sync
reboot

After reboot - there should be log2ram tmpfs filesystem already mounted and log hdd:

cat /proc/mounts
df -h
1 Like

Checking on my r-pi:

[root@raspberrypi2:~]# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               403.5M    304.5M     73.1M  81% /
devtmpfs                458.5M      4.0K    458.5M   0% /dev
tmpfs                   463.0M    512.0K    462.5M   0% /run
tmpfs                   463.0M    152.0K    462.9M   0% /var/volatile
/dev/mmcblk0p1           43.7M     23.7M     20.0M  54% /u-boot
/dev/mmcblk0p4            9.9G     53.8M      9.3G   1% /data
/dev/mmcblk0p4            9.9G     53.8M      9.3G   1% /data/hdd.log
log2ram                 100.0M      2.1M     97.9M   2% /data/log
overlay                 463.0M    512.0K    462.5M   0% /service
[root@raspberrypi2:~]# 
[root@raspberrypi2:~]# 
[root@raspberrypi2:~]# 
[root@raspberrypi2:~]# 
[root@raspberrypi2:~]# cat /proc/mounts 
/dev/root / ext4 rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=469540k,nr_inodes=117385,mode=755 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /var/volatile tmpfs rw,relatime 0 0
/dev/mmcblk0p1 /u-boot vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro 0 0
/dev/mmcblk0p4 /data ext4 rw,noatime 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
/dev/mmcblk0p4 /data/hdd.log ext4 rw,noatime 0 0
log2ram /data/log tmpfs rw,nosuid,nodev,noexec,noatime,size=102400k,mode=755 0 0
overlay /service overlay rw,relatime,lowerdir=/opt/victronenergy/service,upperdir=/run/overlays/service,workdir=/run/overlays/work/service 0 0

Realtime test:

[root@raspberrypi2:~]# logger -s log2ram test
root: log2ram test
[root@raspberrypi2:~]# 
[root@raspberrypi2:~]# 
[root@raspberrypi2:~]# tail /data/log/messages
Jul 16 15:18:49 raspberrypi2 daemon.info connmand[645]: eth0 {add} route 192.168.50.0 gw 0.0.0.0 scope 253 <LINK>
Jul 16 15:18:49 raspberrypi2 daemon.info connmand[645]: eth0 {add} route 192.168.50.1 gw 0.0.0.0 scope 253 <LINK>
Jul 16 15:18:49 raspberrypi2 daemon.info connmand[645]: eth0 {add} route 0.0.0.0 gw 192.168.50.1 scope 0 <UNIVERSE>
Jul 16 15:18:49 raspberrypi2 daemon.info connmand[645]: eth0 {add} route 82.165.8.211 gw 192.168.50.1 scope 0 <UNIVERSE>
Jul 16 15:18:51 raspberrypi2 daemon.info connmand[645]: eth0 {del} route 82.165.8.211 gw 192.168.50.1 scope 0 <UNIVERSE>
Jul 16 15:21:20 raspberrypi2 daemon.info connmand[645]: ntp: time slew +13.213979 s
Jul 16 15:21:34 raspberrypi2 daemon.err connmand[645]: Failed to set hwclock
Jul 16 15:24:10 raspberrypi2 user.notice root: log2ram test
Jul 16 15:24:35 raspberrypi2 user.notice root: log2ram test
[root@raspberrypi2:~]# 
[root@raspberrypi2:~]# 
[root@raspberrypi2:~]# 
[root@raspberrypi2:~]# tail /data/hdd.log/messages
Jul 16 15:18:25 raspberrypi2 daemon.info haveged: haveged: Stopping due to signal 15
Jul 16 15:18:26 raspberrypi2 daemon.info bluetoothd[592]: Terminating
Jul 16 15:18:26 raspberrypi2 daemon.info bluetoothd[592]: Exit
Jul 16 15:18:26 raspberrypi2 daemon.err connmand[624]: D-Bus disconnect
Jul 16 15:18:26 raspberrypi2 daemon.err connmand[624]: Time request for server 192.168.50.1 failed (101/Network is unreachable)
Jul 16 15:18:26 raspberrypi2 daemon.err connmand[624]: Time request for server 192.168.50.1 failed (101/Network is unreachable)
Jul 16 15:18:26 raspberrypi2 daemon.info dnsmasq[597]: exiting on receipt of SIGTERM
Jul 16 15:18:26 raspberrypi2 user.info kernel: [  934.101155] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
Jul 16 15:18:26 raspberrypi2 daemon.info connmand[624]: Remove interface eth0 [ ethernet ]
Jul 16 15:18:26 raspberrypi2 syslog.info syslogd exiting

[root@raspberrypi2:~]# sync
[root@raspberrypi2:~]# reboot
[root@raspberrypi2:~]# 

after reboot:

[root@raspberrypi2:~]# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               403.5M    304.5M     73.1M  81% /
devtmpfs                458.5M      4.0K    458.5M   0% /dev
tmpfs                   463.0M    512.0K    462.5M   0% /run
tmpfs                   463.0M    152.0K    462.9M   0% /var/volatile
/dev/mmcblk0p1           43.7M     23.7M     20.0M  54% /u-boot
/dev/mmcblk0p4            9.9G     53.8M      9.3G   1% /data
/dev/mmcblk0p4            9.9G     53.8M      9.3G   1% /data/hdd.log
log2ram                 100.0M      2.0M     98.0M   2% /data/log
overlay                 463.0M    512.0K    462.5M   0% /service
[root@raspberrypi2:~]# tail /data/hdd.log/messages
Jul 16 15:25:27 raspberrypi2 daemon.info haveged: haveged: Stopping due to signal 15
Jul 16 15:25:28 raspberrypi2 daemon.info bluetoothd[614]: Terminating
Jul 16 15:25:28 raspberrypi2 daemon.info bluetoothd[614]: Exit
Jul 16 15:25:28 raspberrypi2 daemon.err connmand[645]: D-Bus disconnect
Jul 16 15:25:28 raspberrypi2 daemon.info dnsmasq[619]: exiting on receipt of SIGTERM
Jul 16 15:25:28 raspberrypi2 daemon.err connmand[645]: Time request for server 192.168.50.1 failed (101/Network is unreachable)
Jul 16 15:25:28 raspberrypi2 daemon.err connmand[645]: Time request for server 192.168.50.1 failed (101/Network is unreachable)
Jul 16 15:25:28 raspberrypi2 daemon.info connmand[645]: Remove interface eth0 [ ethernet ]
Jul 16 15:25:28 raspberrypi2 user.info kernel: [  407.415117] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
Jul 16 15:25:28 raspberrypi2 syslog.info syslogd exiting
[root@raspberrypi2:~]# tail /data/log/messages
Jul 16 15:25:51 raspberrypi2 daemon.info connmand[646]: Setting hostname to raspberrypi2
Jul 16 15:25:51 raspberrypi2 daemon.info connmand[646]: Setting domainname to sofia.k2intra.net
Jul 16 15:25:37 raspberrypi2 daemon.info dnsmasq[619]: started, version 2.83 cachesize 150
Jul 16 15:25:51 raspberrypi2 daemon.warn dnsmasq[619]: overflow: 3 log entries lost
Jul 16 15:25:51 raspberrypi2 daemon.info connmand[646]: eth0 {add} address 192.168.50.140/24 label eth0 family 2
Jul 16 15:25:51 raspberrypi2 daemon.info connmand[646]: eth0 {add} route 192.168.50.0 gw 0.0.0.0 scope 253 <LINK>
Jul 16 15:25:51 raspberrypi2 daemon.info connmand[646]: eth0 {add} route 192.168.50.1 gw 0.0.0.0 scope 253 <LINK>
Jul 16 15:25:51 raspberrypi2 daemon.info connmand[646]: eth0 {add} route 0.0.0.0 gw 192.168.50.1 scope 0 <UNIVERSE>
Jul 16 15:25:51 raspberrypi2 daemon.info connmand[646]: eth0 {add} route 82.165.8.211 gw 192.168.50.1 scope 0 <UNIVERSE>
Jul 16 15:25:52 raspberrypi2 daemon.info connmand[646]: eth0 {del} route 82.165.8.211 gw 192.168.50.1 scope 0 <UNIVERSE>
[root@raspberrypi2:~]# cat /data/hdd.log/messages | grep log2ram
Jul 16 15:23:43 raspberrypi2 user.notice -s: log2ram test
Jul 16 15:24:10 raspberrypi2 user.notice root: log2ram test
Jul 16 15:24:35 raspberrypi2 user.notice root: log2ram test
[root@raspberrypi2:~]# cat /data/log/messages | grep log2ram
[root@raspberrypi2:~]# 

This proving the log has been synched to disk on shutdown/reboot

Watch out if you are copy-pasting the content of the files above with cat (EOF statements), that the variables (with $ prefix) can get omitted, so check the content and id missing copy-paste the content with some other methods (for example with cat directly).
For some reason I cannot upload the files to the post here, so I will provide link to them later.