Cerbo GX running low on memory

Has anyone else experienced the following issue running Venus OS 3.6x and newer?
I had a VenusGX in my system that I upgraded from 3.5x to 3.6x and it started to reload every 12 to 24 hours due to watchdog issues. I thought it was just my system putting too much strain on it and upgraded to a Cerbo.

Once done the reloads stopped but I noticed that Free Memory on a constant decline. At some point I think it will either run out and reboot OR it’s like Windows that takes memory for caching and releases it when needed. Below I have some stats of the Cerbo.

The first little hiccup (under the popup) is where I disabled Modbus and the second one is where I stopped my pvoutput.org script. It made no significant difference to the trend.

Below is a capture of free memory over time:

root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      445288      383544       10104      200160      545236
Swap:             0           0           0
root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      444200      384636       10096      200156      546332
Swap:             0           0           0
root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      444616      384216       10096      200160      545912
Swap:             0           0           0
root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      444600      384232       10096      200160      545932
Swap:             0           0           0
root@einstein:~# !Modbus Disabled Sunday, 3 Aug 21:00.
-sh: !Modbus: event not found
root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      444460      384372       10096      200160      546072
Swap:             0           0           0
root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      443880      384928       10112      200184      546636
Swap:             0           0           0
root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      480840      347816       10112      200336      509676
Swap:             0           0           0
root@einstein:~# !Stopped PVOutput script.
-sh: !Stopped: event not found
root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      469508      359132       10116      200352      521004
Swap:             0           0           0
root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      469104      359556       10116      200332      521408
Swap:             0           0           0
root@einstein:~# free
              total        used        free      shared  buff/cache   available
Mem:        1028992      483588      345056       10116      200348      506924
Swap:             0           0           0
root@einstein:~#

Next I have a ps output showing venus-platform growing over time.

root@einstein:~# ps | sort -k3 -n -r | head
  816 root     57672 S    /opt/victronenergy/venus-platform/venus-platform
  909 root     56144 S    {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 1729 root     52244 S    {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1730 root     48268 S    /usr/bin/flashmq
  811 root     44236 S    {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
  833 root     41600 S    {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
24089 root     37580 S    python ./dbus-pvoutput-p4.py
  897 root     32952 S    {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
  892 root     31268 S    {netmon} /usr/bin/python3 -u /opt/victronenergy/netmon/netmon
  874 root     29588 S    {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
root@einstein:~# ps | sort -k3 -n -r | head
  816 root     59124 S    /opt/victronenergy/venus-platform/venus-platform
  909 root     56144 S    {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 1729 root     52244 S    {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1730 root     48268 S    /usr/bin/flashmq
  811 root     44236 S    {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
  833 root     41600 S    {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
24089 root     37580 R    python ./dbus-pvoutput-p4.py
  897 root     32952 S    {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
  892 root     31268 S    {netmon} /usr/bin/python3 -u /opt/victronenergy/netmon/netmon
  874 root     29588 R    {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
root@einstein:~# ps | sort -k3 -n -r | head
  816 root     64872 S    /opt/victronenergy/venus-platform/venus-platform
  909 root     56144 S    {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 1729 root     52244 S    {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1730 root     48268 S    /usr/bin/flashmq
  811 root     44236 S    {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
  833 root     41600 S    {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
24089 root     37580 S    python ./dbus-pvoutput-p4.py
  897 root     32952 S    {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
  892 root     31788 S    {netmon} /usr/bin/python3 -u /opt/victronenergy/netmon/netmon
  874 root     29588 S    {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
root@einstein:~# ps | sort -k3 -n -r | head
  816 root     66196 S    /opt/victronenergy/venus-platform/venus-platform
  909 root     56144 S    {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 1729 root     52244 S    {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1730 root     48268 S    /usr/bin/flashmq
  811 root     44236 S    {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
  833 root     41600 S    {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
  897 root     32952 S    {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
  892 root     31788 S    {netmon} /usr/bin/python3 -u /opt/victronenergy/netmon/netmon
  874 root     29588 S    {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
  895 root     28852 S    {dbus_digitalinp} /usr/bin/python3 -u /opt/victronenergy/dbus-digitalinputs/dbus_digitalinputs.py /dev/gpio/digital_input_1 /dev/g
root@einstein:~#

Should I be concerned as this happened after the Venus upgrade to 3.6x. 3.5x was stable. Unfortunately I don’t have the same stats on the Venus.

Ingo

Sounds like Venus-platform has a memory leak. I’ll ask around.

1 Like

This is interesting…running Venus software on a Raspberry pi with 8GB SD card

Crashed, we thought it was because old card and the read/write cycles has been the death knell. Happened just after update to newest software.

Made new SD card…first tried with anothe 8GB card and then 32GB which was floating around and issue solved.

An observation…the software is installed and then the card is partitioned into three equal parts on the leftover space with first start.
Same happened on 32GB card.
That let me to surmise that the 8GB was just getting too small…