Don't upgrade Multiplus II firmware with AC Connected/or a fully charged battery (cautionary tale)

I just upgraded the firmware on a Multiplus II through VRM from 510 to 552. The Pylontech batteries were fully charged and sitting at 52.4V before the firmware update. This is what happened to the battery bank immediately after the upgrade:

image

Voltage spiked to over 55V and then dropped down to 53.2V until I could upload the migrated settings file. Only then did it settle back to the normal 52.4V. Warranty on the batteries is probably gone.

The VRM website claims the inverter is switched off after the upgrade, waiting for reconfiguration, but that is most certainly not the case.

2 Likes

The documentation, located here:
https://www.victronenergy.com/live/venus-os:remote_ve.bus_firmware_updates

States this:

The documentation does not tally with what I experienced today. Looks like a rather serious bug to me.

The Multi is definitely placed in a mode where charging is disabled.

Firmware is updated with a utility called vbdup. This has an option -e which leaves the Multi in Error 16 after an update. This prevents the Multi from starting up at all after the firmware update.

Then, there is a utility called vbsc that restores the “cluster” configuration, eg if you have a three phase system or parallel units, this puts them back the way they were after the firmware update (initially all units in standalone mode). Vbsc also clears error 16, but it sets the charge voltage as low as it will go to prevent charging.

And then your new config is loaded.

So whatever happened here… this is not by design.

1 Like

Been waiting for the explanation.

Now I’m wondering, if the MP was in a upgrade state, could the MPPT be the cause?

There’s no MPPT. It’s just a UPS with batteries, hence why the batteries were at 100%.

Its things like this that make me weary of unnecessary upgrades. I am like a magnet for bad luck.

1 Like

Is there potential for a small window in which the charger is enabled, but the voltage has not been set yet? This all happened in under a minute. Perhaps with a smaller inverter and a larger battery bank this would have been a non-issue, but this is a temporary setup just to get everything updated and configured so there’s only two US2000 batteries on the 8kW MP2. The output of the MP2 is not connected to anything yet.

What was curious was that after it shot up to past 55V it came down to 53.2V for at least 10 seconds, which is the voltage that the Pylontech batteries requests, but of course the Cerbo is supposed to limit Pylontech to 52.4V.

According to the documentation Venus OS will override the “no charging” situation, but it is supposed to respect the voltage and current limits requested by the batteries. If the Multi was not at fault here, is there potential for a bug in Venus OS? The fact that it was stuck at 53.2V for a while is very suspicious.

Please let me know if there’s anything I can do to help get this fixed.

It is possible that something went wrong when the cluster config was restored. So it seems I will have to see if I can reproduce this.

There is a window of a few seconds after the Multi comes up, before it is fully detected by the GX and DVCC kicks in, and this is certainly enough to overvolt a battery. But, it is still supposed to set the voltages to the base voltage (48V in this case) to ensure this cannot happen. This is how it was designed. Maybe there is some kind of stupid regression somewhere.

About the voltage going up to 55V. That’s probably because the BMS disconnected at 54V. After that, you were measuring the voltage of the DC bus, not the actual voltage of the battery. It probably came down to 53.2V because that’s just where the cells happened to be when it reconnected.

1 Like

Are the batteries still off? If you restart them they should boot up. The BMS will record the event but I don’t think your warranty is gone. I’m sure Pylontech support would take this sort of thing into account.
I would record the event and keep it for future if needed.

As an aside… When and only if I upgrade my firmware I normally disconnect the AC and DC charging sources.

OK guys, I reproduced this. On a non-pylontech system, but it is the same. Two things wrong. First, the vbsc backup doesn’t work (this thankfully has no effect on a single-unit system), and second, it charges right after the firmware update. I have the right people on the case. Thanks for reporting.

6 Likes

Man. That tells you a lot, 100% service focused from Victron and staff.

How it’s recreated so quickly by @plonkster is impressive as well :wink:

1 Like

I appreciate the loyalty, but this sort of thing ideally shouldn’t happen. The system thinks the unit is a 1900 model (instead of a 2600 model), and therefore skips some of the steps. One of those steps is the “switch off the charging” step. It’s on the radar now.

3 Likes

But it does … in all industries (service and product) and contexts. Responsiveness to WHEN it happens is the key.

4 Likes

The Batteries seem to be working ok. The cell voltages shot up to 3.8V, which is still below the range where really bad things start happening. It was also for a very short time, so if there is damage it is likely minimal.

The warranty is the only thing I’m really worried about. I hope you’re right about that.

1 Like

Thank you very much.

I have noticed before that it charges immediately after a firmware update, but I assumed it must be Venus OS that is overriding the charging behaviour (as it states in the documentation it would do), and that it would respect the voltage and current limits requested by the battery. Others likely made the same assumption, which is why it was not reported before.

This was the first time that I’ve done a firmware update with both a fully charged battery bank and the AC input connected, and as it exceeded both the current limit (which was 0 at the time) as well as charge voltage, I knew something must be wrong.

1 Like

If it isn’t a regular thing then I think you are OK. Are there no alarms on the Pylontech… You can check on the GX under the battery menu.

I can see all of us getting your fix in our next firmware upgrade which is good for the whole victron community. Your name could be somewhere in the change log :wink:

Must also say it would only be an issue with the scenario you had. No issue otherwise so a very specific use case.

Echoing what Mark said, it is not really loyalty in the true sense of the matter. We are not foaming at the mouth being rabid about Victron either.

It is all about the fact that Victron actually listens, has forums, feedback, you do your part where you can … even the big boss sometimes look into issues mentioned/reported.

This thread just one of many examples.

THAT is what counts the most in my book. Clients books.

Cause shiite always goes wrong man. It is a given … ask me, I’m an expert in that.

1 Like

OK, bad news. The bug I found was a different bug that affects my specific Multi (a 12V 3kVA, model 2600, the lowest one in the range). So we’re looking for a second issue now.

2 Likes

I pulled the logs from one of the batteries. The voltage shot up beyond 56V for a few seconds and the charge current was also beyond the 50A maximum:

Luckily this is the older US2000B which is more resilient against overvoltage. If it was the US2000C the control board could well have been fried.

The overcurrent would have occurred even at a lower SoC, so a more accurate title for this cautionary thread would be “Don’t upgrade Multiplus II firmware with AC connected.”