Two Valid Commands, One Wrong Mode

During integration testing, I noticed that a single On action in a supervisory interface would sometimes leave an inverter/charger in Inverter only. Not every time. Often enough to be real, and rarely enough to be written off as a glitch if nobody was keeping count.

This inverter/charger has one operating mode with four values: off, charger only, inverter only, or on, meaning both. RV-C does not expose that mode as one thing. It gives the charger and the inverter separate command messages, each with its own enable, so a controller that wants the device fully on sends a charger enable and an inverter enable. The interface presented one control to the user. Behind it, that one control issued those two commands back-to-back, and the receiving driver had to take two independent requests and compose them into a single mode value for the device. That composition is where the trouble lived.

The shape of that trouble is old and well named. Each command handler did something like: read the current mode, change the half of it you are responsible for, write the whole thing back. Run one at a time, that is fine. Run two of them close enough together that the second reads the mode before the first has written it, and the second write carries a stale copy of the first handler’s half. The figure walks through it. The charger handler reads off and writes charger only. The inverter handler, having read off a moment earlier, writes inverter only on top of it. The device obeys the last write it hears, and the charger enable has vanished without an error anywhere, because every individual step did exactly what it was supposed to do. Drag the second command to the right, past the first handler’s write, and the inverter handler reads charger only instead; its write then keeps both enables, and the result is on. Whether you see the failure depends on how far apart the two commands land relative to how long a read-modify-write takes, which is why it was intermittent, and why reproducing it reliably took a machine sending the two commands rather than a person pressing a button.

I documented the expected and observed state transitions, narrowed the failure to the interaction between the two commands rather than to either command on its own, and brought the race-condition hypothesis to the manufacturer’s engineering team. They reproduced the behavior on physical hardware, confirmed the race, and built an automated sequence test to exercise it, which is the right tool for a fault whose trigger is a timing window measured in milliseconds. A revised driver is on the test bench now; final validation is still pending.

What I keep returning to is that neither command was wrong. Both were valid RV-C. The failure came entirely from how the driver combined them, and that is the general lesson. When separate asynchronous controls change one underlying operating mode, the receiving implementation has to define which state each handler reads, how nearly simultaneous updates are ordered, and when the combined mode is finally written to the device. There are several honest ways to do that: keep the desired state of each half in one place and derive the mode from it rather than from whatever the device last reported, serialize the handlers so one finishes before the next begins, or hold the write until a burst of commands has settled. Switch the figure’s handlers to serialized and the second read waits for the first write; the result is on wherever the second command lands. Which one the manufacturer chose is their call. Leave the question unanswered, though, and two correct commands can still produce one wrong mode.