The CAN node that went bus-off only on cold mornings

This was some years ago but I still think about it every winter. We had a body control module, nothing special, talking on CAN at 500 kbit/s with the other modules. On the test bench everything perfect. In the field, some customers report that the module "die" in the morning and come back after the car is driving some minutes. Only in the north. Only in winter.

First we suspect the power supply, because cold battery, cranking, voltage drop. We put a logger in a car inside a climatic chamber at minus 20. The voltage was fine. But the CAN error counters, we see the transmit error counter climbing very fast after start, until it pass 255 and the node go to bus-off. After the cabin warm up, the errors stop.

Took us two weeks to find. In the last hardware revision, for cost down, somebody change the external crystal to use the internal RC oscillator of the microcontroller. The internal oscillator is fine for many things but its tolerance over temperature is much worse than a crystal. At room temperature the bit timing was ok. At minus 20 the clock drift enough that our sample point was falling in the wrong place, other nodes see our bits wrong, error frames, error counter go up, bus-off.

CAN bit timing has a budget for oscillator tolerance, it depends of your sample point and the synchronization jump width, and is not so big. At 500 kbit/s you really want a crystal or a good ceramic resonator, not an RC oscillator. The datasheet even say it, in small letters, in a table nobody read.

What we change:

• crystal back on the board (the cost down saved like 15 cents per unit, the investigation cost much, much more)
• sample point at around 87.5%, the usual recommendation for automotive, and the same configuration on all the nodes
• temperature cycling in the validation plan for every hardware change, also the "small" ones
• we log the error counters in the diagnostic memory, so next time we see it from the workshop and not from angry customers

Lesson for me: when a bug only happen at some temperature, some voltage, or some time of the day, don't look only at the software. Is very often physics.