The pump that stopped every 40 minutes: a FreeRTOS priority inversion story

Field complaint: a pressure booster pump in an apartment building stops for 2 seconds roughly every 40 minutes. Water pressure drops, people on the 9th floor shout, the building manager calls our service guy, the service guy calls me. On the bench: nothing. Ran it for 3 days, no stop.

The system

STM32F4, FreeRTOS, three tasks that matter:

• MotorControl, priority 5, runs every 1 ms, kicks the watchdog through a check-in
• Comms (Modbus to the building BMS), priority 3
• Logger, priority 1, writes events to UART for the service tool

Comms and Logger share one UART, protected by a semaphore. And MotorControl also sends a fault frame on the same UART when something is out of range. You can already see it, no? I could not, for 2 weeks.

What was happening

The "lock" was created with xSemaphoreCreateBinary(). A binary semaphore has no priority inheritance. So:

1. Logger (prio 1) takes the UART lock and starts printing a long batch of events
2. Modbus poll arrives, Comms (prio 3) becomes ready and preempts Logger. Logger still holds the lock
3. MotorControl (prio 5) sees a small pressure spike, wants to send a fault frame, blocks on the lock
4. Now the highest priority task is waiting for the lowest priority task, which cannot run because the medium priority task is busy
5. Comms is busy for 180 ms because the BMS on this site polls a big block of registers. MotorControl misses its check-in, watchdog resets the MCU, pump stops 2 seconds

Why every ~40 minutes? The Logger flushes its batch every 40 minutes, and only on this site the BMS poll is big enough. On the bench we had no BMS. Classic Mars Pathfinder story from 1997, but in a building in Coimbatore.

How I found it

Four GPIOs, one per task, toggled at task entry and exit. Scope in normal trigger mode on the watchdog reset line, 4 channels, long memory. After a day of waiting on site with my laptop and a flask of filter coffee: MotorControl pin stops toggling, Logger pin high, Comms pin busy for 180 ms. Picture is very clear when you finally see it.

The fix (3 parts, the third one is the real one)

/* before */
uartLock = xSemaphoreCreateBinary();
xSemaphoreGive(uartLock);

/* after */
uartLock = xSemaphoreCreateMutex();      /* has priority inheritance */
faultQ   = xQueueCreate(8, sizeof(fault_t));

/* in MotorControl, every 1 ms */
if (fault_detected) {
    (void)xQueueSend(faultQ, &f, 0);     /* never block in the control loop */
}

1. Mutex instead of binary semaphore. With priority inheritance, Logger is temporarily raised to priority 5 while it holds the lock, so Comms cannot preempt it
2. Logger formats outside the lock, takes the lock only to push bytes into the DMA buffer. Lock held for microseconds, not milliseconds
3. MotorControl never touches the UART. It drops fault frames into a queue with zero timeout and goes back to work. Comms sends them. The control loop does not wait for anybody, ever

Priority inheritance is a bandage. The real fix is design: the most important task should never share a lock with the least important one.

Pump is running 3 months now without a single stop. The 9th floor is quiet. My service guy bought me a coffee, which in our company is the highest award.