Kotlin coroutines in a Spring service: three leaks we shipped and what structured concurrency fixed

We run our routing service in Kotlin on Spring Boot. Coroutines were introduced two years ago by enthusiastic people, including me. Here are the three mistakes we made, all of them explained in the kotlinx.coroutines docs, which none of us had read properly. Hence the topic.

1. GlobalScope for fire-and-forget

fun ship(order: Order) {
    GlobalScope.launch { auditClient.record(order.id, "shipped") }
    repository.markShipped(order.id)
}

It looks harmless. GlobalScope coroutines are not tied to anything. Nobody cancels them on shutdown, nobody sees their exceptions unless you install a handler, and nothing limits how many exist. One afternoon the audit database was slow and we had 50,000 coroutines waiting in memory. Then the pod restarted and we lost all of those audit records.

Fix: a scope with a lifecycle, owned by the application:

@Component
class AuditScope : DisposableBean {
    val scope = CoroutineScope(SupervisorJob() + Dispatchers.IO + CoroutineName("audit"))

    override fun destroy() = scope.cancel()
}

Honestly, for audit records the real fix was even more boring: write them to an outbox table in the same transaction and let a worker ship them. No coroutine needed. But if you do need background work, give it a scope that someone owns.

2. runBlocking inside a request

Someone needed to call a suspend function from a normal Spring MVC controller, so they wrapped it in runBlocking. That blocks the Tomcat request thread until the coroutine finishes, which defeats the purpose, and nested runBlocking calls on a limited dispatcher can deadlock. Spring MVC has supported suspend fun controller methods for a long time now. Make the controller suspend and delete the runBlocking.

3. Blocking JDBC on Dispatchers.Default

Dispatchers.Default has as many threads as you have CPU cores. It is for CPU work. We were doing JDBC calls on it. Four slow queries on a four core pod and every other coroutine in the service waited. Blocking calls go to Dispatchers.IO, and we limit it to the size of the Hikari pool, so we do not create 64 threads that all wait for 20 connections:

private val db = Dispatchers.IO.limitedParallelism(20) // = hikari maximumPoolSize

suspend fun findRoute(id: Long): Route = withContext(db) {
    jdbcTemplate.queryForObject("SELECT * FROM routes WHERE id = ?", routeMapper, id)!!
}

What structured concurrency gives you

suspend fun quote(orderId: Long): Quote = withTimeout(800) {
    coroutineScope {
        val rates = async { carrierClient.rates(orderId) }
        val customs = async { customsClient.duties(orderId) }
        Quote(rates.await(), customs.await())
    }
}

If the customs call throws, the rates call is cancelled automatically. If the whole thing takes longer than 800 ms, both are cancelled. Nothing is left running in the background. Use supervisorScope instead when one failure should not cancel the siblings.

The one that surprises everyone

Cancellation is cooperative. withTimeout cancels the coroutine, but a blocking JDBC call inside it does not know that. The coroutine stops waiting, the database keeps working, and the thread stays busy until the query returns. We had a "2 second timeout" endpoint with queries running for 40 seconds. Set the timeout on the database side as well: a query timeout on the statement, or statement_timeout in Postgres. Both timeouts, or neither.

All of this is in the docs. I read them on the train last week because it rained too hard even for the bike. Do not tell anyone.