Notes on building an execution engine that never guesses

Every automation platform makes a promise the first time you press Run: that the thing you designed is the thing that will happen. Keeping that promise on the hundred-thousandth run, on a day when a third-party API is slow and a server is being recycled mid-execution, turns out to be most of the actual engineering. This is a look at how we built Lodol's execution engine to keep it.

The guiding idea is short enough to fit on a sticky note. The engine should never guess. Almost everything below is a consequence of taking that seriously.

Keep the model out of the hot path

The most consequential decision we made was to keep the language model out of the execution path entirely. Whatever happens while you're designing a workflow, no model sits in the loop at runtime deciding what to do next.

When you build a workflow, it compiles down to an explicit plan: a graph of steps with defined inputs, outputs, branches, and conditions. Executing that plan is pure machinery. Given the same inputs, a step produces the same outputs, whether it's the tenth run or the ten-thousandth. Nothing is reinterpreting your intent on each execution, so there's no run-to-run drift for you to chase down at midnight. This is the difference between a recipe and a conversation. A conversation can wander somewhere new every time you have it. A recipe, followed exactly, lands in the same place. We wanted the execution engine to be a recipe.

Server-side execution, because a browser tab is a bad place to keep a promise

Early on, workflows executed client-side. It was simple, it demoed fine, and it was quietly fragile, because closing the tab took the run down with it. So we moved execution to the server, and that one change unlocked most of what followed.

Once the server owns execution, a workflow's fate stops depending on someone's laptop, network, or attention span. It can run on a schedule at 3am. It can be triggered by an inbound email. It can be paused from one device and resumed from another. And it can survive the infrastructure underneath it changing shape, which is the part that turns out to matter most.

Checkpointing: surviving the machine that runs you

Here is a scenario every serious execution engine has to answer for. A workflow is twelve steps into a twenty-step run. It has already charged a card and posted a ledger entry. Then the server it's running on gets recycled during a routine autoscaling event.

Both of the easy answers are unacceptable. Restart from the top and you charge the card twice. Drop the run and you've left the books half-written. Neither is a sentence you want to say to a finance team.

So the engine checkpoints. As a run moves forward, its state is recorded durably, step by step. When execution gets interrupted by a deploy, a crash, or an autoscale event, a fresh worker picks the run back up from the last committed checkpoint instead of the beginning. Pair that with steps designed to be idempotent and an interruption costs you a short delay rather than a corrupted outcome. The work that already happened stays happened, and the work that's left gets finished exactly once. The bar we hold ourselves to is simple to state: you should be able to kill a server in the middle of a run and have the result come out indistinguishable from a run that was never touched.

Triggers that hold up when the fleet is moving

A workflow that runs when you click a button is easy. A workflow that reliably fires the moment a Gmail message arrives, while the fleet underneath it is scaling up and down to meet load, is where the interesting failure modes live: the missed event, the duplicated event, the event handed to a worker that no longer exists.

We gave each workflow its own event queue and made trigger delivery robust to the fleet changing shape underneath it. Polling runs on a predictable cadence, streaming connections reconnect on their own when they drop, and scaling events don't turn into missed or double-fired triggers. The outcome we're after is boring on purpose: your workflow fires when it should, once, no matter what the infrastructure happens to be doing that minute.

Make the run legible

Determinism and durability aren't worth much if you can't see what the engine did, so observability isn't a layer we bolted on afterward. It's part of the execution model. Every run produces step-level detail: the path that was actually taken, the time each step spent, and a snapshot of the variables as they flowed through. Open a finished run and you can watch the real executed path, which branch it chose, what each loop iteration did, what a step received and returned. When something needs explaining, the answer is in the run itself rather than in a reconstruction you perform from memory. A divide-by-zero shows up as a plain message instead of a stack trace. A run that was interrupted and recovered reads cleanly rather than as a mystery.

Why boring is the goal

None of this is flashy. There's no demo where "the run resumed correctly after a server recycle" gets applause, and that's rather the point. The teams running payroll, reconciliations, and compliance checks on Lodol don't need their automation engine to be exciting. They need it to be there, to be correct, and to be explainable, on the worst day as reliably as the best one. An engine that never guesses is how you get there. The checkpoints, the server-side model, the per-workflow queues, the traces: all of it is just what it takes to keep one plain promise at scale.

Curious how a run holds up under real conditions?
We're happy to walk through the execution model, checkpointing, and the run traces with your team.