Design durable execution for a crash-prone agent

agents, durability, idempotency

Prompt

Your coding agent runs multi-step tasks (clone, edit, test, commit) that can take ten-plus minutes and involve side-effecting actions. Right now, if the worker process crashes or is killed mid-task, the whole task restarts from scratch, sometimes repeating side effects. Design a durable execution layer for this.

Interviewer's answers to clarifying questionsopen after you have asked your own

Typical task? 5-10 discrete steps, each calling a tool with a real side effect (file write, git commit, API call to a ticketing system).
Crash frequency? Worker processes get killed/restarted on roughly 1-2% of long-running tasks (deploys, OOM, spot-instance reclaim).
Current failure mode? On restart, the agent redoes all steps from step 1, including side effects that already happened (duplicate commits, duplicate ticket comments).
Also observed? A flaky downstream API fails with slightly different error text each time, so naive identical-retry detection never catches the loop, and a run has burned $40 of budget before anyone notices.
Requirement? After a crash and restart, no side-effecting tool call should run twice, and the task should resume rather than restart.

Follow-upsafter your high-level design

Walk through exactly what happens, step by step, when the process is killed between step 4 and step 5 of a 7-step task, and the worker restarts.
How do you tell the difference between "the agent is doing a legitimately long multi-step task" and "the agent is stuck in a loop" automatically?
The flaky API's error text varies on every call. How does your design still recognize this as the same repeated failure and stop it?
Now add human approval before any destructive step. How does that interact with your durability design, given the task might wait hours for a human to respond?

Answer first: out loud, on paper, or with the interview prompt above. Then compare with the rubric.