Long-running AI agent workflows all hit the same first bug in production, and it's never the model. Every one I've shipped worked fine on my laptop, then died the first time a real user ran it, because the process holding the agent stopped existing halfway through the job.
That's the gap nobody demos. A chat completion takes four seconds. An agent that researches a topic, calls three tools, generates a video, and publishes the result takes four minutes on a good day and twenty on a bad one. These things fail for boring infrastructure reasons long before they fail for interesting AI reasons.
Here's what I've learned building them.
The 60-second lie
Your first agent works because it's short. You wire an LLM call into an API route, it returns in a few seconds, everyone claps.
Then you add tools. Then a retry. Then a video model that takes six minutes to render one clip. Now your request is sitting inside a serverless function with a hard execution ceiling. Vercel caps function duration by plan, Lambda stops at 15 minutes no matter what you pay. The function gets killed mid-flight, the user sees a spinner that never resolves, and your logs show nothing useful because the process that had the context is gone.
On Threadovo, one video job fans out into script generation, per-scene image generation, image-to-video, a quality check, and a mux step. There is no timeout on earth that makes that a single HTTP request. So the first real architectural decision wasn't which model to use. It was accepting that the agent does not live inside the request.
The rule I follow now: the HTTP request's only job is to create a record and hand back an ID. Everything else happens somewhere the user's connection can't kill.
Your agent's state belongs in a database, not a variable
Once the work moves out of the request, the next mistake is keeping the agent's state in memory in the worker instead. Same bug, longer fuse. The worker restarts on deploy, or the box gets recycled, and a twelve-minute job evaporates at minute eleven.
Treat every step boundary as a place the machine might die. Each step writes its output before the next step reads it. The job row knows exactly which step it's on. Resuming means reading that row, not replaying from scratch.
The test I use is simple and slightly rude: kill the worker at a random point during a run and restart it. If the job picks up where it left off, the design holds. If it starts over, or half starts over, you don't have a workflow, you have a long function with optimism attached.
This is also what separates durable execution from a plain queue. A queue gets your job off the request. Durable execution, the pattern behind tools like Temporal, Inngest, and Restate, persists state at every step so a crash resumes instead of restarting. Inngest's write-up on durable execution for agents is a decent primer on why retry logic alone doesn't get you there.
Retries will charge your users twice
Here's the failure that actually costs money.
Your job dies after the step that debits credits but before the step that delivers the output. The queue redelivers. The debit runs again. Now a user has paid twice for one video, and you find out through a support email.
Any step with a side effect outside your process needs to be idempotent: billing, publishing to a social account, sending an email, releasing funds. On Threadovo, credits for expensive generation steps are debited once per job, keyed to the job rather than the attempt, and refunded on terminal failure instead of quietly eaten. On Scrivane, the same reasoning covers installment charges. A retried webhook must never turn one installment into two.
The mechanics are unglamorous. Give every side-effecting step a stable key derived from the job. Check "did this already happen" against durable storage before doing it, inside the same transaction where you can. And make failure states explicit, because failed_after_charge is a real state that needs a real resolution path rather than a shrug and a manual refund three days later.
Assume every step runs at least twice and design so only the first one counts.
Sometimes the job needs to sleep for two days
The moment you put a human in the loop, your workflow has to wait, occasionally for days.
Approval gates are the obvious case. An autopilot that drafts posts and waits for the owner to approve them can't hold a connection open for eighteen hours. It has to suspend, persist, and wake on either an approval event or a timeout, then take the default action. Same for anything waiting on an external callback: a rendering provider's webhook, a payment confirmation, a verification window.
If your architecture treats waiting as still running, you'll pay for idle compute and lose the workflow to the next deploy. Waiting should cost nothing and survive everything.
If the user can't see it, it's broken
A four-minute job with no visible progress feels identical to a crashed one. Users refresh. Refreshing spawns a second job if you haven't guarded against it, and now you're debugging duplicate work you created yourself.
So persist the step trail, don't just stream it. Streaming progress over SSE is great until the user closes the tab, at which point the trail is gone and the reload shows an empty screen. I write each step to the job record as it completes and stream from that, so a reconnect replays real history instead of starting blank.
Same principle in Intavue, where a live voice interview has to survive a dropped connection without losing the session. If the only copy of what happened lives in a socket, you're one flaky hotel wifi away from data loss.
What I'd actually reach for
If you're starting today on a serverless stack, don't build the orchestrator yourself. Use something with durable steps, whether that's Inngest, Temporal, or a queue with real scheduling on top, and spend your effort on the parts that are specific to your product.
If you already run your own workers, you can get most of the way there with a job table, explicit step states, and a scheduler that redelivers. That's roughly what sits behind Threadovo's autopilot and settlement flows. It's more code than it looks like, and the part that bites is always ordering. Remove the old scheduled job before adding the new one, or you'll quietly end up with two of them firing.
Keep all of this separate from provider failure, though. An agent that resumes correctly can still be dead in the water when your model vendor goes down. That's a different layer with a different fix, which I covered in keeping your AI feature alive during an LLM provider outage.
My failure drill for long-running AI agent workflows
Before I call one production-ready, I run four tests by hand.
- Kill the worker mid-job. Does it resume at the right step, or start over?
- Deliver the same message twice. Does anything get charged, published, or emailed twice?
- Make a tool call hang. Does the step time out and retry, or does the job sit forever in
running? - Fail the last step. Does the user get their credits back and a real error, or a stuck job and silence?
Every one of those has caught something real for me. Two and four are the ones that catch the bugs that cost money.
None of this is AI engineering, strictly speaking. It's distributed systems work wearing a new hat, and it's why so many impressive agent demos never survive contact with actual users. Everyone focuses on getting the model right. Whether the job finishes is what decides if you ship. If you're also trying to work out whether your agent produces good output end to end, agent evaluation covers the other half of this.
Build for the crash first. The interesting AI problems will still be there afterward.