Making AI Campaign Automation Idempotent and Fault-Tolerant

Imagine a workflow that generates a lead list, then records progress in Google Sheets, then sends outreach emails. If the engine crashes after generating the lead list but before writing to Sheets, a naive retry strategy might re-run the entire workflow from scratch — including sending the outreach emails a second time. For a marketing team, that's an embarrassing bug. For a prospect on the receiving end, it's a bad first impression of your product.

TenseAI was built with strict idempotency standards and stale-run reconciliation specifically to prevent this class of failure.

The Workflow Run State Machine

Every workflow run in TenseAI moves through a well-defined set of states:

[*] -> PENDING
PENDING -> RUNNING            (Acquire Run Lock)
RUNNING -> PAUSED              (Awaiting External Reply)
RUNNING -> COMPLETED           (All DAG Nodes Finished)
RUNNING -> FAILED              (Critical Exception Caught)
PAUSED  -> RUNNING             (Inbound Event Awakening)
FAILED  -> PENDING             (Idempotent Manual Resume)

The critical design decision here is that every transition is explicit and persisted — there's no implicit state inferred from "the last log line we saw." A run is always in exactly one of these states, and every worker in the system agrees on what that state means.

Three Mechanics That Make This Hold Up in Production

Step-Level Execution Tracking

Inside execution_service.py, every individual step execution updates a persistent step_statuses dictionary stored in PostgreSQL. Completed steps are stamped with execution hashes. This means the engine always knows, at a granular level, exactly which steps in a DAG have already run successfully — not just whether the overall workflow is "done."

Preflight Safety Assertions

Before any external API call fires, preflight.py validates the user's credit balance, API credentials, and required input fields. If any of these checks fail, the run transitions cleanly to FAILED before touching any external system. This is a deliberately conservative design: it's far better to reject a run early than to partially execute it and leave external state in an inconsistent condition.

Stale Run Reconciliation

Even with careful state tracking, worker crashes happen — a process can die mid-step, leaving a run stuck in RUNNING with no worker left to finish it. followup_worker.py runs a background reconciler that periodically scans for exactly these stuck runs. When it finds one, it safely reconciles the run's actual state (checking what steps genuinely completed) or marks it PAUSED, so it can be resumed — manually or automatically — without re-executing steps that already succeeded.

Why This Design Choice Matters

Idempotency isn't a feature you bolt on after the fact; it has to be a property of how state transitions and step tracking are designed from the start. By combining a strict state machine, step-level completion tracking with execution hashes, conservative preflight checks, and an active reconciliation worker, TenseAI ensures that high-volume outreach campaigns can fail safely — worker crashes and network blips get recovered from automatically, without ever risking a duplicate email landing in a prospect's inbox.

For teams running outreach at scale, this is the unglamorous engineering work that makes the difference between a tool you trust and one you have to babysit.

This wraps our current architecture series. Together, these five pieces — graph-first compilation, universal envelopes, modular tool nodes, dual-mode event execution, and idempotent state management — form the backbone of how TenseAI runs reliable, reactive, multi-tool AI automations at scale.