Chapter 10. Idempotency, Retries, Rate Limits, and Rollback Boundaries¶

1. Why the Most Expensive Failures Often Look Like "We Just Repeated the Call"¶

Once an agent system starts performing real actions, one of the ugliest incident sources turns out to be very mundane: a repeated call.

From the outside it often looks harmless:

• the request hung;
• the runtime retried;
• the integration was unstable;
• the agent tried to "help" and sent the action again.

In the real world that means:

• two identical tickets;
• two emails to the same customer;
• a repeated charge;
• a duplicate CRM write;
• several modifications to the same object.

So the problem is not the model's reasoning by itself. The problem is that the execution layer cannot live safely in a world of partial failure.

2. Idempotency Is Not Nice-to-Have, It Is Basic Insurance¶

Idempotency exists for one simple question: "If the system repeats this call by mistake, what happens?"

For write operations, a good answer should be one of two things:

• the repeated call does not change the outcome;
• the system can reliably detect the duplicate and avoid creating the side effect twice.

Without that property, any network instability, timeout, or race between runs becomes too expensive.

3. Retry Without Error Classification Multiplies Chaos¶

A very bad strategy looks like this: "if the request failed, retry it three times".

In production that is dangerous, because not all failures are equal:

• validation_failure almost never deserves a retry;
• permission_denied cannot be fixed by repetition;
• retryable_failure may require backoff;
• side_effect_unknown requires caution, not blind repetition.

So retry policy should depend on outcome class, not on the feeling of "maybe it will work now".

4. The Worst Status Is side_effect_unknown¶

There is one especially painful category of failure: you no longer know whether the side effect happened.

Examples:

• timeout after the request was sent to the external service;
• connection loss after commit;
• adapter crash before confirmation was stored;
• an external API response that leaves final state unclear.

This is exactly where naive retry is most dangerous. Sometimes the right behavior is not "retry", but:

• check current state in the external system;
• run reconciliation;
• involve a human;
• stop the workflow and explicitly record the uncertainty.

Reliability case-spine note: retries, rate limits, and rollback boundaries should be tested across all three canonical cases. Support triage needs idempotency keys, duplicate-ticket detection, and reconciliation before retry. Internal knowledge assistant needs rate limits for retrieval fan-out, freshness backoff, and a ban on repeating stale memory writes. Incident coordination needs escalation de-duplication, notification throttling, a clear rollback boundary for incident-state changes, and human confirmation on side_effect_unknown.

4.1. The recovery branch should also be designed explicitly¶

Recent work on tool-failure cases reinforces another practical point: the recovery path should rarely be left as improvisation inside the execution layer.

It is useful to define in advance:

• which outcome classes require reconcile rather than retry;
• where partial success should create a follow-up task;
• when recovery itself requires approval;
• which branches must appear in the eval schema;
• where a dangerous recovery path should stop instead of “trying one more time”.

Many severe incidents happen not on the happy path, but in a poorly designed recovery branch.

5. Rate Limits Are Also Part of Safety Design¶

When teams think about rate limits only as a performance problem, the execution layer underestimates their architectural role.

In reality, rate limits also protect against:

• one runaway agent DDoS-ing an external system;
• cyclic planning turning into a storm of tool calls;
• high-cost capabilities consuming the whole budget;
• retry storms killing an integration.

That is why limits are useful not only at the whole-service level, but also:

• per tool;
• per tenant;
• per workflow;
• per risk class.

5.1. A shared quota needs a shared cooldown, not just local backoff¶

Independent runs can each obey local backoff while making overload worse for peers sharing one provider or gateway quota. Cloudflare's CryptoLabe uses a single global Durable Object to pace every model request, including retries; a rate limit encountered by one scan triggers a cooldown shared by all scans.³

The book's proposed contract is initial request or retry → bounded shared queue → regulator permit → dispatch. Define the coordination scope by the actual quota pool: provider/gateway, account or project, model/group and region—only dimensions that actually determine the limit. Different credentials may consume the same quota, so a limiter per process is insufficient. Do not pause independent pools together. Store a safe pool reference, not secrets.

On a confirmed transient rate limit, the adapter updates shared cooldown_until without shortening an existing pause, respects valid Retry-After, and otherwise uses policy-bounded backoff. After cooldown, release requests gradually with concurrency limits and jitter, not as one simultaneous queue flush. The pause does not cancel requests already sent. Bound queue length, deadlines and cancellation, with fair allocation across runs/tenants; preserve local attempt and cost budgets.

Every actual network attempt must use this path, including hidden SDK/workflow retries and fallback that shares the same pool. Integrate built-in retries with the regulator or disable them in favor of one retry owner. Request pacing does not replace token, concurrency, spending or idempotency controls.

Not every 429 means transient overload: an exhausted spend limit needs an explicit budget decision, not waiting and spending again. Classify the cause using documented provider responses; when unknown, bound further attempts and escalate. Shared cooldown does not reset the task deadline. This is a recommended contract, not an implemented reference-runtime feature; see Chapter 16 for coordinator placement and recovery.

6. Rollback Boundaries Must Be Defined in Advance¶

It is dangerous to discover only during an incident that an operation is "not actually reversible".

For every write capability, it helps to know in advance:

• can the action be undone;
• can the action be safely repeated;
• where the point of no return is;
• what compensating action is acceptable;
• when manual reconciliation is required.

Rollback boundary is not "we will decide later". It is part of the tool and workflow contract.

flowchart TD
    A["Tool request"] --> B["Execute write action"]
    B --> C{"Outcome known?"}
    C -->|Yes, success| D["Store result and continue"]
    C -->|Retryable failure| E["Retry with policy and backoff"]
    C -->|Unknown side effect| F["Reconcile or request human review"]
    C -->|Validation or permission failure| G["Stop and surface error"]

7. A Good Execution Contract Stores Operational Semantics Explicitly¶

Knowing only the input schema and response shape is not enough. For dangerous operations, the contract must also store operational semantics:

• whether the action is idempotent;
• whether an idempotency key is required;
• which failures are retryable;
• what the retry limit is;
• what rate limits apply;
• what to do on unknown outcome;
• whether rollback or compensating action exists.

tools:
  create_ticket:
    mode: write
    idempotent: true
    idempotency_key_required: true
    retry:
      max_attempts: 3
      backoff: exponential
      retry_on: ["retryable_failure"]
    rate_limit:
      per_tenant_per_minute: 20
    rollback:
      strategy: "none"
      reconcile_on_unknown: true

  send_email:
    mode: write
    idempotent: false
    idempotency_key_required: false
    retry:
      max_attempts: 1
      retry_on: []
    rate_limit:
      per_user_per_hour: 10
    rollback:
      strategy: "manual_only"
      reconcile_on_unknown: true

The value of that YAML is that it forces the team to admit an uncomfortable truth: not every action is equally safe to automate.

7.1. A Durable Step Is Not Progress Reporting¶

Cloudflare Agents SDK usefully separates work that stays inside the agent from work that belongs in Workflows: the agent owns live communication and the interaction boundary state, while the workflow owns durable multi-step execution, automatic retries, waiting for external events, and recovery.²

For this chapter, that distinction is a reliability boundary. A durable step should have a stable identifier, idempotency key, replay policy, retry policy, timeout, and audit linkage. Progress reporting — a WebSocket message, streaming update, or UI state — should not become the source of truth. If progress was shown but the durable step was not committed, the runtime should resume from the last durable checkpoint, not trust the last message visible to the user.

8. A Simple Retry Decision Example¶

This is not a production policy engine, but a simple skeleton. Its job is to show that retry must depend on outcome class, not on a generic reflex.

from dataclasses import dataclass


@dataclass
class ExecutionOutcome:
    status: str
    attempts: int
    max_attempts: int


def next_step(outcome: ExecutionOutcome) -> str:
    if outcome.status in {"validation_failure", "permission_denied"}:
        return "stop"
    if outcome.status == "retryable_failure" and outcome.attempts < outcome.max_attempts:
        return "retry_with_backoff"
    if outcome.status == "side_effect_unknown":
        return "reconcile"
    if outcome.status == "success":
        return "continue"
    return "escalate"

The important thing is not the code itself, but the fact that the system now has an explicit operational decision table.

9. The Agent Loop Needs Explicit Stop Conditions¶

Another practical point from the OpenAI guide is worth making formal: a run loop needs clear stop conditions.¹

A good runtime ends a run not “when the model seems calm,” but for explicit reasons:

• a final structured result is produced;
• no more tool calls are needed;
• an unrecoverable error arrives;
• a step or budget limit is reached;
• an approval boundary fires and a human is needed.

This sounds boring, but these rules are exactly what prevent an agent from drifting into endless planning, pointless retries, or decorative tool hops.

10. The Idempotency Key Must Be Part of the Protocol¶

If a write tool "supports idempotency" only on paper, but the key:

• is optional;
• is generated differently in different layers;
• does not survive the retry path;
• is not logged,

then it is hardly real idempotency.

Good practice:

• generate the key at workflow or action boundary;
• carry it through the full execution path;
• log it in audit trails;
• use it for reconciliation and investigations.

Paid calls: request retry is not payment retry¶

A paid tool has at least three independent outcomes: whether the operation executed, whether payment settled, and whether the result was delivered. Monetization Gateway documents this order: authorization verification → origin execution and response preparation → settlement → delivery to the buyer.⁴ A lost response therefore does not prove there was no charge, and settlement failure does not prove that the origin did nothing.

Book recommendation: correlate business-operation, request and payment-authorization identities, but track payment_state, execution_state and result_state separately with explicit unknown values. An unsigned HTTP 402 is an offer of terms, not permission to pay. Retrying with the first approved signature is a protocol step; retrying after a timeout requires reconciliation. Do not issue a fresh payment authorization or repeat the business action before checking settlement, operation status and retrieval of any saved result.

Payment idempotency does not imply tool idempotency. Use keys and reconciliation channels supported by both parties; an arbitrary header does not create deduplication. If the integration cannot determine the outcome, retain uncertainty and escalate rather than automatically charge again. Refunds, business compensation and result redelivery are separate operations with separate authority. This is an architectural recommendation, not an exactly-once guarantee from x402 or Cloudflare.

11. Common Mistakes¶

Typical failures repeat:

• retries are sent to errors that should never be retried;
• unknown outcomes are treated like ordinary failures;
• rate limits exist only at ingress, not on tools;
• rollback is promised but does not actually exist;
• the idempotency key is lost between planner and adapter;
• the agent has too much freedom in repeated attempts.

All of this means the execution layer has not yet grown into a production-grade failure model.

12. What to Do Right Away¶

Start with this short list and mark every "no" explicitly:

• Do write tools have an explicit idempotency strategy?
• Does the system distinguish retryable_failure from side_effect_unknown?
• Are retries bound to policy rather than a generic helper?
• Are there per-tool or per-tenant rate limits?
• Do you understand the rollback boundary for every risky action?
• Can the runtime reconcile instead of blindly retrying?
• Are idempotency keys visible in traces and audit logs?

If the answer is "no" several times in a row, the next unstable integration will almost certainly turn into duplication, noise, or a manual incident review.

13. What to Do Next¶

Part IV now closes the basic execution layer: contracts, sandboxing, capability transport, and side-effect discipline. The next logical move is reliability and observability at the whole agent-system level.

• Chapter 11. Traces, Spans, and Structured Events
• Chapter 9. Sandbox Execution and MCP as an Integration Contract
• Part IV. Tools and Execution
• Sources