Durability & Semantics

The contract behind app.workflow(): what is guaranteed, what is at-least-once, and exactly what happens when a process dies.

Execution model: step memoization#

A workflow is a plain async function. Durability comes from steps: every step.run() result is persisted to a journal keyed by (runId, stepId). Each execution attempt re-invokes the function from the top; completed steps return their recorded result instantly, and execution resumes at the first unrecorded step.

order.ts

const order = app.workflow("order", async ({ step, input }) => {
  // Attempt 1 executes this closure, journals the result.
  // Every later attempt returns the journaled value WITHOUT re-running it.
  const payment = await step.run("charge", () => stripe.charge(input));

  // Suspends the run. Zero resources held. Survives restarts.
  await step.sleep("cooloff", "10m");

  // Crash here? Next attempt fast-forwards: "charge" returns instantly
  // from the journal, "cooloff" is already recorded, execution resumes
  // exactly at this line.
  await step.run("ship", () => shipping.create(payment));
  return { shipped: true };
});

Three consequences to internalize:

Code between steps re-executes on every attempt. Keep it cheap and side-effect free; side effects belong inside step.run.
No determinism rules outside steps — unlike Temporal replay, Date.now(), Math.random(), and fetch are fine between steps. But their values are recomputed per attempt:

// Between-steps code re-executes on EVERY attempt — keep it pure.
// Need a stable value across attempts? Journal it:
const startedAt = await step.run("now", () => Date.now());
const token = await step.run("token", () => crypto.randomUUID());

Step ids are journal keys — unique within a run, stable across deploys. The runtime throws on duplicate ids.

Delivery guarantees#

Thing	Guarantee
Queue job delivery	at-least-once (lease + ack/nack + DLQ)
Run execution attempts	at-least-once
Step result recording	effectively-once — a completed journal entry is never overwritten
Step side effects	at-least-once — can re-fire on crash-after-effect-before-record
Run creation with idempotencyKey	exactly-once per key

The side-effect row is the one that bites people. Make external effects idempotent:

// Run-level dedup: same workflow + same key = same run, forever.
await order.trigger(input, { idempotencyKey: `order-${input.orderId}` });

// Effect-level idempotency: derive a key from runId + stepId so a
// crash-after-effect-before-record can't double-charge.
await step.run("charge", () =>
  stripe.charge(input, { idempotencyKey: `${runId}:charge` }),
);

Suspension#

All three suspension primitives hold zero resources while suspended — no open handles, no JS timers, no memory beyond the database row. They survive restarts and deploys.

API	Suspends until	Mechanism
`step.sleep(id, dur)`	wake time	delayed execution job in the store
`step.waitForEvent(id, event, { timeout })`	app.emit(event) or timeout	persisted wake condition; timeout sweep
`step.invoke(id, wf, input)`	child run terminal state	child created with a deterministic idempotency key

waitForEvent resolves the event payload, or null on timeout.
step.invokereturns the child's output; a failed/cancelled child makes it throwthe child's error in the parent (catchable).
step.all([...]) runs thunks in parallel with independent memoization: if one fails, the retry fast-forwards its completed siblings and re-runs only the failure.

Failure & retry#

A throw inside step.run fails the step: the attempt ends and the engine schedules a retry with exponential backoff (stepRetries, default 2; stepBackoff, default 1s→60s). Completed steps never re-execute on retry.
Retries exhausted → the run fails with step 'id' failed after N attempts: …
A throw outside any step fails the run immediately, no retry. Wrap risky code in step.run.
Process crash mid-attempt → the execution lease (default 60s, configurable via lease) expires and the run is redelivered. While a handler runs, the engine renews its lease at lease/3 cadence, so long steps don't need a long lease.
Clock-skew safety: on Postgres, lease set/renew/expiry use the databaseserver clock, not per-node wall clocks — so a node whose clock drifts can't prematurely expire another node's lease. One authority, no skew.
Fencing tokens:every claim bumps a monotonic token. A paused/zombie worker that wakes after its lease expired (and the job was re-claimed elsewhere) carries a stale token, so its ack / fail / renew is rejected — the late write can't clobber the new owner or double-execute.
workflow.cancel(runId)cancels the run and all descendant runs; pending wake jobs become no-ops, and an in-flight attempt's outcome is discarded.

Versioning#

Workflow definitions are content-hashed at registration (FNV-1a of the function source). Runs pin the version they started with; a mismatch on resume logs a warning, and changing a recorded step's kind throws.

Change to in-flight runs	Safe?
Adding steps after existing ones	✅ safe
Changing between-steps code	✅ safe
Removing / reordering / renaming recorded steps	❌ breaking
Changing a step's kind (run → sleep, …)	❌ throws on resume

Version routing: for a structural change, keep the old function and register it with wf.version(oldFn) — in-flight runs pinned to the old content hash keep executing the old logic, while new runs use the current function. No determinism tax, no forced drain:

evolve.ts

// v2 is the current logic; v1 in-flight runs keep running v1.
const orderV1 = async ({ step, input }) => { /* old steps */ };
const order = app.workflow("order", async ({ step, input }) => {
  /* new steps */
});
order.version(orderV1);   // self-hashed; old pinned runs route here

Rule of thumb

Deploy additive changes freely. For structural changes, route with wf.version(oldFn), drain in-flight runs first, or ship the new structure under a new workflow name.

Encryption at rest#

A durable engine persists every input and output, so for teams handling PII that data sits in the store indefinitely. Set an encryptionKey and ZenZip AES-256-GCM encrypts the four payload columns — job payloads, run inputs/outputs, step results, and event payloads — before they touch storage, decrypting on read. The engine only ever holds plaintext in memory.

app.ts

const app = zenzip({
  // Any-length secret. Load it from the environment or a secret
  // manager — never hard-code it, and never lose it (encrypted
  // payloads are unrecoverable without it).
  encryptionKey: process.env.ZENZIP_ENCRYPTION_KEY,
});

Transparent to enable. Each value is tagged with an enc:1: sentinel, so turning encryption on over an existing database keeps legacy plaintext rows readable while new writes are encrypted — no migration, no downtime.
Authenticated. GCM detects tampering and a wrong key; a per-value random nonce means identical payloads never produce identical ciphertext.
Indexed/control fields stay clear. Queue names, event names, status, timestamps, and waitForEvent match predicates are not encrypted — event matching runs against the in-memory plaintext, so suspension/resume semantics are unchanged.
Scope. Covers the durable payload columns. It is not a substitute for disk/volume encryption or TLS to Postgres — it protects the payloads specifically, including in the dashboard and DLQ views.

How it's tested#

Order demo e2e: charge → sleep → waitForEvent → ship, asserting each step executed exactly once across attempts.
Restart durability: the app is stopped while a run sleeps; a fresh process on the same data dir completes it.
SIGKILL chaos harness: a worker process is killed at four random points mid-workflow and respawned; the run must complete with the correct output, every step executed ≥1 time, and re-execution bounded by the kill count.
Engine-level Rust tests: retries, exhaustion, event timeout, child invoke, idempotency races, cancel-while-sleeping.

See the test files: packages/zenzip/test/workflow.test.ts, test/chaos.test.ts, and crates/zenzip-core/tests/workflow_test.rs. A comprehensive kill-at-every-step-boundary suite is tracked on the roadmap.