Q35 proof · Distributed transactions: saga, compensation, outbox and idempotent effects

35. Distributed transactions

Answer as submitted

“Yes — I have designed and implemented consistency across multiple services and databases, using saga-style coordination rather than fragile two-phase commit.”

Technical decision: expand. The submitted line names the pattern and the rejection. This page is the separate, detailed version: it shows the saga with its state and compensations, why exactly-once is a property of the effect rather than of the delivery, how the outbox removes the need for a distributed lock, and the reconciliation job that exists because compensation is best-effort by nature.

Expanded answer

Yes, and the honest framing is that I do not use distributed transactions across service boundaries at all. Across services I use a saga: a long-running process modelled as a sequence of local transactions, each paired with a compensating action designed at the same time as the step rather than after the incident. Orchestration when the process has more than a handful of steps or a business owner who needs to read it, choreography when the flow is genuinely simple and the participants are already event-driven. The saga state is persisted after every step, so a process that dies mid-way resumes rather than silently stopping, and a participant timeout is treated as a failure because an unbounded wait turns a saga into an orphan holding a lock nobody releases. The transactional outbox is what makes the forward path reliable without a distributed lock: the state change and the outgoing event commit in one local transaction, and a relay publishes afterwards. Exactly-once is a property of the effect, not of the delivery — the transport is at-least-once, so the effect is made idempotent three times over with a client key, a unique constraint as the arbiter, and a conditional write. Finally I accept eventual consistency and engineer around it: read-your-writes where a user would notice the lag, a reconciliation job that compares both sides of every process and repairs the divergences, and a DLQ with an owner. Two-phase commit still has a place inside a single transactional resource; across a service boundary it would make every participant's availability depend on the slowest one.

Concern	Mechanism	What it buys
Multi-step process	saga, orchestrated or choreographed, with persisted state	a business process that resumes instead of stopping halfway
Undoing a step	a compensation designed with the step, run in reverse order	semantic rollback without a lock held across services
State change plus event	transactional outbox, committed in one local transaction	no lost event and no phantom event, without 2PC
Retries and redelivery	idempotency key, unique constraint, conditional write	exactly-once effects on an at-least-once transport
User-visible lag	read-your-writes routing, pending state echoed in the response	eventual consistency the customer never has to notice
Whatever escapes	reconciliation job with an alert, plus a DLQ with a named owner	divergence becomes a repaired incident, not a customer complaint

Live consistency workflow proof

The running component below changes its mechanism/result view through real event binding. Select a mechanism to see how it works, the reasoning behind the choice, and the evidence that it holds.

DISTRIBUTED TRANSACTIONS WORKFLOWRUNNING

Choose a consistency mechanism

Select a mechanism to inspect how it works, the reasoning and the evidence.

$ orchestration: OrderFulfilmentSaga drives 5 steps · choreography: each service reacts to the previous event

A long-running business process spanning payment, stock, shipping and notification is modelled as a sequence of local transactions, each paired with a compensating action, with no lock held across services. Orchestration keeps the process visible in one place; choreography removes the coordinator but makes the flow emergent, so I orchestrate when there are more than three steps or a business owner who needs to read the process.

steps in the fulfilment saga: 5
distributed locks held across services: 0
services that must be available at the same time: 0
process readable in one file: yes (orchestration)

What the approach buys

Rules I apply. Design the compensation with the step, because reverse-engineering the undo during an incident is how money gets lost; persist the saga state after every step, since a process that cannot resume is a process that silently stops; treat a participant timeout as a failure, because waiting forever is how a lock is never released; push non-compensable steps after the last point of failure, since an email cannot be unsent; make the effect idempotent three times over and let the database be the arbiter, because the transport is at-least-once no matter what the documentation promises; and keep a reconciliation job with a real alert, because eventual consistency without reconciliation is just hoping.

Boundary for Q35: this page covers consistency across services — saga, compensation, outbox and idempotent effects. Comparing that with transactions inside a single application or database is the subject of the Q36 question, the messaging and resilience machinery around it is on the Q34 proof page, and the unit-of-work boundary inside one service is on the Q32 proof page.

Source files

Select a file to inspect the proof with syntax highlighting and line numbers. The first file is the code running the component above; the others are the orchestrated saga with its compensations, the saga declared as reviewable data, and the outbox relay with the reconciliation job.

Acceptance criteria

States the consistency model and the mechanism that delivers it.
Pairs every forward step with a compensating action, and orders non-compensable steps last.
Shows an interactive proof of choosing a mechanism and reading the evidence.
Makes effects idempotent and names the arbiter, rather than relying on delivery guarantees.
Includes reconciliation and an owner for what escapes, and says where 2PC still applies.