RTS questionnaire · Q46 · Distributed-system patterns proof

Have you worked with distributed-system patterns such as CQRS, Event Sourcing, Transactional Outbox, Saga, Event-driven architecture, Idempotency, or Retry/Circuit Breaker? If yes, which patterns have you implemented and for how long?

The inventory the question asks for: every pattern in production, what the duration is counted from, how deep the usage actually goes, the failure paths that are tested rather than assumed — and a definition of each pattern, the seven the question names and the rest of the distributed-systems vocabulary they belong to.

Status: reviewed Live pattern inventory 7 patterns asked · 82 defined 6+ years · failure paths tested Jump to live proof ↓

46. Which patterns, and for how long

Answer as submitted

“Yes — I have implemented most of these patterns in production, over roughly 6+ years of distributed-systems work.”
Technical decision: expand. The question asks two things and the second one is the hard part, because a year count alone is not checkable and a per-pattern figure to one decimal would be precise and false. This page states what the count is counted from, which patterns are foundational as opposed to selectively applied, and — the part that separates having read about a pattern from having run one — which failure paths are exercised by tests.

Expanded answer

Yes — all seven, in production. CQRS wherever the read shape and the write shape genuinely differ: a transactional, validated write model through the aggregate, and denormalised projections or materialised views for the reads, so the write model is never shaped by a report. Event Sourcing applied selectively, where the history is the requirement — financial movements, approvals, workflow history — with the log as the source of truth, snapshots so a long-lived aggregate does not replay forever, and upcasting on read so the log is never rewritten; deliberately not applied to CRUD-shaped data, which is the most common way the pattern is over-used. Transactional Outbox on every state change that has to publish an event, combined with an inbox and a dedup key on the consumer side, which is how at-least-once delivery produces exactly-once business effects without a distributed transaction. Saga in both styles — orchestrated where the process has a business owner who needs to read it or where there are four or more steps, choreographed below that — with a compensation designed alongside every step, a timeout that treats a silent participant as failed, and a queue a human can drain, because a compensation that cannot complete is a business problem rather than an exception. Event-driven architecture chosen per boundary rather than applied to everything: schemas published and versioned, a named owning team per topic, replayable consumers, and the backlog with the age of its oldest message as first-class metrics. Idempotency three times over — a client key, a unique constraint as the real arbiter, and a conditional write — because exactly-once is a property of the effect and not of the delivery. Retry and circuit breaker composed in a deliberate order: timeout outermost, then the bulkhead, then the retry, then the breaker, since a retry outside its bulkhead amplifies an outage and a breaker outside its retry trips on a single exhausted request. Durations are counted from the same condition as Q44 — a service that was independently deployable and owned its data — which is 6+ years, with the integration period before that where the messaging and idempotency habits came from.

PatternIn productionDepth and frequency
CQRSyes — 6+ yearsfoundational, wherever the read and write shapes differ
Event Sourcingyes — 6 yearsselective: audit-heavy domains only, never on CRUD-shaped data
Transactional Outboxyes — 6 years (2020 → present)foundational: every state change that publishes an event
Sagayes — 6 years (2020 → present)foundational: order, payment and provisioning flows spanning services
Event-driven architectureyes — 6+ yearsfoundational, chosen per boundary rather than applied to everything
Idempotencyyes — 6+ yearsfoundational: every consumer, every retried write, every payment path
Retry / Circuit Breakeryes — 6+ yearsfoundational: every outbound call, in both the earlier and the current stacks
How the durations are counted. From the same condition used for Q44: a service that was independently deployable and owned its data. The bands overlap and are deliberately approximate, because a per-pattern year count to one decimal would be precise and false. The period before the microservices model — service-oriented, message-driven integration — is considerably longer, and it is where the messaging, idempotency and failure-handling habits originated; those patterns are marked 6+ because they carried across both periods.

Definitions: the seven patterns the question names

A pattern named without its definition proves only that the acronym is familiar, and most of these names are used loosely in circulation: “event” means three different things depending on which part of the system is speaking, “idempotency” is usually described as a property of delivery rather than of the effect, and a saga is often presented as a workflow engine rather than as a sequence of local transactions with designed undos. Each definition below is the one this work is built on, followed by the condition under which the pattern was applied and the duration that is claimed for it.

CQRS — Command Query Responsibility Segregation

Writing through one model and reading through others: commands go through a transactional, validated aggregate and commit one use case at a time, while queries are served by denormalised projections or materialised views shaped for whatever the screen actually needs. The separation exists so that the write model is never designed around a report — and so that the read side can be thrown away and rebuilt.

in production6+ years — wherever the read shape and the write shape genuinely differ

Event Sourcing

Persisting state as the ordered sequence of events that produced it instead of as the current row: the append-only log is the source of truth and the present state is a fold over that log. Running it safely needs three things — an append that declares its expected version so two writers on one stream collide visibly, snapshots so a long-lived aggregate does not replay thousands of events, and upcasters applied on read, because the log is never rewritten.

in production6 years — selectively, where the history itself is a business requirement

Transactional Outbox

Writing the state change and the outgoing message inside the same local transaction — the message lands in an outbox table, not on the broker — and letting a relay publish from that table afterwards, marking rows as it goes. Publication is then at-least-once and can be neither lost nor emitted for a rollback, with no distributed transaction anywhere; the consumer side pairs it with an inbox and a dedup key, because at-least-once is durable all the way to the effect.

in production6 years (2020 → present) — every state change that publishes an event

Saga

A long-running business transaction that spans services and is executed as a series of local transactions, each one carrying a compensating action that semantically undoes it. Orchestrated where a business owner needs to read the process or the flow is long, choreographed where the participants are already event-driven and the flow is short; a timeout treats a silent participant as failed, and a compensation that cannot complete escalates to a queue a human can drain rather than being swallowed.

in production6 years (2020 → present) — order, payment and provisioning flows

Event-driven architecture

Producers publish events to a broker and consumers subscribe to them, with the producer neither knowing nor waiting for its consumers. Running it in production means schemas published and versioned as contracts, a named owning team per topic, replayable consumers, and the backlog plus the age of its oldest message as first-class metrics. It is a decision per boundary rather than a default: a system in which every interaction is an event is a system in which nothing has a clear owner.

in production6+ years — MQ systems before 2020, then Kafka, RabbitMQ, Service Bus and EventBridge

Idempotency

The property that applying the same operation twice has the same effect as applying it once — a property of the effect, not of the delivery, which is why it cannot be delegated to the broker. Applied three times over on the paths that matter: a client-supplied key, a unique constraint so that the database is the arbiter rather than the application’s timing, and a conditional write against the current state.

in production6+ years — every consumer, every retried write, every payment path

Retry / Circuit Breaker

Retry re-attempts a transient failure a bounded number of times with exponential backoff and jitter, and only where the operation is idempotent and still inside its deadline. A circuit breaker is a state machine — closed, open, half-open — that stops calling a dependency which is failing, so callers fail fast or take a fallback instead of queueing behind a dead service. Neither works on its own: they are composed with a timeout and a bulkhead around them, in an order that has been reasoned about and tested.

in production6+ years — every outbound call, in both the earlier and the current stacks

The durations, restated once. They are counted from the same condition used for Q44 — a service that was independently deployable and owned its data — and the bands overlap on purpose, because a per-pattern figure to one decimal would be precise and false. “6+ years” means the pattern was carried across both the earlier integration period and the microservices period; “6 years (2020 → present)” means it arrived with the microservices delivery model. Depth is stated separately from duration, because “used for six years” says nothing about whether it is used everywhere or in the four places that genuinely needed it.

Live pattern-inventory workflow proof

The running component below changes its pattern-group/result view through real event binding. Select a group to see what was implemented, the reasoning behind the choice, and the duration together with the depth of usage.

LIVE PREVIEW
RUNNING
PATTERN INVENTORY WORKFLOWRUNNING

Choose a group of patterns

Select a group to inspect what was implemented, the reasoning and the duration.

$ write model: transactional, validated · read models: denormalised projections and materialised views · an append-only log where the history is the point

CQRS is applied wherever the read shape and the write shape genuinely differ, and event sourcing only where the history is the point: financial movements, approvals, workflow history. The log becomes the source of truth, the write model validates and appends with an expected version, and every read model — including the ones behind the dashboards — is a projection that can be rebuilt from zero.

CQRS in production: 6+ years, wherever the read and write shapes differ
Event Sourcing in production: 6 years, applied selectively to audit-heavy domains
projections rebuildable from the log: all
event sourcing used on CRUD-shaped data: 0 places
justification recorded per adoption: an ADR

Live pattern-definition browser

The component below filters the catalogue by group through real event binding. Select a group to read its summary, the definitions in it, and where each pattern sits relative to this work.

LIVE PREVIEW
RUNNING
PATTERN GLOSSARY WORKFLOWRUNNING

Definitions by group of patterns

Select a group to read the definitions, the group summary and where each pattern sits relative to this work.

$ the seven patterns named in the question · definition, application condition and duration

The definition, the condition under which the pattern is applied, and the duration claimed for it.

definitions in this group: 7
in production: 7
inside a named pattern: 0
platform-provided: 0
deliberately not used: 0
outside these systems: 0

CQRS — Command Query Responsibility Segregation

Writing through one model and reading through others: commands go through a transactional, validated aggregate and commit one use case at a time, while queries are served by denormalised projections or materialised views shaped for whatever the screen actually needs. The separation exists so that the write model is never designed around a report — and so that the read side can be thrown away and rebuilt.

in production6+ years — wherever the read shape and the write shape genuinely differ

Event Sourcing

Persisting state as the ordered sequence of events that produced it instead of as the current row: the append-only log is the source of truth and the present state is a fold over that log. Running it safely needs three things — an append that declares its expected version so two writers on one stream collide visibly, snapshots so a long-lived aggregate does not replay thousands of events, and upcasters applied on read, because the log is never rewritten.

in production6 years — selectively, where the history itself is a business requirement

Transactional Outbox

Writing the state change and the outgoing message inside the same local transaction — the message lands in an outbox table, not on the broker — and letting a relay publish from that table afterwards, marking rows as it goes. Publication is then at-least-once and can be neither lost nor emitted for a rollback, with no distributed transaction anywhere; the consumer side pairs it with an inbox and a dedup key, because at-least-once is durable all the way to the effect.

in production6 years (2020 → present) — every state change that publishes an event

Saga

A long-running business transaction that spans services and is executed as a series of local transactions, each one carrying a compensating action that semantically undoes it. Orchestrated where a business owner needs to read the process or the flow is long, choreographed where the participants are already event-driven and the flow is short; a timeout treats a silent participant as failed, and a compensation that cannot complete escalates to a queue a human can drain rather than being swallowed.

in production6 years (2020 → present) — order, payment and provisioning flows

Event-driven architecture

Producers publish events to a broker and consumers subscribe to them, with the producer neither knowing nor waiting for its consumers. Running it in production means schemas published and versioned as contracts, a named owning team per topic, replayable consumers, and the backlog plus the age of its oldest message as first-class metrics. It is a decision per boundary rather than a default: a system in which every interaction is an event is a system in which nothing has a clear owner.

in production6+ years — MQ systems before 2020, then Kafka, RabbitMQ, Service Bus and EventBridge

Idempotency

The property that applying the same operation twice has the same effect as applying it once — a property of the effect, not of the delivery, which is why it cannot be delegated to the broker. Applied three times over on the paths that matter: a client-supplied key, a unique constraint so that the database is the arbiter rather than the application’s timing, and a conditional write against the current state.

in production6+ years — every consumer, every retried write, every payment path

Retry / Circuit Breaker

Retry re-attempts a transient failure a bounded number of times with exponential backoff and jitter, and only where the operation is idempotent and still inside its deadline. A circuit breaker is a state machine — closed, open, half-open — that stops calling a dependency which is failing, so callers fail fast or take a fallback instead of queueing behind a dead service. Neither works on its own: they are composed with a timeout and a bulkhead around them, in an order that has been reasoned about and tested.

in production6+ years — every outbound call, in both the earlier and the current stacks

The wider distributed-systems pattern catalogue

The question names seven patterns; the vocabulary around them is considerably larger, and a definition of each is more use than a list of acronyms. The five groups below are the whole of it as it is used here: what moves work between services, how state stays correct when more than one participant is involved, how data is spread and cached, how failure is contained, and how the services are put together at the edge. Every entry carries a definition and a statement of where it sits in this work — including the entries that sit nowhere in it, because a catalogue that claims everything proves nothing.

Messaging and delivery — 17 definitions

Broker semantics, consumption and the contract between a producer and its consumers. Every entry here exists because a message can arrive twice, arrive late, arrive after the world has moved on, or never arrive at all. The patterns decide which of those four outcomes is acceptable for a given flow, and what the code does about it when it happens at three in the morning.

PatternDefinitionWhere it sits
Publish–Subscribe (topic fan-out)One published message reaches every interested subscriber, so the producer does not know who consumes it; the broker owns the fan-out, and each subscription keeps its own offset and its own backlog.in production the default integration style between bounded contexts (Q34, Q45)
Competing ConsumersSeveral instances of the same consumer pull from one queue or share a topic’s partitions, which is how a consumer scales out — and the reason the effect has to be idempotent, since one delivery can land on two instances.in production every consumer runs as competing instances; the dedup key is what makes that safe
Queue-Based Load LevelingA queue sits between a bursty producer and a slower consumer, absorbing the burst so it can be drained at a sustainable rate: latency is traded for stability instead of work being dropped.in production background processing and bulk jobs (Q38)
Queue versus Retained LogA queue removes a message once it is acknowledged; a log retains it and lets each consumer hold its own offset, which is what makes replay and a rebuildable read model possible at all.in production both, chosen by whether the consumer needs to be able to replay
Dead-Letter ChannelA message that cannot be processed after its retries is moved to a separate channel with the failure attached, so that one poison message does not block the partition behind it and a human can inspect, fix and replay it.in production every consumer has one, with depth and age alarmed (Q34, Q45)
Poison-Message ClassificationA deterministic failure — a message that will never succeed — has to be told apart from a transient one; retrying the first forever is how a queue quietly becomes a black hole.in production the classification decides retry-then-DLQ or immediate DLQ
Idempotent Receiver (inbox)The receiving end recognises a message it has already processed — a dedup table keyed by message or business id, written in the same transaction as the effect — and returns the previous result instead of repeating the work.inside a named pattern the inbox half of the outbox (Q35)
Claim CheckThe message carries a reference to a payload held in a store rather than the payload itself, which keeps a large document inside the broker’s message-size limit and out of its memory.outside these systems used only where payloads exceed the broker limit; not a general design choice here
Correlation IdentifierA business identifier carried in the message headers and threaded through every hop, so that one logical flow can be reconstructed across services in logs and traces instead of being guessed at.in production required on every event that participates in a multi-step flow (Q45)
Message Sequencing and ResequencingMessages carry a sequence number and a resequencer buffers and releases them in order at the point where order matters, rather than assuming that a brokered topic preserves global ordering, which it does not.in production in its bounded form — a per-aggregate version check rather than a full resequencer
Request–Reply over MessagingA caller that must not block sends a request with a reply address and a correlation id and treats the reply as another asynchronous message, with a deadline enforced on its own side.in production used where a synchronous call would hold a thread across a slow participant
BackpressureThe consumer signals that it is falling behind — by withholding acknowledgement, by a bounded internal queue, or by an explicit credit — so the producer slows down instead of the backlog growing without limit.in production bounded channels and prefetch limits (Q38)
Event Notification versus Event-Carried State TransferTwo different things wearing the same word: a thin “something changed, come and look” ping that keeps the consumer calling back, and an event that carries the state the consumer needs so that it never has to. Choosing the wrong one is how systems end up either chatty or stale.in production thin notifications for reactive flows, state transfer where a consumer needs no call-back
Change Data Capture (log tailing)Deriving events from the database’s own replication log instead of from application code, so that no writer can forget to emit one — at the price of publishing the table rather than the domain, and of a schema that is now an API.outside these systems applied only where a legacy writer could not be changed; the outbox is preferred everywhere else
Message Routing and Aggregation (EIP)The routing vocabulary: content-based router, splitter, aggregator, scatter–gather and wire tap — decide the destination from the content, split one message into many, recombine related messages into one, broadcast and gather the responses, and copy a message to a side channel for inspection.in production used individually where each one pays for itself; scatter–gather for parallel quotes
Priority ChannelTwo or more channels for the same kind of work, drained in priority order, so that an interactive request is never queued behind a nightly batch.in production used where a batch and an interactive flow share a worker pool (Q38)
Broker Delivery GuaranteesIdempotent producers and transactional writes give exactly-once delivery inside the broker’s own boundary — not across the consumer’s database, which is precisely why the dedup key lives with the effect.platform-provided enabled where the broker offers it, and never relied on as the only defence

Transactions and consistency — 20 definitions

How state stays correct when more than one participant is involved. The question of what to do when a unit of work spans two databases is answered here in the negative first: a distributed transaction is rejected, and everything else in this group exists to make that rejection survivable — a compensation, an expected version, a dedup key, a quorum, and a written-down answer for what happens when two writers meet.

PatternDefinitionWhere it sits
Two-Phase CommitA coordinator asks every participant to prepare, then to commit: atomic across services, but blocking, slow, and fragile when the coordinator dies between the two phases — which turns a business flow into an operational incident.deliberately not used across a service boundary — 0 distributed transactions (Q35)
Three-Phase CommitA non-blocking variant of two-phase commit that adds a pre-commit phase to close the coordinator’s blocking window; it still requires every participant to be alive and synchronous.deliberately not used known and rejected along with 2PC
Compensating TransactionAn action that semantically undoes a completed step — a refund, a release, a cancellation — rather than rolling back a transaction, which is why the undo is business logic and has to be designed at the same time as the step it undoes.inside a named pattern one per saga step, designed alongside it (Q35)
Try–Confirm–Cancel (TCC)Three explicit phases: reserve the resource, confirm it, or cancel the reservation — for the cases where a resource can genuinely be held, such as stock or credit, and the split is acceptable to the business.in production the shape of reservation-style steps where inventory has to be held rather than compensated
Delivery Guarantees (at-most-once, at-least-once)At-most-once may lose a message and at-least-once may duplicate it; neither is a business guarantee, which is why the guarantee is manufactured at the effect with a dedup key rather than negotiated with the transport.in production at-least-once transport with idempotent effects, stated as the default (Q34)
Exactly-Once EffectsNot a delivery guarantee but a property of the effect: an operation that is applied exactly once because the duplicates are recognised, with the database as the arbiter rather than the application’s timing.in production the target everywhere; implemented with a unique constraint and a conditional write
Optimistic Concurrency ControlA writer declares the version it read and the store rejects the write if anything changed in the meantime, so that a conflict becomes a visible result instead of a silently lost update.in production aggregate version on every append, row version or ETag on editable records
Pessimistic LockingHolding a lock or a database row for the duration of the work: simple and correct, and it does not survive a slow participant, a lost connection or a process that never comes back.in production used inside a single transaction only — the lock is never held across a network call
Distributed Lock and LeaseA lock in a shared store with a lease or expiry so that a crashed holder eventually releases it — and a fencing token, because a holder that has been paused can come back and write after it has lost the right to.outside these systems used in its bounded form: a short lease on scheduled jobs so only one instance runs the batch
Leader ElectionParticipants agree on one active owner of a responsibility, which is how a scheduled job, a relay or a partition ends up with a single writer instead of several.platform-provided broker partition assignment and the orchestration layer elect, not the application
Consensus (Paxos, Raft)A cluster agreeing on a single order of operations even while members fail, which is what makes a replicated log and a stable leader possible in the first place.platform-provided relied on, not implemented — Raft inside the coordination layer and the broker’s controller
Quorum Reads and Writes (R + W > N)Reading and writing overlapping majorities so that a read always sees the latest write, at the cost of extra latency and of availability when the network partitions.platform-provided a property of the replicated stores in use, not a per-query choice made here
CAP and PACELCUnder a partition you choose consistency or availability; PACELC adds that even without one you are still trading latency against consistency. It is useful as a way to state a decision in writing rather than as a design in itself.in production used as the framing written into the ADR that rejects 2PC (Q35)
Eventual ConsistencyReplicas and projections converge once the writes stop, which is the normal state of a read model; the engineering is in bounding how long that window is and being honest about it in the interface.in production every projection, with lag measured rather than assumed (Q36)
Read-Your-Writes and Monotonic ReadsThe two consistency guarantees a user actually notices: after your own write you must see it, and time must not appear to run backwards on your next read. Implemented by routing the writer’s reads to the primary, or by carrying a version token.in production the write path returns the version that the read path hands back
Logical Clocks (Lamport, vector, hybrid)Ordering without a shared clock: Lamport timestamps give a total order consistent with causality, vector clocks detect writes that were concurrent, and hybrid logical clocks keep physical and logical time together across services.inside a named pattern the reason every event carries a version and a correlation id; a full vector clock has not been needed
Conflict-Free Replicated Data Type (CRDT)Data structures whose merge is commutative, so replicas converge with no coordination at all — at the price of giving up any invariant that needs a global view of the data.deliberately not used the invariants in these domains need one arbiter, which is what the unique constraint is for
Conflict Resolution PolicyA written rule for the same record written twice: version check and retry, last-write-wins with a clock, or a merge function. The wrong answer is “whichever arrives last”, which is a race with no owner and no record.in production version check plus reject-and-retry; never last-write-wins
Write-Ahead Log and DurabilityDurability as a design element rather than an assumption: the log, the flush policy, and exactly what a commit does and does not guarantee — the ground that both the outbox and the event log stand on.in production commit durability and replication settings chosen deliberately
MVCC and Isolation LevelsHow the store actually isolates concurrent work: snapshot isolation lets readers proceed without blocking writers, while locking serialises them — and a long transaction turns either one into a production incident.in production the isolation level is chosen per use case (Q36)

Data distribution and scaling — 14 definitions

Where the data lives, how it is spread, and what it costs to read it back. Distribution is a data question before it is a services question: who owns which rows, what happens when one node can no longer hold them, and how a read is served without asking every participant. The caching entries are here because a cache is a replica with an expiry, and it fails in the same ways.

PatternDefinitionWhere it sits
Database per ServiceEach service owns its schema and is its only writer; everyone else goes through its API or its events. Ownership is what makes independent deployability real rather than aspirational.in production shared database between services: 0 (Q34, Q44)
Sharding / PartitioningSplitting data across instances by a key — range or hash — so that one node does not have to hold everything; the price is that cross-shard queries and transactions become the application’s problem.in production horizontal partitioning and sharding are on Q47 and Q48
Consistent HashingA ring-based mapping from key to node, so that adding or removing a node moves only its share of the keys instead of rehashing the world and invalidating every cache behind it.platform-provided the model behind partition assignment where the number of nodes changes
Read ReplicaA follower database serving read-only traffic, offloading the primary at the price of a replication lag that the read path has to tolerate knowingly rather than discover in production.in production heavy read and reporting paths (Q47, Q48)
Materialised ViewA persisted, precomputed result of a query, refreshed incrementally or rebuilt from source: the cheapest way to serve a dashboard shape without recomputing the join on every request.in production the read side of CQRS (Q36, Q48)
Projection Rebuild by ReplayThrowing a read model away and rebuilding it from the event log, which is only possible when the log really is the source of truth — and is the test of whether it is.in production every projection is rebuildable from zero (Q36)
Leader–Follower ReplicationOne node accepts writes and the followers replicate them, giving a single writer and horizontal read scale, with replication lag as the number to watch.in production the default topology for the relational stores in use (Q47)
Multi-Leader and Leaderless ReplicationSeveral nodes accept writes and the conflicts are reconciled, or quorums are used instead of a leader; it buys availability and locality and costs a conflict story that has to be written down.outside these systems no hand-rolled multi-leader writes; used only where the platform provides it
Index TableA secondary index kept as its own table, so that a lookup by a key that is not the shard key does not have to fan out across every shard.outside these systems recorded as the answer for when the lookup key differs from the shard key
Cache-AsideRead the cache, fall back to the store on a miss, and let the application populate the cache itself: simple, predictable, and the default read strategy for a reason.in production the default read cache (Q48)
Read-Through, Write-Through, Write-BehindRead-through moves the population into the cache layer; write-through writes the cache and the store together; write-behind defers the store write and accepts that a lost cache is lost data.in production write-through where a warm cache matters; write-behind never for anything that must not be lost
Cache Stampede and Thundering HerdWhen a hot key expires, single-flight or a lock around the reload stops a thousand concurrent requests hitting the database at once, and TTL jitter stops the expiries happening together in the first place.in production single-flight on hot keys plus TTL jitter (Q48)
Negative CachingCaching the absence of a record for a short time, so that a lookup for something that does not exist does not reach the database on every single request.in production used with a short TTL on high-volume lookup endpoints
Event-Driven Cache InvalidationInvalidating or updating the cache from the same event that changes the data rather than from a timer, which is what stops a cache from silently serving a lie.in production the write side publishes, the cache listens

Resilience and failure handling — 10 definitions

Bounding the blast radius of a dependency that slows down, fails or disappears. A dependency does not usually fail; it slows down, and the caller runs out of threads. Everything here is about turning that into a bounded, visible, degraded event rather than a cascade — and about proving it with a test instead of trusting the library defaults.

PatternDefinitionWhere it sits
Timeout and Deadline PropagationEvery remote call carries a deadline, and the remaining budget travels with the request so that downstream calls cannot outlive the caller’s patience; a call without one is the classic beginning of a thread-pool death spiral.in production the outermost policy in the pipeline (Q34)
BulkheadIsolating a dependency’s resource pool so that one slow dependency exhausts only its own threads and connections instead of those of every caller in the process.in production a separate pool per outbound dependency (Q34)
Retry with Exponential Backoff and JitterWaiting longer after each failure and randomising the wait, so that a recovering dependency is not immediately re-flooded by every client at the same instant — with a bounded attempt count and a budget per call.in production jittered, bounded, and only on operations that are idempotent (Q34)
Fallback and Graceful DegradationA designed degraded answer — cached, partial, or queued for later — instead of an error, so that a dependency outage is felt by the user as reduced functionality rather than as a failed request.in production a degraded feature, never an empty response (Q34)
Rate Limiting and ThrottlingBounding how much work a caller may send inside a window — token bucket, leaky bucket, sliding window — to protect the service and to keep one consumer from owning it.in production per-consumer limits at the gateway (Q45)
Load SheddingDeliberately rejecting or dropping low-priority work when the system is saturated, because serving everybody badly is worse than serving the requests that matter well.in production used at the entry points where a queue would otherwise grow without limit
Health Endpoint Monitoring, Heartbeat, WatchdogA cheap endpoint reporting whether an instance can serve, used by the load balancer, the orchestrator and the alarm — plus the out-of-band watchdog for the case where the instance cannot answer at all.in production liveness and readiness gates (Q45)
Scheduler–Agent–SupervisorA supervisor distributes work to agents and monitors them, re-dispatching when an agent dies: the topology underneath background processing, and the reason a job does not disappear with its worker.in production the shape of the background-job infrastructure (Q38)
Chaos Engineering and Fault InjectionBreaking a dependency on purpose — kill the broker, add latency, duplicate a delivery, kill a process mid-message — so that the failure paths are proved instead of assumed.in production practised — the failure-path tests inject exactly these cases
Static StabilityThe data path keeps working when the control plane does not, so a failed deployment system, autoscaler or DNS update cannot take the running system down with it.outside these systems designed for in the way services depend on the platform, rather than claimed as a named adoption

Service and API topology — 14 definitions

How the services are put together, reached and changed without a rewrite. These are the patterns that decide whether the rest can be operated: where identity and traffic shaping live, how a caller finds an instance, how a legacy interface is reached without its model leaking into the domain, and how a running system is migrated rather than replaced.

PatternDefinitionWhere it sits
API GatewayOne entry point in front of the services, owning what every service would otherwise repeat: TLS, routing, token validation at the edge, per-consumer rate limits, request size limits and correlation-id injection.in production the edge in the current stack; retries only on idempotent methods (Q45)
Backend for FrontendA gateway per client type that composes and trims the APIs for that client, so that one client’s needs do not bloat an API shared with all of them.in production used where the client shapes diverged (Q45)
API Composition / AggregatorA read served by calling several services and joining the results in memory, for the queries that span boundaries and have no single owner — kept thin so that it does not become a distributed monolith.in production read composition behind the gateway (Q45)
Service Registry and DiscoveryInstances register and callers resolve them by name, so that scaling and redeployment do not require configuration edits; server-side discovery puts the lookup behind the load balancer instead.platform-provided Kubernetes Services and readiness gates, not a hand-rolled registry (Q45)
Client-Side versus Server-Side Load BalancingWhether the caller picks the instance — one hop fewer, and a library in every client — or a proxy does it: simpler clients, one more hop, and one place to change the policy.platform-provided server-side by default; client-side only where the platform provides nothing (Q45)
Sidecar, Ambassador, AdapterMoving a cross-cutting concern into a process beside the service: the sidecar for proxy, mTLS and telemetry, the ambassador as an outbound proxy for a remote service, and the adapter to translate a legacy protocol.in production as the fabric, with the adapter shape used to reach legacy interfaces (Q45)
Service MeshThe sidecar fleet plus a control plane, pushing mTLS, retries, timeouts, traffic splitting and telemetry into the infrastructure instead of into every service’s code.platform-provided used for transport concerns; the application still owns its business-level policies
Strangler FigMigrating by putting the new implementation in front of the old one and routing a growing share of traffic to it, until the old one can be deleted — the opposite of a rewrite, which is how migrations usually fail.in production how the legacy-to-current transition was sequenced (Q44)
Anti-Corruption LayerA translation layer that keeps a legacy or third-party model out of the domain, so that their concepts do not leak in and their next change does not become your change.in production one per legacy integration
Ports and Adapters (hexagonal)The domain at the centre, ports as interfaces and adapters as technology at the edges, which is what keeps the persistence and broker choices swappable and the core testable.in production the structural style of the services in this inventory (Q44)
External Configuration Store and Feature FlagsConfiguration and toggles held outside the artefact so that behaviour changes without a deployment — with the discipline of removing the flag once the decision it guarded has become permanent.in production configuration externalised; flags carry an expiry
Contract-First APIs and Schema RegistryContracts and versions before code: an OpenAPI document or an event schema published, reviewed and validated in CI, so that a breaking change is a build failure instead of an incident.in production clients are generated from the contract (Q37, Q49)
Distributed Tracing and CorrelationOne trace id across every hop, with spans and baggage, so that the path of a saga can be read instead of guessed — the pattern that makes every other pattern on this page debuggable.in production traces plus a correlation id on every event and call (Q37, Q45)
Durable Execution / Process ManagerState kept by an engine so that a long-running orchestration survives a restart and resumes at the step it had reached, instead of being reconstructed or repeated from the beginning.in production used — Step Functions-style orchestration for the long-running flows (Q45)
How to read the relation. Five values are used, and they are not interchangeable.
  • in production — used in the systems described on this page and its sibling pages, which carry the mechanism-level evidence: Q34, Q35, Q36, Q37, Q38, Q44, Q45, Q47, Q48.
  • inside a named pattern — in production, but as a component of CQRS, event sourcing, the outbox, the saga or the resilience pipeline rather than as a separate adoption.
  • platform-provided — the property comes from the broker, the database, the orchestrator or the coordination layer and is relied on rather than hand-written.
  • deliberately not used — known, understood, and rejected, with the reason recorded in an ADR rather than left as an accident.
  • outside these systems — listed so the vocabulary is complete; no production claim is made for it, and the definition says why it is not needed here.
Boundary for this catalogue: a definition is not a claim of tenure. The seven patterns named in the question carry the durations and the depth; the rest carry a relation, and several of them are marked as rejected or as outside these systems on purpose. The mechanism level of the ones that are in production lives on the pages it belongs to: saga state, compensation and the outbox relay on Q35; the resilience pipeline and delivery guarantees on Q34; read models and the isolation choices on Q36; messaging and background processing on Q38; the fabric around all of it on Q45; partitioning, replication and caching on Q47 and Q48.

What the discipline around them buys

Rules I apply. Justify the pattern choice in an ADR that records the alternatives that were rejected, because a pattern with no written reasoning is a pattern nobody can safely change; event-source only where the history is a business requirement, since the log is permanent and the schema becomes an API; never publish inside the transaction and never reach for a distributed transaction across a service boundary; design the compensation at the same time as the step, because reverse-engineering the undo during an incident is how money is lost; make the effect idempotent and let the database be the arbiter rather than the application's timing; compose the resilience policies in an order that has been reasoned about and tested, not assembled in the order the library documents them; and test the failure paths deliberately — duplicate delivery, out-of-order, a compensation that fails, a breaker that opens, a process killed mid-message — because those are the cases that decide whether the pattern is running or merely present.
Boundary for Q46: this page is the inventory — which patterns, where and for how long, plus the failure paths behind them. The mechanism level is on the pages it belongs to: saga state, compensation and the outbox relay on Q35; the resilience pipeline and delivery guarantees on Q34; read models and the isolation choices on Q36; the fabric around all of it on Q45.

Source files

Select a file to inspect the proof with syntax highlighting and line numbers. The first two files are the code running the components above — the pattern inventory and the definition browser; the third is the definition catalogue itself, as data; the rest are the pattern inventory with the durations and trade-offs, the event-sourced aggregate and its append contract, and the tests that exercise the failure paths.

Acceptance criteria