RTS questionnaire · Q34 · Distributed systems proof

Have you worked on distributed systems?

A runnable review of how the parts are divided, how they talk synchronously and asynchronously, what happens when a part fails slowly rather than cleanly, and how the whole thing is operated.

Status: reviewed Live distributed-systems workflow Kafka · gRPC · Kubernetes · OpenTelemetry Jump to live proof ↓

34. Distributed systems

Answer as submitted

“Yes — distributed systems are the core of my architecture experience, across enterprise middleware, microservices and cloud-native platforms.”
Technical decision: expand. The submitted line claims the experience without showing a system. This page is the separate, detailed version: it shows the division into bounded contexts, the synchronous and asynchronous contracts, the resilience pipeline with its ordering explained, and the operational properties that decide whether a service can actually be run.

Expanded answer

Yes, across enterprise middleware and cloud-native platforms. The division follows bounded contexts rather than layering: services own their data, no database is shared, and the event-driven backbone between contexts replaces the synchronous chain that would otherwise couple their availability. Synchronous calls are used only where the caller genuinely needs the answer now, and every one carries a deadline and a retry budget, because a slow dependency without a deadline becomes thread-pool exhaustion in the caller. Asynchronous messaging carries the rest: a schema, a version and an ordering key, partitioned by aggregate id so ordering is per entity rather than global, with the consumer acknowledging only after the side effect is committed. Reliability is a pipeline rather than a collection of tricks — timeout, bulkhead, retry with jitter, circuit breaker, dead-letter queue — whose order is the design, wrapped around consumers that are idempotent by construction because at-least-once is a property of the transport and cannot be negotiated away. Operability is what makes it real: stateless services, scale signals that predict load rather than merely exist (queue depth for consumers, not CPU), rollouts that drain in-flight work, and one correlation id following a request across every hop. The failures I plan for are the unglamorous ones: messages arriving twice, arriving out of order, a consumer killed mid-message, and a dependency that is slow rather than down.

ConcernDecisionWhat it prevents
Divisionbounded context per service, its own data, events between contextsthe distributed monolith: services that must be deployed together
Synchronous callsa deadline and a small retry budget on every outbound callthread-pool exhaustion in the caller when a dependency slows down
Asynchronous messagingschema plus version plus ordering key, partitioned by aggregate idassuming global ordering, which is a guarantee no broker gives
Delivery semanticsassume at-least-once, dedup by message id in the same transaction as the effectduplicate side effects after a redelivery
Failure handlingtimeout, bulkhead, jittered retry, breaker, DLQ in that orderretry storms after an outage and one poison message blocking a partition
Operabilitystateless services, load-predicting scale signals, graceful shutdown, one correlation idbacklogs nobody can see and requests lost during a rollout

Live distributed-systems workflow proof

The running component below changes its concern/result view through real event binding. Select a concern to see the design, the reasoning behind it, and the evidence that it holds.

LIVE PREVIEW
RUNNING
DISTRIBUTED SYSTEMS WORKFLOWRUNNING

Choose a concern of the distributed design

Select a concern to inspect the design, the reasoning and the evidence.

$ three bounded contexts as services · event-driven backbone between them · Oracle SOA on-premises ↔ AWS in the cloud

Microservices where a bounded context and its owning team justify the split, an event-driven backbone between contexts instead of synchronous chains, and an explicit hybrid path where on-premises middleware has to coexist with cloud services. The style follows the boundary rather than the fashion.

services owning exactly one bounded context: 3
shared database between services: 0
on-prem ↔ cloud integration paths: 2 (message bridge, API gateway)

What the distributed design buys

Rules I apply. Let the boundary decide the service rather than the org chart, and give each service its own data so no one is tempted into a distributed join; put a deadline on every synchronous call, because an unbounded wait is a resource leak with a dependency attached; state the delivery guarantee explicitly and assume at-least-once, since exactly-once is not on offer; order the resilience strategies deliberately, because a retry outside its breaker and its bulkhead amplifies an outage instead of absorbing it; make every consumer idempotent and version-aware so redelivery and reordering are ordinary events; and require each alert to have a dashboard, a runbook and a named owner, because an alert nobody can act on is noise that trains the team to ignore the page.
Boundary for Q34: this page covers distributed systems in general — division, communication, resilience and operability. Consistency across services, compensating actions and sagas are the subject of the Q35 question, and the infrastructure side (IaC, CI identity, budgets) is on the Q12 proof page.

Source files

Select a file to inspect the proof with syntax highlighting and line numbers. The first file is the code running the component above; the others are the resilience pipeline with its idempotent consumer, the event contract the consumers see, and the running topology with its scale signals and observability.

Acceptance criteria