45. The service fabric: communication, gateways, identity, discovery, processing
Answer as submitted
“Yes — these are the standard building blocks of the distributed systems I design: communication, gateways, identity across hops, discovery, messaging and distributed processing, all of it observable.”
Expanded answer
Yes, and the fabric is a set of decisions rather than a list of products. Service-to-service communication: typed clients over REST with a deadline and a retry policy as the default, because it is debuggable and every team has the tooling; gRPC where the edge is internal, chatty and latency-sensitive, contract-first so the schema is the interface; and asynchronous messaging wherever the caller does not need an answer now, because a synchronous chain couples the availability of two services. I keep the synchronous hops needed to serve one user request to three or fewer. API gateways own what every service would otherwise repeat — TLS, routing, token validation at the edge, per-consumer rate limits, request size limits, correlation id injection, canary splitting — and hold no business logic, because logic in a gateway cannot be unit tested, versioned or rolled back with the service that owns the rule. One detail worth naming: retries at the gateway are enabled for idempotent methods only, since retrying a POST at the edge is how a gateway creates duplicate orders. Authentication distinguishes three cases: a user request validated at the edge and re-validated by every service downstream with its own policy; a service acting on behalf of that user, propagating the identity so the audit trail names the human; and a pure machine call using client credentials or mTLS with its own narrow scopes. Credentials live in a vault, tokens are never logged, and no service reads an identity from the request body. Discovery is a DNS problem on Kubernetes: a Service resolves to healthy endpoints and unready pods leave the rotation, so a client-side registry only earns its place where the platform provides nothing — and an unhealthy dependency degrades one feature rather than failing the request. Distributed processing is three numbers and a policy: the partition count caps parallelism, the ordering key decides what "in order" means (per aggregate, never global), and the commit point decides whether a crash loses or duplicates work — so commits happen after the side effect, handlers are idempotent, a dead-letter queue carries the failure, and replay is a documented operation rather than an incident.
| Building block | Decision | The mistake it avoids |
|---|---|---|
| Communication | typed clients with deadlines; gRPC for internal latency-sensitive edges; messaging when no answer is needed now | a synchronous chain where one slow service takes its callers down |
| API gateway | transport, identity and traffic shaping only; per-consumer limits; retries on idempotent methods only | business logic in an untestable layer, and duplicate POSTs from an edge retry |
| Identity | three cases stated explicitly: user, on-behalf-of, machine; vault-held credentials; identity never from the payload | a service that trusts its caller, and an authorisation bypass through a body field |
| Discovery | Kubernetes Services and readiness gates; client-side registries only where needed | static host lists that break on every rollout |
| Messaging | partitions and consumer groups; ordering per aggregate key; commit after the side effect | committing before the work lands, which loses messages a broker believes were handled |
| Observability | one correlation id from the edge across every hop, plus per-route and per-queue metrics | a request that cannot be followed past the first service |
Live service-fabric workflow proof
The running component below changes its building-block/result view through real event binding. Select a building block to see the decision, the reasoning behind it, and the evidence that it holds.
Choose a building block of the fabric
Select a building block to inspect the decision, the reasoning and the evidence.
$ typed HTTP clients with a deadline · gRPC where latency and throughput matter · asynchronous messaging for decoupled flows
The default is a typed client over REST carrying a deadline and a retry policy, because it is debuggable and every team already has the tooling. gRPC is used where the call is internal, chatty and latency-sensitive, contract-first so the schema is the interface. Anything that does not need an answer right now goes to a message, because a synchronous chain couples the availability of two services.
synchronous hops to serve one user request: at most 3 outbound calls without a deadline: 0 clients generated from a contract: all transport choice recorded per edge: yes
What the fabric buys
Source files
Select a file to inspect the proof with syntax highlighting and line numbers. The first file is the code running the component above; the others are the gateway policy with its limits and canary weights, the identity handler covering the three call cases, and the consumer-group topology with its partitions, ordering key and commit point.
Acceptance criteria
- Addresses all six elements the question lists, each with a decision behind it.
- Keeps the gateway free of business logic and its retries confined to idempotent methods.
- Shows an interactive proof of choosing a building block and reading the evidence.
- Separates the three identity cases and never derives identity from the request payload.
- States the partition, ordering and commit decisions that make distributed processing correct.