Q38 proof · Asynchronous and background processing: workers, schedulers, queues and batch pipelines

38. Asynchronous and background processing

Answer as submitted

“Yes — asynchronous and scheduled processing is a recurring part of my designs on .NET, Java, Python and cloud platforms.”

Technical decision: expand. The submitted line lists the frameworks, which is the easy half. This page is the separate, detailed version: the rule that decides what leaves the request path, the three questions a cron expression never answers, the platform choice by shape of work, and the job-level reliability concerns — lease, heartbeat, quarantine, backlog age and graceful drain — that separate a job that runs from a job that can be operated.

Expanded answer

Yes, across .NET, Java/Spring, Python and cloud-native platforms, and the framework is the least interesting part of any of it. The first decision is what leaves the request path, and I apply it mechanically: anything whose duration is decided by somebody else — a mail server, a PDF render, an export over a million rows, a third-party sync, an ETL, a model — is enqueued with an id, the response returns that id immediately, and the client polls or subscribes for the outcome. A slow dependency then degrades one screen instead of exhausting the request threads. In-process, a bounded Channel<T> behind a BackgroundService is right for work that must not survive a restart: backpressure instead of unbounded growth, a configurable consumer count, and a shutdown token honoured so a rolling deploy drains and requeues rather than truncating. For anything that has to survive a restart it stops being a channel and becomes a real queue. Scheduled jobs raise three questions a cron expression does not answer: which instance runs it (a distributed lock with a fencing token), what happens if the previous run is still going (a concurrency policy: skip, queue or replace, decided per job), and what happens if the schedule fired while the service was down (an explicit misfire policy). The platform follows the shape of the work: a plain queue for independent tasks (SQS/SNS, Kafka consumer groups, Celery), a state machine for multi-step work with retries (Step Functions), a DAG for a batch pipeline with dependencies and a backfill story (Airflow), and an in-process scheduler only where a failed job is cosmetic. Finally reliability is what makes the queue trustworthy: an idempotent handler because delivery is at-least-once, a visibility timeout extended by a heartbeat so a long job is not redelivered mid-flight to a second worker, a poison message quarantined after its attempt budget instead of retried forever, and the backlog together with the age of its oldest message as a metric with an alert — because a queue that is filling up and a queue that is being worked look identical from the outside until it is too late.

Shape of work	Mechanism	Why not the alternative
Must not leave the request path	enqueue with a job id, return it, expose status	a slow dependency inside a request consumes the request threads of every other user
In-process, may be lost on restart	`BackgroundService` + a bounded `Channel<T>`	an unbounded queue turns a slow dependency into an out-of-memory kill with a delay on the fuse
Recurring, exactly once across replicas	scheduler with a distributed lock and a fencing token	three replicas and a cron expression means three runs and three reports
Independent tasks that must survive a deploy	a real queue with a consumer group	an in-process channel loses accepted work on every rollout
Multi-step with retries and branching	a state machine (Step Functions) or a DAG (Airflow)	chaining jobs by hand hides the dependency graph and the retry semantics
Batch with a backfill story	DAG with partitions derived from the logical date	appending on re-run double-counts the day, silently

Live background-jobs workflow proof

The running component below changes its aspect/result view through real event binding. Select an aspect to see the mechanism, the reasoning behind the decision, and the evidence that it holds.

BACKGROUND JOBS WORKFLOWRUNNING

Choose an aspect of background processing

Select an aspect to inspect the mechanism, the reasoning and the evidence.

$ emails · PDF generation · exports · third-party sync · ETL · AI inference → enqueue + job id → the UI polls or subscribes for status

The rule is mechanical rather than a judgement call: anything whose duration is decided by somebody else — a mail server, a report over a million rows, a model — does not run inside a request. It is enqueued with an id, the response returns that id immediately, and the client polls or subscribes for the result, so a slow dependency degrades one screen instead of exhausting the request threads.

job types moved off the request path: 9
request timeouts caused by background work: 0
jobs that block until they return a result: 0
status visible to the user for every job type: yes

What background processing buys

Rules I apply. Move anything with somebody else's timing off the request path and return an id, because a synchronous dependency is a request thread held hostage; bound every queue so pressure produces a refusal instead of an out-of-memory kill; decide per job whether losing it is acceptable, and let that answer choose between an in-process channel and a real queue; make the handler idempotent rather than wishing for exactly-once delivery; extend the lease with a heartbeat while a long job runs, so it is not handed to a second worker mid-flight; quarantine a deterministic failure after its attempt budget instead of retrying it forever into a backlog; derive batch partitions from the logical date so a re-run overwrites instead of double-counting; and alert on backlog and the age of the oldest message, because filling and draining look the same until the queue is the incident.

Boundary for Q38: this page covers job and batch processing. Service-to-service messaging, delivery guarantees and the resilience pipeline around them are on the Q34 proof page; the outbox relay and the reconciliation job are on Q35; and the read model that a job often feeds is on Q36.

Source files

Select a file to inspect the proof with syntax highlighting and line numbers. The first file is the code running the component above; the others are the .NET worker with its lease, drain and quarantine behaviour, the Airflow ETL DAG, and the job topology with its leases, attempt budgets and alerts.

Acceptance criteria

States the rule that decides what is asynchronous, rather than listing frameworks.
Bounds every queue and drains on shutdown, so a deploy does not discard accepted work.
Shows an interactive proof of choosing an aspect and reading the evidence.
Answers the scheduling questions a cron expression leaves open: instance, overlap and misfire.
Makes jobs idempotent, quarantines poison messages and alerts on backlog age.