MongoDB — all questions in one page

MongoDB is a document-oriented NoSQL database that stores data as flexible JSON-like documents in BSON format. It is schema-flexible, horizontally scalable through sharding, and supports high availability with replica sets. It is commonly used for application backends where data shapes evolve quickly.

In a relational database, data is normalized into tables of rows and columns with a fixed schema, and relationships are expressed with foreign keys and joins. In MongoDB, related data can be embedded inside a single document as nested objects or arrays, so one read often returns a full object graph. Collections do not enforce a fixed schema, so each document can have a different shape.

BSON is Binary JSON, the serialization format MongoDB uses to store and transfer documents. It extends JSON with additional types such as ObjectId, Date, Decimal128, Binary, and 64-bit integers. Drivers convert your language objects to BSON, and mongosh displays them in a readable extended-JSON form.

{ "_id": ObjectId("65f1a2b3c4d5e6f708091234"), "when": ISODate("2026-01-15T10:00:00Z"), "price": NumberDecimal("19.99") }

The most common types are String, Int32, Int64, Double, Decimal128, Boolean, Date, ObjectId, Array, and embedded Document. Specialized types include Binary, Timestamp, Null, Regex, and MinKey/MaxKey. Choosing the right type matters because comparisons and indexes are type-sensitive.

A database is a container for collections, a collection is a group of documents, and a document is a set of field-value pairs. This maps roughly to database, table, and row in relational terms, but without a rigid column definition. Collections are created implicitly on first insert.

The _id field is the mandatory primary key of a document and must be unique within its collection. MongoDB creates a unique index on _id automatically, so lookups by _id are fast. If you omit it on insert, the driver generates an ObjectId.

An ObjectId is a 12-byte value: a 4-byte timestamp in seconds, a 5-byte random value per process, and a 3-byte incrementing counter. This makes it roughly time-ordered and unique without central coordination. The timestamp can be read back with getTimestamp().

ObjectId("65f1a2b3c4d5e6f708091234").getTimestamp()

mongod is the core database server process that stores data and serves queries. mongos is the routing process used in sharded clusters that directs queries to the correct shard. Atlas is MongoDB's fully managed cloud service, which runs these components for you.

Locally you install the official packages or the community server and run mongod with a dbPath. With Docker, you pull the official mongo image and map a volume plus the default port. Both approaches give you a standalone server that mongosh can connect to.

docker run -d --name mongo -p 27017:27017 -v mongo_data:/data/db mongo:8

mongosh is the modern MongoDB shell, a Node.js-based interactive client with syntax highlighting and autocompletion. You connect by passing a connection string. Once connected, you can run JavaScript against the current database.

mongosh "mongodb://localhost:27017/myapp"

Use show dbs to list databases, use <db> to switch, show collections to list collections, and db.stats() for size information. db.help() and db.collection.help() list available methods. These commands help you explore a server quickly.

show dbs
use shop
show collections
db.orders.countDocuments()

Compass is the official graphical client for MongoDB. It lets you browse documents, build queries visually, inspect indexes and schema patterns, and analyze performance. It is useful for exploration and debugging without writing shell commands.

You create a database implicitly by switching to it and writing data; an empty database is not persisted. To drop it, call dropDatabase() while it is the current database. Dropping is irreversible, so confirm the name twice.

use shop
db.dropDatabase()

createCollection() creates a collection explicitly, optionally with options such as a validator or capped size. drop() removes the collection and all its documents and indexes. Inserting into a non-existent collection also creates it automatically.

db.createCollection("audit", { capped: true, size: 1048576 })
db.audit.drop()

insertOne adds a single document and returns its insertedId. insertMany adds an array of documents in one round trip and returns the ids of all inserted documents. insertMany is faster for bulk loading but is not transactional by default.

db.users.insertOne({ name: "Ana", age: 28 })
db.tags.insertMany([{ t: "a" }, { t: "b" }], { ordered: false })

With ordered: true, the default, inserts stop at the first error. With ordered: false, MongoDB continues processing the remaining documents and reports all errors. Unordered is usually better for bulk loads where individual duplicates are tolerable.

find(filter, projection) returns a cursor over documents matching the filter. An empty filter matches everything. The filter is a document where fields are matched by equality or by operators.

db.users.find({ status: "active" }, { name: 1, email: 1, _id: 0 })

A projection limits which fields are returned to reduce network and memory cost. You include fields with 1 or exclude them with 0. You cannot mix inclusion and exclusion except for _id, which can be suppressed in an inclusion projection.

db.users.find({}, { password: 0 })

The comparison operators are $eq, $ne, $gt, $gte, $lt, and $lte. They compare values with BSON type ordering in mind. Multiple operators can be combined on the same field in one condition object.

db.orders.find({ total: { $gte: 100, $lt: 500 } })

$and requires all conditions to match, $or requires at least one, and $nor requires none. $not negates an operator expression on a single field and cannot be used with a plain equality value. Implicit AND already applies when you list several fields in one filter.

db.items.find({ $or: [ { qty: { $lt: 5 } }, { discontinued: true } ] })
db.items.find({ price: { $not: { $gt: 100 } } })

$in matches documents where the field equals any value in a given array. $nin is the negation, matching documents whose value is in none of the listed values or where the field is missing. Both accept regular expressions as members.

db.users.find({ role: { $in: ["admin", "editor"] } })

Use a quoted dotted path that follows the embedded document structure. The path must match the nesting order exactly. Dot notation also works in projections and update operators.

db.users.find({ "address.city": "Bogota" }, { "address.zip": 1 })

When a field holds an array, an equality or operator match succeeds if any element matches. This is implicit any-element semantics. To require several conditions on the same element, you need $elemMatch.

db.posts.find({ tags: "mongodb" })

$elemMatch requires that a single array element satisfies all the given conditions, rather than different elements satisfying different conditions. It is essential for arrays of objects with correlated criteria. It can also cap the projected element with $elemMatch in a projection.

db.orders.find({ lines: { $elemMatch: { sku: "A1", qty: { $gt: 2 } } } })

A query for { field: null } matches documents where the field is explicitly null and documents where the field does not exist. To match only explicit nulls, combine null with $exists: true. To match only missing fields, use $exists: false.

db.users.find({ phone: null, }).count()
db.users.find({ phone: { $type: "null" } })

sort() orders results, skip() discards a number of documents, and limit() caps the result count. MongoDB applies sort first, then skip, then limit regardless of chained call order. Deep skip values are slow because the server still walks the skipped documents.

db.posts.find({}).sort({ createdAt: -1 }).skip(20).limit(10)

find() returns a cursor that lazily fetches results in batches rather than materializing everything at once. batchSize sets how many documents the server returns per round trip. Iterating the cursor pulls subsequent batches until it is exhausted.

const c = db.users.find({}).batchSize(500)
while (c.hasNext()) { const d = c.next() }

countDocuments(filter) runs an aggregation that respects the filter and is accurate but can be slower on large collections. estimatedDocumentCount() reads collection metadata and is very fast but ignores filters and may be approximate after unclean shutdowns. Use the estimate for whole-collection totals in dashboards.

distinct(field) returns an array of the unique values for that field across matching documents. It can take a query filter as its second argument. For array fields, it flattens the elements before deduplicating.

db.orders.distinct("status", { region: "LATAM" })

updateOne modifies the first document that matches the filter; updateMany modifies every matching document. Both return matchedCount and modifiedCount. Always include a filter that is as selective as possible to avoid unintended mass updates.

db.users.updateOne({ _id: id }, { $set: { active: true } })
db.users.updateMany({ country: "CO" }, { $set: { currency: "COP" } })

$set assigns the given value to each listed field, creating the field if it does not exist. It leaves all other fields untouched. It also works with dotted paths to update nested fields.

db.users.updateOne({ _id: id }, { $set: { "address.city": "Medellin" } })

$unset removes the specified fields from a document. The value in the operator document is ignored, so convention is to pass an empty string. Removing a field is different from setting it to null.

db.users.updateOne({ _id: id }, { $unset: { tempToken: "" } })

$inc adds a numeric amount to a field atomically, creating the field with that value if it is missing. It only works on numeric fields and errors if the existing value is not a number. Negative values decrement.

db.counters.updateOne({ _id: "visits" }, { $inc: { count: 1 } }, { upsert: true })

$push appends a value to an array, allowing duplicates. $pull removes all elements matching a condition or value. $addToSet appends only if the value is not already present, which effectively deduplicates.

db.users.updateOne({ _id: id }, { $addToSet: { tags: "vip" } })
db.users.updateOne({ _id: id }, { $pull: { tags: "trial" } })

$currentDate sets a field to the current date, either as a Date or as a Timestamp. You pass true for a Date or a type document for explicit control. It is server-side, so it avoids client clock skew.

db.docs.updateOne({ _id: id }, { $currentDate: { updatedAt: true } })

replaceOne swaps the entire document for the supplied one, keeping only the _id. updateOne with operators changes only the targeted fields. Use replaceOne when the whole record is regenerated by the application, and be aware it drops fields you did not include.

db.users.replaceOne({ _id: id }, { name: "Ana", age: 29 })

An upsert inserts a new document when no document matches the filter, instead of doing nothing. It is enabled with { upsert: true } on update and replace operations. On insert, MongoDB builds the new document from the filter's equality conditions plus the update operators.

db.settings.updateOne({ userId: 7 }, { $set: { theme: "dark" } }, { upsert: true })

deleteOne removes the first matching document and deleteMany removes all matches. Both return a deletedCount. There is no deleteAll shortcut other than deleteMany with an empty filter, which you should use carefully.

db.sessions.deleteMany({ expiresAt: { $lt: new Date() } })

findOneAndUpdate atomically updates one document and returns it, by default the version before the update. Set returnDocument: "after" to get the updated version. This is the standard way to implement counters, queues and optimistic locking in one round trip.

db.jobs.findOneAndUpdate(
  { status: "queued" },
  { $set: { status: "running" } },
  { sort: { createdAt: 1 }, returnDocument: "after" }
)

bulkWrite sends a batch of mixed insert, update, replace and delete operations in one request. It reduces round trips and lets you choose ordered or unordered execution. Each operation reports its own result or error.

db.users.bulkWrite([
  { insertOne: { document: { name: "Luis" } } },
  { updateOne: { filter: { name: "Ana" }, update: { $inc: { age: 1 } } } }
], { ordered: false })

mongoimport reads JSON, CSV or TSV files into a collection, and mongoexport writes a collection to those formats. Both take a --uri or host, a database, a collection and a file path. They are command-line tools, not shell methods.

mongoimport --uri "mongodb://localhost:27017/shop" --collection products --file products.json
mongoexport --uri "mongodb://localhost:27017/shop" --collection products --out products.json

Without an index MongoDB must scan every document in a collection to satisfy a query. An index is a B-tree structure that lets the server jump to matching documents, which changes cost from a collection scan to an index seek. The trade-off is extra storage and slower writes, since each index must be maintained.

Call createIndex with a document describing the fields and direction. A value of 1 means ascending and -1 descending. Creating an index is idempotent, so re-running the same definition is safe.

db.users.createIndex({ email: 1 })

Create an index with the unique option set to true. If duplicate values already exist, the index build fails, so clean the data first. Duplicate inserts then fail with a duplicate key error that your application must handle.

db.users.createIndex({ email: 1 }, { unique: true })

explain() shows the query planner's chosen plan, including whether an index was used, how many documents and keys were examined, and the execution time. Look at the winning plan stage: COLLSCAN means a collection scan, while IXSCAN means an index was used. Comparing totalKeysExamined and totalDocsExamined with nReturned reveals index efficiency.

db.users.find({ email: "a@b.com" }).explain("executionStats")

Embedding keeps related data inside one document, so a single read returns everything and writes stay atomic per document. Referencing stores an id and requires a second query or a $lookup to combine data. Embed when data is read together and bounded in size; reference when it is large, shared, or updated independently.

Model around how your application reads and writes rather than around entity purity. Prefer embedding for one-to-few relationships that are always fetched together, and reference for one-to-many or many-to-many at scale. Keep documents well under the 16 MB limit and avoid unbounded arrays that grow forever.

Common mistakes include treating MongoDB like a relational database with heavy normalization, storing unbounded arrays in a single document, and running updates without a filter. Others are querying without indexes, using skip for deep pagination, and ignoring the difference between null and missing fields. Handling duplicate key errors and validating input also get skipped often.

Install the driver for your language, create a client with a connection string, and reuse that client across requests. The client manages a connection pool internally, so creating one per request is an anti-pattern. Await the initial connection or the first operation before serving traffic.

const { MongoClient } = require("mongodb");
const client = new MongoClient(process.env.MONGO_URI);
await client.connect();
const users = client.db("shop").collection("users");

A connection string starts with mongodb:// or the TLS variant mongodb+srv://, followed by credentials, host or SRV name, optional port, and a default database. Query parameters set options such as retryWrites, w, and readPreference. Credentials should be URL-encoded and never committed to source control.

mongodb+srv://appuser:secret@cluster0.example.mongodb.net/shop?retryWrites=true&w=majority

Users are created per database with createUser and are granted one or more roles. Built-in roles such as read, readWrite, dbAdmin and clusterMonitor cover common needs, and the principle of least privilege applies. Applications should use a dedicated user scoped to the database they need rather than an admin account.

db.createUser({ user: "appuser", pwd: "secret", roles: [ { role: "readWrite", db: "shop" } ] })

Documents in one collection may have different fields and types, so there is no enforced column definition. In practice applications still assume a shape, which is why schema validation is available. Schema-flexible means the database does not block variation, not that design is unnecessary.

Pass a validator with a $jsonSchema to createCollection or to the collMod command. Set validationLevel to strict or moderate and validationAction to error or warn. Validation applies to inserts and updates, not to existing documents unless they are modified.

db.createCollection("users", {
  validator: { $jsonSchema: {
    bsonType: "object",
    required: ["email"],
    properties: { email: { bsonType: "string" } }
  } },
  validationAction: "error"
})

A single BSON document cannot exceed 16 MB, and very deep nesting is discouraged because it complicates queries and updates. If you approach the limit, the data model is usually wrong. Large binary payloads belong in GridFS or object storage.

The legacy mongo shell was a C++ based REPL that is deprecated and no longer shipped with current server versions. mongosh is the supported replacement built on Node.js, offering better autocompletion, syntax highlighting and modern JavaScript. New work should always target mongosh.

The driver generates an ObjectId on the client before sending the write. That means the application already knows the id without a round trip. Only if a raw command omits _id does the server generate it.

Yes, _id can be any BSON type except an array, including a string, number, date or embedded document. A natural unique key such as an email or SKU is a legitimate choice. Compound _id values are possible using a subdocument.

db.users.insertOne({ _id: "ana@example.com", name: "Ana" })

A standalone server has no redundancy or automatic failover. A replica set keeps copies on multiple nodes with an elected primary and automatic failover. A sharded cluster partitions data across shards for horizontal scale and uses config servers plus mongos routers.

A three-member replica set is the usual minimum, providing a primary and two secondaries for failover. Backups, monitoring and authentication should be in place from day one. A single node is acceptable only for development.

Write concern w controls how many nodes must acknowledge a write before it returns; "majority" is the safe default. Read concern controls the consistency of data a read observes, with "majority" avoiding dirty reads. Higher guarantees cost latency.

db.orders.insertOne({ total: 10 }, { writeConcern: { w: "majority", j: true } })

admin holds administrative users and roles plus server-wide commands. local stores node-local data such as the oplog in a replica set member and is not replicated. config holds sharding metadata in a sharded cluster and should never be modified manually.

The client exposes listDatabases(), and a database object exposes listCollections(). These return cursors or arrays of metadata such as names and options. They require the appropriate list privileges.

const dbs = await client.db().admin().listDatabases();
const cols = await db.listCollections().toArray();

findOne returns a single document or null, applying an implicit limit of one. find returns a cursor and is used when you expect multiple documents. Prefer findOne when you only need one record, since it avoids cursor management.

Use countDocuments with a limit of one, or findOne with a projection of only _id. Both stop after the first match. Fetching whole documents just to test existence wastes bandwidth.

db.users.countDocuments({ email: "a@b.com" }, { limit: 1 })

$exists matches documents where a field is present (true) or absent (false). A field present with a null value still counts as existing. It is frequently combined with type checks in data clean-up queries.

db.users.find({ phone: { $exists: false } })

MongoDB compares by type order first, then by value within the same type. Numbers of different numeric types compare by numeric value. A string will never equal a number even if their text looks similar.

Because BSON is type-strict, a query for the number 5 does not match the string "5" or the double 5.0 in all contexts. Data imported from CSV or JSON often stores numbers as strings. Fixing the data with a type conversion is usually the right cure.

$regex matches string fields against a regular expression. An anchored, case-sensitive prefix pattern can use an index, but unanchored or case-insensitive patterns generally cannot. Regex scans are expensive on large collections.

db.users.find({ email: { $regex: "^ana", $options: "i" } })

Use the $size operator with an exact integer, since it matches array length only. $size cannot take a range, so ranges require $expr with $size or a stored count field. Keeping a denormalized count field is a common workaround.

db.posts.find({ tags: { $size: 3 } })

Use dotted notation with the numeric index, as in "scores.0". This targets one concrete position and is rarely index-friendly. Positional queries are useful for debugging more than for production filters.

db.games.find({ "scores.0": { $gt: 90 } })

$all requires the array field to contain every listed value, in any order and with any extra values present. It is the array analogue of a logical AND over membership. It is often confused with $in, which requires only one match.

db.posts.find({ tags: { $all: ["mongodb", "index"] } })

A plain condition on an array of subdocuments can be satisfied by different elements matching different criteria. $elemMatch forces one element to satisfy all criteria. Exact array equality requires the same elements in the same order, which is rarely what you want.

An index supports sorting on its key pattern, and a reverse scan supports the exact opposite direction. A compound index can serve a sort only if the sort fields follow the index prefix order. Otherwise the server performs an in-memory sort, which is limited and can fail on large result sets.

When a sort cannot use an index, MongoDB sorts in memory and aborts if the operation exceeds the memory limit. The classic error is "Sort exceeded memory limit". Fixing it means adding a supporting index or enabling allowDiskUse.

db.logs.find({}).sort({ ts: -1 }).allowDiskUse(true)

Range pagination filters on the last seen sort key instead of skipping N documents. This keeps the query index-supported regardless of page depth. It requires a stable, unique tiebreaker such as _id in the sort.

db.posts.find({ _id: { $lt: lastId } }).sort({ _id: -1 }).limit(20)

By default an idle cursor is closed by the server after about ten minutes of inactivity. noCursorTimeout keeps it alive, at the risk of leaking server resources if you forget to close it. Always close such cursors explicitly.

You can use toArray() for small result sets, or a for await loop for streaming. Streaming avoids loading everything into memory. Remember to close the cursor when you break out early.

const cursor = db.collection("logs").find({ level: "error" });
for await (const doc of cursor) { console.log(doc); }

countDocuments applies the same filter semantics as find, so the numbers should agree for the same filter. Discrepancies usually come from a different filter, from concurrent writes, or from using estimatedDocumentCount by mistake. In a sharded cluster, orphaned documents can also cause differences.

The operation succeeds with matchedCount 0 and modifiedCount 0; it is not an error. Only an upsert creates a document in that case. Applications often need to treat zero matches as a not-found condition themselves.

matchedCount is how many documents the filter selected, and modifiedCount is how many actually changed. Setting a field to its current value matches but does not modify. This distinction matters for optimistic concurrency checks.

A full replacement forces you to send the entire document and risks dropping fields written by other code paths. Targeted operators such as $set are smaller on the wire and safer under concurrency. Replacements also make it easy to lose schema fields added later.

$each appends multiple values at once, and $slice trims the array to a fixed length after the push. Together they implement a bounded recent-items list. Using $sort with them keeps the array ordered.

db.feeds.updateOne({ _id: id }, {
  $push: { recent: { $each: [item], $slice: -50 } }
})

$pull accepts a value or a query condition and removes every element that satisfies it. It removes all matches, not just the first. Use $pull with $elemMatch-style criteria for arrays of subdocuments.

db.orders.updateOne({ _id: id }, { $pull: { lines: { qty: { $lte: 0 } } } })

The positional operator updates the first array element matched by the query filter. It requires the array field to appear in the filter. The filtered positional operator $[<identifier>] with arrayFilters handles multiple or conditional elements.

db.orders.updateOne(
  { _id: id, "lines.sku": "A1" },
  { $inc: { "lines.$.qty": 1 } }
)

arrayFilters let you update specific array elements that satisfy named conditions, without relying on the first match. The update uses $[name] placeholders and the options supply the filter documents. This supports updating several elements or nested arrays in one call.

db.orders.updateOne(
  { _id: id },
  { $set: { "lines.$[l].status": "shipped" } },
  { arrayFilters: [ { "l.qty": { $gt: 0 } } ] }
)

$setOnInsert applies only when the upsert results in an insert, and is ignored on update. $set always applies. Combining them lets you default creation fields while updating mutable fields on every call.

db.stats.updateOne({ day: "2026-01-01" },
  { $inc: { hits: 1 }, $setOnInsert: { createdAt: new Date() } },
  { upsert: true })

You get a write error with code 11000, E11000 duplicate key. The correct handling is to catch it and translate it into a domain response such as "email already registered", not to crash. For idempotent inserts, an upsert often avoids the error entirely.

A write error means an individual operation was rejected, for example by a unique index or validation rule. A write concern error means the write was applied but could not satisfy the requested acknowledgement level. Bulk results report both separately.

Use updateMany with an empty filter and $unset on that field. This touches every document and generates heavy write load, so plan for it on large collections. Confirm the field is truly obsolete before running it.

db.users.updateMany({}, { $unset: { legacyFlag: "" } })

A COLLSCAN stage reads every document in the collection, so cost grows with collection size. An IXSCAN stage walks index keys and fetches only matching documents. FETCH may still appear after IXSCAN when non-indexed fields are needed.

A covered query is answered entirely from the index, with no document fetch. It requires that every field in the filter and projection exists in the index, and that _id is excluded if it is not part of the index. Explain will show no FETCH stage.

db.users.createIndex({ email: 1 });
db.users.find({ email: "a@b.com" }, { _id: 0, email: 1 }).explain()

Embed when the children are always read with the parent, are few in number, and change together with the parent. This gives one atomic write and one read for the whole aggregate. Do not embed when the array can grow without bound.

Use references when the child documents are large, shared across parents, or updated independently and frequently. References also keep the parent document small so it stays in cache. The cost is an extra query or a $lookup to resolve them.

{ "_id": "order_1", "lineIds": [ ObjectId("..."), ObjectId("...") ] }

You store a small copy of the fields you usually need from a related document inside the parent, plus the reference id. Reads avoid a join most of the time, while the source of truth stays in the referenced collection. The copies must be refreshed when the source changes.

There is no hard limit other than the 16 MB document size, but practical arrays should stay in the tens or low hundreds of elements. Unbounded arrays cause document growth, expensive rewrites and poor cache behavior. Use a separate collection or a bucketing pattern when growth is open-ended.

If an update makes a document larger than its allocated space, the server may relocate it on disk. Frequent relocation fragments storage and slows writes. Keeping documents bounded and preallocating fixed-size patterns avoids most of this.

Store an array of ids on one side, usually the side queried more often, and keep the canonical documents in their own collection. For very large or attribute-rich relationships, use a dedicated join collection with one document per pair. This mirrors a junction table but is queried by id arrays.

Yes, and it is normal for polymorphic data such as events or products of different categories. Add a discriminator field such as type and index it for efficient filtering. Keep shapes similar enough that queries and indexes remain useful.

{ "type": "click", "url": "/pricing" }
{ "type": "purchase", "sku": "A1", "amount": 20 }

Split when documents have different access patterns, different retention rules, or wildly different shapes. Separate collections also help when one set needs different indexes or validators. Over-splitting forces application-side joins, so keep boundaries meaningful.

Store the ancestor path as an array of ids on each node, which lets you fetch a whole subtree with one indexed query. The parent reference alone supports only one level at a time. Materialized paths trade write cost for cheap subtree reads.

{ "_id": "laptops", "ancestors": ["electronics", "computers"] }

Use Decimal128 for exact decimal arithmetic, or store integer minor units such as cents. Binary floating point doubles introduce rounding errors that are unacceptable for money. Always store the currency code alongside the amount.

{ "total": NumberDecimal("19.99"), "currency": "USD" }

Store dates as BSON Date values, which are UTC instants, and convert to local time in the application. Keep the time zone identifier separately when the local wall-clock time matters, such as for scheduled events. Avoid storing dates as strings because they sort and compare poorly.

Modern versions allow many such names, but they are strongly discouraged because dot notation and operators rely on those characters. Imported data with dots in keys breaks queries and updates in confusing ways. Rename such fields during ingestion.

Collection and database names are case-sensitive in MongoDB, but on case-insensitive filesystems such as default Windows volumes you cannot create names that differ only by case. Pick one naming convention and stay consistent. This avoids cross-platform surprises.

GridFS splits a file into chunks in a fs.chunks collection and stores metadata in fs.files, bypassing the 16 MB document limit. For most new systems, object storage with a URL in the document is cheaper and simpler. GridFS still makes sense when files must live transactionally with metadata.

A capped collection has a fixed maximum size and behaves as a circular buffer, overwriting the oldest documents. Inserts are fast and preserve natural order, which suits logs and recent-activity feeds. You cannot delete individual documents from a capped collection.

db.createCollection("logs", { capped: true, size: 1048576, max: 5000 })

Time series collections store measurements efficiently by grouping them into internal buckets. They reduce storage and improve range queries over timestamps. You create one with the timeField option and optionally a metaField.

db.createCollection("sensorData", {
  timeseries: { timeField: "ts", metaField: "sensorId", granularity: "minutes" }
})

Each client maintains a pool of connections per server and checks out connections per operation. The pool is bounded by maxPoolSize, which defaults to 100. Reuse a single client per process and size the pool against your concurrency and server limits.

retryWrites lets the driver retry a single write once after a transient network error or failover, using the same transaction number to avoid duplicates. retryReads does the same for reads. Both are enabled by default in current drivers and require a replica set or sharded cluster.

serverSelectionTimeoutMS controls how long the driver waits for a suitable server, and connectTimeoutMS bounds socket establishment. socketTimeoutMS limits how long an operation may sit idle on the socket. Set them low enough that failures surface quickly but high enough to tolerate normal latency.

readPreference decides which replica set members may serve a read: primary, primaryPreferred, secondary, secondaryPreferred, or nearest. Reading from secondaries reduces primary load but may return slightly stale data. Use primary for read-after-write consistency.

db.users.find({}).readPref("secondaryPreferred")

The +srv form looks up DNS SRV records to discover hosts and TXT records for default options, so the URI stays short. It requires TLS and is standard for Atlas. You cannot combine +srv with an explicit port or multiple hosts.

The driver detects the topology change, marks the old primary unavailable, and waits for a new primary to be elected. In-flight operations fail with a retryable error, and retryWrites resubmits them once. This is why applications must still handle transient errors.

Keep them in environment variables or a secrets manager rather than in code or committed config files. Grant the smallest role that works and rotate credentials periodically. Never log a full URI, since it usually contains the password.

Driver errors expose a numeric code and a codeName such as DuplicateKey, plus error labels for retryable cases. Branch on the code rather than on the message text, which can change. Treat unknown codes as unexpected server errors and surface them.

if (err.code === 11000) { /* duplicate key */ }

A write to a single document is always atomic, including updates that touch several fields or array elements. Multi-document operations are not atomic unless you use a transaction. Modeling related data into one document is often the simplest way to get atomicity.

Use a transaction when several documents or collections must change together, such as transferring balance between accounts. Transactions require a replica set or sharded cluster and add latency and locking. Prefer a document model that avoids the need where possible.

const session = client.startSession();
await session.withTransaction(async () => {
  await accounts.updateOne({ _id: "a" }, { $inc: { bal: -10 } }, { session });
  await accounts.updateOne({ _id: "b" }, { $inc: { bal: 10 } }, { session });
});

A compound index covers several fields in a defined order, and the order determines which queries it can serve. The ESR guideline recommends Equality fields first, then Sort fields, then Range fields. Following it lets one index serve both the filter and the sort.

db.orders.createIndex({ status: 1, createdAt: -1, total: 1 })

An index on a field that holds an array becomes multikey, with one index entry per array element. It makes queries on array membership fast. A compound index can have at most one multikey field, and multikey indexes cannot cover queries.

An index serves a sort only when the sort keys follow the index key pattern as a prefix, in the same or fully reversed direction. Otherwise the planner must sort in memory. Mixed directions within a compound sort require a matching index pattern.

A partial index includes only documents that match a filter expression, which shrinks the index and lowers maintenance cost. Queries can use it only when they are guaranteed to match that filter. It is ideal for indexing a rare status value.

db.orders.createIndex({ createdAt: 1 }, { partialFilterExpression: { status: "pending" } })

A TTL index deletes documents automatically after a number of seconds, based on a date field or an array of dates. A background task runs roughly every 60 seconds, so deletion is not immediate. TTL indexes cannot be compound.

db.sessions.createIndex({ createdAt: 1 }, { expireAfterSeconds: 3600 })

A text index supports $text searches with word stemming and stop words across string fields. A collection may have only one text index, though it can span several fields. It is a basic search tool and not a replacement for a dedicated search engine.

db.articles.createIndex({ title: "text", body: "text" })
db.articles.find({ $text: { $search: "mongodb index" } })

hint() forces the query planner to use a specific index by key pattern or name. It is useful when the planner picks a worse plan, or for testing index performance. Hard-coding hints is risky because the collection's data distribution changes over time.

db.users.find({ email: "a@b.com" }).hint({ email: 1 })

If a field has few distinct values, such as a boolean, an index lookup may still match most documents. The planner then correctly prefers a collection scan, so the index only costs storage and write time. Index fields that actually narrow the result set.

Every index must be updated on each insert, update and delete, which increases write latency and disk usage. Redundant indexes also confuse the planner and waste cache. Audit index usage and drop those that are never chosen.

db.users.aggregate([ { $indexStats: {} } ])

queryPlanner shows the winning plan and rejected candidates without executing the query. executionStats executes it and reports counts and timings. allPlansExecution runs all candidate plans, which is heavier and meant for deep tuning.

db.users.find({ email: "a@b.com" }).explain("queryPlanner")

$type matches documents whose field value is of a given BSON type, using either an alias string or a numeric code. It is the main tool for finding badly typed data after an import. Combine it with $exists to reason about missing fields.

db.users.find({ age: { $type: "string" } })

Collation defines language-aware comparison rules for strings, such as case-insensitive or accent-insensitive matching. Without it, comparisons are binary and case-sensitive. Set it per query, per index, or as a collection default.

db.users.find({ name: "ana" }).collation({ locale: "es", strength: 1 })

A unique index enforces uniqueness according to its own collation. Two strings that differ only by case count as duplicates under a case-insensitive collation. Queries must use the same collation for the index to be usable.

db.users.createIndex({ email: 1 }, { unique: true, collation: { locale: "en", strength: 2 } })

$expr allows aggregation expressions inside a find filter, which enables comparisons between two fields of the same document. Plain query operators only compare a field to a constant. $expr is more flexible but usually less index-friendly.

db.orders.find({ $expr: { $gt: ["$paid", "$total"] } })

db.stats() reports database-level figures such as data size and number of collections. db.collection.stats() reports document count, average object size, index sizes and total storage for one collection. These help spot oversized documents or bloated indexes.

db.orders.stats().avgObjSize

remove() and save() are legacy shell helpers with confusing semantics and are deprecated in mongosh. deleteMany/deleteOne and insertOne/replaceOne express intent clearly and return structured results. Using the modern methods avoids surprises during migrations.

No, the unique index on _id is created automatically and cannot be dropped. You can change nothing about its existence, only the values you store. This guarantee is what makes _id lookups reliably fast.

read grants query and read-only commands on a database, while readWrite adds insert, update, delete and index creation on that database. Neither grants administrative control of the server. Applications normally need only readWrite on their own database.

The root role grants full control over the entire deployment, including user and role management. Using it for an application means a single bug or leak can destroy all data. Reserve root for break-glass administration only.

Start mongod with the authorization setting enabled, then create the first administrative user via the localhost exception. After that, all connections must authenticate. Enabling authorization without creating an admin user first locks you out of user management.

mongod --dbpath /data/db --auth

SCRAM is the default challenge-response mechanism for username and password authentication. Credentials are stored hashed per database, and the password never crosses the wire in clear text. TLS should still be used to protect the connection itself.

Roles are granted as { role, db } pairs, so a user created in one database can hold roles on another. The user's authentication database stays fixed at creation time. This lets a single application user work across a few databases without being root.

db.getSiblingDB("admin").createUser({
  user: "reporting",
  pwd: "secret",
  roles: [ { role: "read", db: "shop" }, { role: "read", db: "analytics" } ]
})

A custom role is a named set of privileges you define with createRole, either from scratch or by inheriting built-in roles. It is worth it when built-in roles grant too much, such as an app that only needs to read one collection. Custom roles add operational overhead, so keep them few and documented.

The aggregation pipeline is a server-side framework that passes documents through an ordered sequence of stages, similar to a Unix pipe. Each stage receives the document stream produced by the previous stage, transforms it, and emits zero or more documents to the next stage. Stages can filter, reshape, group, join and compute new values, so the final output is whatever the last stage emits.

db.orders.aggregate([
  { $match: { status: "paid" } },
  { $group: { _id: "$customerId", total: { $sum: "$amount" } } }
])

Stages execute in the exact order they appear in the array; the optimizer may merge or move some stages internally, but the visible semantics follow array order. A few stages have positional rules: $out and $merge must be the last stage, and $geoNear must be the first. $text is only allowed inside a $match that is the first stage of the pipeline.

Each stage processes whatever volume the previous stage produced, so filtering and projecting early keeps later stages cheap. Putting $match and a selective $sort first also lets MongoDB use indexes and avoid a blocking in-memory sort. A pipeline that groups or unwinds before filtering can do far more work than necessary.

$match filters documents using the same query syntax as find(), so operators such as $gt, $in and $elemMatch all work. It should normally be the first stage so that the query planner can use indexes and the smallest possible document set flows downstream. It can also appear later to filter intermediate results after a $group or $unwind.

{ $match: { status: "shipped", amount: { $gte: 100 } } }

Both use the same query operators, but $match cannot use projection options, hints or $where style JavaScript, and it must be a plain query document. $match also cannot contain aggregation expressions such as $expr comparisons directly unless they are wrapped in $expr. Unlike find(), the output of $match continues into further stages rather than being returned to the client.

$project reshapes each document by keeping, dropping or adding fields. Use field: 1 to include and field: 0 to exclude; _id is included by default and must be explicitly excluded with _id: 0. You cannot mix inclusions and exclusions in one specification, except for the _id field.

{ $project: { name: 1, "address.city": 1, _id: 0 } }

Assign an expression instead of 0 or 1, and the field is replaced by the expression result. Expressions can combine operators, literals, field paths and variables, which makes $project useful for unit conversion, string formatting and null handling. In MongoDB 8.x you may also use the shorter { field: "$other" } style aliasing on its own.

{ $project: { total: { $multiply: ["$price", "$qty"] }, label: { $toUpper: "$sku" } } }

$addFields adds or overwrites fields while keeping every other field in the document, whereas $project requires you to list what you keep. $addFields is therefore safer when the document shape should stay mostly intact. $set is an exact alias of $addFields.

$group collapses documents that share the same grouping key into one document per key. The _id field of the output holds the grouping expression, such as a field path, a document of several fields or null for a single global group. All other output fields must be produced by accumulators.

{ $group: { _id: { country: "$country", year: { $year: "$createdAt" } }, n: { $sum: 1 } } }

Both accumulators ignore missing values and values that are not numeric, such as strings or arrays. $sum returns 0 when no numeric value is available, while $avg returns null because it has nothing to divide by. Wrap values in $ifNull or $toDouble if you need a specific fallback behaviour.

$push appends every value in encounter order and keeps duplicates, so it preserves the original sequence. $addToSet keeps only distinct values, which removes duplicates but gives no ordering guarantee and costs more memory on large groups. Use $addToSet for tag-like data and $push when order or repetition matters.

{ $group: { _id: "$sku", tags: { $addToSet: "$tag" } } }

$min and $max compare values across the documents in each group and keep the smallest or largest one. Comparison follows BSON type ordering, so numbers are compared with numbers and strings with strings. They can also be applied to expressions or documents, and $min is frequently used to find the earliest date in a group.

Add a field accumulated with { $sum: 1 }, which increments by one for every document in the group. Using { $sum: "$field" } would add the field values instead of counting documents, and would return 0 if the field is missing. The resulting field is a normal output field that can be renamed or sorted on.

{ $group: { _id: "$status", count: { $sum: 1 } } }

$sort orders the stream by one or more fields using 1 for ascending and -1 for descending. Multiple keys are evaluated left to right, so the first field is the primary order and later fields break ties. Sorting is a blocking operation unless an index can supply the requested order.

{ $sort: { country: 1, revenue: -1 } }

$limit passes only the first N documents of the stream and discards the rest. After a $sort it returns a top-N result, which the engine can compute with a bounded in-memory sort instead of sorting the whole set. Placing $limit before $sort returns arbitrary documents and then sorts only those, which is rarely what you want.

$unwind takes an array field and emits one output document per array element, copying all other fields into each copy. The element itself can be exposed through includeArrayIndex or referenced with $field afterwards. It is the standard way to flatten arrays before grouping on element values.

{ $unwind: { path: "$items", includeArrayIndex: "idx", preserveNullAndEmptyArrays: true } }

By default the document is dropped, because an empty array produces zero output documents. Missing or null array fields are also dropped by default. Set preserveNullAndEmptyArrays: true to keep the document once with the array field missing.

preserveNullAndEmptyArrays keeps documents whose array is missing, null or empty, so counts are not silently reduced. includeArrayIndex adds a field holding the zero-based position of each element, which is useful for ordering or reconstructing the array. It is a common source of inflated document counts when forgotten.

$lookup performs a left outer join with another collection in the same database, matching each input document to foreign documents. The shorthand form needs from, localField, foreignField and as, where as names the array of matched documents. Documents with no match still appear, with an empty array.

{ $lookup: { from: "customers", localField: "customerId", foreignField: "_id", as: "customer" } }

The join runs one query per input document against the foreign collection, so an index on the foreignField is essential; without it each lookup becomes a collection scan. Cost therefore scales with the number of input documents times the per-lookup cost. Filtering the input with an early $match is the most effective optimization.

It returns each input document with new fields added or existing fields overwritten, leaving all other fields untouched. Expressions may reference existing fields, and dotted paths create nested fields. It is usually preferred over $project when only a couple of derived values are needed.

$unset takes a field, a list of fields or an array of field names and removes them from every document. It is shorthand for a $project that excludes those fields. Removing sensitive fields such as password hashes early keeps them from reaching later stages and the client.

{ $unset: ["passwordHash", "internalNotes"] }

$count replaces the stream with a single document of the form { count: n }, where the field name is the argument you supply. It is equivalent to a $group with $sum: 1 plus a $project. An empty input produces no document at all, so clients must handle a missing result.

$facet runs several independent sub-pipelines over the same input documents and returns one document with an array per facet. It lets a single pass produce, for example, paginated results plus total counts plus category breakdowns. Sub-pipelines cannot use $out, $merge or $facet themselves, and the input set is buffered in memory.

{ $facet: { data: [{ $skip: 0 }, { $limit: 10 }], total: [{ $count: "n" }] } }

$bucket groups documents into ranges defined by boundaries, which must be an ascending list of at least two values. It requires groupBy and optionally accepts default for values outside all ranges plus an output document of accumulators. Without a default, documents outside the boundaries raise an error.

{ $bucket: { groupBy: "$amount", boundaries: [0, 50, 100, 500], default: "other", output: { n: { $sum: 1 } } } }

$sortByCount groups by an expression, counts documents per value and sorts the result by count descending. It is shorthand for $group on the expression with $sum: 1 followed by $sort: { count: -1 }. It is handy for quick value distributions during exploration.

$replaceRoot promotes a document-valued expression to be the new top-level document via the newRoot option. $replaceWith is the newer, shorter syntax that takes the expression directly, so $replaceRoot: { newRoot: X } equals $replaceWith: X. The expression must evaluate to a document or the stage fails.

{ $replaceWith: { $mergeObjects: ["$customer", "$$ROOT"] } }

$ifNull returns the first argument unless it is null or missing, in which case it returns the second. In MongoDB 8.x it accepts multiple input expressions and returns the first non-null one. It is the standard way to give defaults to optional fields.

{ $project: { phone: { $ifNull: ["$mobile", "$landline", "unknown"] } } }

$cond evaluates a boolean expression and returns one of two values, either in array form [if, then, else] or with named fields if, then, else. It is the basic if/else of aggregation and can be nested. The condition must evaluate to a boolean, so wrap comparisons explicitly.

{ $project: { tier: { $cond: [{ $gte: ["$total", 100] }, "gold", "standard"] } } }

$switch evaluates a list of branches and returns the value of the first branch whose case is true, falling back to default. It reads more clearly than deeply nested $cond expressions for multi-way logic. If no branch matches and no default is given, the stage raises an error.

{ $switch: { branches: [
  { case: { $lt: ["$n", 10] }, then: "small" },
  { case: { $lt: ["$n", 100] }, then: "medium" } ],
  default: "large" } }

$concat joins any number of string expressions and returns null if any argument is null or missing. Use $ifNull or $toString first when fields are optional or numeric. For joining array elements with a separator, use $reduce combined with $concat.

{ $project: { full: { $concat: ["$firstName", " ", "$lastName"] } } }

$substrCP counts characters, meaning Unicode code points, while $substrBytes counts raw UTF-8 bytes. For text containing accents, emoji or CJK characters, byte indexing can split a character and produce invalid output, so $substrCP is the safer choice. Negative lengths are treated as an error, and out-of-range offsets return an empty string.

$toUpper and $toLower change the case of a string expression, but they only handle ASCII characters reliably. They cannot use an index, so applying them to a field forces per-document evaluation. For real case-insensitive queries, a collation with strength 1 or 2 on the index and the query is the correct tool.

Date part operators such as $year, $month, $dayOfMonth and $hour take a date expression and return an integer. They accept a timezone document so results respect a local calendar rather than UTC. For very large datasets, $dateTrunc on a pre-grouped bucket is more efficient than extracting parts.

{ $project: { y: { $year: { date: "$createdAt", timezone: "America/Bogota" } } } }

$dateToString formats a date into a string using a pattern, with optional timezone and onNull settings. Common specifiers include %Y for the four-digit year, %m for the month, %d for the day and %H:%M:%S for time. It is typically used to build daily or monthly report keys.

{ $project: { day: { $dateToString: { format: "%Y-%m-%d", date: "$createdAt", timezone: "UTC" } } } }

$dateTrunc truncates a date to a given unit such as hour, day or month, returning a date value rather than a string. Because the output is still a date, results can be sorted, compared and indexed normally. It is the recommended way to build time buckets for charts and rollups.

Basic operators include $add, $subtract, $multiply, $divide, $mod, $abs, $ceil, $floor and $round, plus $pow and $sqrt. $add also concatenates when one operand is a date, which makes it useful for date arithmetic. Division by zero raises an error, so guard the divisor.

$filter returns a new array containing only the elements that satisfy a condition, using input, cond and an optional as variable name. The default element variable is $$this. It is the array equivalent of a $match and is often used to trim embedded arrays before returning documents.

{ $project: { active: { $filter: { input: "$items", as: "i", cond: { $eq: ["$$i.active", true] } } } } }

$map applies an expression to every element of an array and returns an array of the results, preserving order and length. It accepts input, as and in. Use it to transform a field inside each array element, for example extracting only product codes.

{ $project: { codes: { $map: { input: "$items", as: "i", in: "$$i.sku" } } } }

$reduce folds an array into a single value by iterating with an accumulator, exposing $$value and $$this in the in expression. It is required for operations like summing array elements, flattening nested arrays or concatenating strings with separators. initialValue defines the starting accumulator.

{ $project: { total: { $reduce: { input: "$items", initialValue: 0,
  in: { $add: ["$$value", "$$this.price"] } } } } }

$size returns the length of an array expression. It raises an error if the value is missing or not an array, so guard it with $isArray and $ifNull on optional fields. In a plain query, $size can only match an exact length inside $expr.

In a find() projection, $slice trims the returned array to the first or last N elements without changing anything else. In aggregation, $slice is an expression that takes an array, a start position and an optional count, and can return a middle subrange using a negative count. The aggregation form can be combined with other expressions, while the projection form cannot.

{ $project: { top3: { $slice: ["$scores", 3] }, last2: { $slice: ["$scores", -2] } } }

$expr lets a query filter compare fields of the same document or use aggregation operators inside find(), $match or deleteMany(). Without it, the query language can only compare a field to a constant. $expr cannot use indexes as efficiently in all cases, so keep it simple and test with explain().

db.orders.find({ $expr: { $gt: ["$shippedQty", "$orderedQty"] } })

$$ROOT references the current top-level document as it enters the stage, which is useful for preserving the original document while reshaping. It is commonly used with $mergeObjects to flatten a subdocument into the root, or with $push to collect whole documents per group. Be aware it holds the full document, so pushing it into an array for many documents can hit memory limits.

{ $group: { _id: "$status", docs: { $push: "$$ROOT" } } }

A let option defines variables that are evaluated once in the stage context and referenced with the double-dollar prefix, for example $$minTotal. Inside $lookup pipelines this is how a local field is passed into the sub-pipeline. Referencing an undefined variable is an error, and variables are scoped to the stage that declares them.

{ $lookup: { from: "orders", let: { cid: "$_id" },
  pipeline: [{ $match: { $expr: { $eq: ["$customerId", "$$cid"] } } }],
  as: "orders" } }

$out writes the pipeline result to a collection, atomically replacing the target collection as a whole. $merge instead inserts, updates or leaves existing documents, so it can incrementally refresh a target. $out is simpler but rebuilds everything, while $merge supports incremental materialized views and can keep existing documents.

$out must be the last stage of the pipeline and writes only to a collection in the same database. It replaces the target collection, so any indexes previously defined on the target are lost and must be recreated. It cannot write to a view, and it does not create a new database.

$merge needs a target collection and an on field that identifies matching documents; the default is _id, and that field must have a unique index. You also choose behaviour with whenMatched and whenNotMatched, and can target a different database. If more than one target document matches, the stage fails, which is exactly what the unique index prevents.

{ $merge: { into: "daily_sales", on: "_id",
  whenMatched: "merge", whenNotMatched: "insert" } }

Run the aggregation that computes the summary and finish it with $merge into a dedicated collection, using whenMatched: "merge" so reruns update the rows. The target collection can then be indexed and queried cheaply, unlike a normal view which recomputes on every read. Applications or a scheduled job decide when to refresh the summary.

Blocking stages that must buffer data are limited to 100 MB per stage by default. Exceeding that limit fails the operation with a memory limit error instead of degrading silently. You either make the pipeline more selective or enable disk use explicitly.

allowDiskUse: true lets blocking stages spill temporary files to disk when they exceed 100 MB of memory. You can pass it as an option to aggregate(), and in MongoDB 8.x find() also accepts it for large sort operations. Spilling prevents failures but is much slower, so it is a safety valve rather than a design goal.

db.orders.aggregate(pipeline, { allowDiskUse: true })

$group and $sort are the classic blockers because they must see many documents before emitting results. $bucket, $facet and $unwind followed by $group can also consume a lot of memory. Reducing input with an early $match and avoiding $push: "$$ROOT" are the usual fixes.

The planner can push a leading $match and a following $sort into a single indexed query plan, so the index both filters and orders documents. It can also cover a pipeline when all needed fields are in the index. Once a stage changes document shape, such as $project or $group, later stages generally cannot use collection indexes.

Run the pipeline with explain("executionStats") and inspect the winning plan, looking for IXSCAN versus COLLSCAN and the number of documents examined. totalKeysExamined close to nReturned indicates a selective index. Metrics such as hasSortStage tell you whether an in-memory sort was needed.

db.orders.explain("executionStats").aggregate([
  { $match: { status: "paid" } },
  { $sort: { createdAt: -1 } },
  { $limit: 20 }
])

Create a text index on one or more string fields with the "text" type, then query with the $text operator and a $search string. $text performs tokenized, stemmed word matching rather than exact substring matching. You can sort by textScore to rank results by relevance.

db.articles.createIndex({ title: "text", body: "text" });
db.articles.find({ $text: { $search: "mongodb index" } },
  { score: { $meta: "textScore" } }).sort({ score: { $meta: "textScore" } });

A collection can have only one text index, though that index may cover several fields. $text must appear inside a $match that is the first stage of a pipeline, and it cannot be combined with hint. Text indexes ignore stop words and stem terms, so exact substring or partial-word searches are not supported.

Wrap the words in double quotes inside the search string to require them as a phrase, and prefix a term with a minus sign to exclude it. Terms separated by spaces are OR-ed by default unless you quote them. This is why "data modeling" -mysql behaves as a phrase match with an exclusion.

db.articles.find({ $text: { $search: "\"data modeling\" -mysql" } })

A 2dsphere index supports queries on GeoJSON geometries and legacy coordinate pairs on a spherical model of the Earth. It enables proximity, intersection and containment queries on points, lines and polygons. The indexed field must contain valid GeoJSON, for example a Point with a coordinates array of longitude then latitude.

db.stores.createIndex({ location: "2dsphere" });

$near returns documents ordered by distance from a point, nearest first, optionally limited by $minDistance and $maxDistance in meters. It requires a geospatial index on the queried field and cannot be used inside an aggregation $match; use $geoNear there. Because results are already sorted by distance, adding your own sort is unnecessary.

db.stores.find({ location: { $near: { $geometry: { type: "Point", coordinates: [-74.07, 4.65] },
  $maxDistance: 2000 } } })

$geoWithin selects documents whose location lies inside a given shape such as a polygon, circle or box, and returns them in unspecified order. $near selects by distance and returns documents sorted nearest first. $geoWithin does not require sorting, which makes it cheaper when you only need membership in an area.

A TTL index is a single-field index on a date or array of dates created with expireAfterSeconds. A background task runs roughly every 60 seconds and deletes documents whose indexed date is older than the configured age. Expiration is therefore not exact and deletions are not guaranteed immediately.

db.sessions.createIndex({ lastSeen: 1 }, { expireAfterSeconds: 3600 });

The indexed field must be a date or an array of dates, and the index must be single-field, so it cannot be part of a compound index. The TTL monitor only removes whole documents, so expiring embedded array elements requires application logic or a capped or time series design. Changing the retention period means using collMod rather than rebuilding the index.

A capped collection is a fixed-size collection that keeps documents in insertion order and overwrites the oldest documents when full. Inserts are very fast and reads in natural order are sequential, which suits logs and queues. The size is declared at creation and cannot be changed without recreating the collection.

db.createCollection("logs", { capped: true, size: 10485760, max: 100000 });

Documents cannot be deleted individually and updates may not increase document size, so they are unsuitable for mutable data. Only the natural order is guaranteed, and you cannot shard a capped collection. MongoDB itself uses one internally for the replica set oplog, which is the classic example of their append-only use case.

A time series collection is a specialized collection type that stores measurements with a timestamp and optional metadata, and internally groups them into buckets. This bucket structure reduces storage and index overhead compared with one document per measurement. It is intended for metrics, sensor data and event streams that are written once and queried by time range.

db.createCollection("weather", { timeseries: { timeField: "ts",
  metaField: "sensor", granularity: "minutes" } });

timeField names the field holding the date of each measurement and is required. metaField optionally names a field that identifies the series source, such as a sensor id, and is used for grouping and bucketing. granularity tells the server the expected interval between measurements so buckets can be sized efficiently.

Updates and deletes are supported but restricted: a delete must match on the metaField and time range, and updates cannot change the time field. Inserting documents with mixed shape in the same series can create many small buckets. Plan retention with TTL on the time field rather than manual deletes where possible.

GridFS is a specification, implemented by drivers, for storing files larger than the 16 MB document limit. It splits a file into chunks and stores metadata separately, so the file can be read in parts. Use it for large binaries such as videos or backups; small files are usually better stored in a normal document or an external object store.

GridFS stores metadata in fs.files and binary chunks in fs.chunks, with an index on the files id and chunk number. The default chunk size is 255 KB, and it can be configured per bucket. Reads reassemble chunks in order, so range requests still fetch whole chunks.

A view is a read-only, named query defined by an aggregation pipeline, stored as metadata in the database. Creating one uses db.createCollection(name, { viewOn: ..., pipeline: [...] }) or db.createView(). Clients query it like a collection, which lets you expose a filtered or reshaped projection without duplicating data.

Standard views are computed on every read and consume no storage, so they are only as fast as their underlying pipeline. To materialize results, end a pipeline with $out or $merge into a real collection and query that collection instead. The trade-off is that the data is only as fresh as the last refresh, which you schedule yourself.

Collation defines language-aware comparison rules for strings, including case and accent handling. To make a query case-insensitive, specify a collation such as { locale: "en", strength: 2 } on the query, and create the index with the same collation so it can be used. The collation must match between index and query or the index is ignored.

db.users.find({ email: "ANA@EXAMPLE.COM" }).collation({ locale: "en", strength: 2 });

Strength sets how many differences are considered significant: 1 ignores case and accents, 2 ignores case but considers accents, and 3, the default, is case and accent sensitive. Higher strengths also consider case variants and, at 4 or 5, punctuation or code point differences. Choose the lowest strength that satisfies the requirement, since it affects index usability.

Read concern controls the durability and isolation guarantees of data returned by a read. "local" is the default and may return data that could still be rolled back, "majority" returns data acknowledged by most replica set members, and "snapshot" provides a consistent point-in-time view, mainly for transactions. "linearizable" exists for reads that must see all prior majority-acknowledged writes.

Write concern states how many members must acknowledge a write before the driver reports success, using w, plus j for journaling and wtimeout for a bound. w: 1 acknowledges from the primary only and is fast, while w: "majority" survives a failover. Stronger write concern costs latency but reduces the chance of rolled-back writes.

db.orders.insertOne(doc, { writeConcern: { w: "majority", j: true } })

A replica set is a group of mongod instances that maintain the same data set for redundancy and high availability. Exactly one member is primary and accepts writes, while secondaries replicate the oplog and can serve reads if configured. Elections promote a secondary automatically when the primary becomes unavailable.

Read preference decides which replica set members a client sends reads to. Modes include primary, primaryPreferred, secondary, secondaryPreferred and nearest. Secondary reads may return slightly stale data, so use primary for read-after-write guarantees.

Retryable writes let the driver automatically retry certain single-document write operations once if a transient network error or failover occurs. MongoDB tracks a session id and transaction number per write so a retried write is applied at most once. They require a replica set or sharded cluster and a supporting write concern, and multi-document operations need a transaction instead.

A session groups related operations so they share causal ordering metadata, which drivers manage automatically. With causal consistency enabled, a read in the same session is guaranteed to see writes performed earlier in that session, even when reading from a secondary. This is achieved by tracking the operation time and cluster time on the session.

Change streams let an application subscribe to a real-time stream of data changes on a collection, database or deployment. They emit resume-token-bearing events such as insert, update, replace, delete and invalidate, without tailing the oplog manually. Because they use the oplog, they require a replica set or sharded cluster and majority read concern.

const cs = db.orders.watch([{ $match: { "operationType": "insert" } }]);
while (!cs.isClosed()) { const next = await cs.next(); printjson(next.fullDocument); }

Each event includes a _id resume token, which you persist after processing and pass back through resumeAfter when reopening the stream. For an event that was not fully processed, startAfter is the correct option. The oplog window limits how far back a resume can succeed, so long outages can make a token unusable.

$jsonSchema lets you attach a JSON Schema validator to a collection so inserts and updates must match declared field types and required properties. You can add it at creation with validator or later with collMod. Validation happens on write, which protects data quality close to the database rather than only in the application.

db.createCollection("users", { validator: { $jsonSchema: {
  bsonType: "object", required: ["email"],
  properties: { email: { bsonType: "string" } } } } });

validationLevel decides which documents are checked: strict validates all inserts and updates, moderate only validates documents that are already valid, and off disables validation. validationAction decides whether a violation is rejected with error or merely logged with warn. Starting with warn is a safe way to roll out a new schema.

Start a session, call startTransaction(), run the operations with that session, then commitTransaction() and end the session. Any error should be followed by abortTransaction(). Transactions require a replica set or sharded cluster and explicit session passing on every operation.

const s = db.getMongo().startSession();
s.startTransaction();
s.getDatabase("shop").orders.insertOne({ _id: 1, total: 10 });
s.commitTransaction();

A transaction has a default lifetime limit of 60 seconds, controlled by transactionLifetimeLimitSeconds, and each operation must fit within the 16 MB oplog entry limit. Transactions block on conflicting writes, so long-running ones increase lock contention. They are also not a substitute for a good schema; cross-document atomicity is expensive compared with single-document atomic writes.

The profiler records slow or all operations in the system.profile capped collection for later analysis. Level 0 is off, level 1 records operations slower than slowms, and level 2 records every operation. Level 2 is heavy, so enable it briefly in production and inspect the results with queries on system.profile.

db.setProfilingLevel(1, { slowms: 100 });
db.system.profile.find().sort({ ts: -1 }).limit(5);

db.currentOp() shows operations currently running on the server, including their namespace, duration, plan summary and waiting state. It is the first tool for diagnosing a slow query or a stuck operation in progress. Filter by secs_running or namespace, and use db.killOp() only for operations you are certain are safe to terminate.

db.currentOp({ secs_running: { $gt: 5 }, active: true });

mongostat prints a per-second snapshot of server counters such as inserts, queries, updates, deletes, connections and page faults. mongotop reports how much time each collection spends on reads and writes, which points to hot collections. Together they separate overall load from per-collection hotspots.

The cluster view shows topology, region, tier and connection string, and supports scaling and pausing the cluster. The backup section manages snapshots and point-in-time restores, while alerts can notify on replication lag, slow queries and disk usage. The built-in Performance Advisor and Query Profiler surface slow operations and index suggestions without shell access.

For a small, bounded number of related items, embed them as an array of subdocuments inside the parent, for example a few addresses on a user. This keeps related data in one document, so a single read returns everything and updates stay atomic. The approach works while the array stays small and is not updated concurrently by many writers.

{ _id: 1, name: "Ana", addresses: [ { city: "Bogota" }, { city: "Medellin" } ] }

When the number of children can grow into the hundreds, store the parent id on each child document and index that field, rather than keeping an unbounded array on the parent. Application code queries children by parent id, optionally with pagination. This keeps documents small and avoids rewriting a growing array on every insert.

db.orders.createIndex({ customerId: 1 });
db.orders.find({ customerId: 42 }).sort({ createdAt: -1 }).limit(20);

For very high cardinality, such as log lines or sensor readings per device, never embed the children; put the parent reference on the child and index it. The parent document then stays small regardless of volume. Time series collections or bucketing are often a better fit than one document per child for this shape.

Embed when the data is read together, the child count is bounded, and the child has little independent life. Reference when the data is large or unbounded, updated independently, accessed separately, or shared by many parents. Always check the 16 MB document limit and the write pattern, since embedding makes every write rewrite the whole document.

The bucket pattern groups many small events into one document, typically by time window, storing an array of measurements plus precomputed counts and totals. It dramatically reduces the number of documents and index entries for high-volume time-ordered data. Reads then fetch one bucket instead of hundreds of documents.

{ sensor: "s1", bucketStart: ISODate("2026-01-01T00:00:00Z"),
  count: 60, total: 1432, samples: [ { t: 1, v: 24 }, { t: 2, v: 25 } ] }

The computed pattern stores a derived value, such as a total, average or count, directly in the document so reads avoid recomputing it. It pays off when reads greatly outnumber writes and the computation is expensive, for example a movie rating average. The trade-off is that the application must update the computed field whenever the underlying data changes.

An extended reference is stale from the moment the source document changes: the copy inside the child keeps the old value until something updates it. The cheap fields to copy are the ones that never change after creation, such as a customer's signup country or a product's SKU prefix. The expensive ones are any field the business can edit, like a name or a price.

Keeping it current needs an explicit writer: either the update path that touches the source also fans out to the copies, or a change stream listener patches them asynchronously. Either way, accept that reads can see a copy a few milliseconds behind, and stamp the embedded block with updatedAt so a background job can find and repair the drift.

A schemaVersion field records which shape a document follows, so application code can handle several versions during a rolling migration. It makes migrations explicit and lets you find documents that still need upgrading. Without it, distinguishing old and new shapes relies on guessing from missing or present fields.

{ _id: 1, schemaVersion: 2, name: "Ana", fullName: { first: "Ana", last: "Diaz" } }

Either migrate lazily, converting documents as they are read or written, or backfill in batches with a script. Lazy migration spreads the work but requires dual-read code paths; batch migration gives a clean end state but must be throttled. Whichever you choose, deploy application code that understands both shapes before and after the change.

Process documents in small batches filtered by an indexed field such as _id, update each batch with bulkWrite, and pause between batches to limit load. Track the last processed key so the job can resume after a failure. Run it during low traffic and monitor replication lag and slow queries while it runs.

db.users.find({ _id: { $gt: lastId }, schemaVersion: { $exists: false } })
  .sort({ _id: 1 }).limit(1000);

A single BSON document cannot exceed 16 MB, which is a hard limit enforced by the server on insert and update. The limit applies to the encoded BSON, including field names and array elements. Exceeding it fails the write, and no index or option can raise the limit.

Split the data using the bucket or subset pattern, move large binaries into GridFS, or promote the embedded array into its own collection with a parent reference. Each option trades a single read for smaller documents and more queries. The important step is recognizing the growth trend before writes start failing.

WiredTiger supports snappy, which is the default and favours speed, zlib for better compression at higher CPU cost, and zstd as a balanced option available in current versions. Indexes use prefix compression by default and can also be compressed. The choice affects CPU versus disk usage, not query semantics.

Pass a WiredTiger storage engine configuration when creating the collection, for example a blockCompressor of zstd. Changing compression on an existing collection requires rewriting its data, typically with an initial sync or a rebuild such as compact or a resync. Verify the effect by comparing storage size and CPU after the change.

db.createCollection("events", { storageEngine: { wiredTiger: {
  configString: "block_compressor=zstd" } } });

When you index a field that holds an array, MongoDB creates one index entry per array element, and the index is marked multikey. A query on an element value can then use the index without scanning the array. Multikey indexes are larger than scalar indexes and have restrictions on how bounds can be combined.

db.articles.createIndex({ tags: 1 });
db.articles.find({ tags: "mongodb" });

Indexing two array fields in the same compound index would produce a cartesian product of index entries for every document, so MongoDB rejects it with a "cannot index parallel arrays" error. The workaround is to index one array field and filter the other in the query, or to restructure the data. A compound index may include at most one array field.

Without $elemMatch, separate conditions on array elements can be satisfied by different elements of the same document. $elemMatch requires a single element to match all conditions, which is usually the intended semantics. It also helps the planner use a multikey index more precisely.

db.orders.find({ items: { $elemMatch: { sku: "A1", qty: { $gte: 2 } } } })

skip(n) makes the server walk and discard the first n documents, so the cost grows with the page number even when an index is used. Deep pages therefore become progressively slower and put load on the server. It is acceptable for small offsets or admin screens, but not for infinite scrolling.

Keyset pagination remembers the last seen sort value and queries for documents after it, for example { _id: { $gt: lastId } } sorted ascending. Each page costs roughly the same because the index seeks directly to the starting point. Include a unique tiebreaker field so pages cannot skip or repeat documents.

db.orders.find({ createdAt: { $lt: lastDate } }).sort({ createdAt: -1 }).limit(20);

ESR stands for Equality, Sort, Range: put fields used in equality filters first, then fields used for sorting, then range-filtered fields. This ordering lets the index satisfy the filter and the sort while scanning a contiguous range. Following ESR usually removes the need for an in-memory sort.

db.orders.createIndex({ status: 1, createdAt: -1, total: 1 });

An index can return documents already ordered if the sort fields form a prefix of the index key pattern and the directions match, either exactly or fully reversed. Equality-constrained leading fields do not break the ordering of the remaining fields. If the sort direction or prefix does not match, the server must perform a blocking sort.

A sort cannot use an index when the requested order does not match an index prefix and direction, when the sort is on a computed or unwound field, or when the pipeline has already reshaped documents. The server then performs an in-memory sort, limited to 100 MB per stage unless disk use is allowed. Large blocking sorts are a frequent cause of slow queries and memory errors.

It stops the moment the planner needs anything the index does not contain. Adding a field to the projection that is not in the index is the usual cause, and so is leaving _id in the results when the index was not built with it. A filter on a field outside the index has the same effect, because the server must fetch the document to test it.

Other quiet breakers: a multikey index where the query has to inspect individual array elements, an $or whose branches use different indexes, and a collation the index was not created with. Check with explain(); if a FETCH stage appears where there was none before, coverage was lost.

Run explain("executionStats") and check that the plan has no FETCH stage and that totalDocsExamined is 0. A plan showing only IXSCAN with PROJECTION_COVERED confirms the index answered everything. If totalDocsExamined is greater than zero, the query is not covered.

Each branch of an $or is planned independently and can use its own index, with the results merged and de-duplicated. If any branch cannot use an index, the whole query degrades to a collection scan. That is why an unindexed clause inside $or can make an otherwise fast query slow.

db.users.find({ $or: [ { email: "a@b.com" }, { phone: "+57300" } ] });

When all values apply to the same field, $in is simpler and lets the planner build a single index scan with multiple bounds. $or is needed when the conditions apply to different fields or contain separate operators. $in also reads more clearly and is easier for the optimizer to handle.

db.orders.find({ status: { $in: ["paid", "shipped"] } });

Negation operators must examine every candidate value to prove that it is not excluded, so they cannot use tight index bounds. With no other selective condition, the server ends up scanning the collection or the whole index. Rewriting the query around positive, selective predicates is usually the fix.

Yes, a leading $match with $or is planned like the same predicate in find(), so each branch can use an index. The same rule applies: an unindexed branch forces a collection scan. Placing the $or after a stage that reshapes documents removes any chance of index use.

countDocuments() runs an aggregation with a $match and $group internally, so it returns an accurate count for a filter but scans matching index entries. The $count stage does the same work inside a pipeline. Neither can rely on collection metadata, which is why both can be slow on very large collections with broad filters.

estimatedDocumentCount() reads collection metadata instead of scanning documents, so it is fast but cannot take a filter. The value can be slightly inaccurate after an unclean shutdown. Use it for dashboards and approximate totals, and use countDocuments() when the number must be exact.

Use $facet with one sub-pipeline for the page using $sort, $skip and $limit, and another for the total using $count. Both run over the same filtered input, which saves a second query and guarantees a consistent snapshot. The facet buffers the input, so keep the pre-facet $match selective.

{ $facet: {
  data: [ { $sort: { createdAt: -1 } }, { $skip: 20 }, { $limit: 20 } ],
  meta: [ { $count: "total" } ] } }

Group by the field and count documents per value, then sort descending by the count; $sortByCount does exactly this in one stage. This gives a quick top-N view of categories, browsers or statuses. Add a $limit afterwards to keep only the most common values.

db.events.aggregate([ { $sortByCount: "$browser" }, { $limit: 10 } ]);

Use $replaceWith with $mergeObjects combining the subdocument and $$ROOT, so the subdocument fields become top-level fields. The nested copy can then be removed with $unset. This is common before exporting data or feeding a downstream system that expects a flat shape.

{ $replaceWith: { $mergeObjects: ["$address", "$$ROOT"] } },
{ $unset: "address" }

Use $switch or nested $cond in a $addFields stage to derive a label, then group by that label. This keeps the range logic explicit and readable compared with deeply nested conditions. For evenly spaced ranges, $bucket is a better fit.

{ $addFields: { band: { $switch: { branches: [
  { case: { $lt: ["$total", 50] }, then: "low" },
  { case: { $lt: ["$total", 200] }, then: "mid" } ], default: "high" } } } }

$$NOW returns the current date and time as a single consistent value for the whole pipeline execution. It is useful for computing ages, retention windows or report periods without sending the timestamp from the client. Because it is evaluated once per operation, all documents are compared against the same instant.

{ $project: { ageDays: { $dateDiff: { startDate: "$createdAt", endDate: "$$NOW", unit: "day" } } } }

$$REMOVE evaluates to a missing value, so assigning it in $addFields or $project removes the field from the output document. It is ideal for conditional exclusion, where a field should disappear only for some documents. It cannot be used to remove fields in a find() projection.

{ $addFields: { internalNote: { $cond: ["$isAdmin", "$note", "$$REMOVE"] } } }

Sort by the timestamp descending before grouping, then take { $first: "$$ROOT" } in the group. The sort must happen before the group, since $first only reflects the incoming order. Sorting ascending and using $last is equivalent but usually less intuitive.

[ { $sort: { createdAt: -1 } },
  { $group: { _id: "$customerId", latest: { $first: "$$ROOT" } } } ]

Collect values with $addToSet and then take $size of the resulting array in a following $project or $addFields. This is a common analytics question because no distinct accumulator exists. The array is held in memory, so for very high cardinality the group can hit the memory limit.

{ $group: { _id: "$country", cities: { $addToSet: "$city" } } },
{ $project: { distinctCities: { $size: "$cities" } } }

After $unwind, group by the parent document id and use $push to collect the elements again, ideally after sorting them into the desired order. This is the standard technique for filtering an array and keeping the result as an array. Remember that $push preserves the incoming order, so sort before grouping if order matters.

[ { $unwind: "$items" }, { $match: { "items.qty": { $gt: 0 } } },
  { $group: { _id: "$_id", items: { $push: "$items" } } } ]

Denormalize the few fields needed at read time using the extended reference or computed pattern, so the common query becomes a single-document read. Keep $lookup for admin views, reports and rare paths where freshness matters more than latency. The cost of denormalization is that changes must be propagated to the copies.

MongoDB writes structured JSON log entries containing a timestamp, severity, component, message id and attributes. Slow operations appear with fields such as planSummary, docsExamined, keysExamined and durationMillis. Filtering on the message id for slow queries and on planSummary: "COLLSCAN" quickly reveals missing indexes.

Select-String -Path mongod.log -Pattern '"planSummary":"COLLSCAN"'

Look for operations with a high ratio of documents examined to documents returned, and for COLLSCAN in the plan summary. Then inspect the filter and sort fields and design a compound index following the ESR rule. Confirm the improvement with explain("executionStats") rather than assuming.

Yes, aggregate() accepts a hint option naming an index, and the pipeline's leading $match and $sort can then use it. Hints are useful for testing or pinning a stable plan, but they become brittle when data distribution changes. Prefer fixing the index design over hard-coding a hint in production.

db.orders.aggregate(pipeline, { hint: { status: 1, createdAt: -1 } });

Projecting only required fields reduces network payload and memory use on both server and client. It also enables covered queries when the projection matches an index. Returning whole documents by default is convenient in development but costly on wide documents with large arrays.

Date operators accept a timezone option using an Olson identifier such as America/Bogota, which shifts the calendar calculations to that zone. Without it, grouping by day happens in UTC and produces off-by-hours buckets for local reports. Store dates in UTC and apply the timezone only when grouping or formatting.

Use $all with an array of values; it matches only documents containing every listed element. A plain array in the query is treated as an exact match of the whole array, which is usually not what you want. $all can use a multikey index for the elements.

db.articles.find({ tags: { $all: ["mongodb", "indexes"] } });

Use $set with the positional $ operator, or arrayFilters with $[elem] when the element is identified by a condition rather than a position. arrayFilters is the reliable choice when several elements could match. Each update rewrites the whole document, so very large arrays are expensive to modify.

db.orders.updateOne({ _id: 1 },
  { $set: { "items.$[i].qty": 5 } },
  { arrayFilters: [ { "i.sku": "A1" } ] });

Use a projection with $elemMatch on the array field, which returns just the first element matching the condition. A plain field projection would return the entire array. This keeps responses small for documents with large arrays.

db.orders.find({ _id: 1 }, { items: { $elemMatch: { sku: "A1" } } });

Store them as Decimal128 when exact decimal arithmetic is required, or as integer minor units when the application can handle scaling. Binary floating point can introduce rounding errors that are unacceptable for money. Aggregation arithmetic on Decimal128 preserves decimal precision.

{ price: NumberDecimal("19.99"), currency: "COP" }

Use a single-document atomic operator such as findOneAndUpdate with $inc, $set or an update pipeline, instead of reading, modifying in the application and writing back. MongoDB guarantees atomicity per document, so concurrent updates cannot interleave inside that operation. Multi-document atomicity requires a transaction.

db.counters.findOneAndUpdate({ _id: "orders" }, { $inc: { seq: 1 } },
  { returnDocument: "after", upsert: true });

Use $push with $each and $slice in an update so the array is trimmed to the newest N elements after insertion. The negative slice count keeps elements from the end of the array. This is a practical way to store a bounded recent-history list without a separate collection.

db.users.updateOne({ _id: 1 },
  { $push: { recentViews: { $each: [ { sku: "A1" } ], $slice: -10 } } });

In the Atlas UI choose Database → Build a Database, pick the M0 (free) tier, select a cloud provider and a region close to your users, name the cluster and deploy. Provisioning takes a few minutes and Atlas walks you through a security quickstart where you create the first database user and add your IP address.

An M0 cluster gives you about 512 MB of storage on shared infrastructure: no dedicated RAM or IOPS, so throughput is modest and noisy neighbours are possible. It is limited to one free cluster per project, has no automated cloud backups, and some features (dedicated tiers, certain backup and scaling options) require a paid tier such as M10 or higher. It is perfect for learning, demos and small prototypes, not for production.

The IP access list (formerly called the whitelist) is a set of IP addresses or CIDR blocks that Atlas accepts TCP connections from. It is enforced by the cluster's front end before authentication, so a correct username and password still fail if your address is not listed: you typically see a timeout or an error such as connection ... refused or not allowed to connect.

Add your current public IP under Network Access → Add IP Address. Cloud runtimes, CI agents and home connections change addresses often, which is the usual reason a working app suddenly cannot connect. You can add 0.0.0.0/0 to allow everything, but that exposes the cluster to the internet and should be limited to throwaway sandboxes.

Atlas database users are separate from your Atlas account: you create them under Database Access → Add New Database User, choosing password (SCRAM) or certificate authentication. Then you assign roles scoped to a database.

Give the application the smallest role that works: readWrite on the single application database covers normal CRUD, while read fits reporting or analytics services. Cluster-wide roles such as readAnyDatabase, dbAdminAnyDatabase or atlasAdmin should be reserved for operational tooling and humans, never for an app server, because a compromised process would reach every database in the cluster.

mongosh "mongodb+srv://cluster0.abcde.mongodb.net/" \
  --username appUser --authenticationDatabase admin

Open the cluster in Atlas, click Connect, and pick the tool: Compass gives you a paste-ready string, Shell gives the mongosh command with your cluster host. The string looks like this:

mongodb+srv://appUser:Secret123@cluster0.abcde.mongodb.net/shop?retryWrites=true&w=majority

Two practical details cause most failures. First, special characters in the password must be percent-encoded (@ becomes %40, : becomes %3A). Second, the path segment (/shop) is the default database, while authentication normally goes against admin. The +srv form resolves the replica set members from DNS, and TLS is enabled for you, so no extra options are needed.

Select the cluster and open the Metrics tab: it charts operations per second, connections, network throughput, disk utilisation, query targeting and WiredTiger cache pressure, with a configurable time range. The Real-Time Performance panel shows currently running operations and slow queries, which is the fastest way to see what is happening right now.

Backups live in the Backup tab, where you can see snapshots, configure a backup policy and restore to a point in time or to a new cluster. Free and shared tiers do not include automated cloud backups, so on M0 you must export your own data with mongodump. Alerts can be wired to email or Slack from the project settings.

Use getIndexes(), which returns one document per index with the key pattern, the generated name and the options.

db.users.getIndexes()

[
  { v: 2, key: { _id: 1 }, name: "_id_" },
  { v: 2, key: { email: 1 }, name: "email_1", unique: true },
  { v: 2, key: { country: 1, createdAt: -1 }, name: "country_1_createdAt_-1" }
]

Read it left to right: key is the indexed field order (1 ascending, -1 descending, "text" or "2dsphere" for special types), name is what you pass to dropIndex(), and options such as unique, sparse, partialFilterExpression or expireAfterSeconds describe behaviour. v: 2 is the current index format version.

Drop an index when it duplicates the prefix of a longer compound index, when it is never used, or when a write-heavy collection pays more in maintenance than the index returns in reads. Check usage first with db.users.aggregate([{ $indexStats: {} }]): an accesses.ops counter that stays near zero after a full workload cycle is strong evidence.

db.users.dropIndex("email_1")
db.users.dropIndex({ country: 1, createdAt: -1 })

Do it in staging first, and remember that dropping frees disk and write overhead immediately but cannot be undone except by recreating it, which on a large collection is an expensive build. The _id index cannot be dropped.

MongoDB has no REBUILD INDEX statement like some relational engines. The reliable approach is to drop the index and create it again with the options you want; on a replica set the build is replicated to secondaries, and from 4.4 you can create an index as hidden: true first, then unhide it once it is built, so it never becomes visible half-ready.

db.users.createIndex(
  { email: 1 },
  { unique: true, hidden: true, name: "email_1_new" }
)
db.users.unhideIndex("email_1_new")

Rebuild only for a concrete reason: changing options (unique, partial, collation), suspected corruption, or a badly fragmented index. Routine rebuilding is not part of MongoDB maintenance and costs a full collection scan plus heavy I/O.

It means the server is scanning the collection to populate the index entries, and until that finishes the index exists in the catalog but the query planner will not use it. Builds run in a hybrid mode: they take a brief exclusive lock on the collection to start and finish, but yield to reads and writes in between, so the collection stays available.

You can watch progress with db.currentOp({ "msg": "Index Build" }), which reports the index name and a progress percentage, or through db.adminCommand({ currentOp: 1, $all: true }) on a replica set. A build that seems stuck is usually I/O bound or waiting for a long-running transaction to finish. Dropping the index while it builds aborts the build.

db.users.stats() reports indexSizes as a document mapping each index name to its size in bytes, plus totalIndexSize. db.users.totalIndexSize() returns just the total, which is handy in a script.

db.users.stats().totalIndexSize   // bytes
db.users.stats().indexSizes
// { _id_: 81920, email_1: 65536, country_1_createdAt_-1: 204800 }

For a single index on a large collection, the aggregation form with $collStats and storageStats avoids pulling the whole stats document. This matters because indexes live in memory as well as on disk: if total index size greatly exceeds the WiredTiger cache, every query risks disk reads, and an oversized unused index is a good candidate for dropping.

Inside the shell, db.version() returns the server version string such as 8.0.4; db.serverBuildInfo() adds the git commit, available storage engines and compile options, and db.hello() tells you whether you are connected to a primary or a secondary. The client version is not a shell method: run mongosh --version in your terminal, or check db.getMongo() in the shell for the connection object.

db.version()          // "8.0.4"  (server)
mongosh --version     // "2.3.1"  (client, from the OS terminal)

The distinction matters during upgrades and when a documented helper is missing: newer mongosh against an old server can behave differently, and features you read about may not exist on the server you reached.

Pass a filter document instead of dumping every operation. The most useful fields are active, secs_running, ns and op.

db.currentOp({
  active: true,
  secs_running: { $gt: 5 },
  ns: { $ne: "" }
})

For each result read opid (needed for killing), op (query, update, command), ns, secs_running, planSummary (COLLSCAN on a huge collection is a red flag), numYields, waitingForLock and client. If nothing shows up, remember that idle connections and drivers' pooled sockets appear as inactive, and that the shell only sees operations on the node it is connected to.

Find the operation id first, then kill it from the admin database. The id is an opid, not a session or thread id.

db.currentOp({ active: true, secs_running: { $gt: 30 } })
db.adminCommand({ killOp: 1, op: 123456 })

The server interrupts the operation at its next yield point, so a long index build or a large update may take a moment to stop, and the client receives an error such as operation was interrupted. Some internal operations cannot be killed safely, and killing a write mid-flight can leave partial work unless it was in a transaction. Always confirm with currentOp afterwards, and prefer fixing the query or index over killing operations routinely.

Start with three areas. Connections: compare connections.current with connections.available; being close to the limit means pool leaks or undersized limits. Queues and locks: non-zero globalLock.currentQueue.readers/writers shows contention. WiredTiger cache: check wiredTiger.cache["bytes currently in the cache"] against ["maximum bytes configured"] and watch the tracked dirty bytes, since a cache at capacity with high dirty pages turns reads into disk I/O.

db.serverStatus().connections
db.serverStatus().globalLock.currentQueue
db.serverStatus().wiredTiger.cache["bytes currently in the cache"]

Then glance at opcounters for traffic shape and asserts for errors. The document is huge, so query the specific path instead of printing all of it.

Rename a collection with renameCollection, which needs the admin database and takes a lock; both names must be in the same database, and dropTarget: true overwrites an existing target.

db.adminCommand({
  renameCollection: "shop.orders_old",
  to: "shop.orders",
  dropTarget: true
})

Rename a field inside documents with the $rename update operator, using a dotted path for nested fields:

db.users.updateMany({}, { $rename: { "name": "fullName" } })
db.users.updateMany({}, { $rename: { "address.zip": "address.postalCode" } })

$rename is a per-document update, not an atomic schema change: it overwrites the target field if it exists, does not work on array elements, and on a large collection should be run in batches so you can throttle and resume.

$slice in a projection trims an array after the document is matched, without changing which documents are returned. Three forms are worth memorising.

db.posts.find({}, { comments: { $slice: 3 } })     // first 3
db.posts.find({}, { comments: { $slice: -3 } })    // last 3
db.posts.find({}, { comments: { $slice: [10, 5] } }) // skip 10, take 5

It is the standard way to avoid shipping a 500-element array to a mobile client, and it can be combined with other projection fields. The catch: the filter runs first, so you cannot ask for documents whose sliced window contains a value; that needs $filter or an aggregation with $slice as an expression. Also remember the document on disk is still read in full.

A view is a saved aggregation pipeline over a collection or another view. It stores no data, is computed on every read, and is inherently read-only: insert, update and delete are rejected against it.

db.createView("publicUsers", "users", [
  { $match: { active: true } },
  { $project: { name: 1, city: 1, email: 0, ssn: 0 } }
])

Grant privileges on the view's own namespace rather than the base collection, so a reporting user can read the projection but not the raw documents:

db.createRole({
  role: "publicUsersReader",
  privileges: [{ resource: { db: "shop", collection: "publicUsers" },
                 actions: ["find"] }],
  roles: []
})
db.grantRolesToUser("reporting", [{ role: "publicUsersReader", db: "shop" }])

Typical uses are hiding PII, freezing a stable field shape for consumers, and pre-filtering tenants.

A namespace is the fully qualified name of a collection or index in the form database.collection, for example shop.orders. It is the identifier the catalog, the query planner and every diagnostic surface use: you see it as the ns field in currentOp, in slow-query log lines, in profiler output, in lock reports and in error messages such as a duplicate key error naming shop.orders.

Namespaces matter operationally because MongoDB documents a length limit (120 bytes for a collection namespace, 255 for index namespaces), so extremely long database or collection names can be rejected. Views, time series collections and GridFS buckets all occupy their own namespaces, which is why a dump of a "single" logical dataset often shows several entries.

Because the two predicates are not the same. { field: null } matches documents where the field is explicitly null or missing, and a normal (non-sparse) index stores missing fields as null entries, so the planner can seek to them. $exists: false matches only missing fields, and those documents are precisely the ones a sparse index omits, so a sparse index cannot answer it and the server falls back to a collection scan.

db.users.find({ deletedAt: null })          // can use index on deletedAt
db.users.find({ deletedAt: { $exists: false } }) // usually COLLSCAN
db.users.find({ deletedAt: { $exists: true } })  // can use a sparse index

That asymmetry is a good reason to write explicit nulls instead of omitting fields when you plan to query on absence.

Store it as BSON binary with the UUID subtype (subtype 4 for a standard RFC 4122 UUID), using the driver's native UUID type rather than a hand-made buffer.

// mongosh
db.devices.insertOne({
  _id: UUID("3f2504e0-4f89-11d3-9a0c-0305e82c3301")
})

// Node.js driver
await devices.insertOne({ _id: new UUID("3f2504e0-4f89-11d3-9a0c-0305e82c3301") })

Binary form is 16 bytes against roughly 36 for the canonical string, so indexes are smaller, comparisons follow byte order, and there are no case or collation surprises. The string form is friendlier for debugging, URLs and CSV round trips, but it doubles storage and invites the classic bug of mixing uppercase and lowercase representations of the same identifier. Whichever you choose, keep one format for the field across the whole system.

WiredTiger stores each index as a B-tree of pages, with internal pages holding separator keys and leaf pages holding the actual key and record-id pairs. Pages are cached uncompressed in memory and compressed on disk, and they are written as immutable blocks so readers never block writers. The B-tree is updated in place within the cache but persisted through checkpointing, which makes index lookups logarithmic in page count rather than linear scans. This structure is why index depth and page size matter for read latency.

Cardinality is the number of distinct values in a field, while selectivity describes how well a predicate filters the collection. A field can have high cardinality but still be a poor index if most queries match a large fraction of documents. Good indexes typically match a small percentage of the collection, so the planner prefers them over collection scans. Measure real selectivity from explain output, not from the number of distinct values alone.

ESR stands for Equality, Sort, Range: fields used in equality predicates should come first, then fields used for sorting, then fields used in range predicates. Following this order lets the index satisfy equality filters, deliver results already sorted, and still bound the range scan efficiently. Placing a range field before a sort field forces an in-memory sort because the index order is broken after the range. The rule is a guideline, and the planner's actual choice should be confirmed with explain.

// Query: status = "active", sort by createdAt, amount > 100
db.orders.createIndex({ status: 1, createdAt: 1, amount: 1 })

A multikey index is created automatically when an indexed field holds an array, and MongoDB writes one index entry per array element. It accelerates queries that match array elements, but a single compound index cannot have more than one array field, because that would create a Cartesian product of entries. Multikey indexes also cannot provide a covered query when the array field itself is returned, since the full array is not stored in the index. Sorting on a multikey field can require a blocking sort because the same document appears under several keys.

A compound index can serve queries that use any contiguous prefix of its fields, starting from the first field. An index on {a: 1, b: 1, c: 1} supports queries on a, on a plus b, and on all three, but not on b alone. This is why field order in a compound index is a design decision, not a formality. An index on the reverse order is a separate index and is not covered by the prefix rule.

A covered query is answered entirely from the index, with no document fetch. Every field in the filter and in the projection must be present in the index, and the query must not return the full document. The plan shows IXSCAN with no FETCH stage. Index-only plans reduce I/O but can be slower than expected if the index entries are large.

db.users.createIndex({ email: 1, name: 1 })
db.users.find({ email: "a@b.co" }, { _id: 0, email: 1, name: 1 })

A unique index rejects duplicate values, but by default only one document may have the field missing, because missing counts as null. A sparse index skips documents where the field is absent, which removes that null collision but also makes the index unusable for queries that need those documents. A partial index uses a filter expression to index only matching documents, giving finer control plus a smaller index. Partial indexes are usually the better choice for selective uniqueness.

db.users.createIndex(
  { email: 1 },
  { unique: true, partialFilterExpression: { email: { $exists: true } } }
)

A TTL index is a single-field index on a date or array-of-dates field with an expireAfterSeconds option. A background thread removes expired documents, typically within about a minute of expiry, so deletion is not exact. TTL indexes cannot be compound, and the field must be a date or an array of dates. Expiry is based on the indexed value, not on document insertion time, so clock skew in application writes matters.

db.sessions.createIndex({ lastSeen: 1 }, { expireAfterSeconds: 3600 })

A text index tokenizes string fields, stems words, and stores term postings so that $text queries can search and rank matches. Weights set the relative importance of indexed fields in the computed text score, so a title match can outrank a body match. A collection may have at most one text index, though it can span many fields. Text indexes are case-insensitive and ignore stop words by default, and they do not support language-specific analysis beyond the specified default language.

db.articles.createIndex(
  { title: "text", body: "text" },
  { weights: { title: 10, body: 1 } }
)

A wildcard index indexes all fields, or all fields under a given path, which suits schemas with unpredictable or highly polymorphic field names. It removes the need to pre-declare an index per attribute, but it writes one index entry per field per document, so it grows faster and consumes more memory than a targeted index. Wildcard indexes do not support shard keys and have restrictions on compound use. They should be a fallback when the schema genuinely cannot be enumerated.

db.products.createIndex({ "attributes.$**": 1 })

A hidden index is maintained by the storage engine but ignored by the query planner. It lets you test the impact of dropping an index without paying the cost of a rebuild if the removal turns out to be harmful. You can unhide it instantly to restore planner visibility. Hiding is useful during index audits when you want to confirm an index is genuinely unused.

db.orders.hideIndex("status_1")
db.orders.unhideIndex("status_1")

A rolling build creates the index on one replica set member at a time, typically starting with a secondary, so the set never loses all redundant capacity at once. Modern MongoDB builds indexes concurrently on secondaries by default, so the classic background-build flag is no longer the mechanism to rely on. Rolling builds still matter for large collections where an index build could saturate I/O or stall replication. Plan them during low-traffic windows and watch replication lag between steps.

WiredTiger keeps hot index pages in its cache, so a large index competes with document data for the same memory budget. When index pages are evicted, lookups require disk reads, which raises latency even if the plan is otherwise optimal. Wide compound indexes and multikey indexes on large arrays amplify this effect. Auditing rarely used indexes is as important as adding missing ones.

MongoDB can intersect results from multiple indexes for a single query, but it rarely does so and the planner usually prefers one well-chosen compound index. Intersection costs extra work per index and per matching document, so it is often slower than a single selective index. You should not design a schema expecting intersection to substitute for a compound index. If explain shows an intersection plan, consider whether a compound index would be simpler and faster.

The planner generates candidate plans, runs them in a trial period against a sample of the work, and keeps the plan that returns results with the least work. The chosen plan is cached by query shape, letting later executions skip the trial phase. The cache is invalidated by index changes, collection drops, and explicit clears. A cached plan can become stale when data distribution shifts, which is why plan cache inspection is part of performance work.

queryPlanner shows the winning plan and rejected candidates without executing the query. executionStats runs the query and reports actual keys and documents examined, timing, and rows returned. allPlansExecution adds execution statistics for the rejected candidate plans, which is useful when you want to know why a plan lost. Use executionStats for routine tuning and allPlansExecution when the winning plan looks wrong.

db.orders.find({ status: "paid" }).explain("executionStats")

totalKeysExamined, totalDocsExamined, and nReturned together reveal how efficiently the plan filters. An ideal selective query has these three numbers close to each other, while a large gap between keys examined and documents returned suggests a weak index or a filter applied after the scan. Also check executionTimeMillis and whether the plan contains a blocking SORT. Comparing ratios is more informative than looking at absolute timings alone.

COLLSCAN reads every document in the collection and evaluates the filter, which is linear in collection size. IXSCAN walks an index to find matching keys, then FETCH retrieves the full documents for those keys. A plan with IXSCAN and no FETCH is covered. Seeing COLLSCAN on a hot query is the clearest signal that an index is missing or unusable.

A blocking sort appears when no index can supply the requested order, so the server must buffer and sort documents in memory. It is especially costly when the sort exceeds the memory limit, which produces an error rather than a silent spill. To remove it, create an index whose key order matches the sort and whose leading fields satisfy the query's equality predicates. Verify success by confirming the plan has no SORT stage and hasSortStage is false.

You can list cached plans with $planCache.list and inspect their scores and whether they are marked inactive. Clearing a single shape with $planCache.clearPlansByQuery or the whole cache with $planCache.clear forces replanning. This helps when a plan was cached under an unrepresentative data distribution, for example right after a bulk load. Clearing the whole cache is a blunt instrument that briefly increases planning cost.

db.orders.getPlanCache().clear()
db.orders.getPlanCache().list()

hint() forces the planner to use a named index or key pattern, bypassing plan selection. It is appropriate when you have proven that the planner consistently picks a worse index, or when you need a predictable plan for a latency-sensitive path. Hints are brittle: dropping or renaming the index breaks the query, and a hint can become suboptimal as data changes. Reassess hints after major data or index changes instead of leaving them indefinitely.

db.orders.find({ status: "paid" }).hint({ status: 1, createdAt: -1 })

$indexStats reports per-index access counters, including the number of operations that used each index since the last restart or index creation. Indexes with zero accesses over a representative period, including a full business cycle, are candidates for removal. The counters reset on restart, so a short observation window can be misleading. Combine the stats with knowledge of ad-hoc or rare queries before dropping anything.

db.orders.aggregate([{ $indexStats: {} }])

A replica set has one primary that accepts writes and any number of secondaries that replicate the oplog and can serve reads. An arbiter participates in elections but holds no data, so it can tip a vote without adding storage or read capacity. Deployments usually use three voting data-bearing members to tolerate one failure. Clients discover topology through the driver, which routes writes to the primary and reads according to the read preference.

The oplog is a capped collection on every member that records idempotent write operations in a total order. Secondaries tail the primary's oplog, fetch the referenced documents when needed, and apply the operations to their own data. Because entries are idempotent, a secondary can re-apply an operation safely after a restart. The oplog is the mechanism for both replication and change stream history.

Lag grows when a secondary cannot apply oplog entries as fast as the primary produces them, often due to slow disk, insufficient CPU, long-running index builds, or network saturation. rs.printSecondaryReplicationInfo() and the replication lag fields in serverStatus show the gap in seconds. Persistent lag risks the secondary falling off the oplog, which requires a full resync. Alert on lag trends rather than single spikes, since brief spikes are normal under bursty writes.

rs.printSecondaryReplicationInfo()

Write concern describes how many members must acknowledge a write before the client considers it successful. w: 1 acknowledges on the primary's memory, w: "majority" waits for most voting data-bearing members, and j: true additionally requires the write to be journaled. Higher acknowledgement reduces the chance of rollback on failover at the cost of latency. Durable critical writes should combine majority acknowledgement with journaling.

db.orders.insertOne(doc, { writeConcern: { w: "majority", j: true } })

local returns data as the queried member currently has it, with no guarantee it will not be rolled back. majority returns data that has been acknowledged by a majority and is therefore durable against failover. snapshot provides a consistent point-in-time view, and outside transactions it is tied to majority-committed data. Use snapshot inside multi-document transactions when you need a stable view across reads.

primary sends all reads to the primary, guaranteeing the freshest data. primaryPreferred falls back to a secondary only when no primary is available. secondary sends reads to secondaries, and secondaryPreferred falls back to the primary when no secondary is reachable. nearest picks the lowest-latency eligible member, which may be the primary. Secondary reads can return stale data, so they suit analytics and reporting rather than read-your-writes flows.

A session tracks an operation time as it performs reads and writes, and subsequent operations in the same session send that time as afterClusterTime. The server then waits until the member has applied at least that oplog timestamp before answering. This guarantees read-your-writes and monotonic reads across members without requiring primary reads. Causal consistency requires majority read concern and is available in sessions on replica sets and sharded clusters.

const session = db.getMongo().startSession();
session.startTransaction({ readConcern: { level: "majority" } });

Members hold elections when they detect that the primary is unreachable, and a candidate needs votes from a majority of voting members to win. Higher priority members are more likely to be elected, and equal priorities are broken by the most recent oplog entry. Elections take seconds but cause a brief window with no primary. Setting priorities deliberately keeps preferred hardware in the primary role.

A stepdown makes the primary relinquish its role, usually triggered by replSetStepDown or by another member with higher priority becoming eligible. In-progress writes can fail with a retryable error while a new primary is elected. Modern drivers automatically retry retryable writes and certain reads once, hiding most of the disruption. Applications should still handle the case where a retry is refused because the operation is not idempotent by design.

rs.stepDown(60)

A hidden member replicates data but is invisible to client read preferences, so it takes no application traffic. A delayed member applies oplog entries after a fixed delay, providing a window to recover from accidental deletes or bad deployments. Both must have priority zero so they never become primary. Delayed members are a safety net, not a backup replacement.

The majority commit point is the most recent oplog timestamp that a majority of data-bearing members have acknowledged. Reads at majority read concern and w: "majority" writes are defined relative to this point. Data beyond it can be rolled back if the primary fails before replication completes. Monitoring the commit point relative to the primary's latest entry reveals replication health.

A sharded cluster consists of shards that hold subsets of the data, mongos routers that direct client operations, and config servers that store cluster metadata such as chunk ranges and shard membership. Applications connect to mongos and are unaware of which shard holds a document. The config servers form their own replica set, and each shard is itself a replica set. Metadata consistency across config servers is essential for correct routing.

A chunk is a contiguous range of shard key values assigned to a shard, and it is the unit of data movement during balancing. MongoDB splits chunks that exceed the configured maximum size and migrates them to even out distribution. Chunk count is metadata, not physical storage granularity, so many small chunks increase metadata overhead. The balancer moves chunks, not individual documents.

A good shard key has high cardinality, low frequency for common values, and a distribution that grows without hotspots. It should also appear in most queries so that operations can be targeted rather than broadcast. Monotonically increasing keys such as timestamps concentrate writes on the last chunk, creating a hot shard. Shard keys are difficult to change, so the decision deserves up-front modelling of query and write patterns.

A hashed shard key distributes documents pseudo-randomly, which spreads writes evenly and avoids monotonic hotspots, but it cannot serve range queries efficiently because ordering is lost. A ranged shard key preserves order, enabling targeted range scans and better locality, at the risk of uneven distribution. Hashed keys are a good default for high-volume insert workloads with point lookups. Choose ranged keys when range queries on the key are common and you can design a compound prefix to avoid hotspots.

A targeted query includes the shard key, so mongos can route it to a single shard or a small set of shards. A scatter-gather query omits the shard key and must be broadcast to every shard, then merged, which scales with shard count. Scatter-gather is acceptable for rare admin queries but damaging on hot paths as the cluster grows. Query shapes, not just index design, determine sharding performance.

The balancer runs on the primary config server and migrates chunks between shards when distribution crosses configured thresholds. Migrations consume network and disk I/O on both source and target shards, which can affect latency. You can schedule balancing windows or temporarily stop the balancer during bulk loads. A cluster with a poorly chosen shard key may balance constantly without ever achieving even distribution.

sh.stopBalancer()
sh.startBalancer()

Zones associate ranges of shard key values with specific shards, letting you pin data to geographic regions or to dedicated hardware tiers. They support data residency requirements and workload isolation, for example keeping EU customer data on EU shards. Each shard can belong to multiple zones and each zone can cover multiple ranges. Every range covered by a zone must have at least one shard in that zone, otherwise writes fail.

Change streams expose a real-time, resumable cursor over oplog-derived events for a collection, database, or deployment. Each event carries a resume token, and passing the last token in resumeAfter lets a consumer continue exactly where it stopped after a disconnect. Tokens remain valid as long as the corresponding oplog history exists, so consumers must persist them durably. Change streams require a replica set or sharded cluster and a read concern that supports majority-committed history.

const cs = db.orders.watch([{ $match: { "operationType": "insert" } }]);

Transactions are scoped to a session, started with startTransaction, and committed or aborted explicitly. They provide snapshot isolation, and conflicting writes cause a TransientTransactionError that should be retried as a whole. Commit itself can fail with an unknown result, which requires retrying commitTransaction with the same transaction number. Transactions have a default lifetime limit and should be short and limited in scope.

The bucket pattern groups many small related documents into a single document, often with a count and a bounded array, reducing document count and index size. It suits high-volume time series or event data where reads fetch a window rather than a single item. Buckets must be bounded so they do not approach the 16 MB document limit. Time series collections apply a similar internal bucketing strategy automatically.

{ sensorId: "s1", start: ISODate("2026-01-01"), count: 3,
  readings: [ { t: 1, v: 20 }, { t: 2, v: 21 }, { t: 3, v: 22 } ] }

The computed pattern precalculates expensive values at write time and stores them on the document, so reads avoid aggregation or joins. Typical examples are running totals, average ratings, or denormalized counts. The trade-off is that every write path must maintain the computed value consistently, which makes atomic updates and transactions important. Use it when reads vastly outnumber writes and the computation is expensive.

The extended reference pattern embeds a small, frequently accessed subset of a referenced document into the referring document to avoid a lookup. For example, an order might store the customer's name alongside the customer id. It reduces read latency at the cost of duplicating data that may drift when the source changes. Keep the duplicated fields few, stable, and non-authoritative.

The subset pattern keeps a bounded, most-relevant portion of a large array in the main document and moves the rest to another collection. A product might embed its fifty newest reviews while older reviews live in a separate collection. This keeps documents small and hot reads fast without losing data. The application must manage the promotion and archival logic explicitly.

The attribute pattern converts a set of sparse, unpredictable fields into an array of key-value subdocuments, often paired with a wildcard or compound index on the key and value. It suits product catalogs where each category has different attributes. Queries then target array elements rather than many optional top-level fields. The cost is slightly more complex query syntax and larger index entries.

{ sku: "A1", attrs: [ { k: "color", v: "red" }, { k: "size", v: "M" } ] }

The polymorphic pattern stores documents of different shapes in one collection, using a shared discriminator field such as type to distinguish them. It keeps related entities together and avoids a proliferation of collections, which simplifies cross-type queries. The cost is that application code and indexes must handle divergent fields, and validation must be written per type. It works best when the variants share most of their structure.

For one-to-few, embed the children because the array stays small and reads become a single document fetch. For one-to-many with a bounded but larger count, embed identifiers or use the subset pattern and reference the rest. For one-to-squillions, always reference from the child side, because an unbounded array risks the 16 MB limit and unbounded index growth. The decisive question is whether the array is bounded and whether it is read together with the parent.

Embedding gives atomic single-document reads and writes, fewer round trips, and no application-level join, but it duplicates data and can grow without bound. Referencing keeps data normalized and avoids duplication, but requires $lookup or extra queries and cannot update related data atomically in one write. Access patterns should drive the decision: model for the queries you actually run, not for abstract purity. Hybrid approaches with small extended references are common in production schemas.

Denormalization is justified when a read path is hot, the duplicated data changes rarely, and the cost of a join or second query dominates latency. It trades write complexity and storage for read speed, so the write paths must be audited to keep copies consistent. Document the authoritative source of each duplicated field to avoid divergent updates. If the duplicated value changes often, denormalization usually causes more bugs than it saves milliseconds.

Add a schemaVersion field to documents and let the application read multiple versions, upgrading lazily on write or through a background job. This avoids a stop-the-world migration and keeps rolling deployments safe. For bulk changes, use batched updates with a filter on the old version and monitor progress. Keep both readers and writers compatible during the transition window.

db.users.updateMany(
  { schemaVersion: 1 },
  { $set: { schemaVersion: 2, "prefs.theme": "light" } }
)

Time series collections store measurements in internal buckets grouped by series and time window, which compresses data and reduces index overhead. They provide specialized indexes and support window functions and downsampling through aggregation. Writes go through a bucketing layer, so updating or deleting individual measurements is more restricted than in a normal collection. They are the recommended choice for metrics, sensor data, and event streams with a time dimension.

db.createCollection("metrics", {
  timeseries: { timeField: "ts", metaField: "sensor", granularity: "minutes" }
})

Atlas Search is a full-text search engine built on Apache Lucene, exposed through the $search aggregation stage. It supports analyzers, fuzzy matching, autocomplete, phrase and compound queries, and relevance scoring that the native text index cannot match. It maintains its own search index separate from MongoDB indexes, defined by a JSON index specification. Native text indexes remain simpler for basic keyword matching inside a self-managed deployment.

db.products.aggregate([
  { $search: { text: { query: "wireless", path: "name" } } }
])

Atlas Vector Search indexes high-dimensional embeddings and supports approximate nearest-neighbour queries through the $vectorSearch stage. It enables semantic search and retrieval-augmented generation by comparing query embeddings with stored document embeddings. You must generate embeddings outside the database and store them in an array field with a matching index definition. Results are ranked by vector similarity and can be combined with filters or a search stage.

{ $vectorSearch: { index: "vec", path: "embedding",
  queryVector: q, numCandidates: 100, limit: 10 } }

Drivers maintain a pool of connections per server in the topology and check out a connection per operation, returning it when the operation completes. maxPoolSize caps concurrent connections, and a value too low causes checkout queueing under load while too high exhausts server file descriptors and memory. Size it from concurrent request volume and average operation duration, then validate with server-side connection counts. In serverless or autoscaling environments, keep pools small per instance and account for total instance count.

mongodb://host/?maxPoolSize=50&minPoolSize=5

WiredTiger uses optimistic concurrency: a write conflict occurs when two operations try to modify the same document concurrently. One operation succeeds and the other receives a WriteConflict error, which is a transient error that the server or driver retries. Long-running transactions and hot documents increase conflict rates. Reducing transaction scope and avoiding read-modify-write patterns on the same document lowers contention.

An upsert is a read followed by a conditional insert, so two concurrent upserts with no matching document can both attempt an insert. A unique index on the filter fields makes one of them fail with a duplicate key error, which the application should treat as a signal to retry as an update. Without a unique index there is no server-side protection. Always pair upsert logic with a unique constraint on the identity fields.

db.users.createIndex({ email: 1 }, { unique: true })
db.users.updateOne({ email: e }, { $setOnInsert: { createdAt: new Date() } }, { upsert: true })

An operation is idempotent if repeating it produces the same final state as running it once. Setting an absolute value with $set is idempotent, while $inc and array $push are not, so a retried increment double-counts. Because drivers retry retryable writes automatically, non-idempotent operations need a deduplication key or an update that guards on current state. Designing for idempotency is the cleanest way to survive failover and network timeouts.

Start from the actual query workload: fields in equality filters, frequent sorts, and join keys on the foreign side are the primary candidates. Weigh each candidate by how selective it is and how often the query runs, since every index adds write cost and memory pressure. Avoid indexing fields that are rarely queried or that match most of the collection. Revisit the set periodically with usage statistics rather than adding indexes reactively.

Older versions offered a background: true option for building indexes without blocking all reads and writes on the collection. Modern versions perform index builds with an optimized process that allows reads and writes during most of the build, making the option largely obsolete. Builds still consume substantial I/O and CPU, and they take a lock briefly at the start and end. On replica sets the build happens on all members, so plan for the aggregate resource impact.

An index can serve a sort if the sort uses the same key pattern or its exact inverse. An index on {a: 1, b: 1} supports sorting by a ascending and b ascending, and also both descending, but not a ascending with b descending. Mixed directions require an index with matching mixed directions. This is a frequent cause of unexpected blocking sorts.

db.events.createIndex({ user: 1, ts: -1 })   // supports sort { user: 1, ts: -1 }

Use dot notation to index a nested field, for example {"items.sku": 1}, and the resulting multikey index stores one entry per array element. A query on items.sku can use it, but a query that requires several fields to match the same array element may need $elemMatch to be correct semantically. Compound indexes with more than one array path are rejected. Consider whether the array is bounded before indexing it.

db.orders.createIndex({ "items.sku": 1, "items.qty": 1 })  // error if both are arrays

$expr compares fields or uses aggregation expressions, and many such predicates cannot be converted into index bounds. Queries that mix $expr with regular predicates may still use an index for the regular part and filter the rest. Where possible, rewrite the condition as a plain field predicate so the planner can bound the scan. Check explain output for keys examined relative to documents returned.

Only prefix-anchored, case-sensitive regular expressions can use an index efficiently, because they define a bounded range of keys. Unanchored patterns such as /foo/ force a scan of every index key or a collection scan. Case-insensitive patterns with the i flag also usually defeat index bounds unless a case-insensitive collation index exists. For flexible search, use a text index or Atlas Search instead.

db.users.find({ name: /^Ann/ })   // index-friendly
db.users.find({ name: /nn/ })     // not index-friendly

Create the index with a collation whose strength ignores case, then run queries with the identical collation so the planner considers the index. The collation must match exactly, including locale, or the index will not be used. This is the standard way to support case-insensitive uniqueness or search on a field. Remember that collation also affects sort order and comparison semantics.

db.users.createIndex({ email: 1 }, { collation: { locale: "en", strength: 2 }, unique: true })

distinct on an indexed field can scan the index directly, avoiding document fetches. countDocuments with a filter can use an index to count matching keys, while estimatedDocumentCount reads collection metadata and ignores filters entirely. An unindexed distinct or filtered count degrades to a collection scan. Verify with explain, since the planner may not always choose the index you expect.

$lookup performs an index-backed lookup on the foreign collection only if the foreign field used in the localField/foreignField pair is indexed. Without that index, each input document triggers a collection scan of the foreign collection, which is catastrophic at scale. Put the $lookup as late as possible and reduce the input set with an early $match. For sharded clusters, the join may require data movement between shards.

db.orders.aggregate([
  { $match: { status: "paid" } },
  { $lookup: { from: "customers", localField: "customerId",
               foreignField: "_id", as: "customer" } }
])

The optimizer can push a leading $match ahead of other stages so it uses indexes, but a $match after $project or $group cannot. Placing filtering stages early reduces the number of documents flowing through later stages and cuts memory use. Use $match plus $sort at the front whenever they correspond to an index. Confirm with explain that the pipeline reports an IXSCAN rather than a collection scan.

Setting slowms makes the server log operations exceeding that threshold, and enabling the profiler records them in system.profile. Level one captures slow operations only, while level two captures everything and adds noticeable overhead. Start with logging rather than full profiling in production. Query the profile collection by duration and examined-to-returned ratio to prioritize fixes.

db.setProfilingLevel(1, { slowms: 100 })
db.system.profile.find().sort({ millis: -1 }).limit(5)

currentOp shows in-progress operations with their namespace, duration, plan summary, and whether they are waiting on a lock. It is the fastest way to identify a runaway aggregation or an unindexed query saturating the server. killOp terminates a specific operation by its opid. Killing operations is disruptive, so confirm the target and understand downstream effects first.

db.currentOp({ secs_running: { $gt: 30 }, active: true })

dbStats reports data size, storage size, index size, and document counts per database. collStats adds per-collection detail including index sizes, average object size, and wiredTiger block-manager statistics. The ratio of storage size to data size indicates compression effectiveness, and a fast-growing index size hints at an unintended multikey index. Watch these over time to catch growth that outpaces data volume.

By default the cache is sized at roughly half of available RAM minus one gigabyte, and it holds both document and index pages. When dirty pages exceed the eviction threshold, WiredTiger background threads write them to disk, and if pressure continues, application threads perform eviction themselves, raising latency. Persistent cache pressure usually means the working set does not fit in memory. Adding RAM or reducing the working set through better indexes and projections is more effective than raising the cache size blindly.

WiredTiger writes a consistent snapshot of data files every checkpoint interval, typically sixty seconds, and records changes in a journal between checkpoints. With journaling enabled, a write acknowledged with j: true can be recovered even if the process crashes before the next checkpoint. Checkpoints bound recovery time, while the journal bounds data loss. Both are separate from replication durability, which depends on write concern.

Not with read preference alone, because a secondary may not have applied the write yet. Causal consistency solves this by passing the client's operation time as afterClusterTime, causing the secondary to wait until it has caught up. Without a session and causal consistency, secondary reads can return stale data even seconds after a write. This is why read-your-writes flows should either read the primary or use causal sessions.

With w: 0 the client does not wait for any acknowledgement, so errors such as duplicate key violations are not reported at all. Writes are fire-and-forget, with no durability guarantee and no ordering feedback. It is only appropriate for loss-tolerant telemetry or logging pipelines. Most applications should use at least w: 1 plus journaling where durability matters.

Retryable writes require a replica set or sharded cluster, a driver that supports them, and a write concern that acknowledges the write. They also require that the operation can be associated with a session and transaction number so the server can detect duplicates. Multi-document operations are retried as a unit only if they are supported retryable write types. Operations like insertMany with unordered writes have specific retry semantics that should be checked.

Writes acknowledged by a majority are past the majority commit point and cannot be rolled back by a subsequent election. Writes acknowledged only by the old primary but not replicated can be truncated during rollback when a new primary takes over. Applications that require durability must therefore combine majority write concern with majority read concern. Rollback files on the demoted member preserve the truncated writes for inspection.

A new member performs an initial sync: it clones existing databases, then applies oplog entries generated during the clone, and finally moves to steady-state replication. During the sync it is in STARTUP2 and cannot serve reads. Cloning is I/O intensive on both the source and the new member, so add members during quiet periods. If the oplog rolls over before the clone completes, the sync restarts.

The oplog must hold enough write history to cover the longest plausible outage of a secondary, so size it from peak write throughput times the acceptable downtime window. Undersized oplogs cause secondaries to fall off and require a full resync. You can resize with replSetResizeOplog in modern versions, which avoids a full resync. Monitor oplog window in hours rather than gigabytes.

db.adminCommand({ replSetResizeOplog: 1, size: 51200 })

Secondary reads reflect whatever the member has applied at the moment of the read, which may lag the primary by milliseconds or much more. There is no ordering guarantee across multiple secondaries, so two reads can see different states. Data on a secondary can also be rolled back if it was never majority-committed. Use secondary reads for analytics and reporting where slight staleness is acceptable.

Failover behaviour depends on election timing, driver retry logic, and application error handling, none of which are exercised in normal operation. Test by stepping down the primary or stopping a member in a staging environment that mirrors production topology. Assert that writes retry successfully and that latency stays within the service level objective. Record the observed failover window so the team knows the real recovery time.

Shard when a single replica set can no longer absorb write throughput, when the working set exceeds what one server's memory and disk can serve at acceptable latency, or when data residency requires regional placement. Sharding adds operational complexity, so vertical scaling and query optimization should be exhausted first. The decision should be driven by measured bottlenecks, not by data volume alone. Plan the shard key before migrating, since it constrains future query patterns.

mongos caches the cluster's chunk-to-shard mapping from the config servers and uses the shard key in the query to compute the target range. If the query includes the shard key, it sends the operation to the owning shard or shards. Otherwise it broadcasts to all shards and merges the results, potentially with a sort or limit applied at the router. Stale routing metadata is refreshed automatically when a shard reports an error.

You can refine a shard key by adding a suffix field with refineCollectionShardKey, which changes the key pattern without a full rewrite. Fully changing the shard key historically required rewriting the collection, and modern versions support resharding as an online operation with significant resource cost. Both options require careful planning and a maintenance window. Choosing well at creation time remains far cheaper than changing later.

db.adminCommand({ refineCollectionShardKey: "shop.orders", key: { customerId: 1, _id: 1 } })

A unique index on a sharded collection must include the shard key as a prefix of the index key, because uniqueness can only be enforced within a shard. The default _id index is the exception, since MongoDB assumes ids are unique. Enforcing global uniqueness on another field requires routing that field into the shard key or handling it in the application. This constraint frequently influences shard key selection.

A $lookup whose foreign collection is sharded and not joined on the shard key requires the router to broadcast the lookup or move data, which is expensive. Joins where the foreign field equals the shard key can be targeted to a single shard. On large clusters, consider embedding the needed fields or pre-joining into a read-optimized collection. Measure with explain before assuming a join is cheap.

moveChunk migrates a chunk from one shard to another and is normally the balancer's job. Manual moves are useful for pre-splitting and pre-placing data before a bulk load, or for evacuating a shard during maintenance. Misuse can create imbalance that the balancer immediately tries to undo. Coordinate with the balancer state to avoid conflicting migrations.

sh.moveChunk("shop.orders", { customerId: "c123" }, "shard02")

A jumbo chunk exceeds the maximum chunk size and cannot be split, usually because too many documents share the same shard key value. The balancer cannot move jumbo chunks, so data can become permanently imbalanced. The underlying cause is a low-cardinality or high-frequency shard key. Fixing it typically requires refining the shard key or resharding the collection.

Consistent cluster backups require coordinating all shards and the config servers, typically by stopping the balancer and using a filesystem or coordinated snapshot approach across components. mongodump against mongos works for logical backups but does not produce a point-in-time consistent snapshot of the whole cluster. Atlas and Ops Manager orchestrate this coordination for you. Always test restores, since partial cluster restores are error-prone.

A compound shard key combines multiple fields, for example a tenant id plus a creation timestamp. It can add cardinality and let the leading field provide locality while the trailing field orders data within it. It also supports more query shapes because the shard key prefix can be targeted. The trade-off is a larger key in every index and more complex routing analysis.

Hashed indexes support only equality matches, because the hash destroys ordering, so range queries cannot use them. They also cannot enforce uniqueness except on the hashed field as a whole and cannot be multikey. Floating-point values are truncated before hashing, which can cause collisions for values differing only in low-order bits. Use hashed indexes for even write distribution and point lookups, not for range access.

$jsonSchema validators enforce field types, required properties, and value constraints on inserts and updates. They can run in strict mode, rejecting invalid documents, or moderate mode, which only warns. Validators apply to new writes, so existing documents must be cleaned up separately. They add a small write cost but prevent a class of data quality bugs.

db.createCollection("users", {
  validator: { $jsonSchema: {
    required: ["email"],
    properties: { email: { bsonType: "string" } }
  } },
  validationLevel: "strict"
})

If the index key order matches the equality predicates followed by the sort keys, the scan produces documents in the requested order while applying bounds on the leading fields. The server then does not need a separate sort stage. The plan shows a single IXSCAN with no SORT. If the sort keys are not a suffix of the key pattern after the equality fields, the sort becomes blocking.

A leading field with few distinct values gives the scan little to discard, so the index may examine far more keys than the query returns. It can still be useful if the query always filters on that field and sorts on the next, since the order is what matters. Otherwise, put the more selective field first if the sort allows it. Judge the outcome from totalKeysExamined relative to nReturned.

Every insert, update, and delete must maintain all affected index entries, which costs CPU, cache space, and journal traffic. Updates to indexed fields also require deleting old keys and inserting new ones, doubling that work. An over-indexed collection can spend more time on index maintenance than on the write itself. This is why unused index audits are a routine maintenance task.

On a replica set, an index build started on the primary is also started on secondaries so that the index exists everywhere. Secondaries build concurrently with replication, and the extra I/O can increase replication lag. In older versions a secondary would restart the build if it received a conflicting write, which made lag dangerous. Watch lag during large builds and consider a rolling approach.

Each update moves the index entry, which adds write amplification and can create contention on hot keys. If the field changes often but is rarely queried, the index is a net cost. For monotonic fields, index churn also concentrates on the rightmost pages. Prefer indexing stable fields that appear in query predicates.

Every collection has a unique _id index created automatically, and it cannot be dropped. The _id value is immutable, so an update attempting to change it fails. It is also the only unique index that does not need to include the shard key in a sharded collection. ObjectId values are roughly time-ordered, which gives mostly ascending inserts.

Identify candidates where one index is a strict prefix of another or where usage statistics show no accesses. Hide the candidate first and observe performance for a full business cycle before dropping it. Confirm no hinted queries reference it, since those would fail immediately. Drop during low-traffic periods and keep the index definition recorded so it can be recreated.

db.orders.hideIndex({ status: 1 })
// after observation window
db.orders.dropIndex({ status: 1 })

A clustered collection stores documents physically ordered by the _id key rather than by a separate index, so range scans on _id read contiguous storage. This removes the need for a separate _id index and improves locality for ordered access. Inserts must respect the clustering order to benefit, and some features behave differently than in standard collections. It suits workloads with naturally ordered keys such as time-based identifiers.

sort({ $natural: 1 }) returns documents in physical storage order and can be served by a collection scan without a sort stage. $natural: -1 returns them in reverse. This is useful for capped collections or for peeking at recently inserted documents, but it does not reflect logical ordering guarantees. Relying on natural order for business logic is fragile because storage order can change after moves or rebuilds.

A sparse index omits documents where the field is missing, so a query for {field: null} or {field: {$exists: false}} cannot be answered from it and will not use the index. The planner must scan documents to find those cases. A partial index with an explicit filter, or a non-sparse index, handles negation correctly. Confirm with explain before assuming a sparse index covers all filter shapes.

MongoDB can execute each $or branch as a separate index scan and union the results, which is a form of index union rather than intersection. Each branch must be independently indexable for this to work. If any branch cannot use an index, the whole query may fall back to a collection scan. Duplicate elimination adds overhead, so $in is often better when the branches target the same field.

$in generates one bound per value, so the scan becomes several point lookups within a single index scan. It is efficient when the list is small and selective, but a large list of values can approach a full index scan. Values are deduplicated and sorted to make the bounds contiguous where possible. Very long $in lists are often a sign that the query should be restructured.

{field: null} matches both documents where the field is null and where it is missing, so a non-sparse index can serve it. {field: {$exists: false}} is a separate predicate that a sparse index cannot serve, because missing values were not indexed. The two look similar but have different index implications and different results on arrays. Always test with real data shapes.

For a single-field index, ascending and descending are equivalent for lookups and for sorting, since the engine can walk the index in either direction. Direction only becomes meaningful in compound indexes, where the relative order of fields determines which sorts are supported. This is why {a: 1} and {a: -1} are redundant with each other. Do not create both.

The build aborts and the partially built index is removed, leaving the collection unchanged. Common causes are duplicate key violations on a unique index, out-of-memory conditions, or a primary stepping down. The error message identifies the failing cause, which should be fixed before retrying. On a replica set, secondaries also abort the build and replication continues without the index.

Blocking sorts and some aggregation stages have an internal memory limit, historically 100 MB, and exceeding it causes an error rather than silent disk spilling. Aggregations can opt into disk use with allowDiskUse: true, while find sorts cannot and must be fixed with an index. A common production error is a sort exceeding the limit after a data growth spurt. The durable fix is an index that provides the order.

db.orders.aggregate(pipeline, { allowDiskUse: true })

Filter and project early so fewer fields and documents flow through the pipeline, and place $match and $sort where an index can serve them. Use $limit before expensive stages when semantics allow, and avoid unbounded $group over the whole collection. Enable allowDiskUse only as a safety valve, not as the primary strategy. Review the explain output for the number of documents entering each stage.

The config server replica set stores cluster metadata and elects its own primary, which is the only member that accepts metadata writes. The balancer runs on that primary and updates chunk placement through it. If the config server set has no primary, metadata changes stall and the cluster can become read-only for routing changes. This is why config servers must be deployed as a properly sized replica set.

Members exchange heartbeats every two seconds by default, and a member that misses heartbeats for the election timeout considers the primary unavailable. The default election timeout is ten seconds, after which an eligible member calls for an election. Lower timeouts detect failures faster but risk unnecessary elections during transient network blips. Tuning them requires care, especially across regions.

A rollback reverts a small number of oplog entries that the demoted primary had applied but that never reached a majority, and the member then rejoins as a secondary. A resync discards the member's data entirely and re-copies it from another member, which happens when the member has fallen off the oplog. Rollbacks are usually fast and expected; resyncs are slow and disruptive. Both indicate that replication durability was not sufficient for that write path.

Add the member with priority zero so it cannot become primary before it is caught up, and let initial sync complete during a low-traffic window. Monitor the sync progress and the source member's I/O, since cloning competes with normal traffic. Once the member is in SECONDARY state and lag is near zero, raise its priority if it should be electable. Verify that the oplog window comfortably covers the sync duration.

The safest logical approach is a filesystem or volume snapshot taken at the same instant across members, or mongodump with --oplog to capture a point-in-time window. Atlas and Ops Manager coordinate consistent snapshots automatically. Restores must replay the oplog to reach a consistent point. Always validate by restoring into a test environment and running application checks.

Point-in-time restore needs a full backup plus a continuous record of oplog entries covering the target time. The restore process applies the base backup and then replays oplog entries up to the chosen timestamp. This is why oplog retention directly determines how far back you can restore precisely. Without sufficient oplog, only full-backup granularity is possible.

Reads at majority may wait for the queried member to catch up to the majority commit point, which adds latency under replication lag. The guarantee is durability rather than freshness, but the wait can be noticeable on a lagging secondary. On a healthy set with low lag the overhead is small. Measure before switching all reads to majority, and reserve it for paths that need the guarantee.

w: "majority" controls how many members must acknowledge the write, while j: true controls whether the primary's on-disk journal must record it before acknowledging. They address different failure modes: member loss versus process crash. Combining them covers both, at the cost of additional latency. Neither protects against application-level errors or bad data.

A writeConcernError means the write was applied on the primary but the requested acknowledgement level was not reached, for example because a secondary was unavailable. The data may still become durable later, so the correct response is usually to retry the write concern check rather than to re-issue the write blindly. Drivers expose this as a distinct error type from duplicate key or validation errors. Treating it as a hard failure can produce duplicate writes.

Drivers implement server discovery and monitoring: they poll each known server with hello commands and react to state changes such as a new primary. When the topology changes, the driver updates its view and reroutes operations. This is why failover is mostly transparent to application code. Stale topology views during a fast election are the usual cause of brief write errors.

maxStalenessSeconds excludes secondaries whose estimated replication lag exceeds a threshold, so reads do not hit badly lagging members. It works with secondary and nearest read preferences and is ignored for primary reads. This is a simple way to bound staleness without implementing causal consistency. Set it generously enough that normal lag spikes do not shrink the eligible member set to nothing.

Members can be tagged, for example by region or tier, and a read preference tag set restricts eligible members to those matching the tags. Tags are tried in order, so the first set with an available member wins. This supports routing analytics to dedicated hardware or reads to a local region. Tags also affect elections when used with replSetTagConfig.

db.orders.find().readPref("secondary", [{ region: "eu" }, {}])

Use a dedicated secondary with tags and a read preference that targets it, or route analytics through a separate connection with secondaryPreferred. Keep long-running aggregations off the primary so they do not hold cache and CPU needed by writes. Consider a delayed or hidden member for heavy reporting. Also cap aggregation memory and runtime so a single report cannot starve the member.

Each shard must produce its sorted stream, and mongos performs a merge sort across shards, which is efficient for small limits. Without a limit, the router may need to buffer large result sets, increasing memory use and latency. Sorts on the shard key can sometimes be satisfied with less coordination. Design shard keys with your dominant sort patterns in mind.

Each database has a primary shard that stores any collections not yet sharded in that database. It is chosen at database creation and can be changed with movePrimary. Unsharded collections in a sharded cluster therefore live entirely on one shard, which can create imbalance. This is a common reason a cluster looks uneven despite a good shard key.

The balancer compares per-shard chunk counts against configured thresholds and migrates chunks from over-loaded to under-loaded shards within a balancing window. It respects zone constraints and skips jumbo chunks. Migration throughput is throttled to limit impact. Persistent imbalance usually points to a low-cardinality shard key or to zone configuration that forbids the needed move.

Pre-splitting creates empty chunk ranges across shards before a bulk load so that writes are distributed from the start. Without it, all initial writes target the single chunk covering the key range, creating a hotspot until the balancer catches up. Pre-splitting is most useful for monotonic keys and large imports. Modern versions reduce the need for manual splitting, but the concept still applies to bulk ingestion planning.

sh.status() shows chunk counts per shard along with balancer state and any jumbo chunks. Aggregating config.chunks gives the same data programmatically for dashboards. Watch for a large skew in chunk counts and for chunks that never move. Imbalance with even chunk counts can still mean uneven data size, since chunks are not equal in bytes.

db.getSiblingDB("config").chunks.aggregate([
  { $group: { _id: "$shard", n: { $sum: 1 } } }
])

Targeted queries to the unavailable shard fail, while scatter-gather queries fail if any shard in the broadcast is unreachable, unless the read preference allows partial results. Writes that need the unavailable shard also fail. A shard is itself a replica set, so a single member loss should not cause this; a full shard outage is a serious event. Design critical paths to be targeted so a single shard failure has limited blast radius.

Cross-shard transactions use two-phase commit coordinated by mongos, with each participating shard running a local transaction. They provide snapshot isolation but incur higher latency and more contention than single-shard transactions. Keep cross-shard transactions short and rare, and prefer shard keys that keep related data together. There are limits on transaction duration and on the number of documents involved in practice.

Use a unique key document holding an owner and an expiry, and acquire it with an atomic findOneAndUpdate that matches either an expired lock or the same owner. The atomic single-document update guarantees only one caller wins. Always include a lease expiry so a crashed holder cannot block forever. Verify ownership before releasing, so a slow worker does not delete someone else's lock.

db.locks.findOneAndUpdate(
  { _id: "job:42", $or: [ { expiresAt: { $lt: new Date() } }, { owner: me } ] },
  { $set: { owner: me, expiresAt: new Date(Date.now() + 30000) } },
  { upsert: true, returnDocument: "after" }
)

findOneAndUpdate with $inc on a counters document returns the new value atomically, giving gap-free-enough sequences per document. A single counter document becomes a hot spot under very high concurrency, so shard the counter by prefix or allocate ranges per process. For high throughput, allocate blocks of ids and hand them out in memory. Never read then write the counter in two operations.

db.counters.findOneAndUpdate(
  { _id: "invoice" }, { $inc: { seq: 1 } },
  { upsert: true, returnDocument: "after" }
)

Upsert is preferable when you want a single idempotent write that both creates and updates, for example recording a user's last login. Insert with duplicate retry is preferable when the operation must fail loudly on conflict, such as claiming a unique username. Upsert still needs a unique index to be safe under concurrency. Choose based on whether an existing document should be updated or treated as an error.

Store arrays of ids on one or both sides when the cardinality is bounded and reads usually start from one side. When the relationship has its own attributes or grows unbounded, use a join collection with one document per pair and a compound unique index. Index both directions you query, for example {a: 1, b: 1} and {b: 1, a: 1}. Avoid bidirectional arrays that must be kept in sync across two documents.

The outlier pattern handles the rare documents that do not fit the dominant schema by moving them to a separate collection or flagging them for special handling. For example, a celebrity user with millions of followers is stored differently from ordinary users. This keeps the common case fast and prevents a few documents from dominating index and document size. Application code must know to check both places for those cases.

The archive pattern moves cold documents to a separate collection, or to cheaper storage, while keeping hot data small and fast. Typically a background job moves documents older than a threshold, either on the same cluster or to object storage. This shrinks the working set and index size and speeds up queries on current data. Queries that need history then target the archive collection explicitly.

Write both the old and new field shapes for a transition period, read preferring the new field with a fallback to the old one, then backfill and finally remove the old field. This keeps every deployable version of the application working during a rolling release. Each phase must be independently reversible. Remove the compatibility code only after the backfill is verified complete.

skip(n) requires the server to walk and discard n documents before returning results, so cost grows linearly with page depth. Range pagination on an indexed, ordered field avoids that by starting after the last seen value. Keyset pagination also stays stable when documents are inserted, unlike offsets. Use skip only for shallow pages or small result sets.

db.events.find({ ts: { $lt: lastSeen } }).sort({ ts: -1 }).limit(20)

Options include a parent reference for simple trees, an array of ancestor ids for fast subtree queries, and a materialized path string for prefix matching. Each has different write costs: ancestor arrays require updating descendants on moves, while parent references move cheaply but query subtrees recursively. Choose based on whether reads or reorganizations dominate. $graphLookup can traverse parent references when the depth is modest.

Soft deletes keep documents with a flag such as deletedAt, so every query must filter them out and indexes must include the flag to stay selective. A partial index with {deletedAt: {$exists: false}} keeps deleted rows out of the hot index entirely. Over time the deleted population can dominate storage and scans. Plan an archival or purge process rather than letting deleted data accumulate indefinitely.

Split the counter into sharded sub-counters, for example per shard of a hash or per time bucket, and sum them on read. This spreads increments across many documents instead of serializing on one. Alternatively, pre-aggregate in memory and flush periodically when exact real-time values are not required. Any single-document counter will eventually become a write bottleneck at high concurrency.

Store tags as an array of strings and index it as a multikey index, which supports equality and $all queries efficiently. For facet counts, a separate aggregation over the same field or a precomputed facet document avoids repeated full scans. The classic attribute pattern is better when tags carry values rather than being plain labels. Watch index entry count when arrays are large.

GridFS splits files larger than the 16 MB document limit into chunks stored in two collections, with metadata in the files collection. Use it when files must live in the same database and be replicated and backed up with it. For most web applications, object storage plus a URL in the document is cheaper and faster to serve. GridFS adds chunk management overhead and is not a content delivery mechanism.

Use a unique partial index whose filter defines the subset, for example only active records or only non-null emails. Uniqueness is then enforced only among documents matching the filter. This avoids the null collision problem of a plain unique index and keeps the index small. Ensure the filter expression matches exactly how the application marks the subset.

db.members.createIndex(
  { groupId: 1, email: 1 },
  { unique: true, partialFilterExpression: { active: true } }
)

Change streams provide a stable, documented event format with resume tokens, filtering, and support for update lookups, and they survive replica set changes. Tailing the oplog directly is unsupported, tied to internal formats, and breaks on version changes. Change streams are the right tool for audit pipelines and downstream integration. Retention is still bounded by oplog size, so consumers must keep up or persist their own history.

Use a text index with $text and $meta: "textScore" for ranking, optionally combined with field weights. For prefix or fuzzy behaviour, a normalized token array plus a multikey index can approximate it. Neither matches Lucene analyzers, so expectations should be set accordingly. If search quality is a product requirement, Atlas Search or an external engine is the better answer.

db.articles.find(
  { $text: { $search: "replica set" } },
  { score: { $meta: "textScore" } }
).sort({ score: { $meta: "textScore" } })

Put the tenant id first in every compound index so each tenant's queries scan only its own key range. This also makes tenant-scoped uniqueness enforceable with a compound unique index. Keep queries always tenant-scoped to avoid cross-tenant scans and accidental data leaks. Index bloat from many small tenants is usually acceptable compared with the isolation benefit.

Including the tenant id in the shard key keeps a tenant's data on few shards, which makes most queries targeted. A pure tenant key can create hotspots for very large tenants, so a compound key such as tenant id plus time or a hashed suffix helps. Very large tenants may need their own shard or zone. The goal is to balance targeting against even distribution.

The working set is the portion of data and indexes actively accessed, approximated by combining hot collection size with index size and the access pattern over a representative period. collStats plus query analysis gives the raw sizes, while cache eviction metrics reveal whether the working set fits. A high page-fault rate with low cache utilization suggests it does not. Provision RAM for the working set, not the whole dataset.

Start with currentOp and the profiler to find the operations consuming CPU, then check whether their plans are collection scans or in-memory sorts. High CPU from many small queries usually points to missing indexes or excessive round trips; from a few operations, to unindexed aggregations. Also rule out compression, checkpointing, and replication apply overhead. Fix the top query shapes rather than adding hardware first.

Look at WiredTiger block-manager statistics for pages read versus cached, along with eviction pressure and checkpoint durations. Rising disk read latency with a stable query mix indicates the working set no longer fits in cache. Bulk index builds and compaction also cause I/O spikes. Confirm with operating system metrics before concluding the database is at fault.

Alert on replication lag, oplog window, member state changes, election count, connections versus limit, cache dirty percentage, and slow query counts. Also track disk space and queue depths, since exhaustion causes outages rather than slowdowns. Alert on trends and sustained breaches, not single samples. Include a check that exactly one primary exists.

All writes for the current interval land in the newest bucket for each series, so bucket count and per-bucket write rate determine contention. Increasing series cardinality through the metaField spreads writes, as does choosing an appropriate granularity. Very high-frequency single-series ingestion is the worst case. Consider bucketing by device and time window rather than one series per metric.

Granularity sets the expected time span per bucket, which controls how many measurements are packed together. Coarser granularity means fewer, larger buckets with better compression but slower per-measurement access and updates. Finer granularity increases bucket count and metadata overhead. Choose based on ingestion rate and typical query window, or set a custom bucketing parameter instead.

Aggregate raw measurements into coarser intervals with $densify, $group, and $merge into a summary collection, then expire the raw data with a TTL index. This keeps long-range dashboards fast while preserving recent detail. Run downsampling on a schedule and make it idempotent by merging on the interval and series key. Store the aggregate definitions so the rollups can be recomputed.

db.readings.aggregate([
  { $group: { _id: { s: "$sensor", h: { $dateTrunc: { date: "$ts", unit: "hour" } } },
              avg: { $avg: "$v" } } },
  { $merge: { into: "readings_hourly", whenMatched: "replace" } }
])

The metaField identifies the series, for example a sensor id or device, and MongoDB groups buckets by that value. It also enables efficient filtering and sorting per series and is the field most worth indexing. Fields in the metaField are stored once per bucket rather than per measurement, which improves compression. Changing it after creation requires creating a new collection.

Filter on the time field and the metaField so the engine can prune buckets, and avoid unbounded time ranges on large collections. Index the metaField and use $group with time-based accumulators for rollups. Project only needed fields, since measurements are nested inside buckets. Avoid per-document updates on time series data, which are restricted and expensive.

$search performs lexical matching with analyzers and returns relevance scores based on term frequency and field weights. $vectorSearch compares embeddings and returns results by semantic similarity, so it can match paraphrases with no shared words. Both must be the first stage in a pipeline and both support filtering and scoring metadata. Hybrid pipelines combine them to balance precision and recall.

Use the filter option inside $vectorSearch so filtering happens during the search rather than after it, which keeps the requested result count filled. Filtering afterwards can silently return fewer results than limit. Pre-filtering on highly selective fields also reduces the candidate set substantially. Verify that the filtered fields are declared in the vector index definition where required.

{ $vectorSearch: { index: "vec", path: "embedding", queryVector: q,
  filter: { tenantId: "t1", status: "active" }, limit: 10, numCandidates: 150 } }

Atlas Search indexes are managed by the service and replicate changes from the collection automatically, so no application code is needed. There is a small replication delay, so searches may briefly lag writes. Index definitions changes trigger a rebuild, which can take time on large collections. Applications should tolerate eventual consistency in search results.

During failover, checked-out connections to the old primary fail and the pool re-establishes connections to the new primary. If maxPoolSize is large and many application instances reconnect at once, the new primary can be overwhelmed. Smaller pools per instance and a sane connection timeout reduce the spike. Monitor connection counts immediately after a failover to confirm the pool behaves as expected.

A connection storm happens when many clients reconnect simultaneously, exhausting file descriptors or overwhelming the newly elected primary. Drivers mitigate this with pool limits and reconnect backoff, but aggressive retry logic in application code can defeat it. Add jitter to retries and cap concurrency at the application layer. Pre-warming and connection limits per host keep recovery predictable.

TransientTransactionError means the transaction did not commit and can be retried from the start, typically due to a write conflict or a network issue. UnknownTransactionCommitResult means the commit may or may not have been applied, so only commitTransaction should be retried, not the whole body. Confusing the two can duplicate work. The standard retry loop handles each label separately.

Include a version field in the update filter and increment it in the same operation, then treat a zero matched count as a conflict. This makes the read-modify-write cycle detect concurrent modification without locking. The application must decide whether to retry, merge, or fail the request. This pattern works well for edit forms and state machines.

const r = await db.docs.updateOne(
  { _id: id, version: expected },
  { $set: { body: next }, $inc: { version: 1 } }
);
if (r.matchedCount === 0) throw new Error("conflict");

Shorten the time a document is held in a transaction, avoid long-running transactions that touch hot documents, and reduce the number of concurrent writers per document by sharding the value. Using $inc in a single atomic update rather than a transaction also avoids conflicts entirely. Retry transient conflicts with backoff as a safety net. If conflicts are constant, the data model needs to change.

The driver attaches a logical session id (lsid) and a monotonically increasing transaction number to each retryable write. The server records the pair, so a retried write with the same identifiers is recognized as a duplicate and not applied twice. This is what makes automatic retry safe for inserts and single-document updates. Sessions expire, so retries must occur within the session timeout window.

Persist the resume token together with the effect of the event in the same transaction, or key the downstream write on a stable identifier from the event so reapplying it is harmless. On restart the consumer resumes from the stored token, which may replay the last event. If the effect is keyed idempotently, replay causes no duplication. Never store the token in memory only, or a crash will replay or skip events.

Store amounts as integers in the smallest currency unit, or as Decimal128 when fractional precision is required, and keep the currency code alongside. Binary floating point introduces rounding errors that are unacceptable for money. Doing arithmetic in aggregation with Decimal128 is supported, but application-level rounding rules should still be explicit. Never store amounts as plain doubles if exactness matters.

{ amount: Decimal128("19.99"), currency: "USD" }
{ amountMinor: 1999, currency: "USD" }

A collection per tenant gives strong isolation and easy per-tenant deletion, but thousands of collections increase metadata, memory, and backup overhead. A shared collection with a tenant id scales better and simplifies operations at the cost of requiring disciplined query scoping. Most SaaS systems use a shared collection plus compound indexes and optional per-tenant sharding for large customers. Revisit the choice when tenant count or size crosses a threshold.

Store the order with embedded line items that capture the product id, name, and unit price at purchase time, so later catalog changes do not alter historical orders. Keep a reference to the product for lookups but treat the embedded values as authoritative for the order. This is deliberate denormalization for immutability and read speed. Totals can be computed at write time and stored for reporting.

{ _id: 1, items: [{ productId: 7, name: "Desk", unitPrice: 199.00, qty: 2 }],
  total: 398.00, currency: "USD", placedAt: ISODate() }

Each driver maintains an internal pool of sockets per server in the topology, opening connections on demand up to a configurable maximum and closing idle ones after a timeout. Creating a new client per request forces TCP, TLS, and authentication handshakes each time, which adds latency and exhausts server-side connection limits. Applications should create a single long-lived client at startup and share it across requests, threads, or worker processes. Pool size should be tuned so total connections across all application instances stay below the server's connection budget.

const client = new MongoClient(uri, { maxPoolSize: 50, minPoolSize: 5 });
await client.connect();

New checkout requests wait in a queue until a connection becomes available or maxWaitTimeMS expires. If the wait expires, the driver raises a pool-cleared or timeout error rather than blocking indefinitely. Persistent saturation usually indicates slow queries holding connections or a pool sized too small for the workload concurrency. Monitoring pool wait time is the quickest way to distinguish a sizing problem from a database performance problem.

Choose the driver that matches the application runtime and its concurrency model, because all official drivers implement the same specification and wire protocol. Node.js suits event-driven services, Java and .NET suit managed enterprise stacks with mature connection and thread pools, Python favors scripting and data workloads, and Go suits low-latency services with static binaries. Evaluate team familiarity, ecosystem library support, and observability hooks rather than raw benchmark differences. Feature parity is high, so driver choice rarely limits access to server capabilities.

Asynchronous drivers return promises or futures and never block the calling thread, which fits event loops and high-concurrency services. Synchronous drivers block until the operation completes, which is simpler to read but ties up a thread per in-flight operation. Python offers both PyMongo and Motor/AsyncIOMotor, and the Java driver offers sync, async, and reactive variants. Mixing blocking calls inside an event loop is a common cause of severe latency spikes.

Retryable writes cover a defined set of single-document write operations that fail with retryable error codes such as network errors or a primary stepping down. Drivers retry once by default when retryWrites is enabled, using a session identifier so the server can detect duplicates. Retryable reads handle a similar narrow set of read failures. Application-level errors like duplicate key violations or validation failures are never retried automatically because the outcome is deterministic.

Enable retryWrites so the server deduplicates the retried attempt using the session and transaction number. Beyond that, make the operation idempotent by using a natural or client-generated unique key so a manual retry cannot insert twice. Retry only errors classified as transient, and use bounded exponential backoff with jitter. Log the retry attempt with a correlation identifier so incident analysis can reconstruct what happened.

await orders.updateOne(
  { _id: clientGeneratedId },
  { $setOnInsert: { status: "new", createdAt: new Date() } },
  { upsert: true }
);

A write is idempotent when repeating it leaves the database in the same state as a single execution. This is usually achieved with upserts keyed on a deterministic identifier, or with conditional updates that include the expected current state. Using $setOnInsert rather than $set inside an upsert prevents overwriting data on a repeat attempt. Idempotency is essential whenever message consumers, retries, or at-least-once delivery are involved.

Include the current version in the query filter and increment it in the update, so the write only matches if nobody changed the document in between. If modifiedCount is zero, another writer won the race and the caller should re-read and decide whether to retry. This avoids long-lived locks and works well for low-contention documents. Keep the retry loop bounded and surface a conflict error to the caller after a few attempts.

const r = await docs.updateOne(
  { _id: id, version: 7 },
  { $set: { title: "new" }, $inc: { version: 1 } }
);
if (r.modifiedCount === 0) throw new Error("conflict");

skip forces the server to scan and discard all preceding documents, so cost grows linearly with page depth and late pages become slow. Concurrent inserts or deletes also shift the offset, producing duplicated or missing rows across pages. Range or cursor pagination anchors on the last seen sort key instead of counting documents. This keeps every page cost roughly constant and gives stable results.

Sort by _id, return a page, then filter the next query with _id greater than the last returned value. Because _id is unique and indexed by default, each page is an index range scan with predictable cost. The client passes the last _id as an opaque cursor token. This pattern is simple and stable but only supports ordering by _id.

db.events.find({ _id: { $gt: lastId } }).sort({ _id: 1 }).limit(50);

Use a compound cursor of the sort field plus a tiebreaker, typically createdAt and _id, and filter with a compound range predicate. The index must match the sort order exactly for the plan to avoid an in-memory sort. Encode both values in the cursor token so the next request can resume precisely. This yields stable ordering even when many documents share the same timestamp.

db.posts.find({
  $or: [
    { createdAt: { $lt: lastTs } },
    { createdAt: lastTs, _id: { $lt: lastId } }
  ]
}).sort({ createdAt: -1, _id: -1 }).limit(20);

If the sort key is not unique, documents with equal values have no defined relative order, and the server may return them differently between executions. Adding a unique tiebreaker such as _id to the sort and index makes the order deterministic. This matters most for cursor pagination, exports, and any diff-based comparison. Sorting stability is a property of the query and index, not of the storage engine.

bulkWrite groups many operations into fewer round trips, which cuts network latency overhead dramatically for batch jobs. It reports per-operation results, including which entries failed and why. Group operations by type and keep batch sizes moderate, for example 500 to 1000 operations, to bound memory and lock time. For very large imports, an unordered bulk write with modest concurrency usually gives the best throughput.

Ordered bulk operations stop at the first error, leaving subsequent operations unexecuted. Unordered bulk operations attempt every operation and collect all errors, which is faster because the server can parallelize internally. Use ordered when later operations depend on earlier ones, such as insert then update of the same key. Use unordered for independent document loads where partial success is acceptable.

await coll.bulkWrite(ops, { ordered: false });

Write concern trades latency against durability guarantees, so it should be chosen per endpoint based on business risk. Acknowledged writes with majority journaling suit financial or irreversible changes, while best-effort writes may suit telemetry or counters. Writes with w:1 are fast but can be rolled back if the primary fails before replication. Expose the trade-off in API documentation rather than silently varying it.

The write is acknowledged only after it has been applied to a majority of voting replica set members and persisted to the on-disk journal on those members. This protects against rollback when a primary fails, provided the majority is still reachable. It increases latency because the primary waits for replication and fsync. It does not protect against application-level mistakes such as writing wrong data.

Reporting queries can be routed to secondaryPreferred so they do not compete with primary write throughput. However, secondaries can lag, so reports may read stale data, and long-running reads hold resources on those nodes. Use a dedicated analytics secondary or a separate cluster for heavy aggregation. Always pair the choice with an explicit staleness tolerance agreed with the business.

db.reports.find({}).readPref("secondaryPreferred");

It helps when read volume is high and the application tolerates slightly stale data, since it spreads load away from the primary. It is misleading when the read path immediately depends on a prior write, because replication lag can make the read miss the write. Causal consistency sessions solve that specific case by guaranteeing read-your-own-writes. Without sessions, secondary reads are eventually consistent by design.

Start a session per logical unit of work, pass it explicitly to every operation that must participate, and end it in a finally block. Sessions are cheap but not free, so avoid creating one per individual query outside transactions. Sessions also carry causal consistency and retryable write state. Leaked sessions accumulate server-side until the logical session timeout cleans them up.

const session = client.startSession();
try {
  await coll.insertOne(doc, { session });
} finally {
  await session.endSession();
}

Every operation must be passed the session explicitly; otherwise it runs outside the transaction and is not rolled back. The transaction must be committed or aborted on every code path, including exceptions. Transactions have a default lifetime limit, so they must not wrap user think time or external calls. Reads inside a transaction use the transaction snapshot and do not see writes committed after it started.

Wrap the whole transaction body in a loop that retries only transient transaction errors, such as write conflicts or a primary stepping down. Each retry must start a new transaction with a fresh snapshot, because the previous one is aborted. Use bounded exponential backoff with jitter to avoid synchronized retry storms. Retrying non-transient errors, like a duplicate key violation, will simply fail again.

for (let attempt = 0; ; attempt++) {
  try {
    return await runTxn(session);
  } catch (e) {
    if (!e.hasErrorLabel("TransientTransactionError") || attempt >= 3) throw e;
    await sleep(50 * 2 ** attempt);
  }
}

A commit can fail with an UnknownTransactionCommitResult label, meaning the outcome is not known to the driver. Retrying commit with the same session and transaction number is safe because the server deduplicates it. This is distinct from retrying the whole transaction, which should happen only for TransientTransactionError. Handling both labels correctly is what makes transaction code robust.

Batching thousands of updates into one transaction holds locks and snapshot history longer, increasing conflict rates and memory pressure. Calling external services inside a transaction extends its lifetime and risks exceeding the transaction timeout. Reading huge result sets inside a transaction wastes cache and can exhaust the 16 MB transaction oplog entry limit. Keep transactions short, touching a small number of documents with no external calls.

MongoDB stores the transaction's changes as a single oplog entry when it commits, and that entry is capped at 16 MB. A transaction that modifies enough data exceeds this limit and aborts. Transactions also hold locks and snapshot history, so long ones amplify write conflicts. This is why the documented guidance is to design schemas so most writes are single-document and atomic without transactions.

Each change stream event carries a resume token, and the consumer persists the last processed token durably alongside its side effects. On restart it reopens the stream with resumeAfter or startAfter using that token, so it continues from the exact point. Tokens are valid only while the corresponding oplog history is retained. If the token is too old, the stream cannot resume and a full resync is required.

const cs = coll.watch([], { resumeAfter: savedToken });
for await (const ev of cs) { await handle(ev); await saveToken(ev._id); }

Opening the stream fails because the required history no longer exists on the server. The consumer cannot silently skip ahead without risking lost events, so it must fall back to a reconciliation process. Typical remedies are a full re-read of the collection, a snapshot comparison, or reprocessing from an archive. This is why oplog sizing and consumer downtime tolerance must be planned together.

Buffer events in memory with batchSize and a time window, then flush the batch and only afterwards persist the token of the last event in it. The order matters: save the token first and a crash before the write loses events; write first and a crash replays them. Replays are unavoidable, so the consumer has to be idempotent, and the buffer must flush on a timer instead of waiting for a full batch.

const batch = [];
for await (const ev of coll.watch([], { batchSize: 200 })) {
  batch.push(ev);
  if (batch.length >= 200) {
    await flush(batch);
    await saveToken(batch[batch.length - 1]._id);
    batch.length = 0;
  }
}

The application writes the business document and an outbox record in the same transaction, so both succeed or both fail. A separate relay watches the outbox collection with a change stream and publishes events to the message broker. After successful publication it marks or deletes the outbox entry. This avoids dual-write inconsistency between the database and the broker.

await db.collection("orders").insertOne(order, { session });
await db.collection("outbox").insertOne(
  { type: "OrderCreated", payload: order, sent: false }, { session }
);
await session.commitTransaction();

Two independent systems cannot be updated atomically, so one can succeed while the other fails. A crash between the two writes leaves the systems permanently inconsistent with no record of the missing message. Retrying can also produce duplicates if the first attempt actually succeeded. The outbox pattern converts this into a single atomic local write plus a reliable relay.

Soft deletes set a flag or timestamp instead of removing the document, preserving referential history and enabling restore. The cost is that every query must filter deleted records, and indexes must include the flag to stay efficient. Collections grow without bound unless a purge job eventually removes old soft-deleted rows. Hard deletes keep data volume and query complexity low but lose recoverability.

db.users.updateOne({ _id: id }, { $set: { deletedAt: new Date() } });
db.users.createIndex({ deletedAt: 1, email: 1 });

Centralize access behind a repository layer that always injects the deletedAt: null predicate, or expose a filtered view. Avoid scattering the condition across hundreds of call sites where one omission leaks deleted data. Add an index that leads with the filter field so queries remain selective. Automated tests and query review should confirm no endpoint bypasses the filter.

Typical fields are createdAt, updatedAt, createdBy, and updatedBy, sometimes plus a schema version. Set timestamps server-side with $currentDate so clock skew and client tampering do not corrupt them. Application code should write actor identity from the authenticated context, never from client-supplied input. Consistent field names across collections make downstream analytics and audits far easier.

{ $set: { updatedBy: actorId }, $currentDate: { updatedAt: true } }

Application servers can have skewed or misconfigured clocks, which produces timestamps that are out of order or wrong. $currentDate uses the server's clock consistently across all writers. It also removes the risk of a client sending an arbitrary or malicious timestamp. The trade-off is that the value reflects when the server applied the write, not when the request originated.

Database-per-tenant gives the strongest isolation and per-tenant backup or restore, but multiplies metadata, connections, and operational overhead. Collection-per-tenant is a middle ground but still strains namespace and catalog limits at high tenant counts. A shared collection with tenantId scales to many tenants cheaply but requires disciplined query filters and index design. Most large SaaS systems choose shared collections and reserve dedicated clusters for premium tenants.

Prefix every compound index with tenantId so queries stay selective and cannot scan across tenants. Enforce the tenant filter in a repository layer rather than trusting individual call sites. Consider a schema validator that requires tenantId on every document. For strict isolation needs, combine this with collection-level or cluster-level separation for sensitive tenants.

db.orders.createIndex({ tenantId: 1, createdAt: -1 });

CSFLE encrypts specific fields in the driver before they reach the server, so the server stores and can compare ciphertext without seeing plaintext. It protects sensitive values such as tax identifiers or medical data from database operators and backups. Encryption keys are managed externally in a KMS, and the driver needs a key vault collection. The trade-off is that encrypted fields cannot be arbitrarily queried or aggregated.

Queryable Encryption stores encrypted fields with a structured encrypted index so equality queries can run server-side without decrypting. Classic CSFLE with deterministic encryption supports equality only, and only when the field is encrypted deterministically, which leaks frequency patterns. Queryable Encryption provides stronger leakage guarantees and supports a defined set of query types. It requires newer drivers and explicit encrypted field metadata at collection creation.

SCRAM-SHA-256 is the default username and password mechanism and suits most application deployments. x.509 certificate authentication fits service-to-service identity with an internal CA and avoids shared secrets. LDAP and Kerberos integrate with enterprise directories for centralized identity, typically in on-premises environments. Atlas additionally supports federated identity through an identity provider.

SCRAM-SHA-1 remains for compatibility with old drivers and existing user credentials. New deployments should use SCRAM-SHA-256, which relies on a stronger hash and is the default in modern servers. User credentials created for one mechanism must be recreated to use another. Leaving legacy mechanisms enabled unnecessarily widens the authentication surface.

Built-in roles such as readWrite or dbAdmin cover common job functions and are maintained by the server. Custom roles let you grant a precise set of privileges on specific collections, which supports least privilege. The cost is more upfront design and maintenance as schemas evolve. Many teams start with a narrow built-in role and move to a custom role when the built-in is too broad.

db.createRole({
  role: "ordersApp",
  privileges: [{ resource: { db: "shop", collection: "orders" },
                 actions: ["find", "insert", "update"] }],
  roles: []
});

The application user should have only the actions it actually performs, on only the collections it touches, in only its own database. It should not have cluster-wide administration, user management, or the ability to create and drop collections unless the application truly needs it. Separate credentials should exist for migrations and for operational tooling. Rotating and scoping credentials per service limits blast radius after a compromise.

Bind to specific private interfaces rather than all interfaces, and never expose the port to the public internet. Enable TLS with a trusted CA and require client certificates where appropriate. Use private endpoints or VPC peering plus IP access lists so only known networks can connect. On Atlas, restrict access list entries to specific CIDR ranges and service accounts.

net:
  bindIp: 10.0.4.12
  tls:
    mode: requireTLS
    certificateKeyFile: /etc/ssl/mongodb.pem

Audit logging records administrative and authentication events such as user changes, role grants, and connection attempts, plus optionally CRUD operations. It is essential for compliance investigations and detecting unauthorized access. The cost is storage volume and write overhead, which grows sharply if CRUD auditing is enabled. Most organizations audit administrative events always and CRUD only for sensitive collections.

Store credentials in a dedicated secrets manager and inject them at runtime, never in source control or container images. Rotate database users on a schedule and support overlapping validity during rotation. Scope each secret to a single service and environment so leakage is contained. Avoid logging connection strings, since they often embed passwords.

Do not copy production data verbatim into test environments; instead generate synthetic data or apply irreversible masking during export. Masking must preserve referential relationships so joins and tests remain meaningful. Encryption at rest does not satisfy this requirement because the data is still recoverable with keys. Track which fields are classified as PII so masking rules are maintained as schemas evolve.

Identify every collection and embedded document that holds the subject's personal data, then delete or anonymize them in a tracked job. Keep a minimal record of the erasure itself for compliance evidence, without retaining the erased personal data. Cover derived stores such as analytics exports, search indexes, and caches, not just the primary database. Document retention exceptions, such as legal hold, before deleting.

Deleted documents can persist in backups, oplog history, and derived copies for the retention period. Point-in-time restore could reintroduce the data, so backup retention policy must be part of the compliance design. WiredTiger may also retain deleted data in free space until overwritten. Compliance therefore requires a documented lifecycle across primary storage, backups, and downstream systems.

mongodump produces a logical BSON export and works across storage engines but is slower and heavier on a live system. Filesystem or volume snapshots are block-level and fast, and can be restored quickly, but require coordination with the storage layer. Snapshots are only crash-consistent unless the journal is included and the snapshot is taken correctly. Many organizations use snapshots for fast recovery and logical dumps for portability and long-term retention.

A base backup is restored, then oplog entries from that backup's timestamp forward are replayed to reach a target moment. This lets you recover to just before a bad deployment or accidental deletion. The achievable window depends on how much oplog history is retained. Atlas continuous backups automate this process and present a selectable restore time.

A backup that cannot be restored within the required recovery time provides no real protection. Restore testing validates the procedure, credentials, tooling versions, and the actual recovery time objective. It also exposes gaps such as missing indexes, users, or dependent collections excluded from the backup. Schedule regular drills, including one full restore into an isolated environment.

Feature compatibility version, or FCV, tells the server which feature set to enable, allowing a binary upgrade and a feature upgrade to be separated. After upgrading binaries you set the FCV to the new version only when you are confident rollback is no longer needed. Lowering FCV is possible but limited, which is what enables a downgrade window. Managing FCV explicitly turns a risky jump into two controlled steps.

db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 });
db.adminCommand({ setFeatureCompatibilityVersion: "8.0" });

Upgrade secondaries one at a time, waiting for each to rejoin and catch up before moving to the next. Then step down the primary so an upgraded secondary takes over, and finally upgrade the former primary. Keep the FCV at the old version until the whole set runs the new binaries. Client drivers must already support the target server version before the upgrade starts.

Confirm every driver and library in use supports the target server version, and review removed or deprecated server features. Verify the backup is restorable and that a downgrade path exists. Check that all replica set members can run the new binaries with the same configuration and storage engine. Read the official upgrade notes for the exact version to catch behavior changes.

Raising FCV enables new behaviors and removes some backwards compatibility, which makes rolling back the binaries unsafe. Keeping the old FCV for a soak period lets you observe the new version in production while retaining a downgrade option. Raise the FCV only after the upgrade is proven stable. This separation is the core reason the FCV mechanism exists.

Full Time Diagnostic Data Capture samples server statistics every second and writes them to compressed diagnostic files. It captures metrics such as operation counters, queue depths, cache usage, and replication state, plus some server status detail. After an incident, these files reconstruct what the server was doing before and during the failure. Support engineers routinely ask for them first when diagnosing performance problems.

mongostat shows a per-second snapshot of server-level activity such as inserts, queries, updates, connections, and page faults. mongotop shows how much time is spent reading and writing each collection and database. Use mongostat to see whether the instance as a whole is saturated, and mongotop to identify which namespace is responsible. Both are quick triage tools, not long-term monitoring.

mongostat --uri "$URI" 1
mongotop --uri "$URI" 5

Level 0 disables profiling, level 1 logs only operations slower than slowms, and level 2 logs every operation. Level 2 is extremely expensive and should be used only briefly for targeted diagnosis. Level 1 with a well-chosen slowms threshold is the normal production setting. Profiling output lands in the system.profile collection, which should be capped and monitored.

db.setProfilingLevel(1, { slowms: 100 });

Find the slow operation in the profiler or logs, then run explain with executionStats to inspect the winning plan, documents examined, and keys examined. Compare examined documents to returned documents to detect a missing or unselective index. Check whether the plan used a collection scan, an in-memory sort, or an inefficient index. Confirm the fix with a repeat explain and monitor after deployment.

db.orders.find({ status: "open" }).explain("executionStats");

Keys examined counts index entries scanned, while documents examined counts full documents fetched to satisfy the query. A large gap between the two often means the index is not covered and the server fetches many documents. Both numbers should be close to the number of documents returned for an efficient query. High values indicate the index is not selective enough for the predicate.

A query is covered when the index alone satisfies the filter, the sort, and the projection, so no document fetch is needed. This requires every referenced field, including any excluded _id, to be part of the index. Covered queries can be dramatically faster because they avoid random document reads. The trade-off is wider indexes, which cost memory and write throughput.

Alert on sustained lag above a threshold that matters to the business, not on a single spike, because brief lag is normal during bursts. Correlate lag with secondary CPU, disk, and network saturation to distinguish load from a failing member. Also alert when a secondary's optime stops advancing at all, which indicates a stalled or disconnected member. Lag alerting must be tuned to the application's staleness tolerance.

Connection exhaustion causes hard failures that CPU graphs may not predict, especially with connection storms from restarted services. Tracking current connections against the configured limit gives early warning of pool leaks or per-request client creation. Alert on both total connections and connection churn rate. A spike in churn often reveals an application-level anti-pattern.

Alert on projected time to full based on growth rate, not just on a static percentage threshold. Separate warnings for data volume growth, journal volume, and log volume, since they fail differently. Include the disk containing the oplog and the temp directory used by index builds. A static ninety percent alert often fires long after action is needed or constantly on stable systems.

The working set is the portion of data and indexes actively accessed during normal operation, and ideally it fits in RAM. When it does not, reads hit disk and latency rises sharply. WiredTiger uses the cache for recently used pages, so total data size can far exceed RAM as long as the hot set fits. Capacity planning should measure actual access patterns rather than total collection size.

If cache eviction rates and disk reads rise while CPU stays moderate, the working set no longer fits and either RAM or a hotter data model is needed. If latency correlates with disk queue depth and IOPS saturation at steady cache hit ratios, storage throughput is the limit. Measure cache dirty percentage and eviction pressure, plus disk utilization during peak hours. Choose based on which resource saturates first under representative load.

Snappy is the default collection compression and offers fast compression with moderate space savings. zstd compresses better, especially for structured or repetitive data, at some CPU cost during reads and writes. Choose zstd when storage or IO bandwidth is the constraint and CPU headroom exists. Compression is configured per collection at creation time, so changing it requires a rebuild.

db.createCollection("logs", {
  storageEngine: { wiredTiger: { configString: "collection.blocks_compressor=zstd" } }
});

The journal is a write-ahead log that WiredTiger uses to recover committed writes after a crash. With journaling enabled and j:true, an acknowledged write is recoverable even if the data files were not yet updated. Journal commits are grouped, so batching writes improves throughput without sacrificing durability. Disabling journaling trades crash recovery guarantees for write speed and is rarely appropriate.

The server groups writes and fsyncs the journal on a short interval, so j:true waits for the next group commit rather than forcing an immediate flush. This means concurrent writers share the fsync cost, which is why throughput stays reasonable. Latency per write is still bounded by that interval. Understanding grouping explains why j:true is far cheaper than it appears in isolation.

Index builds consume CPU, memory, and disk IO, and on secondaries they are triggered by replicated index creation commands. Builds can slow replication and increase lag, and they hold resources for their duration. Modern versions support building on rolling members and resuming interrupted builds, which reduces risk. Plan large builds for low-traffic windows and monitor lag throughout.

Use a rolling approach where possible, building on secondaries before the primary, and avoid building many indexes at once. Ensure sufficient memory and disk headroom, since builds spill to disk when they exceed limits. Monitor replication lag and be ready to pause or abort. Index builds are resumable, so an interrupted build does not always restart from zero.

Every index must be updated on each relevant write, consuming CPU, cache, and IO even if no query uses it. Unused indexes also inflate storage and slow down backups and restores. The $indexStats aggregation stage reveals usage counts per index. Review index usage periodically and drop indexes that are clearly redundant.

db.orders.aggregate([{ $indexStats: {} }]);

List the exact operations, their expected duration, rollback steps, and the owner for each. Confirm backups are current and that monitoring and alerting will remain active during the window. Communicate expected degradation or downtime to dependent teams in advance. Keep a live log during execution so the postmortem can reconstruct timing.

Connection exhaustion from clients creating new connections per request is a frequent cause, as are unindexed queries saturating IO. Disk full conditions, often from logs or the oplog, cause abrupt failures. Network partition or misconfigured election settings can leave a replica set without a primary. Many incidents trace back to a change, such as a deployment that altered query patterns or index coverage.

A good postmortem reconstructs a precise timeline from logs and metrics, identifies contributing causes rather than a single scapegoat, and produces concrete follow-up actions with owners and dates. It distinguishes detection time from resolution time so monitoring gaps become visible. It is blameless, which is what allows people to report what really happened. Follow-up actions must be tracked to completion or the same incident recurs.

Start in a non-production environment and inject one fault at a time, such as killing a secondary, pausing a member, or adding network latency. Verify that the application's retry logic and driver behavior handle the fault without data loss or duplicate writes. Measure failover time and confirm alerts fire as expected. Only then run controlled experiments in production with a rollback plan and a defined blast radius.

The driver detects the topology change, marks the old primary unavailable, and retries operations against the new primary after the election completes. With retryable writes enabled, brief elections are usually invisible to the application. Without retries, operations fail and the application must handle the error. Election duration depends on election timeout settings and member health.

Use firewall rules or a proxy to block traffic between specific members for a bounded period, then remove the rule. Verify that the majority side elects a primary and the minority side steps down. Confirm the application observes the expected errors and recovers after healing. Always have out-of-band access so you can restore connectivity if the test goes wrong.

Right-size cluster tiers based on measured working set and IOPS rather than peak guesses, and scale during known traffic patterns. Use lower-tier nodes for non-production environments and pause them when idle. Review storage growth, backup retention, and data transfer costs, which are often overlooked. Reserved capacity or committed contracts reduce cost for stable, long-running workloads.

Cluster tier and node count dominate, followed by storage, backup retention, and cross-region data transfer. Continuous backups with long retention and many restore points add measurable cost. Read-only analytics nodes and multi-region deployments multiply infrastructure spend. Cost reviews should map each line item to a business requirement.

Archive cold documents to cheaper storage and keep only the hot window in the primary collection. Apply a TTL index to automatically expire time-series or log data that has no long-term value. Consider time-series collections for high-volume timestamped data, since they compress better. Deleting data alone does not immediately shrink files, so compaction or a resync may be needed to reclaim disk.

db.events.createIndex({ createdAt: 1 }, { expireAfterSeconds: 2592000 });

WiredTiger marks space as free within its files but does not return it to the operating system immediately. New writes reuse that free space, so steady-state size may stabilize. To actually shrink files you need compaction, or an initial sync into a fresh member. This distinction matters when planning storage capacity after a large purge.

Retry only errors known to be transient, such as network failures, primary stepdowns, and transient transaction errors. Use exponential backoff with jitter and a hard attempt cap, and respect the overall request deadline. Never retry non-idempotent operations without a deduplication key. Emit metrics for retry counts so rising retries surface as an early warning.

const delay = Math.min(cap, base * 2 ** attempt) * (0.5 + Math.random() / 2);

Without jitter, many clients that failed at the same moment retry at the same moment, producing synchronized waves that keep the system overloaded. Randomizing the delay spreads retries over time and lets the system recover. Jitter matters most when a large fleet shares a dependency, such as a single primary. Combine jitter with a cap so delays do not grow unbounded.

Propagate a request or trace identifier into the operation comment so it appears in the profiler and logs. This lets you connect a slow user request to the exact database command that caused it. Pair comments with structured application logging that includes the same identifier. Keep comments short, since they are transmitted with every command.

await coll.find(filter, { comment: "req-8f21c" });

maxTimeMS bounds how long the server spends on an operation, after which it aborts with a MaxTimeMSExpired error. This protects the database from runaway queries that would otherwise consume resources indefinitely. API endpoints should set a timeout aligned with their own request deadline. Catch the resulting error and return a controlled response rather than a generic failure.

db.reports.find(query).maxTimeMS(2000);

Always apply a limit and a projection so the query returns only needed fields, and paginate at the API layer. Enforce a maximum page size in the service rather than trusting client-supplied limits. Stream results with a cursor when bulk processing is genuinely required instead of materializing everything. Unbounded result sets are a leading cause of memory pressure in both driver and server.

A cursor timeout means the server closed an idle cursor after its inactivity period, so continuing to iterate fails. A socket timeout means the driver gave up waiting on the network for a response. The first is a server-side resource policy, the second is a client-side network condition. Confusing them leads to the wrong fix, such as raising timeouts when the real issue is a slow query.

An $or query can use indexes per branch only if every branch is indexed; otherwise the server falls back to a collection scan. $in on a single indexed field handles the common case more efficiently than $or across different fields. For multi-field alternatives, consider a normalized schema or separate queries merged in the application. Always verify with explain which branches used indexes.

Negation predicates match a large, unbounded portion of the collection, so an index scan offers little benefit over a collection scan. The optimizer often chooses a collection scan because most documents qualify. Rewriting the query as a positive predicate on an indexed field is usually the fix. If negation is truly required, ensure another selective predicate drives the index.

Embedding works while the array stays small and is read together with the parent. When the many side can grow large, store references and query the child collection with an indexed parent identifier. A hybrid approach keeps a bounded summary array on the parent and full detail in children. The decision hinges on access patterns, document size limits, and update contention.

A single BSON document is limited to 16 MB, which caps how much can be embedded. Arrays that grow continuously risk hitting the limit and cause document rewrites that hurt write performance. Large binary content belongs in GridFS or object storage with a reference in the document. Designing around the limit avoids painful migrations later.

Use $inc, which is atomic at the document level, rather than read-modify-write in application code. A single hot counter document can become a write bottleneck, so shard the counter across several documents and sum on read. Pre-aggregated counters can also be updated asynchronously from an event stream when slight lag is acceptable. Always verify the accuracy requirement before choosing approximate approaches.

db.stats.updateOne({ _id: "orders" }, { $inc: { count: 1 } });

findAndModify, exposed as findOneAndUpdate in drivers, atomically updates a document and returns either the old or new version. A separate read followed by a write has a race window where another writer can change the document. The atomic variant is essential for queues, leases, and state transitions. Choose the returnDocument option deliberately based on whether the caller needs pre- or post-update state.

db.jobs.findOneAndUpdate(
  { status: "queued" },
  { $set: { status: "running", leasedUntil: future } },
  { sort: { priority: -1 }, returnDocument: "after" }
);

Use findOneAndUpdate with a status filter and a lease expiry timestamp so a worker atomically claims one job. Workers that crash leave an expired lease, which another worker can reclaim by matching on an expired timestamp. Include an attempt counter to avoid infinite retries of poison messages. Consider a TTL index or a reaper job for abandoned jobs.

A unique index enforces the constraint at the storage level, so concurrent requests cannot both succeed. Application-level existence checks have a race window under concurrency. Handle the duplicate key error explicitly and translate it into a meaningful API response. Relying only on validation logic leaves the invariant unprotected against concurrent writers.

db.users.createIndex({ email: 1 }, { unique: true });

Catch the specific duplicate key error code rather than any write error, then return a conflict response such as HTTP 409. For idempotent creates, treat a duplicate on the same client key as success. Avoid retrying blindly, since the same conflict will recur. Log the index name from the error detail to make diagnosis quick.

Swallowing all exceptions hides data corruption, validation failures, and permission problems as generic errors. It also prevents the driver's retry logic from seeing errors it could handle correctly. Handle specific error categories and let unknown errors propagate with context. Structured error mapping at the service boundary keeps behavior predictable.

Inject failures deterministically, for example by stubbing the collection to throw a transient error on the first call. Assert that the operation produced exactly one document and one side effect after retries. Test concurrent duplicate submissions, not just sequential retries. Include a test where the process crashes between the write and the acknowledgement.

Stop accepting new requests, allow in-flight operations to complete within a grace period, then call close on the client to release sockets and background monitors. Skipping this leaves the server holding connections until timeouts expire and can complicate rolling restarts. Coordinate shutdown with the orchestrator's termination grace period. Log shutdown completion so deployment timing is verifiable.

process.on("SIGTERM", async () => { await client.close(); process.exit(0); });

Each new instance creates its own pool, and during a rolling deploy both old and new instances hold connections simultaneously. If maxPoolSize is generous and replicas are many, the total can exceed the server limit. Size the pool with total instance count in mind, and ensure old instances close clients promptly. Autoscaling makes this worse because pool count becomes dynamic.

Local returns data as the queried node has it, which may later be rolled back if that node was not majority-acknowledged. Majority returns only data that has been acknowledged by a majority of members and therefore cannot be rolled back. Majority costs more because it may wait for replication progress. Use it when reading data whose durability matters, and local for tolerant workloads.

Snapshot reads present a consistent point-in-time view across multiple documents and collections outside a transaction. This is useful for reports that must not mix pre- and post-update states. It is available on replica sets and sharded clusters and has performance implications, so it is not a default. Inside transactions, snapshot semantics apply automatically.

A session tracks an operation time and cluster time, and subsequent reads in that session wait until the target member has caught up to that point. This makes reads after writes consistent even when routed to a secondary. The guarantee applies only within the session, not across different clients. It is the correct tool when combining secondary reads with immediate reads of just-written data.

const s = client.startSession({ causalConsistency: true });
await coll.insertOne(doc, { session: s });
await coll.findOne({ _id: doc._id }, { session: s });

Retryable writes require a replica set or sharded cluster and a supported storage engine; standalone servers do not provide them. They also do not cover multi-document transactions as a whole, which need their own retry logic. Bulk writes with unordered semantics have per-operation retry behavior that must be understood. Some write concern configurations also affect retry support.

A cluster-wide stream is simpler to operate and gives a single ordered resume point, but it processes events from all collections. Per-collection streams isolate consumers and allow independent scaling, at the cost of more cursors and more resume tokens to manage. Filtering early with a pipeline reduces network and processing overhead. Choose based on how many consumers need independent progress tracking.

Delete or archive outbox entries after successful publication, ideally in bounded batches so deletion does not create large spikes. A TTL index on a sentAt field works when a grace period is acceptable. Monitor backlog size and consumer lag as a health signal. Never delete entries that have not been confirmed as published downstream.

db.outbox.createIndex({ sentAt: 1 }, { expireAfterSeconds: 86400 });

Validators enforce structure at the database level, so any writer, including scripts and migrations, obeys the same rules. They catch regressions when a code path bypasses application validation. Validation is applied on insert and update and can be set to warn or error. The cost is a small write overhead and the need to migrate existing documents to comply.

db.createCollection("users", { validator: { $jsonSchema: {
  bsonType: "object", required: ["email", "tenantId"],
  properties: { email: { bsonType: "string" } }
} }, validationLevel: "moderate" });

Strict applies the validator to all inserts and updates, including updates to documents that already violate it. Moderate skips validation for documents that were already invalid before the update, easing migration of legacy data. Setting the level to off disables validation without removing the rules. Most teams start moderate during migration and move to strict once data is clean.

A background thread scans the TTL index roughly every minute and removes documents whose indexed date is older than the expiry. Deletion is not immediate or exact, so expiry may lag by minutes and longer under load. TTL indexes must be single-field date or array-of-date indexes. They are for lifecycle cleanup, not for precise scheduling.

db.sessions.createIndex({ lastSeen: 1 }, { expireAfterSeconds: 3600 });

A partial index includes only documents matching a filter expression, so it stays smaller and cheaper to maintain. This suits queries that always target a small subset, such as active orders or unprocessed jobs. The query predicate must imply the index filter for the planner to use it. Partial indexes also reduce the storage footprint of large collections with skewed access.

db.jobs.createIndex({ createdAt: 1 }, { partialFilterExpression: { status: "queued" } });

A sparse index simply omits documents where the indexed field is missing, without a filter expression. A partial index is more expressive because it can filter on any predicate, including field values. Sparse indexes can produce surprising query plans because the planner must know the field exists. Prefer partial indexes for new designs.

String comparisons follow the collation of the operation, and an index only supports queries whose collation matches. A case-insensitive query needs an index created with the same case-insensitive collation. Using a different collation forces a collection scan or an in-memory sort. Define collation deliberately per collection or per index for user-facing text fields.

db.users.createIndex({ name: 1 }, { collation: { locale: "en", strength: 2 } });

The planner evaluated candidate plans in a trial period and discarded the slower ones before executing the winner. Rejected plans appear in the allPlansExecution output with their trial statistics. This shows the optimizer considered alternatives, which helps confirm whether an index was even a candidate. If no index plan appears, the index likely does not match the predicate or sort.

The server can combine results from two indexes when no single index covers the predicate fully. This requires scanning both indexes and intersecting document identifiers, which adds overhead. A well-designed compound index usually outperforms intersection. Intersection plans often indicate a missing compound index for the query pattern.

A hint forces the planner to use a specific index, useful when the optimizer chooses a plan you know is wrong for a particular data distribution. It is a sharp tool: data changes can make the hinted index a bad choice later. Prefer fixing the index design or statistics first. If you hint, document why and revisit it periodically.

$lookup performs a join per input document, so cost scales with the number of outer documents and the indexed lookup efficiency. Ensure the foreign field in the joined collection has an index, and filter the outer pipeline before the lookup. Denormalizing frequently joined data is often better than repeated joins. Use explain to confirm the index is used inside the lookup.

Run the count and the page query separately, and cache or approximate the count when it changes slowly. $facet can compute both in one round trip, but it processes the full pipeline and can be costly on large collections. For UI purposes an estimated count is often sufficient. Avoid recomputing an exact count on every keystroke or page change.

batchSize sets how many documents the server returns per network round trip, not how many total documents the query returns. Larger batches reduce round trips but increase memory per fetch on both sides. Tuning it helps for large scans and streaming exports. The default is usually reasonable for interactive queries.

A projection that excludes large fields reduces network transfer and can enable a covered index scan when the remaining fields are all indexed. Excluding fields still requires reading the full document from storage unless the query is covered. Thoughtful projections also reduce driver memory pressure. Avoid projecting nothing and filtering in application code.

appName identifies the client in server logs, currentOp output, and monitoring tools, which makes it possible to attribute load and connections to a service. Without it, operators can only guess which application holds a connection or ran a query. It is also used by Atlas to group activity by application. Use a stable, meaningful name per service and environment.

mongodb://host/?appName=orders-api-prod

It bounds how long the driver waits to find a suitable server matching the read preference and topology requirements before failing. A short timeout fails fast but may error during brief elections or network blips. A long timeout hides problems by making requests hang. Choose it to align with the caller's deadline and the expected election duration.

Command monitoring events expose each command's duration, success, and failure, which feeds metrics and tracing. Enable it selectively with sampling or filters, because full logging of every command is expensive. Never log full command payloads in production, since they may contain personal data. Aggregate metrics rather than emitting one log line per operation.

Add the new CA certificate to the trust stores of all members and clients before issuing new leaf certificates. Then rotate member certificates one at a time, allowing each to rejoin and serve traffic. Rotate client certificates after the server trusts the new CA. Remove the old CA only once nothing still presents certificates signed by it.

Kerberos provides ticket-based single sign-on with mutual authentication, which suits environments already standardized on it. LDAP is simpler to integrate for directory-based username and password checks. Kerberos has stricter operational requirements, including time synchronization and service principal management. The choice usually follows the organization's existing identity infrastructure.

Each client presents a certificate signed by a CA the server trusts, with a subject matching the MongoDB user's name. Server certificates must have the correct subject alternative names for the hostnames clients use. Certificate expiry and revocation handling must be part of operations. x.509 is well suited to service identity because no shared secret is transmitted.

Filters often contain personal data, tokens, or identifiers that should not appear in log aggregation systems. Log shapes and durations instead of full payloads, and redact sensitive fields explicitly. Log retention and access controls should match the sensitivity of what is logged. A leak through logs is as serious as a leak through the API.

Enable encryption at rest for backup storage and restrict restore permissions to a small operational group. Encrypt backups with keys managed separately from the database credentials so a single compromise does not expose both. Test that restores work with the documented key access procedure. Audit who restores data and where restored copies land.

A hidden member replicates data but is invisible to client read preferences, so it can serve dedicated workloads such as backups or analytics. It still votes in elections unless configured otherwise. Hidden members can be configured with a delayed priority to provide protection against operator error. They consume the same resources as any other member.

A delayed member applies oplog entries behind the primary by a configured number of seconds, so an accidental deletion is not immediately replicated there. You can recover data from it before the delay window elapses. The window must exceed the time it takes to detect the mistake. Delayed members cannot be used for timely reads.

cfg.members[2].priority = 0;
cfg.members[2].hidden = true;
cfg.members[2].secondaryDelaySecs = 3600;

The oplog is a capped collection recording every write, and secondaries replay it to stay in sync. Its size determines how long a member can be offline before it needs a full resync, and how far back point-in-time recovery can go. Too small an oplog causes frequent resyncs after maintenance. Size it for the longest expected outage plus peak write volume.

If the secondary's last applied oplog entry has already been overwritten in the primary's oplog, it cannot continue incrementally. The member then performs an initial sync, copying all data and rebuilding indexes. Large datasets make this slow and IO-heavy, which can affect the whole set. Adequate oplog sizing and prompt recovery prevent most resyncs.

Identify the operation with currentOp, verify it is safe to terminate, then call killOp with its operation identifier. Killing a write can leave partial work, so confirm the operation's semantics first. Prefer maxTimeMS-based limits so runaway operations self-terminate. Record what was killed for the postmortem.

db.adminCommand({ currentOp: true, secs_running: { $gt: 30 } });
db.adminCommand({ killOp: 1, op: 12345 });

compact rewrites a collection's data files, reclaiming space freed by deletions and defragmenting storage. It requires resources and blocks some operations on the collection while running. Rolling maintenance across replica set members limits the impact. Modern WiredTiger releases also reclaim space during normal operation, so compaction is not always necessary.

The validate command checks collection data and index consistency and reports errors. It can be expensive on large collections, so run it during a maintenance window or on a secondary. Use it after suspected corruption, storage failures, or unexpected server crashes. A clean validate is a prerequisite before declaring an incident resolved.

db.orders.validate({ full: true });

Pre-images contain the document before an update or delete, and post-images contain it after. They are disabled by default because they add storage and write overhead. Consumers that need to compute diffs or audit previous values must enable them on the collection. Enable only when a consumer genuinely requires them.

db.createCollection("orders", { changeStreamPreAndPostImages: { enabled: true } });

Transactions must run against the primary, so read preference is effectively primary within a transaction. Reads inside the transaction see the transaction's snapshot, not the latest committed state. This isolation is why transactions are unsuitable for long-running reporting. Design read-heavy analytics to run outside transactions.

Use majority write concern with journaling so the commit is durable across a majority before the application proceeds. The write concern is set at transaction start and applies to the commit. Weaker concerns risk acknowledging a commit that could be rolled back. Align the choice with the same durability standard used for non-transactional critical writes.

session.startTransaction({
  readConcern: { level: "snapshot" },
  writeConcern: { w: "majority", j: true }
});

Sessions consume server-side resources and are tracked until the logical session timeout expires. Long-lived idle sessions accumulate and can contribute to resource pressure, especially if transactions remain open. Always end sessions in a finally block and avoid caching them across requests. Monitor for open transactions that never commit or abort.

Inspect currentOp for operations with a transaction marker and a long running time. These hold snapshot history and locks, and can prevent oplog truncation. Alert when transaction age exceeds a threshold well below the configured limit. The usual root cause is a missing commit or abort on an error path.

db.adminCommand({ currentOp: true, "transaction.timeInactiveMicros": { $gt: 60000000 } });

Transactions add coordination overhead and have hard limits, so using them to compensate for a poorly modeled schema scales badly. Embedding related data so a single document update is atomic is usually faster and simpler. Transactions remain valuable for genuinely multi-document invariants such as ledger transfers. Choose the model first, then apply transactions where atomicity truly spans documents.

Write new documents in the new shape while reading code tolerates both shapes, then backfill existing documents in batches. Use a background job with bounded batches to avoid saturating IO, and track progress idempotently. Once the backfill completes and old-shape reads are gone, enforce the new shape with a validator. This expand-migrate-contract sequence avoids a hard cutover.

Backfills run for hours and may be interrupted by deploys, failovers, or rate limits. A resumable job records its position so it can continue without reprocessing everything. Idempotent updates make reruns safe when the position is uncertain. Together they turn a fragile one-shot script into an operable migration.

db.users.updateMany(
  { schemaVersion: { $lt: 2 } },
  { $set: { schemaVersion: 2 } },
  { hint: { schemaVersion: 1 } }
);

Process in small batches with a deliberate pause, and run during lower-traffic periods. Target a specific index and predicate so each batch is efficient. Monitor replication lag, cache pressure, and application latency, and pause automatically when thresholds are breached. Never run an unbounded updateMany across an entire large collection during peak hours.

Restoring into a live database can conflict with existing documents and indexes, producing duplicate key errors or partial state. Prefer restoring into a separate database or cluster, then migrating or swapping deliberately. Verify index definitions, users, and roles are restored as expected. Confirm that the restore does not silently overwrite newer data.

Compare collection counts and document counts between source and restored copy, and check that all expected indexes exist. Verify users, roles, and any auxiliary collections such as configuration data. Run application-level smoke tests against the restored instance. Document the verification steps so restores are repeatable under pressure.

The short form is a correlated equality join: localField is matched against foreignField and MongoDB can always use an index on the foreign field. The pipeline form is a full sub-query per input document. You pass outer values through let and read them inside the sub-pipeline as $$ variables, so you can filter, project, sort and limit the foreign side before it is attached.

db.orders.aggregate([
  { $lookup: {
      from: "customers",
      let: { cid: "$customerId" },
      pipeline: [
        { $match: { $expr: { $eq: ["$_id", "$$cid"] } } },
        { $project: { name: 1, tier: 1, _id: 0 } }
      ],
      as: "customer"
  } }
])

The catch: an $expr comparison is not automatically indexable. Put an equality $match on an indexed field first, or the sub-pipeline becomes a collection scan per document. The pipeline form is the right tool when you need a top-N join, not a full array of matches.

$unionWith appends the output of another collection or pipeline to the current stream. It is the clean way to query a live collection and an archive collection as one logical set, or to combine pre-aggregated rollups with raw data.

db.events.aggregate([
  { $match: { type: "click" } },
  { $unionWith: {
      coll: "events_archive",
      pipeline: [ { $match: { type: "click" } } ]
  } }
])

What it does not do: no deduplication, no schema reconciliation, no ordering guarantee, and both branches must live in the same database. Duplicate _id values from both sides are returned twice. Each branch is read in full, so an unindexed $match in either branch costs a collection scan. Treat it as a UNION ALL, and add a $group or $sort afterwards only when you actually need it.

$graphLookup performs a breadth-first search inside one collection. startWith seeds the search, connectFromField and connectToField define the edge, and as names the resulting array. Documents are de-duplicated by _id, so cycles terminate.

db.employees.aggregate([
  { $graphLookup: {
      from: "employees",
      startWith: "$reportsTo",
      connectFromField: "reportsTo",
      connectToField: "_id",
      as: "chain",
      maxDepth: 5,
      depthField: "level",
      restrictSearchWithMatch: { active: true }
  } }
])

Cost control: maxDepth caps recursion, depthField lets you stop or filter later, and restrictSearchWithMatch prunes branches early. Crucially, an index on connectToField turns each hop into an index lookup instead of a scan. Without maxDepth on a deep or cyclic graph, the working set grows quickly per input document.

$facet receives the documents produced so far and feeds the same input to every sub-pipeline in one pass. It is the standard way to return a page of results, a total count and per-facet counts in a single round trip.

db.products.aggregate([
  { $match: { status: "active" } },
  { $facet: {
      items: [ { $sort: { price: 1 } }, { $skip: 20 }, { $limit: 20 } ],
      total: [ { $count: "n" } ],
      byBrand: [ { $group: { _id: "$brand", n: { $sum: 1 } } } ]
  } }
])

Limits worth knowing: the input is materialised before the facets run, so put $match and $project before it; sub-pipelines cannot contain $out, $merge or a nested $facet; and each facet's output document is capped at 100 MB, which allowDiskUse does not relieve. Indexes only help the stages before $facet.

$bucket takes explicit, sorted boundaries that must all share one BSON type. Every document whose group-by value falls in [lower, upper) lands in that bucket, and you control the labels and the accumulators.

{ $bucket: {
    groupBy: "$price",
    boundaries: [0, 50, 100, 500],
    default: "other",
    output: { n: { $sum: 1 }, avgPrice: { $avg: "$price" } }
} }

$bucketAuto computes the boundaries itself from a requested bucket count, trying to distribute documents evenly; granularity (R5, R10, 1-2-5, POWERSOF2) snaps the edges to human-friendly numbers. Use $bucket for fixed business bands such as price tiers or SLA ranges, where the boundaries must stay stable across runs. Use $bucketAuto for exploratory histograms, accepting that boundaries shift as data changes and that documents outside the computed range can land in an overflow bucket.

$setWindowFields keeps every document and adds computed fields over a window, instead of collapsing rows into groups. That gives you running totals, moving averages, ranks and period-over-period comparisons in one stage.

db.orders.aggregate([
  { $setWindowFields: {
      partitionBy: "$customerId",
      sortBy: { createdAt: 1 },
      output: {
        runningTotal: { $sum: "$amount",
                        window: { documents: ["unbounded", "current"] } },
        last30d: { $sum: "$amount",
                   window: { range: [-30, "current"], unit: "day" } }
      }
  } }
])

Key mechanics: documents windows count positions, range windows work on the sort key's values (so unit applies to dates), and the default window is ["unbounded", "unbounded"]. Because it needs ordered input, it sorts and can spill; it is the natural replacement for self-joins and correlated sub-queries used to compute deltas.

$densify creates the documents that are missing from a sequence, and $fill assigns values to them. Together they turn sparse series into a continuous timeline, which matters when a chart or a downstream calculation must show zero-value periods rather than skip them.

db.metrics.aggregate([
  { $densify: { field: "ts", range: { step: 1, unit: "hour",
                 bounds: "full" }, partitionByFields: ["host"] } },
  { $fill: { sortBy: { ts: 1 }, partitionBy: "host",
             output: { value: { method: "linear" } } } }
])

$densify either extends the observed range or you supply explicit bounds. $fill supports a constant value, locf (last observation carried forward), or linear interpolation across the sort key. Both are partition-aware, so run them per tenant or per device. They generate rows, so the cost grows with the gap size: densifying years of minute data explodes the pipeline.

$out writes the whole pipeline result into a collection, atomically replacing whatever was there. It is simple and readers never see a half-written collection, but it rewrites everything on each run, so the cost is proportional to total output size.

$merge is incremental: it matches on an on field (default _id, which needs a unique index), then applies whenMatched (merge, replace, keepExisting, fail, or a pipeline) and whenNotMatched (insert, discard, fail). That makes on-demand materialised views practical: recompute only a recent slice, merge it into the summary collection, and keep history.

{ $merge: {
    into: "daily_sales",
    on: "_id",
    whenMatched: [ { $set: { total: "$$new.total", refreshedAt: "$$NOW" } } ],
    whenNotMatched: "insert"
} }

Both are write stages, so they cannot appear in a view definition or inside $lookup/$facet.

They are system variables, not fields. $$ROOT references the whole input document, which is handy for keeping a snapshot while reshaping: { $push: "$$ROOT" } inside a $group collects full documents, and { $sortArray: { input: "$items", sortBy: { price: 1 } } } style reshaping often starts from $$ROOT.

$$NOW returns the same date value for the entire pipeline, evaluated once at start, so every document sees an identical timestamp. That makes age calculations safe: { $dateDiff: { startDate: "$createdAt", endDate: "$$NOW", unit: "day" } }.

$$REMOVE removes a field conditionally inside $project or $addFields: { secret: { $cond: [ "$isPublic", "$secret", "$$REMOVE" ] } }. It is far cleaner than computing a value and then unsetting it in a later stage.

By default $unwind drops the document entirely when the field is an empty array, missing, or null, which silently loses rows in reports.

{ $unwind: {
    path: "$items",
    includeArrayIndex: "idx",
    preserveNullAndEmptyArrays: true
} }

preserveNullAndEmptyArrays keeps those documents with the array field absent, and includeArrayIndex gives the array position (null for preserved documents), which is useful for "position in cart" logic. Also note that a non-array value is treated as a one-element array, so string fields pass through unchanged. And unwinding multiplies document count, so a later $group or $sort processes far more rows than you might expect. Often $filter plus $map avoids the explosion.

$function runs JavaScript on the server for every document. It is expressive, but it is the wrong default for three reasons.

• Cost: the JS engine is invoked per document, so it is far slower than native operators and cannot use indexes at all.
• Opacity: the optimiser cannot reason about the expression, so stages around it cannot be pushed down or reordered.
• Portability: server-side JavaScript can be disabled at deployment level, so an expression that works in staging may fail in a hardened production tier.

Most requests that reach for $function are satisfied by $switch, $cond, $getField, $regexMatch, or $dateTrunc. If the logic truly belongs in code, compute it in the application on write and store the result, so reads stay indexable and cheap.

Memory-hungry blocking stages such as $sort, $group, $bucketAuto and $setWindowFields are limited to 100 MB of RAM per stage. Exceeding it raises an error unless the operation may spill to disk.

db.events.aggregate(pipeline, { allowDiskUse: true })

Since MongoDB 6.0 the server parameter allowDiskUseByDefault is true, so pipelines spill instead of failing unless a deployment turns it off. Spilling prevents the error but does not make the query fast: disk-backed sorts are an order of magnitude slower, and the real fix is a supporting index, a preceding $match, or a $project that shrinks documents. The 100 MB bound is per stage; a $facet result is still capped at 100 MB per facet regardless of allowDiskUse.

Not in general, and this is a common misconception carried over from SQL engines. $group is a blocking hash aggregation: it keeps one entry per distinct group key in memory, so its footprint scales with the number of distinct groups, not with input order. Pre-sorting by the group key does not remove that hash table, and it adds a full blocking sort of its own.

What actually helps:

• Filter first with $match so fewer documents reach the stage.
• Reduce cardinality: grouping by userId over millions of users costs far more than grouping by country.
• $project only the fields the accumulators need.
• Bound the work with $limit or a time window where the business question allows it.

Check explain for spill indicators; a $group that writes to disk is a signal to redesign the pipeline, not to add a sort.

$sample with a size below roughly 5% of the collection uses a pseudo-random cursor, which jumps to randomly chosen positions in the storage engine rather than reading every document. That is cheap on a large collection but it is not a strict uniform sample, and on small collections the server switches strategy.

For larger sample sizes it performs a top-k random sort, which scans the collection and keeps the best candidates in memory, so cost grows with collection size and the 100 MB stage limit applies.

db.users.aggregate([ { $sample: { size: 100 } } ])

Consequences: never put $sample after an expensive $group and expect it to save work, and do not use it for statistical guarantees. For "N random documents matching a filter", apply $match first; the sample is then drawn from the filtered stream, not the whole collection.

$redact walks the document recursively and evaluates an expression per level, returning $$DESCEND, $$PRUNE or $$KEEP. It is the right tool when the decision depends on a field's own value at any depth, for example hiding every sub-document whose classification is internal.

{ $redact: { $cond: [
    { $eq: [ "$classification", "internal" ] }, "$$PRUNE", "$$DESCEND" ] } }

$project with an inclusion list is explicit, predictable, and cheap: you state exactly which fields leave the database. It is the safer default for API responses, because a new sensitive field added later is excluded unless someone opts it in.

Note that $redact cannot use an index for the fields it inspects and it runs after other stages, so it does not reduce the data read from disk. For a hard guarantee, restrict with a $project and enforce field-level access with views or roles.

Pass explain: true to aggregate(), or call .explain("executionStats") on the cursor. The output nests stages rather than showing one flat plan, so you read it inside-out: the innermost $cursor shows the collection access, and each stage reports its own timing and document counts.

db.orders.explain("executionStats").aggregate([
  { $match: { status: "open" } },
  { $group: { _id: "$customerId", n: { $sum: 1 } } }
])

Look for totalKeysExamined versus totalDocsExamined at the cursor, nReturned per stage to spot where rows explode, usedDisk to catch spilling, and whether the $match was pushed into the $cursor or sits above it. A $match that appears after $cursor instead of inside it means the filter was applied in memory, not by the index.

Because order of stages defines meaning, not just performance. $sort then $limit gives the top N by that key; $limit then $sort sorts an arbitrary subset, which is almost never what a report wants.

Performance-wise, the pair $sort + $limit is optimised into a top-k sort: the server keeps only N candidates in memory instead of materialising the full sorted set, so a $limit of 10 over millions of documents is far cheaper than sorting them all. The optimiser can also coalesce a $sort immediately followed by $limit when the source is a single collection.

{ $sort: { score: -1 } }, { $limit: 10 }   // top 10, bounded memory

If the sort is placed after a $group or $unwind, no index can help and the top-k optimisation still applies, but the input has already been materialised.

The $documents stage lets you define the input inline, which turns a pipeline into a self-contained, runnable test fixture.

db.aggregate([
  { $documents: [
      { sku: "A", qty: 2, price: 10 },
      { sku: "A", qty: 1, price: 10 },
      { sku: "B", qty: 5, price: 2 }
  ] },
  { $group: { _id: "$sku", revenue: { $sum: { $multiply: ["$qty", "$price"] } } } }
])

This keeps the logic under test identical to what runs in production, with no fixture collection to create or clean up. Two caveats: $documents must be the first stage, and it accepts only constant expressions, so you cannot reference collection data. For deeper coverage, keep pipelines in one place in code and run them against a seeded test database in CI, asserting on exact output rather than on document counts.

arrayFilters lets you update specific elements of an array without knowing their positions. Each filter gets an identifier, and the update path references that identifier with $[id].

db.orders.updateOne(
  { _id: 42 },
  { $set: {
      "items.$[i].status": "backordered",
      "items.$[i].notes.$[n].flagged": true
  } },
  { arrayFilters: [
      { "i.sku": { $in: ["A-1", "B-2"] } },
      { "n.visibility": "public" }
  ] }
)

Each identifier is an independent predicate on its own array level, so the two filters do not interfere. An element matches only if it satisfies the filter at that level. Rules that trip people up: you must declare an identifier for every $[...] placeholder, identifiers cannot be reused across levels, and the filter field paths are relative to the array element, not to the document root.

The positional $ updates the first array element matched by the query filter. Its big limitation is that it is ambiguous with multiple array conditions, which is why pairing it with $elemMatch is the safe pattern.

db.orders.updateOne(
  { _id: 42, items: { $elemMatch: { sku: "A-1", qty: { $lt: 2 } } } },
  { $inc: { "items.$.qty": 1 } }
)

Without $elemMatch, the two predicates could match different elements and the position is undefined. The alternatives are $[] (all elements) and $[id] (elements matching an arrayFilters entry). Prefer the identifier form in new code: it is explicit, handles multiple conditions cleanly, and does not depend on the query filter's shape, so refactoring the filter cannot silently change which element is updated.

It turns a read-modify-write into a single atomic update, which is exactly what a bounded leaderboard or recent-activity list needs.

db.boards.updateOne(
  { _id: "weekly" },
  { $push: {
      top: {
        $each: [ { player: "ana", score: 91 } ],
        $position: 0,
        $sort: { score: -1 },
        $slice: 10
      }
  } }
)

Order of application inside the modifier is fixed: $each inserts, $position places them, $sort orders the whole array, then $slice trims to the first or last N (negative keeps the tail). Because $slice runs last, the array never grows past the cap, so the document stays far below the 16 MB limit and the write stays cheap.

Note that $sort on the array elements requires the field to exist; missing sort keys are ordered by BSON type rules, so normalise the documents before relying on the ordering.

$addToSet adds a value only if an equal value is not already present, using BSON equality rather than a field-level key.

db.users.updateOne(
  { _id: 7 },
  { $addToSet: { tags: { $each: ["beta", "beta", "gamma"] } } }
)

Surprises: the comparison is order-sensitive for embedded documents, so {a:1, b:2} and {b:2, a:1} are different values and both get added. Numeric types compare by value, so 1 and 1.0 do not duplicate, but NumberInt(1) and NumberLong(1) are distinct types. And each element is checked against the array, so a large $each on a huge array is O(n·m) and can be slow on hot documents.

If order and uniqueness matter more than array semantics, model membership as an object keyed by the value, or keep a separate collection with a unique index, which also removes the document-growth problem.

$setOnInsert applies only when the upsert inserts a new document; on a match it is ignored. It is the standard way to stamp creation metadata without a second round trip.

db.users.updateOne(
  { email: "ana@example.com" },
  { $set: { lastSeen: new Date() },
    $setOnInsert: { createdAt: new Date(), role: "member" } },
  { upsert: true }
)

Update pipelines (an array of stages such as $set and $replaceWith) are a different execution mode: the stage list is applied to the matched document and there is no notion of "insert only", so $setOnInsert, $push, $inc and friends are rejected there.

If you need computed creation values in a pipeline upsert, branch inside the pipeline on a field that only exists on insert, or simply use the classic modifier form for upserts and reserve pipelines for recomputation.

Instead of modifier operators you pass an array of stages, which lets the new value depend on other fields of the same document, computed on the server.

db.products.updateMany(
  { category: "sale" },
  [ { $set: { finalPrice: { $round: [
        { $multiply: ["$price", { $subtract: [1, "$discount"] }] }, 2 ] },
      updatedAt: "$$NOW" } } ]
)

Allowed stages are the ones that reshape a single document: $set/$addFields, $project/$unset, and $replaceWith/$replaceRoot. The write is still atomic per document, so readers never observe the intermediate state.

The trade-off is that each document runs the pipeline, so a large updateMany with heavy expressions costs more CPU than a constant $set, and the expressions cannot use indexes. Use it for deterministic recomputation during backfills and schema migrations, not as a general substitute for computing values in the application.

$inc is applied by the storage engine while the document is locked for that write, so concurrent increments serialise and no update is lost. There is no read-then-write window for two clients to interleave, which is why counters should never be implemented as find then save.

db.stats.updateOne({ _id: "page:home" }, { $inc: { views: 1 } })

What it does not protect: numeric type overflow. A 32-bit integer at 2147483647 will not silently become a long; the update fails with an error, so initialise counters with NumberLong when they can exceed that. Nor does it help when the same document is the target of thousands of increments per second: the document becomes a hotspot and writes queue behind each other, so shard or bucket the counter instead.

Because you need to know what you claimed, and the claim must happen in the same atomic step as the state change. With returnDocument: "before" you get the document as it was prior to the update, which is the only way to prove you are the client that moved it from queued to processing.

const job = await jobs.findOneAndUpdate(
  { status: "queued", runAt: { $lte: new Date() } },
  { $set: { status: "processing", lockedBy: workerId,
            lockedUntil: new Date(Date.now() + 60000) } },
  { sort: { priority: -1, runAt: 1 }, returnDocument: "before",
    projection: { payload: 1 } }
);

If the result is null, someone else won the race: no exception, no duplicate processing. With the default "after" you cannot tell whether you performed the transition or merely re-read a job another worker had already taken, which is the classic double-processing bug.

Only per document. Each document's modification is atomic, but the set of documents as a whole is not: concurrent readers can observe some documents updated and others not, and if the operation fails partway, the earlier writes remain applied.

Two practical consequences:

• The update must be idempotent, because a retry after a partial failure will re-apply it to documents that already succeeded. Prefer absolute assignments or guards such as { migrated: { $ne: true } } over relative operators like $inc.
• A long updateMany yields to other operations between documents, so it does not block the collection, but it also gives you no isolation boundary.

If the whole set must land together, wrap it in a transaction — subject to the 16 MB oplog entry size and the 60-second default lifetime, which means batching anyway. Otherwise accept eventual consistency and make progress resumable.

$bit performs bitwise and, or and xor on an integer field in place, atomically. Classic uses are permission masks, feature flags packed into one integer, and compact presence bitsets.

db.users.updateOne(
  { _id: 7 },
  { $bit: { perms: { or: NumberInt(4) } } }
);
db.users.updateOne(
  { _id: 7 },
  { $bit: { perms: { and: NumberInt(~4) } } }
);

Constraints: the field must be an int or long. A missing field is treated as zero, but a field holding a double, string or decimal raises an error, so the schema must be consistent. Because the operation is read-modify-write inside the storage engine, concurrent flag flips do not clobber each other — unlike fetching the value and writing it back. For bitsets spanning more than 64 flags you need multiple fields or an array of longs.

It matches three cases: the field is explicitly null, the field is absent, and the field is an array that contains null. That last case is the one people forget, because equality against an array matches if any element equals the value.

db.users.find({ nickname: null })
// matches: {nickname: null}, {}, {nickname: [null, "ace"]}

A normal index stores a null entry both for explicit nulls and for missing fields, so the query can be answered with an index scan on the null key rather than a collection scan. That is why { field: null } is indexable while { field: { $exists: false } } historically was not. If you want strictly null and not missing, query { field: { $type: "null" } }, and if you want "present", use { field: { $ne: null } } knowing it excludes nulls but includes arrays.

$type matches by BSON type, either by string alias or numeric code. The alias "number" is a shorthand that expands to double, int, long and decimal, which is why it is the right choice when a field's numeric type drifted over the years.

db.orders.find({ total: { $type: "number" } })
db.orders.find({ total: { $type: ["int", "long"] } })
db.orders.find({ total: { $type: "string" } })   // data-quality check

Two behaviours to remember: for an array field, $type matches if any element has that type, and the aggregation expression $type returns an array of the distinct types present, which is useful for finding mixed-type fields before a migration. $type on an indexed field can use the index, unlike a $where-style check.

Cross-type comparison follows a fixed order, roughly: MinKey, Null, Numbers, String/Symbol, Object, Array, BinData, ObjectId, Boolean, Date, Timestamp, Regular Expression, MaxKey.

The important consequence is type bracketing: a range predicate only compares against values of the same type bracket. { price: { $lt: 100 } } will never return a document where price is the string "50", even though "50" sorts after numbers overall. That is deliberate — it stops queries from returning nonsense — but it also means a single stray string breaks range reports silently instead of loudly.

db.items.find({ price: { $lt: 100 } })   // numbers only
db.items.find({ price: { $type: "string" } })  // find the strays

Note that numbers compare by numeric value across int, long, double and decimal, so 1 equals 1.0.

Three distinct problems, and only one of them is about indexes.

• Injection: characters like ., *, ( change the meaning of the pattern, so a search for "a+b" quietly matches "aab". Escape the input before building the pattern.
• ReDoS: a nested quantifier such as (a+)+$ can make the matcher backtrack exponentially. A short malicious string can pin a CPU core on the server, so cap input length and reject patterns with nested quantifiers, or avoid user-supplied patterns entirely.
• Cost: an unanchored pattern scans every candidate document; only ^prefix patterns get index bounds.

const safe = input.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
db.docs.find({ title: { $regex: "^" + safe } })

For real search requirements, use a text index or Atlas Search rather than regex.

It usually collapses the whole query into a collection scan. To answer $or with indexes, the planner needs a usable index for every branch, then it runs them as sub-plans and deduplicates the results by record id. If any branch has no index, the planner compares costs and normally concludes that scanning once is cheaper than scanning once anyway plus doing index lookups for the other branches.

db.orders.find({ $or: [
  { customerId: 42 },        // indexed
  { legacyRef: "X-9" }       // no index -> whole query scans
] })

Read explain for SUBPLAN/OR stages; seeing COLLSCAN at the top confirms it. Fixes: index every branch, split into two queries and merge in the application, or restructure the data so one indexed field covers both cases. Also watch duplication — the same document can match several branches, which the OR stage removes.

Dot notation with several predicates can be satisfied by different array elements; $elemMatch requires one element to satisfy all of them.

// different elements can match each predicate
db.orders.find({ "items.sku": "A-1", "items.qty": 5 })

// the same element must match both
db.orders.find({ items: { $elemMatch: { sku: "A-1", qty: 5 } } })

For a single predicate the two are equivalent, and dot notation is preferable because it produces tighter index bounds and lets the planner use separate index entries per condition. Reach for $elemMatch whenever two or more conditions must describe the same element, and remember that $elemMatch on a single condition disables the usual multikey bounds optimisation, so it can be slower for simple equality.

Because an index stores one entry per array element, not the array length. { tags: { $size: 3 } } is a predicate on the array as a whole, so the server must examine every candidate document and count elements.

db.posts.find({ tags: { $size: 3 } })   // COLLSCAN

Workarounds:

• Maintain a tagCount field with $inc on push/pull and index it — the standard approach for "exactly N" queries.
• For "at least N", an index on the array field with a range predicate can bound the candidates, then filter with $expr: { $gte: [ { $size: "$tags" }, 3 ] }.
• For "has exactly one element", an index on tags.1 plus { "tags.1": { $exists: false } } is narrow and cheap.

Any denormalised count must be kept consistent by every write path, or it silently drifts.

$expr lets a query use aggregation expressions, so you can compare two fields of the same document or branch on a condition with $cond.

db.orders.find({ $expr: {
  $gt: [ "$shippedAt", { $cond: [ { $eq: ["$express", true] },
                                   "$promisedAt", "$createdAt" ] } ]
} })

The expression is evaluated per candidate document, so the planner cannot derive index bounds from it and you get a collection scan (or an index scan on whatever separate predicates exist). $cond additionally makes the predicate non-sargable: even a supporting index on the compared fields cannot be ranged, because the comparison target varies per document.

If this shape is hot, compute the threshold on write into a field and index it, or restructure to a plain range predicate. Reserve $expr for ad-hoc analysis and admin queries.

A blocking in-memory sort is capped at 32 MB of data. Beyond that the operation fails with QueryExhausted-style sort errors unless it can spill or use an index.

An index avoids the sort entirely when the requested order matches the index order for the scanned bounds: the plan shows no SORT stage, just IXSCAN followed by FETCH. The equality prefix can be in any order, but the sort fields must appear in index order with matching direction (or the exact reverse).

db.orders.createIndex({ customerId: 1, createdAt: -1 });
db.orders.find({ customerId: 42 }).sort({ createdAt: -1 }).limit(20);
// IXSCAN supplies the order: no SORT, no 32 MB ceiling

Since 4.4 find accepts allowDiskUse, which lets an oversized sort spill instead of failing — a safety valve, not a design. If explain shows SORT with usedDisk: true, the index is missing or the direction is wrong.

maxTimeMS puts an upper bound on server-side execution and aborts the operation with error 50 (MaxTimeMSExpired) when exceeded, which stops one bad query from occupying a connection indefinitely. $comment attaches an arbitrary string that the server records in the profiler, in currentOp and in the slow-query log, which is how you connect a database operation back to the code path that issued it.

db.orders.find({ status: "open" })
  .comment("OrdersApi.listOpen v3 requestId=8f2c")
  .maxTimeMS(800)

Then db.system.profile.find({ "command.comment": /OrdersApi/ }) gives you the exact plans and timings for that endpoint. Treat the timeout as a contract: the application must catch error 50 and return a controlled response rather than a 500, and the budget should be lower than the client's own timeout so the server gives up first.

Sorting uses the same cross-type order as comparison: missing fields sort as null, and null sorts before numbers and strings. So a document without a score field appears before every document that has one when sorting ascending, which usually surprises people building leaderboards.

db.players.find().sort({ score: -1 })   // nulls land last here
db.players.find({ score: { $ne: null } }).sort({ score: -1 })

For arrays, ascending sort uses the smallest element of the array as the key and descending sort uses the largest, so an array-valued field can appear in an unexpected position. Sorting on an array field also prevents the plan from being a pure index-order scan when the index is multikey in ways that break the order guarantee, so verify with explain that no SORT stage appears. If ordering must be deterministic, add a tiebreaker such as _id.

It works on small, append-only collections and it is a trap at scale. $natural bypasses indexes and returns documents in storage order; -1 walks that order backwards, so on a fresh collection the newest inserts come first.

It stops being true once documents are updated or deleted. Updates can relocate a document inside the storage engine, and deleted space is reused by later inserts, so reverse natural order is no longer insertion order. A capped collection or a clustered collection keyed by a monotonic value does preserve insertion order, but a normal collection does not.

db.events.find().sort({ $natural: -1 }).limit(20)   // works, but not "latest"

The reliable version is an index on a monotonic field: find().sort({ createdAt: -1 }).limit(20), ideally with _id as a tiebreaker. The index scan returns exactly the newest documents without touching the rest of the collection.

Comparisons work numerically across numeric types, so a Decimal128 10.00 and a double 10 compare equal and sort together. Arithmetic is where the trouble starts: the result takes the wider type, so mixing decimal and double promotes to decimal, but the double has already lost precision before the conversion happens.

db.invoices.aggregate([
  { $group: { _id: null, total: { $sum: "$amount" } } }
])
// mixed types -> decimal result, but double inputs carried their rounding error in

Ordering also has a subtlety: 0.1 as a double is not exactly 0.1, so it can compare unequal to NumberDecimal("0.1") in edge cases, which breaks equality lookups and unique indexes over the field.

Practical rule: pick one type per field and enforce it with $jsonSchema bsonType: "decimal". Never convert currency with $toDouble in the middle of a pipeline.

A BSON Date is a signed 64-bit integer of milliseconds since the Unix epoch, always UTC. There is no timezone stored with the value — the offset only exists when you format it for display or parse it from a string.

new Date("2026-03-01T00:00:00-05:00")   // stored as 05:00Z
ISODate("2026-03-01T00:00:00Z")

Consequences: millisecond resolution only, so sub-millisecond ordering needs a separate sequence field; two events in the same millisecond are indistinguishable by date alone. Local midnight is not a fixed instant, so "today" must be computed with a timezone-aware library or an explicit offset, never by string concatenation on the client.

Do not confuse Date with Timestamp, which is an internal BSON type used by the oplog with second and increment components, and is not meant for application timestamps. Store UTC, convert at the edge, and keep a separate field for the user's declared timezone when it matters.

Twelve bytes: a 4-byte Unix timestamp in seconds, a 5-byte random value generated once per process, and a 3-byte incrementing counter.

ObjectId("65f1c2a4b7e3f10a9c8d4e21").getTimestamp()
// ISODate("2024-03-13T09:12:04Z")

Approximate order, yes: sorting by _id generally groups documents by insertion time, which is why it works as a cursor for pagination. Strict order, no — two processes can produce ids whose timestamps differ from the actual write order, the counter is per-process, and clock skew breaks the assumption entirely. Within a single process the counter makes ids monotonically increasing, which is why _id is usually a good tiebreaker.

Also note that ids are not secrets: the embedded timestamp leaks creation time. If that is sensitive, use a random UUID or a custom string key instead.

A UUID is 16 bytes stored as BinData, and the subtype determines byte order. Subtype 4 is the standard, but drivers historically wrote subtype 3 with a different byte order, so the same logical UUID can end up as two distinct BSON values that never compare equal.

db.users.insertOne({ _id: UUID("3b241101-e2bb-4255-8caf-4136c566a962") })
// BinData(4, "OyQRAdK7QlWMr0E2xWapYg==")

Traps to watch: configure the driver's UUID representation explicitly (in the Node.js driver, uuidRepresentation: "standard"); a legacy subtype 3 value inserted by an old service will not match a new subtype 4 query; and a unique index on the field will happily accept both representations as different keys. Migration requires rewriting values, not just changing the driver setting.

Also consider index locality: random UUIDs scatter inserts across the whole index, unlike monotonic ids.

The hard limit is 16 MB per BSON document, and the failure mode is ugly: a $push or $set throws once the new version would exceed it, which typically happens in production on a document that has been growing quietly for months.

db.articles.aggregate([
  { $project: { size: { $bsonSize: "$$ROOT" } } },
  { $match: { size: { $gt: 8000000 } } },
  { $sort: { size: -1 } }
])

Remediation depends on the shape: for unbounded arrays, apply the subset or bucket pattern and keep a bounded recent slice in the parent; for large embedded blobs, move them to GridFS or object storage and keep a URL; for wide, uneven documents, split rarely-read fields into a companion collection keyed by the same _id. Note the limit also applies to aggregation output documents, so a $group that builds a huge array fails the same way.

Yes: BSON documents support a maximum nesting depth of 100 levels. Beyond that, writes are rejected. In practice you hit design problems long before the number.

Deeply nested paths are painful because every query, index and update has to spell out the full path, and a single mistyped segment silently matches nothing. Indexes on deep paths also grow, because the whole path is part of the index key metadata. Updates to a field at depth 8 are hard to express with positional operators, and $lookup or $unwind over such structures gets unreadable.

{ a: { b: { c: { d: { e: 1 } } } } }   // "a.b.c.d.e" everywhere

If a document needs more than three or four levels, that is usually a signal to flatten it, extract a sub-document into its own collection, or use an array of objects with a type discriminator instead of nested type-specific containers.

Because JavaScript has no 64-bit integer. A BSON long arrives as a driver Long object, and treating it as a plain number silently truncates anything above 2^53.

const doc = await coll.findOne({ _id: 1 });
doc.seq.toNumber();                  // may lose precision
Long.fromString(doc.seq.toString()); // safe round trip

Typical failures: JSON serialisation turns a Long into { low, high, unsigned }; == comparisons against numbers fail; arithmetic on the Long object yields nonsense. Numeric fields also drift in type — an int that exceeds 2,147,483,647 stops accepting $inc with an overflow error, and a value written as a double when it should be a long breaks equality with existing documents.

Decide the type per field, set bsonType validation, and configure the driver to promote longs consistently rather than relying on defaults.

BSON stores fields in the order they were written and MongoDB does not reorder them, so { a: 1, b: 2 } and { b: 2, a: 1 } are different byte sequences. That matters because equality on an embedded document is order-sensitive: the query below matches the first document and not the second, even though the values are the same.

db.docs.find({ sub: { x: 1, y: 2 } })   // does NOT match { sub: { y: 2, x: 1 } }

Two practical consequences. Any hash or checksum computed over serialized BSON changes when a field is renamed, because $rename moves it to the end of the document. And an import that rebuilds documents in a different order silently breaks keys built from those hashes. When identity should ignore order, compare field by field with $expr instead of relying on document equality.

Each level needs its own properties block, and array elements are validated with items. A common mistake is declaring required at the top level and assuming it applies inside the array.

{ $jsonSchema: {
    bsonType: "object",
    required: ["orderId", "items"],
    properties: {
      orderId: { bsonType: "string" },
      items: {
        bsonType: "array", minItems: 1,
        items: {
          bsonType: "object",
          required: ["sku", "qty"],
          additionalProperties: false,
          properties: {
            sku: { bsonType: "string" },
            qty: { bsonType: "int", minimum: 1 }
          }
        }
      }
    }
} }

additionalProperties: false makes the schema strict, which is powerful but painful when a new field is rolled out — plan it deliberately. Validation applies to inserts and updates on the collection, including updates through aggregation pipelines.

Deploy it in warn mode first. validationAction: "warn" lets every write through but logs violations, so you can measure how much existing or in-flight code would fail before you enforce anything. "error" is the default and rejects the write with DocumentValidationFailure.

db.runCommand({ collMod: "orders",
  validator: { $jsonSchema: { bsonType: "object",
                              required: ["status"] } },
  validationLevel: "moderate",
  validationAction: "warn" });

Sequence: warn with validationLevel: "moderate" so already-invalid documents can still be updated if the update does not touch the missing field; watch the logs until violations stop; backfill the bad documents; then flip validationAction to "error". The "strict" level applies the rules to all documents, including previously invalid ones, which reverts an unrelated field update — useful only once the data is clean.

collMod modifies collection metadata without recreating the collection, which makes it the safe way to change rules on a live collection.

• Validation: validator, validationLevel, validationAction.
• TTL: change expireAfterSeconds on an existing TTL index by referring to its key pattern, including setting it to a much shorter value to purge old data quickly.
• Views: replace the pipeline of a view in place.
• Storage and indexing options: such as changeStreamPreAndPostImages and index hints.

db.runCommand({ collMod: "sessions",
  index: { keyPattern: { lastSeen: 1 }, expireAfterSeconds: 3600 } })

Changing expireAfterSeconds does not trigger an immediate mass delete: the TTL monitor runs about once a minute and removes in batches, so a large purge still takes time and produces oplog traffic.

A view is a stored pipeline evaluated at read time, so a $lookup inside it re-runs for every query — the join is never precomputed and no index can span the joined collections.

db.createView("order_summary", "orders", [
  { $lookup: { from: "customers", localField: "customerId",
               foreignField: "_id", as: "customer" } },
  { $unwind: "$customer" },
  { $project: { customerName: "$customer.name", total: 1 } }
]);

Limits worth stating: the pipeline cannot contain $out or $merge, including inside an embedded $lookup sub-pipeline; a view must live in the same database as its source; it is read-only; and the definition is public — getCollectionInfos and explain expose the pipeline, so never embed literal secrets or references to sensitive fields. Also, the view has its own collation, which must match if it reads another view.

It pre-allocates a fixed byte size and keeps documents in insertion order. When the space is full, the oldest documents are overwritten automatically, so the collection acts as a ring buffer with no cleanup job.

db.createCollection("recent_logs",
  { capped: true, size: 10485760, max: 100000 })

Guarantees and restrictions: reads in natural order return insertion order, deletes are not allowed, updates that would grow a document beyond its allocated space fail, and the collection cannot be sharded. Because writes are append-only, they are fast and predictable, and a tailable cursor can follow new inserts.

Where it does not fit: anything needing deletion, resizing documents, secondary indexes with heavy query patterns, or TTL-based retention. For log retention with flexible queries, a normal collection with a TTL index is usually the better choice; capped collections are for fixed-size, in-order streams.

$jsonSchema validates structure and per-field types, but it has no expression language, so it cannot say "endDate must be after startDate" or "total must equal sum of line items".

The trick is that a validator is a query filter, not just a schema: you can combine $jsonSchema with query operators, including $expr.

db.runCommand({ collMod: "bookings", validator: { $and: [
  { $jsonSchema: { bsonType: "object", required: ["start", "end"],
                   properties: { start: { bsonType: "date" },
                                 end: { bsonType: "date" } } } },
  { $expr: { $gte: ["$end", "$start"] } }
] } })

This runs per write on the server and rejects violations with DocumentValidationFailure. It costs a small amount of CPU and cannot reference other documents, so cross-document invariants still belong to unique indexes or transactions.

The field must hold a GeoJSON object with a type and coordinates: Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon or GeometryCollection.

db.places.createIndex({ location: "2dsphere" });
db.places.insertOne({ name: "Cafe",
  location: { type: "Point", coordinates: [-74.0721, 4.7110] } });

Rules that cause silent failures: coordinates are [longitude, latitude], in that order, not lat/lng; longitude must be within -180 to 180 and latitude within -90 to 90; a polygon's linear ring must be closed, with the first and last positions identical; and coordinates must be numbers, not strings, so a value imported as "4.7110" makes the document unindexable.

An insert with invalid geometry is rejected when the index exists, which is good — but a 2d index without GeoJSON validation accepts far looser input and behaves differently.

$near answers "nearest to this point": results come back sorted by distance, and it supports $minDistance and $maxDistance. That ordering is exactly what you want for "closest stores", but it also means the server must compute and sort distances.

db.stores.find({ location: { $near: {
  $geometry: { type: "Point", coordinates: [-74.07, 4.71] },
  $maxDistance: 2000
} } }).limit(10);

$geoWithin answers "inside this shape": a polygon, a circle via $centerSphere, or a multi-polygon. The result is unordered, which makes it cheaper and composable — it can appear inside an aggregation $match, be combined with other predicates, and be used when order is not part of the question.

Rule of thumb: distance-ranked search uses $near (or $geoNear in aggregation); containment or region filtering uses $geoWithin.

Because it relies on the geo index to produce documents in distance order, and once other stages have transformed or filtered the stream the index can no longer be used. The server therefore rejects the pipeline if $geoNear is not first.

db.stores.aggregate([
  { $geoNear: {
      near: { type: "Point", coordinates: [-74.07, 4.71] },
      distanceField: "dist.meters",
      maxDistance: 3000,
      query: { open: true },
      key: "location",
      spherical: true
  } },
  { $limit: 10 }
])

Notes: distanceField is required and receives the computed distance; query adds non-geo predicates that are applied alongside the index scan; key disambiguates when several geo indexes exist; spherical: true is required for 2dsphere semantics. It cannot be used inside $lookup or $facet, and unlike $near it works in aggregation at all.

Meters. With a 2dsphere index, $maxDistance, $minDistance and the value in distanceField are all in meters on the WGS84 sphere, so a 5 km radius is 5000.

{ $geoNear: { near: point, distanceField: "d",
              maxDistance: 5000, spherical: true } }
// d ~ 5000 means 5 km

Conversions are on you: divide by 1000 for kilometres, by 1609.34 for miles. $geoNear also offers distanceMultiplier, which scales the value written to distanceField — handy for returning kilometres directly, but be aware it does not change the unit of maxDistance, which stays in meters.

With a legacy 2d index the unit is not meters at all: it is the coordinate system's own unit, and distances come back in radians for spherical queries. Mixing a 2d index with GeoJSON input is the classic source of "my radius is 100x too small" bugs.

Several shapes look reasonable but cannot be answered by the index:

• Negations: $not, $nin or $nor around a geo predicate, since the index finds what is inside, not what is outside.
• Computed distance: an $expr or $function that calculates a haversine distance per document instead of using $near/$geoNear.
• Index type mismatch: a 2d index queried with GeoJSON geometry, or a 2dsphere index queried with legacy coordinate pairs and $box-style operators.
• Wrong field or representation: coordinates stored as strings, or as {lat, lng} objects instead of a GeoJSON array, which the index simply does not contain.

Confirm with explain: a geo query should show IXSCAN with 2dsphere in the index name, not COLLSCAN. If it scans, the predicate is not index-eligible regardless of how it reads.

indexSizes gives the on-disk bytes per index, which is the only honest measure of what an index costs you in cache and storage.

const s = db.orders.stats();
Object.entries(s.indexSizes).forEach(([k, v]) =>
  print(k.padEnd(40), (v / 1048576).toFixed(1) + " MB"));

Interpretation: an index larger than the data it serves is usually a candidate for removal, especially if it is a prefix of another compound index. Compare with $indexStats access counts and with the working set — a rarely used index that fits in RAM is still cheap, one that pushes the working set out of cache is expensive.

Before dropping: check for hints referencing it, unique or TTL semantics it enforces, and whether a query relies on its sort order. Hide the index first (db.collection.hideIndex()) so you can restore it instantly if a rarely-hit path degrades.

Level 1 records only operations slower than slowms, which is the right setting for production triage. Level 2 records every operation, including sub-millisecond ones, so it captures the full workload but adds real overhead to every read and write and fills system.profile quickly.

db.setProfilingLevel(1, { slowms: 100 });
db.system.profile.find(
  { ns: "app.orders", "command.comment": /OrdersApi/ },
  { op: 1, millis: 1, planSummary: 1, keysExamined: 1, docsExamined: 1 }
).sort({ ts: -1 }).limit(10)

Read the entries for planSummary (was it COLLSCAN or IXSCAN?), the ratio of keysExamined/docsExamined to nreturned, hasSortStage, and appName when appName is set in the connection string. Level 2 is for short, targeted windows on staging or a single replica, never a permanent production setting.

Use getParameter and setParameter admin commands. Only parameters marked as runtime-settable can be changed this way, and the change does not survive a restart, so it must also be written into the configuration file for permanence.

db.adminCommand({ getParameter: 1, slowms: 1 });
db.adminCommand({ setParameter: 1, logLevel: 1 });
db.adminCommand({ setParameter: 1, slowms: 200 });

Typical uses during an incident: raise logLevel briefly to capture verbose logs, adjust slowms without touching the profiler, or check limits such as maxTransactionLockRequestTimeoutMillis. Both commands require elevated privileges (clusterMonitor to read, clusterManager/hostManager to write).

Because settings are per node, in a replica set you must apply them to every member; a change on the primary alone makes behaviour inconsistent after failover.

The profiler shows individual slow operations; $collStats aggregations show the distribution of all of them. latencyStats returns read and write latency histograms in microseconds, so you can see whether the 95th percentile degraded even when no single query crossed slowms.

db.orders.aggregate([
  { $collStats: { latencyStats: { histograms: true } } },
  { $project: { reads: "$latencyStats.reads.latency",
                writes: "$latencyStats.writes.latency" } }
]);

This is how you answer "is the collection slower than last week?" with per-collection evidence rather than a server-wide average. Combined with storageStats in the same stage you get document counts, average object size and free space in one call. It is a cheap, read-only command, safe to run on production, and the histograms reset on restart — so capture them periodically if you need a trend.

Because it is a capped collection. Once it fills, new profile entries overwrite the oldest, so a long-running level-2 session can erase the very evidence you were collecting, and on a busy node the default size covers only seconds of activity.

db.setProfilingLevel(0);
db.system.profile.drop();
db.createCollection("system.profile",
  { capped: true, size: 50 * 1024 * 1024 });
db.setProfilingLevel(1, { slowms: 100 });

Profiling must be off while the collection is dropped, and it is recreated implicitly if you skip the explicit step. Practical routine: capture a bounded window, export the entries you need, then reset. Keep level 1 permanently with a sensible slowms, and reserve level 2 for minutes-long investigations on one node. Also remember that profiling writes are themselves writes, adding load to the same instance you are diagnosing.

mongosh is for humans running a bounded administrative action: creating an index, adjusting a validator, sampling documents, checking a plan. It connects, does the work, and exits; there is no pool to size, no retry policy, and no deployment pipeline around it.

// idempotent by construction
const name = "orders_status_createdAt";
if (!db.orders.getIndexes().some(i => i.name === name)) {
  db.orders.createIndex({ status: 1, createdAt: -1 }, { name });
}

Driver code is for anything that must run repeatedly, be tested, retry on transient errors, or participate in a transaction. Scripts that mutate data at scale need the same discipline as application code — idempotent operations, a filter that can resume, batching, and a dry-run mode — but they should not become the permanent integration path. If a script is scheduled in cron and has error handling, it is application code that happens to be written in JavaScript.

Take a 12 million document orders collection. A shard key of {country: 1} has roughly 200 distinct values, so at least 60,000 documents per value on average, and if 55 percent of orders sit in one country, that single value is a range the balancer can never split below its concentration. A key of {customerId: 1} looks far better with two million distinct values, but if your largest customer owns 400,000 documents, that one value is an unsplittable 400,000 document range and becomes a permanent hot chunk. A key of {createdAt: 1} has excellent cardinality and even distribution over time, yet every insert lands on the trailing chunk because the values increase monotonically.

The three properties answer different questions: cardinality bounds how finely you can split, frequency bounds how evenly chunks fill, and monotonicity decides whether inserts spread across chunks at all. High cardinality alone guarantees neither of the other two.

Hash the key. sh.shardCollection("app.events", {_id: "hashed"}) spreads writes uniformly, but range targeting is gone, so every range query on _id becomes scatter-gather and insertion order no longer matches key order.

Compound the key with a high-distribution prefix, for example {tenantId: 1, _id: 1} or {deviceId: 1, createdAt: 1}. Inserts spread across as many chunks as there are active prefix values and equality queries on the prefix stay targeted; the trade-off is that queries without the prefix fan out, and one huge prefix value recreates the hotspot.

Generate the identifier in the application instead of relying on the timestamp prefix inside ObjectId: UUIDv4, or the random component of a ULID. Range targeting is preserved, but you lose natural time ordering and any locality that came with it.

Take {tenantId: 1, createdAt: 1} across 5,000 tenants. Each insert is routed by tenant, so writes spread over as many chunks as there are tenants with data in the active range instead of all landing on the highest chunk. Within a single tenant the key is still monotonic, so you get a per-tenant hotspot rather than a global one, which is acceptable while no tenant dominates.

Unlike a hashed key, range targeting survives: {tenantId: "acme", createdAt: {$gte: from}} resolves to one shard and one bounded chunk range, and per-tenant time scans stay efficient on that shard. The price is that any query without tenantId fans out to every shard, and a tenant large enough to fill more than one chunk reintroduces the original concentration.

Resharding does not rewrite the collection in place. The cluster creates a new collection under a temporary UUID, runs a donor-to-recipient pipeline that clones documents under the new key, and applies concurrent changes so writes continue while the clone proceeds. When the copy catches up, a short blocking phase performs the swap, rename, and cleanup.

The cost is roughly one full extra copy of the collection on the receiving shards, the network traffic to move it, sustained read load on donors, and oplog pressure on every shard. For a 2 TB collection on a busy cluster, plan hours of degraded but not stopped service. You also need free disk on the recipients of about the collection size, and the operation is metadata-heavy on the CSRS because it tracks resharding progress and index builds for the new collection.

Clients keep reading and writing the original collection throughout the clone and the oplog application phases; mongos routes on the old shard key, so all existing queries behave normally. The resharding machinery captures those writes and replays them into the new collection. What changes is the background load: donors serve extra reads, recipients absorb a large write stream, and the oplog on both sides grows faster than usual.

Certain operations are blocked for the duration, notably some DDL on that namespace and creation of conflicting indexes, and the final commit takes a brief cluster-wide lock on the collection. If the clone cannot keep up with the write rate, resharding never commits, which is exactly when you call abortReshardCollection. Watch $currentOp for the resharding entry and the oplog window, which shrinks under the extra traffic.

Run db.adminCommand({abortReshardCollection: "app.orders"}) from a mongos. Abort is accepted during the clone and oplog application phases; once the operation enters the commit phase it can no longer be aborted and must run to completion or be recovered by the cluster.

After a successful abort the original collection is untouched and still sharded on the old key, clients see no change, and the temporary collection and its indexes are cleaned up asynchronously. That cleanup is the part people forget: the temporary data occupies disk until the range deleter finishes, so free space does not return instantly. If you aborted because the copy could not keep pace, the honest conclusion is usually that the target key or the timing was wrong, not that you should retry identically during peak hours.

The shard key is the routing identity of a document. Chunk ranges are defined over it, every index on a sharded collection is prefixed with it, and mongos decides where a document lives purely from those values. Changing a field that participates in chunk boundaries would mean a document could belong to a different chunk than the one holding it, so the cluster would have to physically relocate it and rewrite every index entry that encodes the key.

No in-place update can do that atomically across shards, which is why the key was immutable: the only paths were dump and restore into a new collection. MongoDB 5.0 did not make the field mutable. It added reshardCollection, which builds a new collection with the new key and swaps it in, so the observable effect is a changed key while the underlying documents are still copied, not mutated.

The shard key index must not be a multikey index. Routing assumes one document maps to exactly one point in the key space, which is what lets mongos translate a predicate into a single chunk range. An array produces several index entries per document, so a document could legitimately belong to several ranges on different shards, and the invariant that a chunk owns its documents exclusively collapses.

When the natural key really is multi-valued, model around it rather than forcing it. Promote the array element to an entity of its own, for example one document per (tag, entityId) pair in a linking collection whose key is {tag: 1, entityId: 1}. Alternatively pick a scalar field that is functionally determined by the same access pattern, such as the owning account, and index the array separately as a normal multikey index for filtering.

Two consequences matter. First, routing: a lookup by _id alone can no longer be resolved to a shard, so findOne({_id: x}) fans out to every shard, which is painful for the most common access pattern in many applications. Second, uniqueness: when _id is not the shard key, the _id index is only unique per shard, so the cluster can technically hold two documents with the same _id on different shards.

It is acceptable when the dominant access pattern is a tenant or account predicate and _id lookups are rare or always accompanied by that predicate, which is typical of multi-tenant SaaS. If point lookups by _id dominate, put _id in the key, for example {_id: "hashed"} or {tenantId: 1, _id: 1}.

Use explain at the mongos, not at a shard, and read the mongos-level stage. SINGLE_SHARD means routing resolved to one shard; SHARD_MERGE means it fanned out. The shards object lists exactly which shards were contacted, and each entry carries its own winning plan and execution stats.

db.orders.find({tenantId: "acme", status: "open"})
  .explain("executionStats").queryPlanner.winningPlan.stage
// "SINGLE_SHARD"  -> targeted
// "SHARD_MERGE"   -> scatter-gather

Confirm with executionStats.totalDocsExamined per shard and with the nShards counter, and cross-check against the shard key pattern from db.getSiblingDB("config").collections.findOne({_id: "app.orders"}).key. Targeting is a property of the predicate versus that key, so if the key is {tenantId: 1, createdAt: 1}, a query filtering only on createdAt can never be targeted.

Refining only appends fields to the existing key, turning {tenantId: 1} into {tenantId: 1, createdAt: 1}. It is metadata-only: no documents are copied, no chunk data moves, and the operation completes in seconds because the old key is a strict prefix of the new one, so existing chunk boundaries remain valid.

db.adminCommand({
  refineCollectionShardKey: "app.orders",
  key: {tenantId: 1, createdAt: 1}
})

Constraints: you need an index that supports the new key before you run it, you cannot reorder or remove existing fields, and you cannot convert a hashed component into a ranged one. Refining is the right answer when the current key already routes correctly but chunks are too coarse, for example one tenant whose data never splits. If the leading field itself is wrong, refinement cannot help and you need reshardCollection.

Design for the growth you expect, not the data you have. If one tenant is 2 percent of documents now and is growing at three times the average, that tenant will eventually own a whole chunk, so include a discriminator that subdivides large tenants, such as {tenantId: 1, bucket: 1} or {tenantId: 1, createdAt: 1}, where the suffix is high cardinality and present in the main query.

The signal to watch is per-range concentration, not the global average. db.printShardingStatus() shows chunk counts per shard, and $shardedDataDistribution shows the size of each shard's holdings for a namespace, so a single value approaching the chunk size shows up as a chunk that never splits while its neighbours move. Alert when the largest single shard-key value exceeds a meaningful fraction of the chunk size, and treat a repeating jumbo chunk for one value as the concrete warning that refinement or resharding is due.

The default maximum chunk size is 128 MB in modern releases; older deployments used 64 MB. It is a ceiling on how large a chunk may grow before a split is attempted, not a fixed size, so most chunks sit below it.

use config
db.settings.updateOne(
  {_id: "chunksize"},
  {$set: {value: 64}},
  {upsert: true}
)

Lowering it gives finer granularity, which helps a cluster that cannot balance because it has only a few oversized chunks, but it multiplies the number of chunks, the routing table each mongos must hold, split activity, and metadata churn on the config servers. Raising it reduces metadata and migration overhead but makes balancing coarse: with 20 chunks on 5 shards, one chunk of imbalance means 5 percent skew. Change it only with a specific reason, and remember existing chunks are not re-split immediately; only future growth follows the new ceiling.

A chunk is flagged jumbo when it exceeds the maximum chunk size and cannot be split, because too many documents share the same shard key value, or because it holds a single document that is itself larger than the chunk size. sh.status() displays such chunks with a jumbo marker on the range, and the balancer treats them as immovable in most versions, which is why one hot value can pin hundreds of gigabytes to one shard.

Remediation, in order of preference: change or refine the shard key so the offending values can be subdivided, which is the only real fix; split the range manually at hand-chosen points with sh.splitAt if the data actually has splittable boundaries; or clear the flag with clearJumboFlag only after you have genuinely reduced the chunk below the ceiling, because clearing it on an oversized chunk moves the problem rather than solving it.

Auto-split is driven by mongos: on writes and updates it tracks chunk growth and asks the shard to split a chunk once it crosses the maximum size, using key boundaries derived from the actual data. Auto-merge, introduced in 7.0, is the inverse operation run by the balancer, which merges adjacent chunks that live on the same shard when their combined size stays well below the threshold.

Merging matters because deletes and archival shrink data without ever shrinking the routing table. A collection that once held a billion rows and now holds ten million can still be spread over thousands of tiny chunks, each costing metadata on the CSRS, memory in every mongos, and a unit of work for the balancer. Auto-merge reclaims that. It is also why heavy bulk deletes are no longer followed by a permanent excess of chunks, though merges are deliberately conservative and will not chase every small chunk immediately.

The window is a config database setting, evaluated in the cluster's timezone:

use config
db.settings.updateOne(
  {_id: "balancer"},
  {$set: {activeWindow: {start: "23:00", stop: "06:00"}, mode: "full"}},
  {upsert: true}
)

Inside the window the balancer migrates chunks; outside it, imbalance simply persists. The latency cost is not only inside the window. A migration holds a brief critical section on the donor chunk where writes to that range are blocked, and during the catch-up phase the recipient applies the remaining change stream, so clients touching a migrating range can see elevated latency and occasional write conflicts. Migrations also consume network and IO that your daytime workload needs. That is the trade: a predictable nightly disturbance in exchange for steady daytime latency, at the price of accepting skew for up to eighteen hours.

Justified cases are narrow: draining a shard ahead of decommissioning, evacuating a specific hot range off a struggling node, restoring a layout after a zone range change, or seeding distribution before a launch. It is an operator override of a decision the balancer normally makes.

sh.moveChunk("app.orders", {tenantId: "acme"}, "shard02")
sh.moveChunk("app.orders", {tenantId: "acme"}, "shard02", {_waitForDelete: true})

Risks: the command blocks until the migration finishes, so a 90 GB chunk can hang your shell for a long time; interrupting it can leave orphaned documents on the donor until the range deleter runs; moving a chunk whose range is pinned by a zone violates the zone layout and the balancer will simply move it back; and jumbo chunks will refuse to move. Always check zone coverage and available disk on the recipient first.

An orphan is a document that physically sits on a shard which no longer owns the chunk range containing it. They appear when a migration is interrupted after the recipient has taken ownership but before the donor has deleted its copy, or after a failed moveChunk, a crashed mongos, or a botched restore of a single shard.

Normal queries do not see them, because mongos routes by range metadata, but a direct connection to the shard does, and they consume disk and distort counts on that shard. cleanupOrphaned runs on the shard's mongod and deletes documents outside every range the shard owns; in 8.0 it can be scoped to a single collection and to a starting key, and it can run on a secondary, which makes it far less disruptive than the older whole-shard sweep.

db.adminCommand({
  cleanupOrphaned: "app.orders",
  startingFromKey: {tenantId: "m"}
})

db.printShardingStatus() describes the layout: databases with their primary shard, collections with their shard key, chunk counts per shard, whether the balancer is enabled, and a truncated list of ranges. It answers "where do chunks point" and is the fastest way to see whether a collection is evenly divided in count and whether zones are configured.

$shardedDataDistribution answers "how much data does each shard actually hold", per namespace and per shard, including owned document count, owned logical size, and orphaned documents. That distinction is decisive, because chunk counts can look balanced while sizes are wildly unequal when chunks have very different document counts.

db.getSiblingDB("admin").aggregate([{$shardedDataDistribution: {}}])

Use the first for structure and zones, the second for capacity and disk planning. A cluster is balanced only when the second says so.

Stopping the balancer prevents it from initiating new migrations. It does not freeze the cluster. Auto-split continues, because splits are triggered by mongos on writes, so a growing chunk can still divide while the balancer is off; the range deleter continues cleaning up after completed migrations; and a migration already in flight when you stop the balancer is allowed to finish rather than being killed.

Explicit moveChunk calls still work, since they bypass the balancer's scheduling, which is exactly why teams that "stopped the balancer" during an incident are surprised to find chunks moving. Also remember that balancerStop does not persist across a config server failover in the way people assume; verify state with sh.getBalancerState() and sh.isBalancerRunning() rather than trusting that your earlier command is still in effect.

It is a per-shard server parameter, defaulting to 1, that sets how many chunk migrations a shard will run in parallel as a donor or recipient. Raising it drains a large imbalance faster, which matters when you have just added a shard and need to move terabytes before peak hours.

db.adminCommand({setParameter: 1, chunkMigrationConcurrency: 4})

It is a mistake when the shard is already saturated. Each concurrent migration adds a read stream on the donor, a write stream on the recipient, oplog generation on both, and a critical section that briefly blocks writes to the affected range. Raising it on a cluster whose disks are near their IOPS ceiling converts a slow rebalance into a latency incident, and on a link with limited bandwidth it just splits the same throughput across more partially finished migrations. Change it deliberately, watch disk latency and replication lag, and put it back.

In almost every case the data was loaded without going through mongos. Splits are initiated by mongos when it observes a chunk growing during routed writes; if you restore with a direct connection to one shard, or bulk load into a shard's replica set before sharding the collection, the shard happily writes 400 GB into one range and nothing ever asks it to split. The same symptom appears when zones map the entire key space to a single shard, or when the collection was sharded with a key whose only values all fall at one end.

Fix it by splitting before or after the load: sh.splitAt at chosen boundaries, or a controlled reshard. For future loads, pre-split the empty collection and always write through mongos. Confirm the diagnosis in config.chunks, which will show one document for that namespace.

That message means the balancer compared chunk counts and found the difference within its threshold, not that sizes are equal. The usual causes are chunks of wildly different sizes, jumbo chunks that cannot be moved, ranges pinned by zone definitions, or a collection whose chunks are all on one shard while the balancer considers other namespaces more urgent.

Start with the size view rather than the count view:

db.getSiblingDB("admin").aggregate([{$shardedDataDistribution: {}}])
db.getSiblingDB("config").chunks.aggregate([
  {$match: {ns: "app.orders"}},
  {$group: {_id: "$shard", chunks: {$sum: 1}}}
])

If counts are equal but owned size differs by an order of magnitude, you have a granularity problem: the chunks are too large to balance meaningfully, so lower the chunk size or refine the key. If some chunks are jumbo, no balancing will move them until the key is fixed.

On the recipient, every migrated document is inserted and indexed, so it generates oplog entries just like application writes; the donor later deletes its copies, generating more. A migration of 50 GB therefore produces tens of gigabytes of oplog traffic on both sides, and because migrations run in batches with majority write concern for the metadata steps, they can grow the oplog faster than steady-state traffic. If the migration rate outpaces replication, secondaries fall behind and the oplog window shrinks exactly when you need it most.

On the network, the cost is roughly the collection's compressed data plus index rebuild traffic, which is why moving a large range over a cross-region link takes far longer than the byte count suggests. Throttle with chunkMigrationConcurrency, schedule migrations off-peak, and never let a rebalance compete with replication on a bandwidth-constrained link.

mongos keeps an in-memory routing table per sharded namespace, loaded from the config servers and refreshed on change. For each query it extracts bounds on the shard key from the predicate, intersects those bounds with the chunk ranges it knows, and contacts only the shards owning the resulting ranges. A full shard key equality resolves to one chunk and therefore one shard; a prefix or range resolves to a contiguous set of ranges and possibly several shards; anything without the shard key covers every chunk in the namespace.

Requests carry a shard version. If the mongos's view is stale, the shard replies with a stale-config error, mongos refreshes from the CSRS, and retries, which is why a metadata change does not require manually flushing every router. Operations on unsharded collections simply go to the database's primary shard.

Two things: the router's decision, and the per-shard execution. The mongos-level winning plan uses stages that only exist in a sharded cluster, such as SINGLE_SHARD, SHARD_MERGE, SHARD_MERGE_SORT, and SHARD_WRITE. Those stages tell you whether routing collapsed to one shard or fanned out, which is usually the first question in a sharding investigation.

The shards object then contains an independent plan and execution stats per contacted shard. That is where sharding-specific problems appear: one shard doing a COLLSCAN while another does an IXSCAN, one shard examining 40 million keys while the rest examine thousands, or one shard returning in two seconds while another takes forty. A replica set explain can never expose that skew, and it is often the actual cause of a slow sharded query.

mongos pushes the sort and an extended limit down to every shard, then merges the sorted streams at the router, keeping only the first skip + limit documents overall. Each shard must therefore produce a locally sorted stream: cheap if a local index supplies the order, expensive if it does not, because each shard then runs a blocking sort over its entire matching set. With ten shards you pay ten sorts, ten sets of memory pressure, and the merge waits for the slowest one.

// pushdown per shard: sort + limit(skip + limit)
// mongos: SHARD_MERGE_SORT with a blocking SORT stage

The practical rule: give each shard an index matching filter plus sort, and if the query is latency-critical, include the sort key in the shard key so the query targets a single shard and no merge is needed at all.

Co-sharded means the same shard key pattern and, in practice, that the join predicate includes the shard key. Then each shard can execute the lookup locally against its own portion of the foreign collection, because the matching foreign documents are guaranteed to live there. The plan stays shard-local, no documents cross the network, and the cost is proportional to a normal indexed join.

Otherwise the join cannot be satisfied locally. If the foreign collection is unsharded it lives entirely on one shard, so every other shard must fetch from that shard to complete its part of the join, which turns a local operation into a remote one. If it is sharded on a different key, the work fans out further and results must be merged at mongos. This is why denormalizing the join into the document, or co-locating both collections on a tenant key, is often the right architecture rather than tuning the lookup.

$out cannot write to a sharded collection, since it replaces the target wholesale and would have to destroy and rebuild the shard layout; the target must be an unsharded collection. Both stages also must be the final stage of the pipeline and cannot run inside a transaction.

$merge can write into a sharded collection, but the on field must contain the collection's shard key, otherwise there is no way to route each output document deterministically. The other structural cost is that the write-side stage runs on a single shard rather than distributed: for sharded input, mongos gathers the pipeline's output and sends it to one shard to perform the merge, so throughput is bounded by one node and one connection path. For large analytical rebuilds, write to an unsharded staging collection with $out and swap, or use $merge keyed by the shard key to keep document updates routed.

mongos routes each branch of an $or independently and contacts the union of the resulting ranges. If both branches constrain the shard key to the same shard, the query stays targeted. If they resolve to different shards, the union is broader, and if any single branch has no shard key predicate, that branch alone expands the union to every shard, which drags the whole operation into scatter-gather.

// targeted: both branches carry tenantId
{$or: [{tenantId: "acme", status: "open"},
       {tenantId: "acme", status: "closed"}]}

// scatter: second branch has no shard key
{$or: [{tenantId: "acme"}, {priority: "urgent"}]}

Rewrite as $in on the shard key when possible, or split the query into two targeted requests in the application and merge the results yourself, which is usually faster than one fan-out.

Because routing happens before any index is consulted. mongos does not maintain a global index catalogue and has no idea that email is indexed on all ten shards; it only knows chunk ranges over the shard key. Without a shard key predicate there is no range to compute, so every shard must be asked, and only then does each shard's local index make its share of the work fast.

So the index reduces work per shard but cannot reduce the number of shards involved. The fix is not another index; it is a predicate on the shard key, a different shard key chosen around that access pattern, or an application-level routing table that maps an entity such as a user to the tenant that owns it. Understanding this distinction is what separates tuning a shard from tuning a replica set.

In practice it degrades targeting. The router extracts bounds from ordinary match expressions such as {tenantId: "acme"}, where the value is a literal it can place on the key space. Inside $expr the comparison is an expression tree, and the shard-key bounds the router can derive are far more limited, so a query that looks equivalent fans out to all shards even though every constant is known at parse time.

// fan-out
{$expr: {$eq: ["$tenantId", "acme"]}}

// targeted
{tenantId: "acme"}

The same reasoning explains the general advice about $expr and indexes: it is a tool for genuinely dynamic comparisons between fields, not for values you could have written as literals. On a sharded collection that mistake costs you a factor of the shard count rather than just an index scan.

No. The read preference is a property of the operation, and mongos forwards it to each shard's replica set, where it is evaluated against that set's members and tag sets. There is no per-shard override in a single query; the routing target set is computed from the shard key, not from where you would prefer to read.

The legitimate alternatives are architectural. Use zones to pin a key range to a region, so a targeted read naturally lands on the shards in that region, and let read preference choose a nearby secondary within each set. For offline analytics, connecting directly to a specific shard's secondary is possible and sometimes done, but it is outside the cluster's consistency model: you will see orphaned documents, no cluster-wide snapshot, and no protection from migrations. That is a reporting pipeline decision, never an application read path.

The from collection in $graphLookup cannot be sharded, and the stage cannot run inside a multi-document transaction. It is also memory-bounded, returning an error above its internal limit unless allowDiskUse is enabled, which turns a fast graph traversal into a spill-to-disk operation.

Design around it rather than fighting it. If the graph is small and read-heavy, keep it in an unsharded collection on a dedicated shard and accept the fan-in, since the stage already funnels results through one place. For hierarchies that are traversed constantly, precompute the closure: store ancestor paths or a materialized ancestors array, index it, and answer with a targeted equality query instead of a recursive traversal. Recursive lookups are one of the few places where sharding forces a modelling decision rather than a tuning decision.

The merge performed by mongos discards documents with an _id already seen from another shard. That is deliberate and historically existed to hide orphaned documents: if a stale copy exists on a donor, the client sees one document rather than two. It also means the merge is not a pure concatenation and that duplicates caused by migration artifacts are silently swallowed.

It breaks down when the pipeline changes what identity means. A $group or $bucket runs at mongos on the merged stream, so it sees deduplicated input, which is usually what you want but can mask data problems. A $project that removes _id before the merge, or a $unionWith that combines two logically distinct sets, can produce duplicates the router has no reason to collapse. If logical uniqueness matters, enforce it explicitly with $group on your own key rather than relying on merge semantics.

mongos rewrites the request so each shard returns skip + limit documents after sorting, then discards the first skip documents of the merged stream. Nothing is skipped at the shard. With skip: 100000 and limit: 20, every shard must produce and transmit 100,020 documents, so both network traffic and per-shard work scale with the skip value multiplied by the number of shards.

// shard-local: sort + limit(100020)
// mongos: merge, discard 100000, return 20

On a single replica set the cost is one bounded scan; here it is one per shard plus a merge that cannot return until the slowest shard finishes. Replace it with keyset pagination on a stable, indexed sort key, and if that key includes the shard key the page can even be targeted to a single shard.

They store every piece of cluster metadata and no user data: the shard list, each database's primary shard, each collection's shard key and options, the full set of chunk ranges with their owning shards, index definitions for sharded collections, and cluster-wide settings such as the chunk size and balancer window. Authentication data for the cluster also lives there.

The replica set form matters for two reasons. Availability: metadata must survive the loss of a node, and a majority write concern on metadata is the only way to guarantee that a migration or split is durably recorded before clients are told it happened. Atomicity: a single chunk migration updates several metadata documents at once, and those changes must commit together, which requires multi-document transactions, available only on a replica set. A standalone config server could do neither, which is why the CSRS replaced them.

With no primary, cluster metadata becomes read-only. Running mongos instances keep serving from their cached routing tables, so queries and writes to shard data continue as long as the shards themselves are healthy. What stops is everything that needs to write metadata: splits, chunk migrations, so the balancer idles, DDL such as creating or sharding collections, index builds on sharded collections, and zone changes.

Two operational traps. First, do not restart the mongos fleet during this state, because a fresh mongos cannot load a routing table and will fail to start. Second, do not assume the cluster is fine because traffic flows; if the outage outlives your patience you may drift into a state where a shard's data no longer matches metadata. Recovery is restoring a majority of config servers, then verifying with sh.status() and a targeted query before re-enabling the balancer.

Two is the minimum defensible number, and three is common so that maintenance does not reduce you to one. A single mongos is a hard dependency for every client that points at it, even though the data remains safe in the shards.

mongos holds no persistent state; its routing table is a cache that can be rebuilt from the CSRS at any time. That is why it is not a single point of failure for the cluster: you can lose one, add one, or replace one without touching the data, and clients that use a multi-host connection string or a DNS seed list simply reconnect to a surviving router. The true single points of failure are the CSRS majority and the shards themselves. The practical caution is that adding routers does not add routing throughput for a single query; it adds concurrency and failure tolerance.

Each application instance opens its own pool to each mongos in the seed list, so application-side connections are roughly instances times routers times maxPoolSize. That multiplication is the one teams underestimate, because a rolling deploy doubles it temporarily.

The mongos-to-shard hop is different and is the reason mongos exists as a multiplexer: mongos maintains its own pool per shard replica set, shared across all clients, so shard connection counts scale with the number of routers and shards, not with the number of application clients. Inbound connections at a shard are therefore approximately routers times per-host pool size, plus internal replication and migration connections. When a shard reports connection exhaustion, the cause is usually too many routers, a pool sized too large, or an application that bypassed mongos and connected directly.

mongos is CPU-bound on query parsing and result merging, memory-bound on the routing table, which grows with the number of chunks, and network-bound on scatter-gather results. A router that must merge millions of documents from ten shards is doing real work, and an undersized one becomes the bottleneck in front of a healthy cluster.

Placement matters as much as size. Routers should sit close to the shards on a low-latency link, because every query pays that round trip, and they should not share hosts with shards that need memory for the WiredTiger cache. For multi-region applications, put routers in each region so the application's first hop is local and only the shard hop crosses the WAN. Since routers are stateless, scale them horizontally when scatter-gather concurrency, not throughput per query, is the limit.

sh.addShard("rs3/host:27017,host:27017") registers the shard and the balancer begins moving chunks to it, because a new empty shard is maximally imbalanced. Expect sustained migration traffic for hours or days depending on data volume, plus a burst of metadata writes on the CSRS as each migration commits.

Watch four things: free disk on the new shard, since it will receive roughly an equal share and needs headroom beyond that; replication lag on both donor and recipient, because migrations generate oplog; network saturation if the shard is in another rack or region; and client latency on the ranges being migrated, where the critical section briefly blocks writes. If zones are configured, only ranges belonging to the new shard's zone will move, so verify the zone tags before assuming it will fill.

db.adminCommand({removeShard: "shard03"}) starts a drain: the balancer moves every chunk off the shard. The command is asynchronous and must be re-issued to observe progress; the response reports a state of started, ongoing, or completed and lists remaining chunk and database counts.

Two things block completion. First, databases whose primary shard is the one being removed appear under dbsToMove and require an explicit movePrimary for each, which relocates their unsharded collections. Second, chunks that cannot move: jumbo chunks, or ranges pinned by a zone whose only member is this shard, will keep the drain at ongoing forever. Draining terabytes also takes real time, so schedule it. Only when the response reports zero remaining chunks and no databases does the shard get removed from the metadata.

Consistency here means the config metadata describes exactly the data that exists on the shards at the captured instant. So the sequence is: stop the balancer, confirm no migration is in flight, then capture the config server replica set and all shard replica sets at the same logical point. In practice that means parallel mongodump with --oplog per shard plus a dump of the config database, or filesystem snapshots taken within the same window on every node.

What breaks it is a migration that commits between capturing two shards: metadata then says one shard owns a range whose documents are still on another, and a restore produces missing or duplicated data. Backing up each shard at leisure with the balancer running is the classic mistake. Also remember to restore config and shards together; a config restore from a different point in time is worse than no restore.

flushRouterConfig forces a mongos to discard its cached routing table and reload metadata from the CSRS. Since 4.4 routers refresh automatically when a shard returns a stale-config error, so it is rarely needed in normal operation. It remains a recovery tool: after manual metadata repair, after restoring a config server, or when a router is demonstrably serving a layout that no longer exists.

db.adminCommand({flushRouterConfig: 1})                    // this mongos
db.adminCommand({flushRouterConfig: "app.orders"})         // one namespace

balancerStop and balancerStart control migration scheduling only; they change nothing about routing. Confusing the two leads to the classic incident where someone stops the balancer, still sees wrong routing, and concludes the command failed. Verify state with sh.getBalancerState() and sh.isBalancerRunning().

Symptoms include repeated stale-config errors that refresh does not fix, queries that fail to target any shard, sh.status() listing a chunk on a shard that no longer exists, mongos refusing to start because it cannot load a routing table, and documents appearing or disappearing depending on which router you hit.

Recovery is conservative. First stop the balancer to prevent further metadata writes, then compare config.chunks against config.shards and against what each shard actually reports owning. If the divergence came from a bad restore, restore the CSRS as a set from a backup known to be consistent with the shards, then flush routing on every router. Do not hand-edit config.chunks unless you fully understand the range invariants; a wrong edit turns a routing bug into data loss, and vendor support should be involved before that step.

There is no small documented ceiling, but several practical constraints converge. Every mongos holds the full routing table in memory, so chunk count, not shard count, drives router memory; more shards usually means more chunks. Every query that is not targeted multiplies its work by the shard count, so a fan-out pattern gets slower as you add shards, not faster. Metadata writes on the CSRS grow with migration activity, and each shard adds its own replica set to operate, monitor, and back up.

The cost model is the real limit: sharding scales writes and storage on the axis of the shard key, and scales nothing else. If your workload is mostly scatter-gather reads, adding shards adds cost and latency. Teams that reach twenty or thirty shards usually discover their growth is in one collection and a better key or a data lifecycle policy would have been cheaper.

serverStatus().shardingStatistics is the sharding-native source: chunk migration counters for donor and recipient roles, split activity, and catalog cache statistics that reveal how often routers are invalidating and reloading metadata. Rising catalog cache refreshes mean frequent metadata churn, which is usually migration or DDL storms rather than a query problem.

db.adminCommand({serverStatus: 1}).shardingStatistics

Pair that with the distribution view, refreshed on a schedule, so you can see skew in bytes rather than chunk counts. Add balancer state and window, per-shard disk and replication lag, migration concurrency, and the count of jumbo chunks per namespace. Finally track SINGLE_SHARD versus SHARD_MERGE proportions from sampled explains: a rising scatter-gather share is the earliest signal that a new access pattern has outgrown the shard key.

A zone is a label attached to a subset of shards, plus a set of shard-key ranges assigned to that label. You tag shards with sh.addShardToZone and assign ranges with sh.updateZoneKeyRange. A shard may belong to several zones, but any given range belongs to at most one, and chunks whose range is in a zone may only be placed on shards tagged with that zone.

What it constrains is placement, nothing else. It does not change routing logic, does not affect how a query is planned, and does not isolate traffic. It simply tells the balancer "chunks in this range are only allowed on these shards", which is how residency and locality are enforced. If your predicate does not include the key the zone ranges are defined on, queries still fan out across every zone, so a zone layout is a storage guarantee rather than a performance guarantee.

Tag the shards first, then define ranges over the shard key, then shard the collection:

sh.addShardToZone("shard-eu", "EU")
sh.addShardToZone("shard-us", "US")
sh.updateZoneKeyRange("app.orders",
  {region: "EU", _id: MinKey},
  {region: "EU", _id: MaxKey}, "EU")
sh.updateZoneKeyRange("app.orders",
  {region: "US", _id: MinKey},
  {region: "US", _id: MaxKey}, "US")
sh.shardCollection("app.orders", {region: 1, _id: 1})

Mistakes that quietly break it: bounds written in the wrong order relative to the shard key's sort order, overlapping ranges, lower bound treated as exclusive when it is inclusive and upper as inclusive when it is exclusive, and values not covered by any range at all, which then have no zone and can land anywhere. Also verify that the strings written by the application match the range exactly, because "eu" and "EU" are different key space.

Because routing and placement use different inputs. Placement uses the zone ranges defined over the shard key. Routing uses bounds on the shard key extracted from the predicate. If you zone by country but shard on _id, residency can still be correct as long as country is part of the key, but the moment it is not, there is no way to express "routes to the EU zone" at all.

The concrete consequence is a transatlantic query: a European user's request fans out to shards in every region, and the merging mongos waits for the slowest one. Writes are worse, since they must travel to whatever shard owns the key. The test is simple: run explain for the real application predicate and check whether the stage is SINGLE_SHARD and whether that shard is the one in the expected zone. If not, the zone layout is decorative.

It gives residency and locality in one key. Zone ranges over {region: 1, createdAt: 1} pin each region's data to that region's shards, while the date suffix makes time-range queries targeted and keeps chunk sizes bounded as data grows, since a busy region splits along time rather than by luck.

sh.shardCollection("app.events", {region: 1, createdAt: 1})

The cost is a per-region hotspot: within each region the key is monotonic in time, so inserts concentrate on that region's trailing chunk, which is fine for a handful of regions but not for thousands of tenants. Queries without region, such as a global admin report, fan out to every region and pay cross-region latency. And if you later need to subdivide a single busy region, you cannot reorder the key, so you refine it with a further suffix or reshard.

They are zone sharding with the operational work managed: each region gets its own shards tagged into a zone, and zone ranges are defined over a location field in your documents. Reads that include the location field resolve to the local zone, which is where the latency win comes from.

Writes do not become globally fast. A write is routed to the shard that owns its key, so it must reach that zone's primary. If a user in Singapore writes a document whose location is EU, the write crosses the WAN twice, once to the EU primary and once for the acknowledgement. What global clusters give you is local latency for local data, plus residency, not universal low latency for arbitrary writes. If your access pattern writes across regions, the honest design decision is either to keep writes regional, or to accept the round trip and measure it rather than assume.

You prove it structurally, not by assertion. Show the shard-to-region mapping, the zone tags, and the zone ranges covering exactly the regulated key values, then show that the collection is sharded on the field those ranges are defined over. Then show the data distribution per shard to confirm the bytes really are only on in-region shards.

sh.status()
db.getSiblingDB("admin").aggregate([{$shardedDataDistribution: {}}])

Then close the gaps auditors find: unsharded collections live on a database's primary shard, which may be in another region, so verify every regulated collection is sharded or its primary shard is in-region. Backups, oplog copies, and monitoring snapshots also travel, and a multi-region CSRS replicates metadata everywhere. Finally, ranges not covered by any zone range are the silent hole: verify coverage is total for the regulated key space, not just for the values you tested.

Zones are metadata, so the switch itself is instant and online; the data movement that follows is not. The safe sequence is: ensure the destination zone's shards have capacity and are tagged, add the new range assignment, then let the balancer migrate the chunks, and only then remove the old assignment.

sh.updateZoneKeyRange("app.orders",
  {region: "LATAM", _id: MinKey},
  {region: "LATAM", _id: MaxKey}, "US-EAST")

Downtime risk comes from three places: removing the old assignment before data has moved leaves a window where no zone covers those chunks, so they may be balanced anywhere; a migration storm saturating the link to the new zone; and jumbo chunks that refuse to migrate and pin the range to the old zone indefinitely. Move ranges in slices, during a low-traffic window, watching replication lag on both sides.

It costs a wide-area round trip to every zone plus the transfer of every matching document to a single router, because the merge must happen in one place. If three zones are 80 ms apart and each returns 20,000 documents, the query pays at least the slowest zone's latency and moves all 60,000 documents across the WAN, then returns perhaps twenty to the client. Egress charges and jitter usually hurt more than the average latency suggests.

Avoidance is design, not tuning: include the location field in the predicate so the query targets one zone; keep a regional read model that a regional service queries; or run a mongos per region and merge in the application when a genuinely global answer is needed. Alerting on the ratio of SHARD_MERGE to SINGLE_SHARD plans for each critical query keeps this from creeping back in as new features ship.

Zones provide placement, not disaster recovery. If every shard tagged into the EU zone is unavailable, the chunks in that zone have no owner: the data is not automatically relocated to another region, and queries for those ranges fail even though the rest of the cluster is healthy. A zone is not a failover unit, and no balancer action can create capacity where there is none.

Design for it deliberately. For ranges that must survive a regional loss, tag shards from two regions into the same zone, so the zone spans regions and each shard's replica set can elect a primary in the surviving region, accepting cross-region replication lag as the price. For genuinely region-local data, accept the outage and make the application fail cleanly rather than hang. Test both paths, because the failure mode people discover in production is a query that waits on a dead region until timeout.

They compose in two independent layers. Zones decide which shards hold the data, so a targeted query on the location key lands on in-region shards. Read preference then decides which member of each of those shards' replica sets serves the read, so nearest or a tag set picks a local secondary. Together they give a read that never leaves the region.

db.orders.find({region: "EU", _id: id})
  .readPref("nearest", [{region: "eu"}])

What they do not guarantee: they do not target the query for you, so a predicate without the location field still fans out to every region and then applies read preference per shard. They do not provide cross-shard snapshot consistency. And nearest is evaluated per replica set, not globally, so it cannot compensate for a badly chosen shard key. Zones plus read preference reduce latency only when the predicate already routes correctly.

An empty zone means its shards hold no chunks. Common causes: the range was defined but no documents have values inside it yet; the application writes "eu-west" while the range covers "EU", so nothing ever matches; the range bounds are in the wrong order and therefore cover nothing; chunks in the range are jumbo and cannot migrate in; or the shards lost their zone tag during a topology change.

An unbalanced cluster is merely slow. An empty zone is a silent correctness problem: writes for the intended region may be landing on shards in another jurisdiction, which is a residency violation nobody notices because everything succeeds. Detect it by comparing each zone's assigned ranges against actual chunk ownership in sh.status() and by watching per-zone document counts. Coverage gaps in the zone map, not just emptiness, deserve the same alert.

Look at the ratio of work to results, not at the data size. If totalDocsExamined is thousands of times nReturned, if plans contain COLLSCAN, or if one query shape accounts for most of the latency, you have an index problem and sharding will multiply it across shards rather than fix it.

db.orders.find({...}).explain("executionStats")
// ratio totalDocsExamined : nReturned, and winningPlan.stage

A volume problem looks different: every hot query already uses a tight IXSCAN, examined-to-returned is close to one, CPU is moderate but disk reads and cache eviction are high, and latency degrades as the working set outgrows RAM. Writes may be capped by a single primary. That is when more machines, not more indexes, is the answer, and only then does sharding become a legitimate conversation.

Storage is rarely the first limit. The binding constraint is usually memory: the working set, meaning the indexes plus the hot portion of documents that queries actually touch, must stay resident in RAM, or reads turn into random disk I/O. A node holding 4 TB of data is fine if the active month is 60 GB; a node holding 300 GB with a 300 GB working set will struggle regardless of CPU.

The second limit is write throughput, because a replica set has exactly one primary accepting writes; adding secondaries scales reads and resilience, not writes. Watch wiredTiger.cache bytes in cache against the configured maximum, cache eviction pressure, and disk utilization on reads. When both the working set and write rate are near the ceiling of the largest sensible node, you have a real sharding case with evidence behind it.

You add a config server replica set, at least two routers, and every shard as its own replica set, each needing monitoring, backups, upgrades, and a restore test. Upgrades become an orchestrated sequence across components rather than a rolling restart. Backups require quiescing the balancer so metadata and data are consistent.

Then there are the costs that show up as engineering time: a shard key that is expensive to change and constrains every future access pattern; queries that were fine on one node now fanning out; migration incidents during rebalances; and debugging that requires reasoning about routing as well as execution. You also lose some simplicity in unique constraints and transactions. A team should be able to name who owns that operational surface before the first shard is added, because it never gets cheaper to add later.

It front-loads complexity against a workload that did not need it. With a few million documents the whole dataset fits in RAM on one node, so sharding cannot help, and the small per-shard caches may even reduce the hit ratio compared with a single large cache. Meanwhile every query without the shard key is now slower than it was on a replica set.

The subtler cost is that the shard key gets chosen with almost no data to inform it. You pick on intuition, discover the real access pattern a year later when there is 50 times more data, and now the correction is a reshard with a full copy and a maintenance plan. The disciplined alternative is to design the schema so the future key is available and indexed, keep the option open, and shard when a measured limit is actually reached.

When the workload is read-dominant, reads can tolerate slight staleness, and writes comfortably fit on one primary. Add secondaries and route reporting or geographically distant reads with read preference, then put a cache in front of the hottest documents or aggregate responses so the database sees a fraction of the requests.

This scales read throughput and read latency without touching routing, and it keeps unique constraints, transactions, and backups simple. It does not scale write throughput, storage beyond a single node's practical limit, or a working set that cannot fit in memory, so those are the conditions that force sharding instead. Be honest about the cache's own costs: invalidation bugs and stampedes after expiry can make a cache look like a database problem, and hiding a bad query behind a cache just delays the fix.

Separate the data that must be fast from the data that must merely exist. Keep the last 60 to 90 days in an operational collection sized so its indexes and documents stay in RAM, and move older documents to an archive collection, object storage, or a managed archival tier. The operational working set stays small and predictable, so one replica set keeps serving with low latency while total storage grows without bound.

The application routes by date: recent queries hit the hot collection, historical queries hit the archive with a different, slower service-level objective. What you must design is the seam, since a query spanning the boundary needs two reads and a merge, and that is usually acceptable for reports but not for interactive screens. This is often the cheapest capacity fix available, and it is routinely skipped in favour of sharding.

Because report queries are usually the worst possible fit for sharding: broad date ranges, aggregations without the shard key, and heavy sorts. Building a materialized result into a single unsharded collection means the expensive scatter-gather runs once on a schedule, and every dashboard read afterwards is a targeted read of a small precomputed set on one shard.

db.orders.aggregate([
  {$match: {createdAt: {$gte: startOfMonth}}},
  {$group: {_id: "$region", revenue: {$sum: "$total"}}},
  {$out: "reporting.monthly_revenue"}
])

Trade-offs: the data is stale by the refresh interval, it costs extra storage, and the rebuild itself is a heavy job that must run off-peak or incrementally with $merge. It also concentrates reporting load on one shard, which is fine for a few gigabytes and wrong for a hundred.

A chunk is split at key boundaries, not document boundaries, so if one shard-key value maps to documents that are each several megabytes, the chunk fills quickly and cannot be split without breaking that value apart, which is impossible. You get jumbo chunks, and with them a range that the balancer can never move. A single document larger than the chunk size guarantees a permanently unsplittable chunk.

Large documents also slow every migration, inflate oplog entries, and make the shard key value itself costly if it is a long string included in every index prefix. The remedies are schema-level: keep documents well below the limit, use the bucket pattern or GridFS for blobs, and avoid storing megabytes of arrays that also block multikey indexing. Sharding rewards small, uniform documents far more than it rewards tuning.

Growth rate, working set, and workload shape.

db.stats(1024)                       // dataSize, indexSize, per-collection growth
db.orders.stats(1024)                // count, avgObjSize, totalIndexSize, wiredTiger
db.serverStatus().metrics.document   // inserted, updated, deleted, returned

dbStats and collStats give you total data and index size in kilobytes, average document size, and per-collection index overhead, which lets you project when a node's RAM stops covering the hot subset. serverStatus().metrics gives inserted, updated, deleted, and returned counts, so you can see whether the pressure is read volume or write volume, and metrics.queryExecutor.scanned shows how much work those reads cost. Add disk utilization and cache eviction rate. If the projection says you are eighteen months from a wall, you have time to design the key properly instead of guessing.

When tenants are few and individually large, with contractual isolation, per-tenant retention, and the ability to drop or restore one tenant's data independently. A collection per tenant gives you that directly: dropping a tenant is a drop, restoring is a restore, indexes can be tailored, and one tenant's runaway query is easier to attribute.

It is not a scaling strategy on its own, since all those collections still live on the same replica set and compete for the same cache and IO. Its limits appear as namespace and file-handle overhead, memory per collection, and the impossibility of cross-tenant aggregation. The strongest pattern for a large multi-tenant system is often both: collection-per-tenant for the few large customers, and a shared collection sharded on {tenantId: 1, ...} for the long tail, which keeps the key simple because tenant identity is already the boundary.

The migration path is deliberate: deploy the config server replica set and mongos routers, then add the existing replica set as the first shard, so no data moves. Sharding a collection at that point is instant because all chunks already live on that one shard. Only when you add a second shard does the balancer begin distributing, which is the expensive part and belongs in a planned window.

sh.addShard("rsMain/host1:27017,host2:27017,host3:27017")
sh.shardCollection("app.orders", {tenantId: 1, createdAt: 1})

Traps: connection strings and DNS must change to point at mongos, and any client that keeps a direct shard connection bypasses routing. Users and roles live in the CSRS, so authentication must be set up there. Unsharded collections stay on the primary shard, so residency assumptions may break. And unique indexes must be prefixed by the shard key, which can invalidate an existing constraint you relied on.

Strong signals: a single shard-key value produces a chunk that cannot split and keeps reappearing as jumbo; the balancer reports balance in chunk counts while $shardedDataDistribution shows one shard holding most of the bytes; a growing share of critical queries plans as SHARD_MERGE because the dominant access pattern no longer includes the key; and write concentration on one chunk shows up as sustained latency for one tenant while the cluster looks idle.

Weaker but relevant: the key field is being deprecated in the schema, or a new product line groups data by a dimension that did not exist when the key was chosen. Before committing, quantify what the new key would fix and what it would break, since resharding moves the hotspot rather than eliminating it. If the new key would still be monotonic, or still route the main query badly, the work buys nothing.

There is no monolithic BEGIN on the server. startTransaction() is client-side bookkeeping; the first statement assigns a txnNumber and registers the transaction. From then on one node is the coordinator: the mongos the session is pinned to in a sharded cluster, or the primary in a plain replica set.

Commit is two-phase. Phase one, prepareTransaction goes to every participant: each shard takes its locks, persists a prepared record and refuses to rewind. Phase two, the coordinator durably writes the decision — commit or abort — into config.transactions, and only then tells participants to commit. That ordering is the guarantee: once the decision exists, the outcome is fixed, and a crash mid-broadcast is recoverable by replaying it. Participants hold their own record until the decision reaches them.

config.transactions is an ordinary replicated collection, and that is the design: the decision must survive failover like any other data. Each document is keyed by lsid plus txnNumber and carries the state (prepared, committed, aborted), lastWriteDate, and for prepared entries the oplog slot and write concern the coordinator committed with.

Records are not deleted at commit. A cleanup thread removes one only when no further message about that transaction can plausibly arrive — anchored to the majority commit point plus a grace period, tunable through transactionRecordMinimumLifetimeMinutes (30 minutes by default). That retention window is what makes a retried commit safe: the second commitTransaction finds the record, returns the stored outcome, and applies nothing twice. On a sharded cluster the coordinator's record lives with the config server, and each participant shard keeps its own.

Every transaction reads from one WiredTiger snapshot taken at its first read, and its writes become visible only at commit. That eliminates dirty reads, non-repeatable reads and phantoms inside the transaction — re-running the same query returns the same rows — and it eliminates lost updates on documents you actually write, because a concurrent modification of a document you hold produces a WriteConflict and aborts you.

What survives is write skew. Two transactions can read overlapping sets, each check an invariant, and write disjoint documents. Nothing conflicts, both commit, the invariant is broken. MongoDB never takes predicate or range locks, so it cannot notice that the two read sets intersected. Serializability is not offered, and no isolation level setting buys it. The engineering consequence: invariants must be materialized onto a document that both transactions must write.

Constraint: at least one engineer per team stays on call.

// T1 and T2 run concurrently, both read onCall: 2
const n = await coll.countDocuments({ team: "pay", onCall: true });
if (n > 1) {
  await coll.updateOne({ _id: myId }, { $set: { onCall: false } });
}

T1 takes Alice off, T2 takes Bob off. Different documents, so no conflict, both commit, nobody is on call.

Three honest fixes. Move the invariant onto a document both must write: a { team, onCallCount } row they $inc with $expr guarding the floor, so the second one gets WriteConflict and retries. Or hold an explicit lock document per team for the critical section. Or restate the rule so a unique index can enforce it. Retrying harder does nothing — the anomaly is logical, not a race the engine can see.

transactionLifetimeLimitSeconds defaults to 60. A background cleanup task, running on a fixed interval, finds transactions past their lifetime, aborts them and releases their locks and snapshot. In a sharded cluster the parameter must be set on every shard replica set member, not just on mongos, or shards disagree about who is still alive.

The client does not get a clean "timeout". The next statement on that session fails with NoSuchTransaction (code 251), which carries the TransientTransactionError label. A driver running the standard retry loop will therefore silently start the whole transaction over. That is correct for a transaction that was simply too big, and pathological when the transaction was slow because it was queueing behind a lock: the retry has the same footprint and will hit the same wall at t+60s, forever. Alert on repeated retries of the same txnNumber chain.

You cannot tell from the error, and that is precisely what UnknownTransactionCommitResult means: the commit may have been applied and the reply lost. Three defences, in order of value:

1. Retry the commit only — never the body — while the label is UnknownTransactionCommitResult, with w: "majority" and a bounded maxTimeMS. When the budget is exhausted you are in genuinely unknown territory.
2. Make the body idempotent through a natural key, so a blind replay cannot double-apply.
3. Read back with readConcern: majority on the key you wrote. That is the only authoritative answer.

Note the asymmetry: a TransientTransactionError arriving after a commit that already returned ok: 1 is noise — the commit reply is final.

The aggregate cap is gone. The server writes as many oplog entries as necessary to describe a transaction, so a 40 MB transaction is not rejected for total size the way pre-4.2 folklore claims. What remains is the per-entry BSON limit: each applyOps entry must stay under 16 MB, and the server splits the write set across entries to respect that.

Size still bites in three places. A single statement whose write set cannot fit one entry — a giant $push or a huge bulk update inside the transaction. The WiredTiger cache ceiling: a transaction too large to ever fit aborts with TransactionTooLargeForCache, which is a cache error wearing a size costume. And replication: secondaries replay those entries one at a time, so one enormous transaction becomes a catch-up burst on every member rather than a single atomic step.

The premise is stale. Since 4.4 you can create collections and indexes in a transaction, with hard conditions: the transaction must not be a cross-shard write transaction, and the collection must not be capped. So on a replica set it works; in a sharded cluster it works only while the transaction stays on one shard.

Cross-shard DDL is forbidden for good reasons. A metadata change has no document identity to conflict on, so there is nothing for the two-phase machinery to lock per participant; the catalog write would have to be visible to other sessions watching that namespace, which breaks the snapshot the transaction is built on; and a failing build would have to be rolled back on shards that already published the namespace. The practical rule stands: create the namespace first, with w: "majority", before you open the transaction that uses it, or the transaction can fail to acquire its lock.

WiredTiger is MVCC: it keeps old versions of changed documents so a reader holding an older snapshot can still find them. The transaction's snapshot pins that point in time. Any document the transaction may still read cannot have its obsolete versions discarded, and the pages holding them cannot be evicted freely.

So the cost is indirect but real. Updates made by other sessions accumulate in the history store for as long as your snapshot lives, and cache that would hold hot working-set pages is spent holding versions nobody will read again. Long transactions therefore degrade the whole node, not just themselves — you see rising cache pressure, more eviction, friendlier-looking latency for everyone else. That is why the docs tell you to abortTransaction() rather than simply abandon a session, and why the 60-second lifetime limit exists at all: it is a cache-protection mechanism before it is a fairness mechanism.

A replica set commit is one-voter: the primary writes the decision into its own config.transactions and the oplog does the rest. A distributed commit has independent failure domains, so mongos becomes an actor in its own right.

The coordinator — chosen at the first statement, and sticky for that transaction — drives prepareTransaction to each participating shard, so each holds its locks in a prepared state; persists the decision on the config server; then broadcasts commit and cleans up. It also owns recovery: if the mongos dies mid-commit, the decision already on the config server lets another mongos finish the job after the client retries commit. Two practical consequences follow. Every statement in a sharded transaction costs extra round trips, and losing the mongos between prepare and commit is a real latency spike, not a theoretical one. Also, all participating shards must be up in the same window — a shard down means the transaction cannot prepare.

It bounds how long an operation inside a transaction waits for a lock before giving up. Default is 5 milliseconds, and the failure surfaces as a write conflict carrying TransientTransactionError. Setting it to -1 means wait indefinitely, which is almost always wrong.

You raise it for one recognizable pattern: a fast metadata operation, a short index build or a create that momentarily holds a collection lock, aborts every in-flight transaction that touches the same namespace. Bumping the value to a few tens of milliseconds lets those transactions ride out the blip instead of retrying.

But be honest about the trade. This parameter only delays the abort; it does not resolve deadlock, and a large value turns a fast failure into a slow one, which is worse under load because lock queues grow. It also has to be at least as long as the in-progress transaction plus the pending DDL needs, otherwise transactions waiting behind the DDL abort anyway.

No. Uncommitted writes are private to their transaction: they are not in any snapshot another session can acquire, so a dirty read is structurally impossible rather than merely discouraged. Snapshot visibility is taken at commit.

What a transaction can do is read stale data, and this trips people up because it looks like the opposite failure. Because the snapshot is fixed at the first read, a write that commits outside your transaction after that point is invisible to you for the rest of the transaction. The documented version of this is stark: another session deletes a document, and your transaction still reads it happily. If a read must see the latest committed state, take a write lock with a findOneAndUpdate that touches the document — that both refreshes visibility and makes the conflict explicit.

Separately, readConcern: local outside a transaction can return data that is later rolled back. That is not a dirty read either; it is uncommitted-by-majority data.

Because MVCC cleanup is global. WiredTiger cannot discard an old document version while any active snapshot might still need it. One transaction holding a snapshot from ten minutes ago forces the engine to retain every version of every document touched since then, in the history store, on disk and in cache.

The symptoms look unrelated to transactions: cache fill ratio climbing, eviction pressure, reads that suddenly need disk, write latency creeping up as updates must chase older versions. Teams spend hours tuning indexes for a problem caused by an abandoned session in a debugger.

Operationally: always abortTransaction() in a finally block, keep transactionLifetimeLimitSeconds at a value you can defend, and treat "open transactions older than N seconds" as a first-class alert rather than a curiosity. The limit exists to cap this blast radius.

The case for it: if legitimate transactions genuinely need 90 seconds — a batch reconciliation, a report that aggregates then writes — then a 60-second abort is a coin flip that produces retry storms and duplicate work.

The case against, which usually wins: the limit is a proxy for cache protection, and raising it multiplies the history-retention cost described above across every concurrent transaction. Ten-minute transactions on a busy primary will show up as cache pressure and secondary lag, not as clean errors. Worse, a longer limit converts fast failures into slow ones: a transaction stuck behind a lock now occupies a session slot for ten minutes instead of one.

My answer in a design review: keep 60, and move the long work out of the transaction — precompute in a staging collection, then commit a small delta. If the parameter must move, change it on shards and coordinator together and re-measure cache pressure afterwards.

Three conditions converge. The updater holds a snapshot, and the target document's current version was committed after that snapshot. The engine walks the update chain for the key, finds a version newer than the reader's snapshot, and cannot legitimately apply the change without violating isolation. It returns WT_ROLLBACK, which mongod translates into code 112, WriteConflict.

Two nuances separate seniors from people who have read one blog post. First, the storage engine retries certain conflicts internally before surfacing one — so the error the application sees means repeated attempts already failed. Second, behaviour differs by context: inside a transaction the conflict aborts the transaction and is labelled TransientTransactionError; outside a transaction the server retries the operation itself a bounded number of times, which is why a non-transactional $inc rarely reports one. The document lock itself is short — conflicts come from snapshots, not from lock hold time.

WiredTiger admits only a fixed number of concurrent operations into the storage engine — the classic defaults are 128 read tickets and 128 write tickets, and the server exposes them under serverStatus().wiredTiger.concurrentTransactions with available, out and totalTickets.

Tickets are a throttle, not a lock: an operation that cannot get one queues in the server, invisible to the query plan. That is why a node can show low CPU, no lock waits worth mentioning, healthy disk, and still serve 200 ms reads — the time is spent waiting for admission. The signature is available pinned at zero with out at the ceiling during the slow window.

Do not "fix" this by raising the ticket count casually. More concurrency into a saturated cache or disk usually lowers throughput. Read it as evidence that you have more concurrent work than the node can service, then attack the real cause: too many connections, unindexed scans holding tickets longer, or hot documents serializing writes.

An operation that would otherwise hold a lock for a long time periodically releases it so others can make progress — the classic triggers are a page fault that forces a disk read, or a large number of documents modified in one statement. The engine must then verify nothing changed underneath before resuming; if it did, the operation can be rolled back and retried.

The observable cost: numYields in the profiler and in slow-query logs. A high yield count means the statement kept surrendering its position — typically a scan without a supporting index, or a mass update racing other writers.

The correctness lesson matters more than the metric. Because an update can yield mid-flight and be retried, an operation is atomic per document, but two operations on the same document are not ordered the way wall-clock logs suggest. Never infer ordering from application timestamps; use a monotonic counter or the server's own ordering to reason about "which write won".

They serialize, necessarily. WiredTiger takes an exclusive lock on the document (internally, on the update chain for that key) for the duration of the modification. The second updater blocks on that lock — but only briefly, because the critical section is short: append a new version, release.

The interesting part is what happens next. Once it acquires the lock, the second updater re-examines the document against its own snapshot. If the first write is now newer than that snapshot, the second cannot apply blindly, and it either retries with a fresh snapshot (non-transactional path, server-side) or aborts with WriteConflict (transactional path).

So "serialized" does not mean "both applied in order". One of them may fail and be retried, and an operation that read the document earlier in the same transaction will be aborted rather than allowed to write over newer data. That is the mechanism behind lost-update protection on a single document.

async function runTxn(fn, { attempts = 5 } = {}) {
  for (let i = 0; i < attempts; i++) {
    const session = client.startSession();
    try {
      let out;
      await session.withTransaction(async () => {
        out = await fn(session);
      }, {
        readConcern: { level: "snapshot" },
        writeConcern: { w: "majority" },
        maxCommitTimeMS: 5000
      });
      return out;
    } catch (e) {
      if (!e.hasErrorLabel("TransientTransactionError")) throw e;
      const base = Math.min(2 ** i * 50, 1000);
      await sleep(base / 2 + Math.random() * base); // jitter
    } finally {
      await session.endSession();
    }
  }
  throw new Error("transaction retry budget exhausted");
}

Why each piece: withTransaction retries the commit on UnknownTransactionCommitResult for you, so your loop only handles the body. Jitter prevents every replica of your service retrying in lockstep and re-creating the same conflict storm. A bounded attempt count stops the pathological case where the transaction is simply too contended. A fresh session per attempt avoids inheriting a poisoned transaction state.

Trust the labels, not the message. TransientTransactionError means "the world moved, the same transaction could work now": write conflicts, lock timeouts, NoSuchTransaction, a stepdown during the transaction, TransactionTooLargeForCache when contention inflated the working set. UnknownTransactionCommitResult means "commit outcome indeterminate" and only the commit is retried. RetryableWriteError is the non-transactional sibling.

Everything else is logic: DuplicateKey from a real duplicate, schema validation failures, NamespaceNotFound, BSONObjectTooLarge, "cannot use $expr here", anything you can reproduce deterministically. Retrying those is a latency amplifier that hides the defect.

The trap is a transaction that always conflicts under some traffic shape: it is labelled transient, so the loop spins, and you get 500s only after the budget drains. Log the attempt count. A transaction that needs three retries is a design smell, not bad luck.

You invent a natural key from the request, and you let a unique index enforce it. The idempotency key should be stable across retries and distinctive per logical operation — derived from something the caller supplies, not from a timestamp or a random UUID generated inside the retry block, which would defeat the whole exercise.

// unique index: { accountId: 1, requestId: 1 }
try {
  const r = await coll.insertOne({ accountId, requestId, amount, status: "posted" });
  return { created: true, id: r.insertedId };
} catch (e) {
  if (e.code !== 11000) throw e;
  return { created: false, doc: await coll.findOne({ accountId, requestId }) };
}

The duplicate key error becomes the success path for a replay. Two details: the index is the mechanism, not a hint — without it you get two documents; and the document must be written in the same statement as the state it describes, otherwise a crash between operations leaves the key recorded and the work undone. For multi-document effects, the key belongs inside the transaction.

The race is that upsert is not omniscient. The server matches the filter, finds nothing, and inserts. Two concurrent upserts on the same filter can both observe "no match" before either insert lands. Without a constraint on the filter fields, you get two documents and no error — the worst outcome, because it looks like success.

With a unique index on the fields the filter matches, one insert wins and the loser gets DuplicateKey (11000). That error is not a bug, it is the coordination primitive: catch it and retry the same upsert, which now takes the update path. That is the documented, safe pattern, and it is why "use upsert" without an index is bad advice.

Note the subtlety inside transactions: a duplicate key error inside a multi-document transaction kills the transaction, so you must retry the whole transaction, not just the statement. And the filter must match the index exactly, or the constraint does not apply.

const now = new Date();
const lease = new Date(now.getTime() + 30_000);

const doc = await locks.findOneAndUpdate(
  { _id: "invoice:42", $or: [ { owner: null }, { expiresAt: { $lt: now } } ] },
  { $set: { owner: workerId, expiresAt: lease, fence: { $add: ["$fence", 1] } } },
  { returnDocument: "after", upsert: false }
);
if (!doc) throw new Error("lock held by someone else");

The filter must express ownership and expiry in the same predicate — that single atomic compare-and-set is the entire lock. returnDocument: "after" gives you the lease you actually hold rather than the one you hoped for. The fence increment is what makes the lock usable: downstream writes must carry the fence value they were granted, so a worker that lost its lease cannot overwrite the new owner's work.

Keep upsert: false. An upsert here would happily create a second lock document under a different filter shape, or resurrect a deleted one. Provision the lock rows in a migration instead.

Because it inherits every failure mode of a lease, and adds a few of its own. Three killers:

• Process pause. A GC pause, a VM freeze, a SIGSTOP. The worker wakes up believing it holds a lease that expired two seconds ago and writes over the new owner's work.
• Clocks. Lease expiry compares timestamps. Skewed client clocks make a holder expire early or never. Never send new Date() from the app for expiry arithmetic; use $$NOW or $currentDate so the server's clock is authoritative.
• TTL is not immediate. A TTL index deletes on a background sweep, roughly every 60 seconds, so "the lock expires at T" is really "somewhere after T".

The mitigation is fencing, not tighter timing: monotonically increasing tokens that the resource itself validates, rejecting stale writers. If the destination cannot validate a fence, MongoDB is the wrong lock. Use a system that offers consensus semantics, or redesign so the operation is idempotent and the lock is only an optimization.

The application reads, computes, writes back:

const acc = await coll.findOne({ _id: "A" });   // balance 100
// ... 40 ms of business logic, another writer deposits 50 meanwhile
await coll.updateOne({ _id: "A" }, { $set: { balance: acc.balance - 30 } });
// balance becomes 70, the +50 is gone

The $set carries a value computed from a stale read, so it overwrites whatever happened in between. The document was never concurrently written while the update ran — there was no conflict to detect. That is why this bug survives load testing.

Fix one, express the delta: { $inc: { balance: -30 } } — server-side arithmetic on the current value, atomic, no stale input.

Fix two, guard the write with $expr so it only applies when the precondition still holds: { $expr: { $gte: ["$balance", 30] } } with $inc. Check matchedCount to learn whether the guard held. Both keep the decision on the server, which is the actual fix.

const res = await coll.updateOne(
  { _id: id, version: expected },
  { $set: { status: next }, $inc: { version: 1 } },
  { writeConcern: { w: "majority" } }
);

if (res.matchedCount === 0) {
  const exists = await coll.countDocuments({ _id: id }, { limit: 1 });
  return exists ? 409 : 404;          // conflict vs missing
}

The version lives in the same document as the state it protects, so the check and the write are one atomic operation. Increment always, never set the version to a literal — that reintroduces the race you are removing.

The follow-up read is honest only because "not found" is stable: a document that exists cannot spontaneously become the wrong version for long, though it can be deleted between the two calls. If that distinction matters, wrap both in a transaction with readConcern: snapshot. Also note matchedCount vs modifiedCount: a no-op update can match without modifying, so branch on matched.

Because you are trying to make a database commit and a message publication agree, and they are different systems with independent failure modes. Write the order, then insert the outbox row, and a crash between them loses the event. Insert the outbox row first and a rolled-back order emits a phantom event.

await session.withTransaction(async () => {
  await orders.insertOne({ _id: orderId, ... }, { session });
  await outbox.insertOne({
    _id: new ObjectId(), topic: "order.created", payload: { orderId },
    createdAt: new Date(), status: "pending"
  }, { session });
}, { writeConcern: { w: "majority" } });

One transaction, one commit point: either both documents exist or neither does, and the relay can never publish something that did not happen. This is the one place where a transaction buys you something a change stream cannot — a change stream on orders has no way to know a second write was supposed to accompany it.

A saga is right when the work spans services or genuinely cannot fit in one transaction's time and cache budget: charging a card at a third party, provisioning in another system, migrating ten thousand documents. Holding a MongoDB transaction open across a network call to a payment provider is the anti-pattern that produces 60-second aborts and locks held for no reason.

The price is that you no longer have atomicity, you have compensations, and every one of them must be designed, tested and idempotent. Concretely you owe: a persisted saga log with a state per step, forward recovery that resumes from the log after a crash, compensating actions for each completed step (a refund, not a "delete the charge"), and a monotonic guard so a compensation cannot fire after the saga already succeeded.

Also decide the semantics out loud. Between steps the system is visibly inconsistent; someone will query it. Either hide that state from readers or make it legitimate in the model.

Very little on its own, which is why the name misleads people. A session is a server-side identity — the lsid UUID — that lets the server correlate operations from one client over time, across connections and across failover. That correlation is the substrate for everything else: retryable writes, causal consistency, transactions.

Without opting into causal consistency, a session guarantees no ordering across members. It does not pin you to a server, it does not hold a snapshot, and it does not create a transaction. What it does give you is a stable handle the driver can re-send operations under so the server can recognise a retry.

The lifetime is worth memorising: sessions expire server-side after logicalSessionTimeoutMinutes, 30 minutes by default, and the driver refreshes activity on use. A session idle for longer is silently forgotten — this is why a long-parked session's txnNumber history no longer protects you.

The mechanism is a timestamp, not a routing trick. Every write returns an operationTime. In a causally consistent session the driver remembers the highest one seen and attaches it to subsequent reads as readConcern: { level: "majority", afterClusterTime: <ts> }.

The receiving member then refuses to answer until its applied oplog timestamp reaches that value. A secondary that is 300 ms behind will wait 300 ms, not return stale data. That is the whole guarantee, and it is a floor, not a snapshot: you get "at least as new as what I wrote", not "a consistent point in time across all my reads" — that needs snapshot.

const s = client.startSession({ causalConsistency: true });
await coll.insertOne({ _id: 1 }, { session: s });
await coll.findOne({ _id: 1 }, { session: s }); // waits for the write

Cost: you have traded a stale read for a blocking wait, so a lagging secondary converts into tail latency. Alert on it.

Two channels, and conflating them causes real bugs. First, operationTime: a per-operation value in the reply that the driver advances its session watermark with. Second, $clusterTime: a cluster-wide gossip value that members exchange in every reply and every heartbeat, so time-like knowledge propagates even without client traffic.

The driver's rule is simple — take the max it has seen and send it on the next read as afterClusterTime. It never needs to reason about which node served what.

What breaks causality in practice: doing part of the work outside the session. If one statement runs on a session-less implicit session, its timestamp never enters the watermark, and the following read can legally return a state that predates it. The same applies when you hand a write's result to a different session or a different client instance. Causal consistency is a chain, and every un-sessioned call is a broken link.

Every operation you issue without a session still gets one — the driver creates an implicit session, uses it for that single operation (or that cursor's lifetime), and returns it to a pool. That is what makes retryable writes work by default: without an lsid and txnNumber, the server could not recognise a duplicate.

It bites in three places. Cursors: an implicit session is attached to the cursor, and if the cursor lives long, the session is held, which matters on servers with a low session pool. Read-your-own-writes: implicit sessions are independent, so operation two has no idea what operation one wrote — you need one explicit session spanning both. And transactions: a transaction requires an explicit session, full stop.

Explicit sessions are cheap, so the senior habit is to pass one session through a request's unit of work rather than sprinkling session-less calls and hoping causality sorts itself out.

endSession() returns the session to the client-side pool and, on client close, the driver sends endSessions to the server so records can be reclaimed promptly. It does not abort an open transaction; if you end a session with a live transaction, the server eventually expires it, but until then it holds locks and a snapshot.

If you never call it: nothing dramatic at small scale, because sessions expire after logicalSessionTimeoutMinutes (30 by default) and the server reclaims them. At scale, the failure is a session-pool squeeze — leaked sessions per request exhaust the pool, and new operations block waiting for one, which presents as unexplained latency under concurrency.

Two habits: always endSession() in a finally block, and prefer withSession/withTransaction helpers that do it for you. If you ever lower logicalSessionTimeoutMinutes, remember the driver caches session state for that window — a mismatch between driver and server values causes retryable writes to silently stop being retried.

Retryable by the driver: single-document writes — insertOne, updateOne, replaceOne, deleteOne, findOneAndUpdate and friends, plus insertMany and bulk writes with ordered: false. Not retryable: multi-document updates and deletes, and ordered: true bulk inserts, because the driver cannot prove partial application is safe.

Errors: anything the server labels RetryableWriteError — primary stepdown, NotWritablePrimary, InterruptedDueToReplStateChange, shutdown in progress — plus network errors where the write may or may not have arrived. A WriteConcernError is not a reason to re-send the write; the write landed, only the acknowledgement is short.

The trap: a retry can turn a timeout into a duplicate if the operation was not idempotent. Retries protect you from transport, not from design. And a retryable write still needs w: > 0, a replica set, and a server that has not expired the session record.

Retry safety needs an acknowledgement. With w: 0 the driver fires and forgets: there is no reply carrying the lsid/txnNumber outcome, so nothing distinguishes "the server applied it and the reply was lost" from "the socket died before it arrived". Re-sending would risk a duplicate, so the driver refuses to retry at all — and the server cannot help, because the whole dedup mechanism is keyed on a statement it already recorded.

Consequences people miss: w: 0 also silently disables transaction support (a transaction needs an acknowledged commit), and it gives you no operationTime, which breaks causal consistency on that session.

Where it is legitimate: high-volume telemetry or metrics where losing a sample is acceptable and latency dominates. Even then, ask whether w: 1 with retries is not simply the better trade — the failure mode of w: 0 is silent data loss, which nobody ever alerts on.

Not if the driver retried it as a retryable write. The server records the lsid plus txnNumber of the last applied statement, and a repeat of the same pair returns the cached result without re-executing. So $inc: { count: 1 } stays at one increment even though the command went over the wire twice. This is a genuinely non-idempotent operation made safe by bookkeeping.

It breaks in exactly these cases: the driver has retryable writes disabled (retryWrites=false in the URI), the deployment is a standalone, w: 0 was used, the operation is a multi-document update, or your code retried — an application-level catch that re-issues the updateOne gets a fresh txnNumber and does double-count.

So: $inc plus driver retries is safe; $inc plus your own retry loop is a double-count waiting for a slow network. Wrap app-level retries in a documented key or a transaction.

withTransaction maintains one commit attempt loop. If commitTransaction fails with UnknownTransactionCommitResult, it re-sends only the commit — the body is never re-run, because the body may already have been applied. It retries until the error stops carrying that label.

Practical settings worth setting explicitly: writeConcern: { w: "majority" } on the transaction so a commit that returns really is durable on a majority, and maxCommitTimeMS so a commit that is stuck behind replication lag cannot hang the request thread indefinitely. A commit that times out is not a failure — it is an unknown, which is a third state your code must handle.

The senior point: the retry is safe because the server remembers the decision in config.transactions. If you "helpfully" add your own commit retry around the driver's, you can interleave commits from two code paths on one session, which is the kind of bug that only appears during a failover test.

lsid identifies the session; txnNumber is a monotonically increasing counter per session, allocated by the driver, distinguishing one logical retryable unit from the next. For retryable writes the server also tracks a statement id inside the transaction number, so a multi-statement retryable sequence is deduplicated statement by statement.

For single writes, the memory is the last applied txnNumber per session — cheap. For transactions, it is the record in config.transactions described earlier, which is why a transaction can be recovered after a stepdown and a plain write cannot be resumed mid-flight.

Retention is bounded by session lifetime (30 minutes idle) and by transaction-record cleanup. Both matter to you: a client that pauses longer than the session timeout and then replays a write is issuing a new write as far as the server is concerned. Any idempotency guarantee beyond that window has to come from your own unique key, not from the driver.

It converts replication lag into client latency. A read carrying afterClusterTime must wait until the serving member has applied through that timestamp. Write in Frankfurt, read in Singapore, and the read blocks for one inter-region round trip plus whatever the primary needs to replicate — not because the data is unavailable, but because causality says "not yet".

It is unacceptable on user-facing paths where the reading region is far from the writing region and the value written is the only thing that matters. The alternative is not "stale reads"; it is scoping the dependency: keep writes and their immediate reads in the same region, use a local cache keyed to the write, or move the read to the primary for the short window after a write.

What you should never do is enable causal consistency globally because it sounds safer. It is a per-session contract, so enable it on the sessions that own a read-after-write dependency, and leave analytics and listings on plain secondary reads with a bounded maxStalenessSeconds.

The secondary may not have applied the write yet, so you read the old value. With a 200 ms replication lag and an immediate read, this is not a rare edge — it is the normal case during any write burst.

Four fixes, in increasing cost. Use readPreference: primary for the read-after-write path — correct and boring, and the right default for anything user-facing that immediately confirms an action. Use a causally consistent session so the read waits for the required timestamp — correct but turns lag into latency. Use readConcern: majority without causality — helps only if the write itself was majority-committed and you are willing to read from a member that has reached the commit point. Or restructure so the client already has the value it wrote and does not need to read at all.

The wrong answer is hoping. Catching the user's "I saved it and it's gone" ticket after launch is far more expensive than routing one query to the primary.

majority promises you only see data that has been acknowledged by a majority of voting members, so it cannot be rolled back later. The mechanism is the majority commit point: the highest oplog timestamp that a majority has durably written. A read with this concern waits for the serving member's applied timestamp to reach that point before answering.

Consequences worth stating in an interview. It prevents dirty reads from the durability side — a write that could still be rolled back by an election is invisible. It costs latency equal to however far the commit point trails the primary, which is usually small and occasionally not. It does not give you a consistent snapshot across several reads; each read may see a different commit point unless you use snapshot or a transaction.

Also note majority and w: "majority" are opposite sides of one coin: a read threshold and a write acknowledgement. Mixing w: 1 writes with majority reads is fine, and buys less than you think.

Yes, on find, aggregate and distinct, since 5.0. The point is repeatability across a single operation, and the ability to pin a read to a specific point in time with atClusterTime — which is how you take a coherent multi-collection read without opening a transaction: read once, capture atClusterTime from the cursor, then read the second collection at that same timestamp.

const c1 = db.orders.find({}, { readConcern: { level: "snapshot" } });
// capture the timestamp, then reuse it
db.orderLines.find({}, { readConcern: { level: "snapshot", atClusterTime: ts } })

The catch: it is not a transaction. Each statement sees its own snapshot unless you pass atClusterTime, there is no isolation from concurrent writes appearing between them in a way you can lock against, and you cannot write. And the same MVCC cost applies — a long-lived snapshot for a big aggregation pins history just as a transaction would. Use atClusterTime deliberately, and keep the window short.

Because $inc removes the round trip that creates the race. The application never carries a value into the write; the server applies an arithmetic delta to the current version of the document under the document lock. Nothing can be lost, because no stale number ever leaves the server.

Two further effects explain the throughput difference. First, the update is in place in the storage engine — WiredTiger appends a new version of one small document rather than rewriting a large one, so the critical section is short and the write path stays compact. Second, a single-document update is atomic and retryable, so the driver can absorb a stepdown without the application re-reading.

What $inc does not fix is contention: every increment to the same document still serializes on that document. Throughput plateaus at whatever one hot document can absorb, and past that you see WriteConflict under transactions or a lock queue without them. That is a schema problem, not an operator problem — which is why hot counters get sharded.

Split the counter into N documents and add them up when you read. Writes pick a bucket pseudo-randomly, so contention is divided by N and the hot document disappears.

// N bucket docs: { _id: { c: "page:42", b: 0..N-1 }, n: NumberLong(0) }
await buckets.updateOne(
  { _id: { c: "page:42", b: Math.floor(Math.random() * N) } },
  { $inc: { n: 1 } }
);

const [ { total } ] = await buckets.aggregate([
  { $match: { "_id.c": "page:42" } },
  { $group: { _id: null, total: { $sum: "$n" } } }
]);

The trade-offs are real: a read now costs an index range scan plus a group instead of one lookup, the read is only eventually consistent unless you wrap it in a snapshot read, and you pay N times the storage for many counters. Choose N by measuring per-document write throughput, not by taste — and never delete a bucket, or the total silently drops.

Because the comparison that decides expiry must use one clock. If the app writes expiresAt: new Date(Date.now() + 30000), that value came from the application host, and the next claim compares it against another host's clock. Two machines 400 ms apart is enough for workers to disagree about who holds a lease — and container clocks drift after suspend far more than that.

const job = await jobs.findOneAndUpdate(
  { _id: jobId, status: "pending" },
  { $currentDate: { startedAt: true }, $set: { status: "running", worker: id, leaseUntil: { $add: ["$$NOW", 30000] } } },
  { returnDocument: "after" }
);

$currentDate and $$NOW are evaluated server-side at execution, so every writer and every comparison shares the primary's clock. It also removes a subtle bug: with a client-supplied timestamp, a retried operation rewrites "now" as the old now, and the document records a start time that never happened.

One atomic find-and-modify on a predicate that encodes the precondition. The second worker's filter no longer matches, so it gets null and moves on.

const job = await jobs.findOneAndUpdate(
  { status: "pending", runAt: { $lte: "$$NOW" } },
  { $set: { status: "running", worker: workerId },
    $currentDate: { startedAt: true },
    $inc: { attempts: 1 } },
  { sort: { priority: -1, runAt: 1 }, returnDocument: "after" }
);
if (!job) return; // nothing to do

The critical property is that match-and-modify is a single document-level atomic operation. A find followed by an updateOne has a window in which both workers read the same pending job — that pattern is the bug, not a detail.

Two supporting pieces: the worker must be idempotent, because a crash after claiming means the job runs again on lease expiry; and the claim predicate must include every condition the handler depends on, or you will dispatch work whose dependencies changed.

In one document, one statement. Keep per-SKU availability in a single document and make the decrement conditional on the condition that must hold:

const res = await stock.updateOne(
  { sku: "A-1", available: { $gte: 3 } },
  { $inc: { available: -3, reserved: 3 } },
  { writeConcern: { w: "majority" } }
);
if (res.matchedCount === 0) throw new Error("insufficient stock");

The filter carries the invariant, so the check and the decrement cannot be separated by another writer. No transaction is needed for the reservation itself — and that matters, because a transaction here would just add the same document lock with more overhead.

Where you do need a transaction: reserving several SKUs at once, where partial success would strand inventory. Then all the decrements go in one transaction with w: "majority", and a conflict on any document aborts the whole set cleanly.

Also model release explicitly: reserved plus available sums to a constant you can assert on, so a leaked reservation is detectable instead of invisible.

$expr lets the update predicate compare two fields within the document, which a normal filter cannot do:

await items.updateOne(
  { sku, $expr: { $gte: ["$stock", "$requested"] } },
  { $inc: { stock: -1 } }
);

The obvious filter form, { stock: { $gte: requested } }, requires the app to read requested first — and that read is the race. $expr evaluates server-side against the current version, so the decision is atomic with the write.

The cost is index support. An $expr comparing two fields is generally not indexable, so the predicate is evaluated per candidate document after the index narrows what it can. Always pair it with an indexed equality on the same filter (sku above) so you scan one document, not a collection. If you check explain and see a broad scan, the fix is a different filter, not a different expression. And verify matchedCount, since zero matches here is a business outcome, not an error.

The resource exposes an ETag derived from the document's version — a strong ETag built from _id and version, not a hash of the representation, which changes for cosmetic reasons. GET returns it; the client sends If-Match: "<etag>" on PUT or PATCH.

const r = await doc.updateOne(
  { _id: id, version: etagVersion },
  { $set: patch, $inc: { version: 1 } }
);
if (r.matchedCount === 0) return res.sendStatus(412);  // Precondition Failed

Common mistakes. Returning 200 on a lost race instead of 412, so the client believes it won. Accepting a missing If-Match and silently doing a blind write — that is the lost update coming back through the front door. Computing the ETag from a timestamp with second granularity, so two edits in the same second share a version. And forgetting that the response must carry the new ETag, otherwise every client retries with a stale one. Return 428 if you want to require the header explicitly.

The unique index is the deduplication — not a findOne first, which reintroduces the race you are trying to close.

// unique index on { eventId: 1 }
try {
  await inbox.insertOne({ eventId, type, payload, receivedAt: new Date(), state: "new" });
} catch (e) {
  if (e.code === 11000) return { duplicate: true };  // already handled or in flight
  throw e;
}
// then apply the effect, then mark state: "applied"

Now the honest part: the insert only reserves the id. If you crash between reserving and applying, a redelivery hits the duplicate path and the effect never happens. So the row needs a state, and redelivery of a "new" row must attempt the effect rather than return success — the transition new → applied must itself be a conditional update so only one worker performs it.

For multi-collection effects, put the inbox insert and the effect in one transaction. Then duplicate detection and application commit together.

Store amounts as Decimal128, not double — binary floating point cannot represent 0.10. Integers in minor units are also fine; pick one and enforce it with schema validation, because mixed representations are worse than either.

await session.withTransaction(async () => {
  await accounts.updateOne({ _id: from, balance: { $gte: amt } },
    { $inc: { balance: NumberDecimal(amt).negate() } }, { session });
  await accounts.updateOne({ _id: to },
    { $inc: { balance: NumberDecimal(amt) } }, { session });
  await ledger.insertOne({ from, to, amt: NumberDecimal(amt), at: new Date() }, { session });
}, { writeConcern: { w: "majority" } });

The transaction gives you all-or-nothing across the two accounts and the ledger. It does not give you idempotency: a retried transfer that bypasses the driver's dedup moves the money twice. Add a unique transferId written inside the same transaction. And it does not give you overdraft protection unless the $gte guard is inside the transaction — a check done before the transaction is a check against stale data.

Write path: one transaction, so the header and its lines commit together. Read path: never require a transaction. That asymmetry is the whole design.

You can go further and make the invariant structural. If lines always sum to a total stored on the header, the transaction can assert it, and any reader that trusts the header is consistent by construction. Writes that append a line must then update the header in the same transaction — which is fine, because they are rare compared to reads.

Where people hurt themselves: putting the lines in an unbounded array so the header document grows toward 16 MB and every line write rewrites a large document, converting a cheap append into a contended write. Keep lines as their own collection with orderId indexed, and accept that a reader needing both sides does two queries with a bounded window of inconsistency — or one $lookup if the access pattern demands it, never one transaction per page view.

Atomic: the tombstone on the document and every side effect that must agree with it — revoking a shared grant, removing the item from a saved list, releasing a reserved SKU. If any of those can happen without the others, you get an object that is deleted for its owner and live for everyone else.

await session.withTransaction(async () => {
  await docs.updateOne({ _id, deletedAt: null },
    { $set: { deletedAt: new Date(), deletedBy: actor } }, { session });
  await shares.deleteMany({ docId: _id }, { session });
  await reservations.updateMany({ docId: _id, state: "held" },
    { $set: { state: "released" } }, { session });
});

Two things it does not solve. Queries: every read path must filter deletedAt: null, and a transaction cannot enforce that — use a partial index and a validator or a repository layer that refuses unfiltered access. Growth: soft-deleted documents still occupy indexes and the working set, so a tombstone without an archival job is a slow-motion capacity problem, not a delete.

Because atomicity in MongoDB is per document per operation. The unit of isolation is one update statement, and your business logic spans three statements separated by network time.

const d = await c.findOne({ _id });          // t0
const next = compute(d);                      // t1  -- CPU, GC, HTTP call
await c.updateOne({ _id }, { $set: { v: next } }); // t2  -- overwrites t0..t2

Anything committed between t0 and t2 is erased by the $set, and no error is raised, because at t2 there was no concurrent writer holding the document — the conflict window closed long before. Replica sets do not help; there is nothing to detect.

The three cures, in order of preference: express the change as a delta the server can apply ($inc, $push, $addToSet, $max); express the precondition in the filter ($expr, version field) and treat zero matches as a conflict; or, when the logic genuinely needs a read, wrap it in a transaction so the snapshot aborts you on conflict.

A bulk write is a batching convenience, not an atomic unit: each operation applies independently, and a failure leaves everything before it applied.

ordered: true stops at the first error; ordered: false continues and reports every failure in writeErrors. Choose by what the batch means. Independent upserts from a sync job: unordered maximises throughput, reconcile errors afterwards. A sequence where later operations depend on earlier ones — parent then children — needs ordered, and even then you own the cleanup.

Ordering rules that keep failures recoverable: put the operations that establish identity first, destructive deletes last, and never mix a delete of a document with an insert that reuses its unique key in the same batch, because order then decides whether you get a duplicate key error.

Always inspect result.writeErrors and the difference between insertedCount and what you sent. A bulk write that "succeeded" with three errors is the classic silent data gap. If the batch must be all-or-nothing, it belongs inside a transaction — and then it must be small.

When the cost of strong consistency exceeds its value to the reader. Feeds, counts, dashboards, search indexes, recommendations, audit trails consumed by humans hours later — all fine. Money movement, inventory, uniqueness, and anything a user is about to act on — not fine.

Making it safe is mostly about honesty in the product and the schema, not the database. Three practices that matter:

• Bound it. A lag figure you can state and alert on, not "eventually". maxStalenessSeconds turns an unbounded window into a contract.
• Make the read model self-describing. Return the as-of timestamp or version so a UI can say "as of 3 minutes ago" instead of implying real time.
• Never let a stale read feed a decision. Every write path re-reads authoritative data inside its own atomic operation; only display paths get the replica.

The failure mode to avoid is accidental eventual consistency: nobody chose it, one query just happened to hit a secondary, and the bug shows up as an intermittent "my change disappeared" report.

Persist the intent before the effect, and make every transition a conditional update on the state you expect. The document is the state machine; the process is disposable.

// 1. claim the transition atomically
const claimed = await ops.findOneAndUpdate(
  { _id: opId, state: "pending" },
  { $set: { state: "applying", worker: id }, $currentDate: { claimedAt: true } },
  { returnDocument: "after" }
);
if (!claimed) return;               // taken, or already done

// 2. do the external effect with an idempotency key = opId
await gateway.charge(opId, claimed.amount);

// 3. record completion
await ops.updateOne({ _id: opId, state: "applying" }, { $set: { state: "applied" } });

The invariant is that the external call is idempotent under opId, so a crash between 2 and 3 is recoverable: a recovery sweep finds applying rows with a stale claimedAt, re-issues the call, and the gateway deduplicates. Without that key, step 2 becomes the money-losing step.

Note what is absent: no long transaction held across the network call. That is the point of the pattern.

Do not read secs_running alone; a healthy batch transaction can legitimately be 20 seconds old. Read the shape:

• transaction.parameters.txnNumber and lsid — the identity you will need to kill it.
• transaction.timeOpenMicros — age of the transaction, distinct from the age of the current statement.
• transaction.timeInactiveMicros — the discriminator. High and growing means the session is idle with an open transaction: nobody is doing work, the locks and snapshot are just held.
• locks and waitingForLock — whether it is a victim or a blocker.
• numYields, planSummary — whether the current statement is scanning.

A transaction with timeInactiveMicros climbing past tens of seconds is almost always an application bug: a session leaked, or an exception path with no abortTransaction. Correlate with clientMetadata and appName to find the offender, and check prepareReadConflicts before concluding it is harmless.

db.getSiblingDB("admin").aggregate([
  { $currentOp: { allUsers: true, idleSessions: true } },
  { $match: { transaction: { $exists: true } } },
  { $project: {
      opid: 1, appName: 1, client: 1, lsid: 1,
      secs: { $divide: ["$transaction.timeOpenMicros", 1000000] },
      idle: { $divide: ["$transaction.timeInactiveMicros", 1000000] },
      state: "$transaction.parameters.state", txnNumber: "$transaction.parameters.txnNumber"
  } },
  { $match: { secs: { $gt: 30 } } },
  { $sort: { secs: -1 } }
])

idleSessions: true is the essential flag — without it, sessions that are idle but holding an open transaction are omitted, which is exactly the population you are hunting. allUsers: true is required to see other users' operations, which means the admin privilege to run it.

transaction.parameters.state distinguishes inactive from prepared; a lingering prepared state usually points at a coordinator that died between phases. Sort by age, then look at appName — the same offender repeating is a code defect, not a one-off.

Two levers, and they are not interchangeable. db.killOp(opid) kills the current operation; for a transaction you also want the session gone, so prefer killing the session:

db.adminCommand({ killSessions: [ { id: UUID("..."), txnNumber: NumberLong(7) } ] })
db.adminCommand({ killAllSessionsByPattern: [ { "lsid.id": UUID("...") } ] })

Killing the session aborts any open transaction and releases its locks and snapshot — which is what you actually needed. Killing just the opid can leave the transaction open, and the next statement re-establishes it.

Client-side: the in-flight statement fails, and subsequent work on that session gets NoSuchTransaction with TransientTransactionError, so a well-behaved driver retries from the top. That is fine. What is not fine is killing sessions as routine operations — you have just converted your problem into the client's retry storm. Kill, then fix the leak. Log the lsid before killing so you can prove which code path it was.

Three families, and they only mean something in combination.

• serverStatus().transactions — totalStarted, totalCommitted, totalAborted, plus the gauges currentOpen, currentActive, currentInactive, currentPrepared.
• WiredTiger transaction statistics, including the write-conflict counter, which is the direct measure of two writers colliding on the same document.
• Lock and queueing signals — globalLock.currentQueue, and the ticket gauges from wiredTiger.concurrentTransactions.

Read them as a ratio, not absolutes. totalAborted / totalStarted climbing is the number that predicts an outage: it means work is being done and thrown away. Rising currentInactive alongside a flat commit rate points at leaked sessions rather than contention. And a high conflict counter with low abort count usually means lock waits are being absorbed by the server's internal retry — latency is rising before errors appear. Use a derivative over a window; a single sample tells you nothing.

You cannot see the byte size of an individual in-flight transaction as a first-class metric, so you infer it. The leading indicators are the per-transaction gauges plus cache behaviour: currentOpen and currentActive rising while totalCommitted stays flat means transactions are accumulating, not flowing. On the WiredTiger side, watch cache fill and the bytes-dirty track — a growing transaction pins versions and dirties pages, so the store climbs without a matching increase in application write volume.

The hard ceiling is transactionSizeLimitBytes, derived from the cache size, and the error when you exceed it is TransactionTooLargeForCache. Two readings of that error matter. If it fires on a transaction whose size is intrinsic — a bulk migration — it is a design signal: chunk the work. If it fires only under load, contention has inflated the working set and the fix is retry with backoff plus a smaller write set.

The cheap preventative: log the document count and payload size each transaction touches, and alert when p99 crosses your own budget.

The gauges describe a population of live transactions; the counters describe throughput. currentInactive is the one I look at first, because it is the pathological state: a session that opened a transaction and walked away. It holds locks and a snapshot while doing nothing, so it costs you cache and blocks writers without contributing throughput.

Then the ratio: totalAborted against totalStarted. Some aborts are normal — optimistic patterns abort and retry by design. A ratio that climbs from 2% to 30% over an hour is contention that your retry loop is hiding from the error rate but not from the latency graph.

Finally currentPrepared sitting above zero for more than a moment. Prepared transactions are a transient state between the two phases; a stable non-zero value means coordinators died mid-flight or a shard is unreachable, and those transactions are holding locks indefinitely. That is a page-someone condition, not a dashboard curiosity.

The log line for a transaction carries the session identity and the transaction context: lsid, txnNumber, and statement-level fields, plus the usual planSummary, keysExamined, docsExamined, numYields and durationMillis. That identity is the difference between guessing and knowing, because you can join the log line to the application log line that emitted the same request id.

slowms still governs what gets written, so the trap is thresholding by statement duration: each statement in the transaction can be 5 ms while the transaction takes 400 ms, and nothing is logged. For transactions, log by transaction duration in the application, not by statement duration in the server.

Also watch slowms being lowered during an incident: on a busy primary it is self-defeating, because the log volume itself becomes the bottleneck. Sample instead — log a fraction of the slow path, or use the profiler at level 2 into a dedicated collection for a bounded window.

Each profiled entry has the usual op, ns, command, planSummary, keysExamined, docsExamined, nreturned, millis — plus lsid and txnNumber for anything inside a transaction, which is how you filter:

db.system.profile.find({ lsid: { $exists: true }, millis: { $gt: 50 } })
  .sort({ ts: -1 }).limit(20)

You can then group by lsid and sum millis to reconstruct a transaction's real cost, which is the analysis the per-statement view hides.

What it cannot tell you: whether the transaction committed or aborted. There is no single entry representing the transaction as a unit, and aborted work is profiled like any other. That has two consequences — wasted work looks identical to useful work, so you cannot measure your conflict rate from the profiler, and the audit trail cannot prove an outcome. For those, use serverStatus().transactions and application-side instrumentation. Also remember the profiler writes into a capped collection, so a hot window evicts its own evidence.

Not in the general case. explain() is not part of the command set you can interleave into an open multi-document transaction; the driver will run it as its own operation, so what you get is a plan for the query shape, not for the transaction.

So you get the plan three ways and cross-check. Run explain("executionStats") outside the transaction with the identical filter, projection and sort, and read totalKeysExamined against nReturned. Reproduce the transaction on a staging copy with the same documents and the same readConcern, where you can also inspect the plan cache entry that will be selected. Then confirm on the real system from the profiler: planSummary on the actual transactional entries is the ground truth for what the server really ran.

The senior habit is the second step, because a plan that is fine on an empty collection is often a COLLSCAN ten million documents later, and the staging reproduction is the only place you will see that before production does.

Four reasons, and only the first is obvious.

1. Different cache state. The plan cache is keyed on query shape, and shapes are shared across sessions. A plan cached by a non-transactional read can be selected for the transactional one, or rejected because a competing plan won more trials.
2. Different read concern. A snapshot read executes against a fixed point in time; the planner's selectivity estimate comes from a snapshot that may not match the current distribution, and a plan that was optimal five minutes ago is not necessarily optimal now.
3. Different data volume. Inside a transaction you usually touch fewer documents with more selective filters, so plan choice rarely matters — but when it does, one bad COLLSCAN holds locks for the whole transaction.
4. Sharding. Each shard plans independently, and the mongos-level plan is a merge of those, so a sharded transactional query can degrade on one shard while the other three are fine.

Practical rule: pin the access path with a supporting index, not with a hint, and verify with $indexStats.

Real. One minute default lifetime (transactionLifetimeLimitSeconds), enforced by a cleanup process, with NoSuchTransaction as the client-visible symptom. Five milliseconds default lock-request wait (maxTransactionLockRequestTimeoutMillis). 16 MB per document and per oplog entry — not per transaction; the server splits a transaction across as many entries as needed. The WiredTiger cache ceiling behind TransactionTooLargeForCache. No cross-shard DDL. No capped collections. A shard with an arbiter cannot take part in multi-shard transactions.

Folklore. "16 MB total per transaction" — obsolete since 4.2. "1000 documents per statement" — that number circulates from a 4.0-era guidance note about keeping transactions small; it is a design heuristic, not an enforced limit, and conflating the two leads people to build batching logic for a constraint that does not exist while ignoring the cache ceiling that does. Verify against the production-considerations page for your exact version before you quote any number in a design review.

NoSuchTransaction (251). The transaction is gone: aborted by the lifetime cleanup, killed, aborted on a previous statement, or a shard stepped down. Carries TransientTransactionError, so the standard loop restarts from the beginning — which is only correct if your body is idempotent and cheap. Recurring 251s mean either a too-long transaction or an exception path that aborts and then keeps issuing statements.

TransactionTooLargeForCache. The write set cannot fit the storage cache budget, or contention inflated it. Chunk the work; retrying the same transaction unchanged is a loop with a 100% failure rate.

LockTimeout (24) / lock request timeout. The transaction waited longer than maxTransactionLockRequestTimeoutMillis for a lock — usually DDL pending behind another transaction, or a hot document. Raising the parameter is legitimate only when the blocker is transient metadata; if the blocker is an application write, raising it converts an error into latency and then into a queue.

In all three, the useful telemetry is the retry count per logical request, not the error count.

Two sessions, interleaved by hand:

// session A
const sA = db.getMongo().startSession();
sA.startTransaction();
sA.getDatabase("t").c.updateOne({ _id: 1 }, { $set: { x: 1 } });

// session B -- outside a transaction
db.c.updateOne({ _id: 1 }, { $set: { y: 2 } });   // waits for A's lock

// back in A
sA.getDatabase("t").c.updateOne({ _id: 1 }, { $set: { x: 2 } });  // WriteConflict
sA.abortTransaction();

What it proves, and why it is worth doing once by hand: the conflict is not about timing but about snapshot versus write. B's write committed after A's snapshot, so A can no longer modify that document. It also shows the asymmetry in the docs — B, outside a transaction, blocks waiting for the lock, while A, inside one, fails fast rather than waiting.

Run it with retryWrites off and a transaction, then again as two plain updates, and you will see why the same contended workload behaves completely differently depending on whether you wrapped it in a transaction.

Necessity. Can the invariant live in one document instead? A transaction is the expensive answer; a conditional single-document update is usually the correct one.

Footprint. How many documents, how many shards, how many bytes at p99? Budget it explicitly, because the cache ceiling and the 60-second lifetime are the two things that will bite.

Duration. Any network call inside the body is a defect. So is any unbounded query. Measure the p99 body time under load, not in staging.

Contention. Do all requests touch the same few documents? If yes, the design converts throughput into aborts. Shard the hot document or change the model before shipping.

Failure semantics. Idempotency key written inside the transaction, retry loop with jitter and a bounded budget, UnknownTransactionCommitResult handled, abortTransaction in finally, w: "majority".

Observability. Retry counts, abort ratio, timeInactiveMicros alert, and a load test that pushes past the conflict knee. Then re-check whether the endpoint needed a transaction at all.

A collection is not a heap of documents; it is a B-tree whose pages are the unit of I/O, caching, and compression. Internal pages hold only keys plus child page addresses — they are the routing layer, never user data. Leaf pages hold the encoded records (for MongoDB, the BSON payload keyed by record id) and are where all reads land.

In practice the tree is shallow: leaf pages are sized in the tens of kilobytes (roughly 32 KB uncompressed by default) and internal pages are only a few kilobytes, with a 4 KB allocation unit underneath. That means a collection with hundreds of millions of documents still has a root-to-leaf depth of about three, so a point lookup costs three page accesses at most, and the top one or two levels are almost always resident in cache.

The consequence for interviews is this: the cost of a query is not "documents scanned" at the storage layer, it is pages touched. When you explain a plan, every FETCH is potentially a leaf-page read.

WiredTiger is MVCC in memory. A leaf page in cache holds a set of keys (record ids) each pointing to an update chain: a list of versions, newest first, with the transactions that created them. A read walks that chain and returns the first version visible to its snapshot.

This has three practical consequences. First, an updateOne does not rewrite the page; it prepends a version, so a document updated thousands of times in a tight loop accumulates a long chain and slows every reader of that document. Second, a delete is not a page edit but a tombstone appended to the chain, which is why deleteMany burns CPU and journal rather than freeing anything. Third, cache accounting reflects this: serverStatus().wiredTiger.cache["bytes currently in the cache"] grows with versions and tombstones, not just with live documents.

Long chains are reclaimed only when no snapshot needs them, so a stuck open transaction or a lagging cursor can pin history and inflate cache pressure.

WiredTiger compresses whole pages, not documents, so compression is a property you set per collection at creation time:

db.createCollection("events", {
  storageEngine: { wiredTiger: { configString: "block_compressor=zstd" } }
})

snappy is the default: fast, symmetric, moderate ratio — it mainly removes BSON key repetition and padding. zstd trades CPU for a much better ratio and, usefully, supports a dictionary so small documents compress well. zlib exists but is rarely the right answer.

Prefix compression is a separate mechanism that applies to index keys, not collection data, and it is on by default. Because index keys in a B-tree are sorted, consecutive keys share long prefixes (a tenant id, a status, a date); prefix compression strips the shared part, which is why an index on a long string field is smaller than you would estimate from the key length alone.

Compression changes disk footprint and I/O, not the size of the working set in cache: pages are decompressed before use.

Choose zstd when the read path is dominated by I/O rather than CPU, and when the data compresses well but is read rarely relative to its size — archives, log-like documents, wide documents with repeated field names and enum values, historical partitions. The win is fewer bytes per page fault and a smaller working set on disk, which matters when your IOPS budget is the binding constraint.

Stay with snappy when the collection is hot, when writes dominate (every page write pays compression), or when the primary is already CPU-saturated by application query work. zstd also raises checkpoint cost, since checkpoints write compressed pages, so a write-heavy collection can see eviction and checkpoint latency increase.

Measure rather than guess: compare db.collection.stats().size and storageSize per compressor on a representative sample, then watch wiredTiger.cache["pages read into cache"] and CPU after the switch. A 2:1 ratio on cold data is worth it; a 1.3:1 ratio on a hot collection usually is not.

The default is 50% of (total RAM minus 1 GB), with a 256 MB floor. On a 64 GB host that is roughly 31.5 GB of cache; the subtracted gigabyte is reserved for the rest of the process — connections, sorting buffers, aggregation, the TTL monitor, and the OS page cache that still serves the journal and the compressed files.

Override it explicitly with storage.wiredTiger.engineConfig.cacheSizeGB when mongod shares a host (containers, co-located monitoring, Atlas-side overhead) or when you run several instances per machine. Do not raise it toward 90% and expect a proportional gain: beyond the working set, cache is idle, and a larger cache means longer eviction and checkpoint work per cycle.

Two caveats worth stating in an interview. The cache is not the only memory consumer, so RSS exceeding cacheSizeGB is normal. And on Linux the filesystem cache still holds compressed pages, which is why db.serverStatus().mem.resident can be well above the configured cache.

The working set is the set of pages — collection leaves, index leaves, and internal pages — that must be resident for the latency target to hold. It is not the dataset size, and it is not the cache size either. Cache is capacity; the working set is demand.

A generous cache only helps if the demand actually fits and stays stable. Three things break that assumption. Skewed access patterns make a small hot subset but a long cold tail, so every scan of the tail evicts hot pages. Index-heavy schemas inflate the resident footprint, since each index has its own B-tree and its own internal pages. And unbounded queries read cold pages faster than eviction can cope, producing the classic signature: high cache fill, dirty percentage climbing, and read latency rising even though RAM is "plenty".

So the sizing question is not "how much data do I have" but "how many distinct pages do my top query shapes touch per hour". That number, plus index footprint, is what you compare against cache.

Writes land in memory first. Eviction is the background machinery that reconciles dirty pages to disk and then frees them, and it runs at two levels of urgency: a target (pages are evicted opportunistically to keep the cache below ~80% full) and a trigger (~95%, where application threads are forced to do eviction work themselves).

Dirty bytes have their own thresholds: roughly 5% as a target and 20% as a trigger. When wiredTiger.cache["tracked dirty bytes in the cache"] stays above 20%, you have a reconciliation backlog — the writers are producing dirty pages faster than the eviction workers can flush them. The usual causes are a write burst, slow fsync on the storage, a checkpoint in progress, or too few eviction threads for the core count.

The symptom users report is not "eviction is slow", it is a latency spike: once application threads join eviction, a simple insert starts waiting on I/O. Watch pages evicted by application threads — if it is non-zero and growing, the cache is the bottleneck, not the query.

A checkpoint is a consistent snapshot of the data files. WiredTiger performs one roughly every 60 seconds, or earlier if the journal has grown to about 2 GB, and at the end of it writes a checkpoint record into the journal naming the point in the journal up to which the data files are consistent.

That record is the hinge between the two durability mechanisms. Recovery reads the data files as of the last checkpoint, then replays journal records from that checkpoint forward — not from the beginning of the journal. Without a valid checkpoint record the journal cannot be applied, which is why a corrupted checkpoint is a much worse incident than a corrupted tail of the journal.

Operationally: checkpoints are what make the data files eventually correct, the journal is what covers the gap since the last one. Both matter — a long checkpoint due to large dirty pages widens the window of work that a restart must replay, which is why crash recovery time tracks journal size and checkpoint frequency, not total data size.

They solve different problems and live in different places.

• The journal is a per-node, per-storage-engine write-ahead log in journal/. It records page-level physical changes and exists so that a single mongod can recover its own data files after a crash. It is internal, idempotent, and never replicated.
• The oplog is a capped collection in the local database. It records logical, idempotent operations (applyOps-style) and exists so that other members can replay them. Change streams read it; replica set consistency depends on it.

Practical consequences follow directly. Losing the journal loses local durability; losing the oplog tail forces a resync. You size the journal by write volume and checkpoint frequency, and the oplog by write volume and the recovery window you promise. And j:true says nothing about replication — it only guarantees the local journal entry is on disk, not that any secondary has seen it. Confusing the two is one of the most common senior-level mistakes.

The journal is flushed to disk on an interval — on the order of 100 ms — rather than synchronously per write. With j:false, the write is acknowledged once it is in the WiredTiger cache and the in-memory journal buffer, so a crash can lose the last window of writes: everything after the last flush, up to roughly that interval's worth of operations.

What can never be lost: anything already checkpointed, and anything acknowledged with j:true. What j:false does not mean is corruption — recovery is still consistent, you just roll back to a slightly older point, and the data files remain structurally valid.

Note the interaction with replication: w:"majority", j:false still guarantees the write is on a majority of nodes' memory, so a single-node crash survives it; a full-cluster power loss does not. That is why j:false is defensible for high-volume, re-derivable telemetry and indefensible for money. Also remember that on a replica set the rollback after such a crash is applied by the replication protocol, so application-visible effects can be subtler than "one document vanished".

The default (about 5% of free disk, with a floor around 1 GB) is a guess about your workload and is almost never the right number for a real one. Size it from measured write volume and a stated window:

// current oplog window, in hours
db.getSiblingDB("local").oplog.rs.stats().maxSize
// and the actual time span it covers
db.getSiblingDB("local").oplog.rs.find().sort({$natural:-1}).limit(1)
// compare first.ts and last.ts

Then: required size = peak GB/hour × target hours × safety factor. Target hours must cover the worst realistic outage of a secondary — a maintenance window, a node rebuild, a cross-region link failure — so 24 to 72 hours is typical, more if you promise change-stream resume across a weekend.

Two refinements: size for the primary's peak, because every secondary must absorb that stream; and remember that bulk operations, index builds, and large backfills generate oplog entries at rates far above steady state, so measure during those events, not on a quiet Tuesday.

The oplog is capped and circular: when it fills, the oldest entries are overwritten regardless of whether every member has consumed them. A secondary that falls behind the oldest remaining entry gets OplogTruncatedException and can no longer apply changes incrementally — its position simply no longer exists.

Recovery is a full resync: the member discards its data files and re-copies the dataset, which means hours of network transfer plus index rebuilds, and sustained read load on the sync source. If the whole cluster's oplog is too small, the primary itself eventually cannot satisfy w:"majority" waits for a lagging member — the write concern blocks, latency climbs, and the incident looks like "writes hang" rather than "oplog too small".

Detection: monitor rs.printReplicationInfo() for the configured window and the actual first-to-last span, and alert when the window shrinks below your defined recovery budget. The fix is replSetResizeOplog, applied to every member, not just the primary.

WiredTiger limits how many operations may be inside the engine at once: by default 128 read tickets and 128 write tickets, adjustable via wiredTigerConcurrentReadTransactions and wiredTigerConcurrentWriteTransactions. A ticket is taken on entry to the storage engine and returned on exit, so the limit caps concurrency inside WiredTiger — not connections, not application threads.

Exhaustion does not produce an error; it produces queueing. Operations wait in the storage layer for a ticket, so latency rises on operations that look cheap in explain, while CPU may be moderate. The tell is here:

db.serverStatus().wiredTiger.concurrentTransactions
// { write: { out: 128, available: 0, totalTickets: 128 },
//   read:  { out: 128, available: 0, totalTickets: 128 } }

available: 0 held for seconds is the signature. Usual cause is not too few tickets but slow ticket holders: fsync-bound writes, eviction stalls, or a long-running operation pinning a read ticket. Raising the limit is legitimate once the storage is healthy, but it hides the real problem if it is not.

Three distinct constraints get conflated: throughput (MB/s), IOPS, and fsync latency. MongoDB is usually hurt by the last two. Each committed write waits for the journal flush; each checkpoint writes dirty pages; each eviction writes a page before it can be reused. If a single fsync takes 10 ms, then a serialized journal flush caps your write rate no matter how many IOPS the device advertises.

Symptoms move in a fixed order when the disk is the constraint: dirty bytes in cache climb past the 20% trigger, eviction starts running on application threads, tickets stay at available: 0, and then latency rises across unrelated query shapes. Query tuning at that point is wasted effort.

Evidence to collect: the storage latency histogram in FTDC, db.serverStatus().wiredTiger.cache["pages evicted by application threads"], and OS-level await/queue depth. On cloud volumes the classic trap is burst credits: throughput looks fine in testing and collapses under sustained checkpoint load, so test with throttled baseline performance, not burst.

drop() is a metadata operation. WiredTiger unlinks the collection's files and its index files, and the space returns to the filesystem immediately. There is no per-document work, no journal entries per document, and no oplog storm. Cost is proportional to the number of files, not the number of documents, which is why dropping a 500 GB collection takes about a second.

deleteMany({}) is the opposite: every document becomes a tombstone in its update chain, gets journaled, is replicated through the oplog, invalidates index entries one by one, and must be reconciled and evicted before the space can be reused. On a large collection this means hours of write amplification, a pinned dirty cache, replica lag, and the risk of the oplog wrapping before secondaries catch up.

So the correct pattern for "clear this data" is almost always drop() or a time-partitioned collection swap — not a mass delete. If you must delete a majority of documents, deleting in bounded batches reduces the blast radius but never makes it cheap.

Deleted space becomes reusable inside the data file, not outside it. WiredTiger keeps the freed extents on a free list; new inserts and updates of similar size will reuse them before the file grows again. That is why dbStats().dataSize falls while storageSize and the file on disk stay flat — the gap is the free list.

Reuse is not perfect, because pages are fragmented and reused only when the size fits. A collection that shrinks to 10% of its peak may still hold a mostly-empty file, and a capped or heavily fragmented collection degrades further.

This is expected behavior, not a leak, and it is the argument for partition-by-time designs: rotate to a new collection instead of deleting from an old one, then drop the old one and get the bytes back instantly. If you genuinely need the file to shrink in place, compact is the only lever, and it is expensive — which is exactly the trade-off the next question covers.

compact rewrites the collection's data files and index files, keeping only live records and discarding free space and fragmentation. It is a full physical copy: it reads every live document, writes it into new extents, and rebuilds the files. It requires a primary, it can be interrupted and restarted, and it is I/O heavy enough to look like a backup job from the storage's point of view.

Blocking behavior matters: modern versions block operations on that collection for the duration rather than the whole database, but the collection is still effectively offline for writes from the application's perspective — plan it as a maintenance action with a window, not an ad-hoc fix.

Justify it when a collection shrank dramatically and you need bytes back, when a large delete left the file badly fragmented, or when an internal check finds excessive free space. Do not justify it as routine hygiene: for time-series or append-heavy data, partitioning and dropping is cheaper, faster, and has none of the availability cost.

A columnstore index stores values column-by-column rather than document-by-document. That layout is useless for point lookups — you cannot seek to one document — but it is excellent for scanning a few fields across many documents, which is precisely the aggregation shape that normally forces a COLLSCAN.

Build behavior: the index is created by scanning the collection and accumulating per-field value runs, which needs working memory proportional to the number of distinct values and the batch being sorted, and spills beyond the index-build memory budget (maxIndexBuildMemoryUsageMegabytes, 500 MB by default). Builds are therefore heavier than a normal B-tree build and should be scheduled like one.

Maintenance: unlike a B-tree, a columnstore is not updated in place per write. There is no cheap way to insert a single value into a compressed column, so it is designed around batched/segment maintenance, and it remains a specialized tool: use it when the dominant workload is a small number of analytical aggregations, not as a general replacement for compound indexes.

Each index is its own WiredTiger B-tree in its own file (or its own set of extents inside the collection's file group). It is not a structure embedded in the collection's tree, and it is not derived at read time. The collection B-tree maps record id → BSON; the index B-tree maps key → record id.

An entry contains the indexed values in the index's sort order, the record id as the payload, and, for multikey indexes, a per-key discriminator so that the same record can appear multiple times. Because the record id is the payload, an index lookup returns a pointer and nothing else — the document still has to be fetched from the collection tree, unless the query is covered and the projection can be satisfied from the index alone.

Two consequences worth stating: index entries are subject to the same page/compression machinery as data, including prefix compression on sorted keys; and an index update is a second B-tree modification, which is why every extra index multiplies write cost rather than adding to it linearly-ish.

A multikey index stores one entry per array element, not one per document. A document with 50 tags produces 50 entries, each a full key plus a record id, all in the same B-tree. Multiply by array length and you get index size that has nothing to do with document count.

The pathological cases are arrays that are long and low-selectivity (thousands of strings, or an array of subdocuments where the index is on a nested field), and documents where several indexed arrays coexist — a compound multikey index on two array fields cannot be bounded efficiently, and the planner will often refuse to use it well because the bounds on the second field cannot be derived independently.

Diagnostics: db.coll.stats().indexSizes next to avgObjSize, plus $indexStats for access counts. If one index is larger than the collection, you are paying for an index that is probably being scanned rather than seeked. The fix is usually a computed summary field (a bounded set of derived keys) instead of indexing the raw array.

A compound index may contain at most 32 fields in its key pattern. There is also a limit on the total size of an index key (see the next question) and, separately, on the size of the index key pattern document itself. These are engine limits, not planner heuristics, and creation fails with an error rather than silently truncating.

In practice 32 is never the constraint you hit first; key size and write cost are. Every additional field widens every entry, multiplies the B-tree's fan-out requirement, and makes prefix compression less effective when the added fields are high-entropy. A 12-field index is a design smell: it usually means the query patterns were never consolidated.

The boundary issue to raise in an interview is the prefix rule under the limit. A 32-field index can serve queries on any prefix of its fields, so one wide index can cover several shapes — but only leading prefixes, and only if the sort directions and collation line up. Beyond that, a wide index is dead weight on every write to the collection.

An index entry must fit within about 1 KB of key data (1024 bytes, including the per-type overhead WiredTiger adds). Exceed it and the write fails — you get a "key too large" error from the storage engine rather than a silent truncation, and the document insert or update is rejected. That is important: the failure is on the write, so a document that is perfectly valid on its own can be rejected because you indexed a huge string or a big array.

Typical triggers: indexing a field that holds base64 blobs, a description field with unbounded length, or an array of long identifiers. Note that multikey indexes are counted per element, so a single monstrous element is enough to break inserts even though the array is short.

Mitigations: index a deterministic hash of the value (and query by the hash), index a prefix field you maintain yourself, move the payload out of the indexed document, or drop the index and accept the scan. Checking before deploy matters because the failure appears only when a specific document arrives.

A partial index indexes only documents matching a filter expression:

db.orders.createIndex(
  { customerId: 1, createdAt: -1 },
  { partialFilterExpression: { status: "OPEN" } }
)

Every document excluded by the filter simply has no entry, so size scales with the matching subset — often a tiny fraction of the collection. Lower size means better cache residency, faster builds, and less write amplification, since updates to excluded documents no longer touch this B-tree (though the filter field itself must still be evaluated).

The cost is a correctness constraint the planner enforces strictly: the index is usable only if the query is provably a subset of the filter. A query with no status predicate cannot use it, even though the results would be identical. That means the filter must be part of your standard query shape, and any ad-hoc query without it silently falls back to a collection scan. Design the filter around an invariant the application always enforces, not around a convenience.

db.coll.hideIndex("idx_name") makes the planner ignore the index while the index continues to be maintained on every write. Reads behave as if it were gone; you cannot hide _id, and you cannot hide an index that is still needed to enforce a unique constraint in some setups.

The lifecycle is: hide it, then watch the workload for a full business cycle — including month-end jobs and rarely-hit admin screens — comparing latency and plan shapes on the queries that used it. If nothing degrades, dropIndex and reclaim the space. If something degrades, unhideIndex restores it instantly, with no rebuild cost.

That last point is the whole argument. Dropping and re-creating an index on a large collection means a full build: a collection scan, memory pressure, replication impact, and minutes to hours of vulnerability. Hiding gives you a reversible experiment at zero cost. The observable signal is $indexStats: a hidden index still reports accesses, but the accesses.ops counter stops incrementing in practice because the planner never selects it.

A modern build is a hybrid: brief exclusive locks at the start and end, with a long middle phase where reads and writes are allowed. During that middle phase, writers must keep the half-built index consistent, so each write to the collection records the change in a pending side-table and the build thread later applies it.

Consequences you should be able to state. Writes pay an extra cost for the duration of the build, proportional to write rate. The build holds memory up to the budget (maxIndexBuildMemoryUsageMegabytes, 500 MB default) and spills to temp files beyond it, so watch disk. On a replica set, secondaries start their own build when they see the commit, so index builds ripple through the cluster and can widen replication lag. And the final phase briefly blocks writes to take the lock and mark the index ready, which is the small latency spike users notice.

Practical rule: build during low write volume, one index at a time, and prefer a short-lived secondary-first rollout for very large collections.

No — the flag is accepted for compatibility but no longer selects a different algorithm. The old distinction was: foreground builds took an exclusive lock for the whole build (fast, but writes blocked), background builds sampled the collection and re-scanned for changes (slow, writes allowed). Both behaviors are gone.

Since the hybrid build was introduced, all builds behave in the "background-ish" way: they yield to writes for the bulk of the work and only take an exclusive lock at the beginning and end. Removing the option removed a real footgun, because "background" used to mean a build that could take many times longer and never quite converge on a hot collection.

So the correct answer to "should I build in the foreground or background" is that the choice no longer exists; the meaningful levers are now the timing relative to write volume, the memory budget, and whether to build on secondaries before the primary with a rolling approach. Passing {background: true} is harmless but signals you learned the older documentation.

The build fails as a whole; it does not create a partially unique index. The build scans the collection in key order, and if two documents share the same key with a different record id, it aborts with a duplicate key error (code 11000) naming the offending key value. The index is not left in a usable state — on failure the partially built index is removed, so you are back to where you started, minus the time and I/O spent.

This matters operationally because the failure happens at the end of a long build on a large collection. The correct sequence is to find the duplicates first, cheaply:

db.orders.aggregate([
  { $group: { _id: "$externalId", n: { $sum: 1 } } },
  { $match: { n: { $gt: 1 } } },
  { $limit: 20 }
], { allowDiskUse: true })

Also note the interaction with multikey fields: a unique index on an array field enforces uniqueness per element across documents, so duplicate array values in different documents collide, while a repeated value inside one document's array does not create a second entry.

The TTL monitor is a background thread on the primary that wakes roughly every 60 seconds. Each pass, it queries each TTL index for documents whose indexed date is older than expireAfterSeconds and issues deletes in batches until the pass budget is used, with delete operations that appear in the oplog like any other write.

Consequences that follow directly from "it is a batched delete on a 60-second timer":

• Expiry is not exact. A document can live up to roughly one monitor cycle plus however long the backlog takes, so never treat TTL as a scheduling mechanism with second-level accuracy.
• Deletes are real deletes — journaled, replicated, and index-maintaining. A large expiry burst is a write burst, and it can show up as replication lag and dirty cache pressure.
• The delete rate is throughput-limited, so a collection receiving 50k expiring documents a minute may never catch up.

If the exact time matters (billing cutoffs, leases), enforce it in queries and let TTL handle cleanup only.

A wildcard index ({ "$**": 1 }) creates index entries for every field path present in each document, recursively. Entries include a $_path component identifying where the value came from, so the index is one B-tree holding every field combination in the collection. Size therefore scales with total field count across all documents, not with document count.

Costs stack up: write amplification grows with each document's field count; the same query-shape benefit can be had from one targeted compound index for a fraction of the space; and you lose the ability to reason about selectivity, because a wildcard entry for a low-cardinality field is just as large as for a unique one. Comparisons across different paths are also weaker for the planner.

It earns its keep in genuinely schemaless or polymorphic collections with unpredictable ad-hoc queries — an exploration-heavy product catalogue, tenant-defined custom fields. Constrain it: { "customFields.$**": 1 }, or add wildcardProjection, so it covers only the unbounded part of the schema.

A text index is effectively a multikey index over the terms produced by the analyzer, not over the original string. For each indexed field, MongoDB applies language-specific processing: splitting into tokens on word boundaries, case-folding, removing stop words (the, and, of — which is why they never match), and stemming so that "running" and "run" collapse to a common root. The resulting terms become keys pointing at the document.

Consequences: size scales with distinct terms per document times average token length, so a text index on long prose is one of the largest indexes you can create, often exceeding the collection. Only one text index is allowed per collection, so it must cover every searchable field, with weights (1 to 99,999) biasing relevance scores rather than filtering.

The default language drives stemming and stop words, and can be set per document via a language field. Prefix and phrase semantics are limited, which is the honest reason to look at Atlas Search for anything beyond keyword matching.

2dsphere indexes GeoJSON geometry on a sphere. Internally it converts each shape into a covering of S2 cells — a hierarchical subdivision of the sphere into quadtree cells with 64-bit identifiers. The index keys are those cell ids, which makes proximity queries a matter of finding cells at the right level and then verifying exact geometry.

That two-phase design (index lookup on cells, then a precise geometry check) explains the behaviors you see in explain: the index narrows candidates via bounds, and a filter step rejects shapes whose cell covering overlaps but whose actual geometry does not. It also explains why 2dsphere supports polygons, points on a sphere, and legacy coordinate pairs, while the older 2d index treats coordinates as a flat plane with geohash cells and cannot represent real curvature.

For queries to use the index, the predicate must be a supported geo operator ($near, $geoWithin, $geoIntersects) on the indexed field, and $near additionally requires the index and sorts by distance, so it cannot be combined freely with other sorts.

A clustered collection stores documents in _id order inside the collection's own B-tree, rather than in record-id order with a separate _id index pointing into it. It must be declared at creation time — you cannot convert an existing collection — and it requires a unique _id.

What disappears: the _id index itself (with its storage and its per-write maintenance), and the second B-tree traversal on every _id lookup. A range scan on _id becomes a contiguous leaf scan instead of a scatter of fetches, which is a large win for append-heavy collections and for anything using _id-based pagination.

What you give up: insert locality now depends entirely on _id monotonicity, so random _id values (hashes, UUIDs) cause page splits in the middle of the tree, exactly like a random primary key in a relational engine. Size hints and capped-style use cases must be reconsidered, and secondary indexes still exist separately.

Every collection except a clustered one has a unique _id index, created automatically, and it cannot be dropped. It exists because _id is the document's identity: replication, retryable writes, and findOne({_id: ...}) all depend on a guaranteed unique, always-present key. If a document arrives without one, the driver or server generates an ObjectId before the write.

Cost per insert is one extra B-tree insert into a unique index — plus the uniqueness check, which is not free under concurrency. This is where the classic anti-pattern lives: a random _id (UUIDv4, random string) inserts into random positions of the _id B-tree, dirtying pages across the whole index and inflating cache pressure, whereas ObjectId is roughly monotonic and appends to the right edge.

So the practical guidance: keep _id monotonic where possible, or use a clustered collection if _id-ordered access is the dominant pattern. Do not try to drop it — you cannot, and you should not want to.

Index A is redundant if index B has A's key pattern as a leading prefix with the same collation, because B can serve every query A serves (modulo sort direction and unique/sparse semantics). {a:1} is covered by {a:1,b:1}, but not by {b:1,a:1}.

Tooling in order of usefulness: $indexStats for usage counts, db.coll.getIndexes() plus a script that groups key patterns by prefix, the Performance Advisor in Atlas, and db.coll.aggregate([{$indexStats:{}}]) to confirm the "redundant" index really has near-zero accesses. Careful with the numbers: accesses.ops resets on restart and on index rebuild, so a low count right after a deploy is meaningless.

Then be conservative. Two indexes that look like prefixes may differ in collation, in uniqueness, or in a partial filter, and a "redundant" index may be the only one that supports a covered query. Hide before dropping, and check for hint() references in code and stored aggregations — a hint on a dropped index throws.

When a query shape is new, the planner does not trust its cost estimates. It generates candidate plans (one per usable index plus a collection scan), runs them concurrently in a race against the real data, and lets them compete. During the trial period each candidate gets a number of "works" — historically on the order of a hundred evaluation cycles — and the plan that returns results fastest, weighted by work completed, wins and is cached for the shape.

Two consequences. The first execution of a shape is slower than steady state because of the race, which is why a p99 spike right after a deploy or a cache flush is often benign. And the winner is chosen on a small sample of the data, so a plan that wins on a mostly-empty collection can lose badly later — the classic reason a query degrades as data grows.

Modern versions shorten this by using statistics and heuristics to prune candidates before racing, and SBE-era planning caches both the winning plan and its trial results.

A plan is cached only if the shape is cacheable: the query must be a simple enough find/update/delete with a stable parse tree, and the cache honors a bounded number of entries per collection. Multi-planning must have completed, so the entry stores the winning plan plus performance statistics. Aggregation pipelines and shapes the planner cannot safely reuse are simply not cached, and are re-planned each time.

Invalidation events:

• An explicit planCacheClear or a write operation on the collection — creating or dropping an index, dropping the collection, or a change in the index catalog in some versions.
• The collection being rebuilt or restored, and collMod-style metadata changes.
• Cache eviction when the per-collection entry cap is reached and a new shape arrives.
• Server restart, which empties the cache entirely.

Diagnostic note: explain reports planCacheKey and, in execution stats, the cached plan's works. If planCacheKey changes between runs of what you think is one query, the shapes differ by something subtle — usually literal types.

The key is derived from the parse tree with literal values replaced by placeholders: field names, operator structure, sort, projection, collation, and the collection. Two queries with the same shape share one cached plan, which is the whole point — thousands of parameterized queries cost one planning exercise.

It goes wrong in four predictable ways. Type variance: {a: 5} and {a: "5"} are different shapes, so a collection serving both doubles entries and can cache a plan optimized for the wrong type. Value skew: {tenantId: X} with X matching 90% of documents gets the plan that was tuned for a rare X, because the shape is identical. Entry cap pressure: many shapes evict each other, so planning happens constantly. Stale statistics: nothing forces a re-plan when the data distribution changes, so a bad cached plan can persist indefinitely.

db.coll.getPlanCache().clear() forces re-planning, and .list() with planCacheKey filters shows what is cached. Clear it deliberately: after a bulk load, a major index change, or when you have evidence a specific shape is stuck.

Because the plan was chosen on a small sample and then frozen in the cache. A shape planned against 10k documents may reasonably prefer an index scan with a FETCH per result; the same shape at 10M documents, where the predicate now matches 40% of the collection, should switch to a collection scan — but the cached winner is never re-evaluated.

The mechanism is cost, not correctness: the planner's estimate of documents matched comes from index bounds and histograms/sampling, and when the estimate is wrong the cost comparison inverts. Growth also changes the physical picture — the index no longer fits in cache, so each FETCH becomes a random disk read, and an index scan that was cheap becomes worse than a sequential scan.

Signals to look for: totalDocsExamined far above nReturned, a rising executionTimeMillis with unchanged query text, and a plan that suddenly differs right after a restart (when the cache was empty). Remedies: drop and recreate the index to trigger re-planning, clear the plan cache, or — better — restructure the query so its shape reflects the real selectivity.

Estimation runs off index bounds plus statistics. From the bounds the planner knows how many distinct key ranges the predicate touches; it then estimates how many documents each range returns using sampled statistics and heuristics about key distribution. For a collection scan it estimates total documents. Everything downstream — whether to use the index or scan — is a comparison of those estimates.

The failures are structural. Statistics are sampled, so rare values are invisible until they are not. Correlated predicates are treated as independent, so {status:"A", region:"EU"} is costed as if the two were unrelated. Skew is invisible: an index on tenantId has one key holding 80% of documents, and the estimate for that key is a uniform-distribution guess. And $or, negation, and regex predicates have bounds the planner cannot bound precisely, so it falls back to a fixed guess.

That is why the honest diagnostic is empirical: compare totalKeysExamined and nReturned against the estimate. When they diverge by orders of magnitude, the plan was built on a bad estimate.

SBE replaced the classic tree-walking model with a virtual machine over slots: each operator produces values into numbered slots, and a compiled plan is a linear program of instructions. No per-document virtual dispatch, no heap-allocated BSON objects flowing between stages, far better cache locality and less allocation pressure.

What it changed for practitioners: throughput on scan-heavy and aggregation-heavy work improved substantially, and explain gained a second section — queryPlanner still shows the tree, while SBE plans report per-stage metrics under executionStats.executionStages with an nReturned/executionTimeMillisEstimate shape that is subtly different from the classic output.

What survived: the planner and its cost model, the index B-trees, and the plan cache. So the classic reasoning about index selection, bounds, and blocking sorts remains valid; only the execution machinery underneath is different. Many older blog posts describe operator names from the classic engine that no longer appear — that mismatch is a good sign someone is quoting pre-6.0 material.

The express path is a fast lane that bypasses most of the normal execution machinery for very simple reads. If the query is a pure equality/range find on an indexed field, needs no sorting beyond the index order, no blocking stage, no complex projection or collation work, and no shard-side merging, the server can answer it directly from the index with a minimal execution plan.

You see it in explain as a noticeably short plan — often just EXPRESS_IXSCAN or an IXSCAN with no FETCH and no SBE wrapper stages — and in the numbers as a large drop in per-document overhead. That is why a covered equality lookup can run in microseconds while the same logical query with one extra field in the projection costs an order of magnitude more.

The lesson for design: shape point lookups to hit the express path. Keep the projection inside the index, keep the predicate a simple equality, avoid $expr, avoid mixed collations, and do not add a sort the index does not already provide.

Read them as a chain: keys examined → documents fetched → documents returned. The diagnostic ratio is totalDocsExamined / nReturned. Ideal is 1.0 for a selective query; anything above roughly 10 starts to matter, and above 100 you are paying for a full scan's worth of work through an index, which is worse than a scan because fetches are random.

The second ratio, totalKeysExamined / totalDocsExamined, tells you whether the index bounds are tight. A large value means the index scan is walking keys that the query then rejects after fetching — a low-selectivity prefix, a multikey expansion, or a range whose bounds are too wide.

"executionStats": {
  "nReturned": 5,
  "totalKeysExamined": 48000,
  "totalDocsExamined": 47000
}

This plan is a disaster on both counts: the index barely narrowed anything, and almost every examined key caused a fetch. The fix is a compound index whose prefix matches the actual predicate, not a hint.

seeks counts how many separate positions in the B-tree the executor had to jump to. A single range predicate on a sorted key is one seek regardless of how many keys it then reads; $in with 50 values is up to 50 seeks; a multikey index or an $or across branches multiplies further.

The distinction matters because seeks are the expensive part of index access on cold pages: each seek is a new descent and likely a new page. A plan with seeks: 1 and 10,000 keys examined is a wide range, cheap per key. A plan with seeks: 5000 and 5,000 keys examined is doing random accesses, and will behave far worse than the totals suggest.

Compare against nReturned for the efficiency picture: many seeks for few results means the index is being used as a filter for sparse hits, and a different key order (or a compound index putting the selective field first) would collapse the seeks. This is the number that explains "the index exists but the query is still slow on a cold cache".

It is a per-stage attribution of time in the winning plan, measured with a coarse timer, and it is an estimate by design: stages report time including the work of their children, so numbers do not sum to the total, and very fast stages often report 0 because the clock resolution is coarser than the work.

Traps worth naming: it reflects only the plan that ran, not the planning cost or the time spent in the race between candidate plans; it excludes client-side time, network, and driver deserialization; and it can be dominated by the first stage that had to touch disk, so a FETCH at 900 ms and IXSCAN at 890 ms describe the same wait, not 1.8 s of work.

Use it for relative shape, not absolutes. If SORT is the biggest slice and its spills is non-zero, that is actionable. If FETCH dominates after a huge IXSCAN, the index is wrong. But never quote it as the query's latency — measure end-to-end time for that, and read executionTimeMillis for the whole execution.

A spill means a blocking stage exceeded its in-memory budget and wrote intermediate data to temporary files on disk. In aggregation the budget is 100 MB per stage and spilling requires allowDiskUse: true; without it, the operation fails instead. In a find with a blocking sort, the sort memory limit applies and no disk fallback exists — the query errors with QueryExceededMemoryLimitNoDiskUseAllowed.

In explain look for "spills": 1 (or a non-zero count) plus spilledDataStorageSize and spilledRecords on the offending stage:

"stages": [{ "stage": "$group", "spills": 1,
             "spilledRecords": 4200000 }]

A spill is a latency cliff, not a small penalty: it converts a cache-resident hash table into random I/O, and it competes with the working set for the same disk. If you see spills on a recurring pipeline, the answer is structural — reduce the rows entering the blocking stage with an earlier $match or better index, or pre-aggregate.

They are the two halves of an in-memory sort in the classic operator naming, and they still appear in explain output for sorted plans.

SORT_KEY_GENERATOR extracts and materializes the sort key for each incoming document and computes the sort pattern metadata — it is the preparation step, and its cost scales with documents entering the sort. SORT_MERGE is the actual sort: it consumes those keys, orders them (with a top-k heap when a limit is present), and hands results upward.

Reading them correctly matters: a SORT_MERGE with a limit nested under it is the good case — bounded memory, top-k only. A bare SORT_MERGE with no limit and a large nReturned is the dangerous case: unbounded memory and the 100 MB-style failure mode. And if SORT_MERGE appears above an IXSCAN that already produced sorted output, the index was not usable for the sort — usually a direction or prefix problem in the compound key pattern.

IXSCAN walks the index B-tree and yields record ids. Those ids are pointers, not documents, so any field the query needs that is not in the index requires a lookup in the collection B-tree — that is the FETCH stage. It is not an inefficiency by itself; it is the cost of reading the actual document.

IXSCAN { a: 1 }  →  FETCH  →  LIMIT

It is avoidable only in two situations. A covered query — every field in the filter, sort, and projection is in the index, and _id is either projected out or in the index. Or a plan that needs nothing beyond the keys themselves, such as a count-style or existence check where the server can answer from the index.

The performance argument against FETCH is cache behavior, not CPU: each fetch is a random access into the collection tree, so 10,000 fetches on a cold collection means 10,000 potential page faults. That is why "index exists but query is slow" almost always resolves to a FETCH volume problem, and why the covered-query design (trailing fields added to the key) is a real optimization.

Both are projection stages, but they describe different work. PROJECTION_SIMPLE means the document was fetched and the server is emitting a subset of its fields — an inclusion/exclusion projection applied to a full BSON object. PROJECTION_COVERED means the projection was satisfied entirely from index keys, so no FETCH occurred at all.

So the appearance of PROJECTION_COVERED (or, in SBE output, an IXSCAN with no FETCH) is the explain-level proof that a query is covered. It matters for three reasons: latency and cache (no random collection lookups), read amplification (fewer pages touched per result), and capacity (more queries per node before the working set thrashes).

Conditions to keep it: the projection must be a subset of the index fields; no field needing the full document; the filter and sort must also be index-satisfiable; _id must be excluded or present; and array fields change the semantics — projecting a multikey index field cannot be covered because the array shape is lost in the index.

On a sharded cluster the query is routed by mongos and the explain shows shard-side plans under shards[], with mongos-level stages on top: SINGLE_SHARD when routing was targeted, SHARD_MERGE when several shards returned sorted streams that had to be merged, and SHARDING_FILTER when the query was broadcast and orphaned or non-matching documents had to be filtered out after the fact.

What they hide: the per-shard numbers are separate, so a merged total of 100 ms can be 99 ms on one shard and 1 ms on the other nineteen — you must read shards[s].executionStats individually to find the straggler. And a SHARDING_FILTER in the plan is a red flag: it means the operation could not be targeted by shard key, so every shard scanned and the router discarded. That is the explain signature of a scatter-gather, and the merge stage's memory is bounded, so broadcast queries with sorts are where sharded clusters fall over.

indexBounds shows, per indexed field, the interval the scan will walk:

"indexBounds": {
  "tenantId": ["[\\"acme\\", \\"acme\\"]"],
  "createdAt": ["(new Date(1750000000000), new Date(1760000000000)]"]
}

Exact equality shows up as a closed interval with identical endpoints. An open interval is a range. [MinKey, MaxKey] means the field was not constrained at all — the index is scanning that field's entire key space, and any selectivity is coming from the fields before it.

Wide bounds are the single most useful explain signal for "why is this index slow": they show that the predicate did not translate into a narrow interval. Common causes are a leading field with no predicate, a range predicate on an earlier field that prevents later fields from tightening (bounds are only derived for the prefix), a multikey expansion, an $in that turned into many small intervals, or a regex that is not left-anchored and therefore becomes [MinKey, MaxKey] on that field.

The planner treats $or as a union of branches. Each branch that can use an index gets its own sub-plan — in explain you see an OR stage whose children are SUBPLAN nodes, each holding its own IXSCAN with its own bounds:

OR
  SUBPLAN
    IXSCAN { status: 1 }      bounds: ["open","open"]
  SUBPLAN
    IXSCAN { priority: 1 }    bounds: ["high","high"]

Branches are executed and their results are unioned, then deduplicated by record id — otherwise a document matching both predicates (a high-priority open item) would be returned twice. Deduplication costs memory proportional to the overlap, which is why an $or over low-selectivity predicates can buffer heavily.

Rules that follow: every branch should be independently indexed, or one branch becomes a COLLSCAN that drags the whole union to a scan; the sub-plans cannot share a sort, so an $or plus a sort usually means a blocking sort on top; and if the branches are on the same field, $in is strictly better than $or.

hint() is correct when you have evidence the planner's choice is wrong and you need a bounded, reversible mitigation now: a cached bad plan, a shape whose estimate collapses under skew, or a query where the trial-period winner is measurably worse. During an incident, pinning {tenantId: 1, createdAt: -1} on the pathological shape restores latency in seconds and buys time to fix the real problem.

It becomes debt the moment it outlives the incident, because a hint hard-codes a decision whose premise can change: data grows, the hint's index becomes non-selective, a better index is added and never used, and the query silently keeps the old plan forever. Worse, hints are fragile — dropping or renaming the hinted index turns the query into a hard error (BadValue: hint provided does not correspond to an existing index), so index cleanup becomes a code change.

The disciplined pattern: hint to stabilize, then replace it with a fixed query shape or a query setting, and record why it existed. A hint with no ticket number is a hint nobody will ever dare remove.

Query settings attach behavior to a query shape rather than to a call site, so the server can enforce policy on queries the application cannot be redeployed to fix. Two levers matter operationally: reject, which refuses to execute matching shapes at all, and maxTimeMS, which caps their runtime.

db.adminCommand({
  setQuerySettings: {
    find: "orders",
    filter: { status: "ARCHIVED" },
    settings: { reject: true }
  }
})

In an incident caused by an unbounded report query hammering the primary, this is exactly the tool: reject the shape and the pressure stops immediately, without killing the application or dropping an index. For shapes that must remain available, a maxTimeMS cap converts an outage into a bounded failure the caller can handle.

Caveats: the shape must match precisely (field names, operators, no literal values), rejected queries surface as ordinary command errors that the application must tolerate, and the settings persist — so they must be inventoried and removed once the code path is fixed.

The pipeline is a demand-driven chain of stages: the last stage pulls, each stage produces a document and hands it upstream. A streaming stage transforms one input document into one output document and can emit immediately — $match, $project, $addFields, $limit, $unwind (per element). Such a stage never needs the whole input, so memory is O(1) per document and latency is low.

A blocking stage cannot emit until it has seen all input (or a defined subset): $group, $sort, $bucketAuto, $facet, $graphLookup, $setWindowFields. Its memory is O(distinct groups) or O(rows), which is where the 100 MB budget and spilling come from.

This is why stage order is a performance decision, not a style one. Every $match you move above a blocking stage reduces its input, and every blocking stage you can eliminate by letting an index provide the order or the grouping removes a memory ceiling entirely.

A stage blocks when its output depends on input it has not seen yet. That is a semantic property, not an implementation choice:

• $group must aggregate across the entire input stream — a group's final sum is unknown until the last document.
• $sort without an index-provided order must order everything before emitting the first result.
• $facet runs several sub-pipelines over the same input, so the input must be buffered and replayed.
• $graphLookup expands recursively; depth is data-dependent.
• $setWindowFields needs the whole partition before it can compute a running or lagging value.
• $bucketAuto needs the distribution to pick boundaries.

Two escape hatches exist: get the order from an index so $sort disappears ($sort + _id on an indexed prefix), or restructure so you never need global aggregation — filter into the blocking stage first, or pre-aggregate into a summary collection. Everything else is paying the memory cost honestly.

Each blocking stage has an in-memory budget of 100 MB. Cross it without allowDiskUse and the operation fails with QueryExceededMemoryLimitNoDiskUseAllowed — not a partial result, a failure. With allowDiskUse: true, the stage writes its intermediate state to temporary files under the storage directory and continues.

It is a safety valve, not a fix. Spilling converts memory pressure into disk I/O on the same device serving your working set, so a spilling pipeline slows down every other query on the node. It also means the operation is now competing for the same IOPS as checkpoints and eviction.

Also note what the limit is per stage, not per pipeline: a chain of three blocking stages can hold 300 MB. And the budget applies to memory, not to output size — a pipeline returning gigabytes through a streaming tail is fine, while a modest result set behind a large $group can fail. That distinction is what most candidates miss.

$group builds a hash table keyed by the group key, holding the accumulators ($sum, $push, $addToSet, $first) for each distinct key. Memory is proportional to the number of distinct keys times the size of the accumulator state — which is why grouping by a high-cardinality field (a request id, a document id) is effectively a materialization of the whole collection.

When it exceeds 100 MB and allowDiskUse is set, the hash table is partitioned and spilled: keys are written to bucket files by hash, then each partition is processed separately. The output is identical, but you have paid a full write and read of the intermediate data. Two accelerators exist and both are worth naming: a $match before the $group to shrink input, and a $sort on the group key before it so the stage can use a sorted (streaming) grouping instead of a hash table — the classic optimization when input is already index-ordered.

$push and $addToSet are the accumulator traps: they hold every value.

When a $limit sits directly after a $sort, the optimizer coalesces them and the sort becomes a top-k heap of size k instead of a full ordering. Memory is bounded by k, not by input size, so $sort + $limit 10 over 50 million documents holds ten documents in memory. That is why "latest 20 events" pipelines are safe while "all events, sorted" is not.

Two ways this breaks. If anything sits between the sort and the limit — a $addFields, a $project the optimizer will not push through — the coalescing may not happen and you get a full blocking sort. And if the sort can be served by an index (an IXSCAN producing the requested order), there is no sort stage at all: the pipeline is streaming and the $limit lets the executor stop early.

db.events.aggregate([
  { $match: { tenantId: "acme" } },
  { $sort: { createdAt: -1 } },
  { $limit: 20 }
])  // index { tenantId: 1, createdAt: -1 } makes this streaming

Design for the second case; the first is only a mitigation.

Only the $match, $sort, and $geoNear stages that appear before any stage that transforms documents can be pushed down to the query planner. That leading prefix compiles into an ordinary find-style plan — visible in explain as a $cursor stage with its own IXSCAN and FETCH — and everything after it runs as an in-memory pipeline.

The reason is structural: once a stage has produced new documents ($group output, $project with computed fields, $unwind expansions), those documents have no index. An index maps stored fields to record ids; a computed document is not in any B-tree. So $match after $group filters a derived stream and cannot be indexed at all.

The practical skill is knowing which rewrites keep the pipeline indexable: put every filter that references stored fields first, put $sort before any reshaping, and avoid a $project that renames the field you later filter on — that alone can turn a covered index scan into a full collection pass.

$lookup is a nested-loop join: for each document arriving from the left side, it runs an inner query against the foreign collection and attaches the matching documents as an array. The shape of that inner query is what determines the cost.

With an index on the foreign field (foreignField), the inner query is an index seek — one B-tree descent per left document, roughly O(1) each, so total cost is linear in the left side. Without an index, the inner query is a collection scan per left document: 10,000 left documents means 10,000 full scans of the foreign collection. That is the single most common cause of a "the aggregation was fine in staging" incident.

db.orders.aggregate([
  { $match: { status: "OPEN" } },              // shrink the left side first
  { $lookup: { from: "customers",
               localField: "customerId",
               foreignField: "_id",
               as: "customer" } }
])

Verify with explain: on the shard/collection side the $lookup stage reports its own totalDocsExamined, and a value far above the left-side count means the inner query is scanning.

$lookup produces one output document per input, with an array of matches. $unwind then flattens that array into one document per matched element. So the cardinality changes from N to N × matches — and every downstream stage, including another $lookup, now operates on the expanded stream.

That is the quadratic trap: a chain of two $lookup + $unwind pairs multiplies fan-out at each hop. A left row with 10 matches, each of which has 10 matches, is 100 rows, and each of those rows may trigger its own inner query. Nothing in the pipeline warns you; the plan just gets slow and starts spilling.

Mitigations, in order of effectiveness: filter or project the array down (use a $lookup with a sub-pipeline and $limit: 1 when you only need the top match, or $arrayElemAt instead of $unwind when you need one element); apply $match before the join so you expand fewer left rows; and if the fan-out is inherent, denormalize the fields you actually need into the parent document.

$facet buffers its entire input and replays it into each sub-pipeline, so its memory is proportional to the input size and it is a blocking stage. Each sub-pipeline's own stages are subject to the 100 MB per-stage budget, and — critically — allowDiskUse does not rescue a $facet whose buffered input exceeds the limit; you get a failure rather than a spill.

That makes $facet safe only if the input is already small. The standard fix is to put everything that reduces cardinality before the facet:

[ { $match: { tenantId, createdAt: { $gte: since } } },
  { $facet: {
      byStatus: [{ $group: { _id: "$status", n: { $sum: 1 } } }],
      total:    [{ $count: "n" }] } } ]

If the input cannot be reduced — a global reporting facet over millions of documents — the honest answer is to pre-aggregate nightly into a summary collection and facet over that, instead of buffering a production collection in memory on demand.

$graphLookup performs a recursive breadth-first expansion inside the pipeline, following connectFromField to connectToField up to maxDepth. Each level runs a query against the from collection, so the number of queries grows with the frontier size, not with depth alone: a node with 100 children generates 100 lookups at the next level.

Constraints to state: the stage is blocking and has a memory limit on the order of 100 MB for the accumulated visited set and results, spilling only if allowDiskUse is enabled; maxDepth is your only bound on runaway expansion, and unbounded is the default; and depthField adds bookkeeping per result.

The decisive limitation is that $graphLookup cannot use sharding in a targeted way — it queries from from the current context — so on a sharded collection it is closer to a broadcast per level. For deep or dense hierarchies, a materialized closure table or an array of ancestors beats recursion every time.

$setWindowFields requires a partitionBy, a sortBy, and an output window. Internally it must group the stream into partitions, order each partition by the sort key, and then evaluate the window function for each document — so it is a blocking stage with a sort underneath, and memory scales with the size of the largest partition.

Cost drivers, in order:

• Partition cardinality. Many small partitions means the engine manages many independent buffers; one giant partition means it must hold it entire.
• Sort. If partitionBy plus sortBy can be served by an index prefix, the sort disappears; otherwise it is a blocking sort with the 100 MB budget and possible spill.
• Window bounds. An unbounded window (range: ["unbounded", "current"]) over a large partition is the streaming-friendly case; arbitrary documents: [-1000, 1000] frames force buffering.

Practical rule: keep partitions small and index-ordered ({ tenantId: 1, ts: 1 } matching partitionBy: tenantId, sortBy: ts), and it behaves well; ignore both and it will spill.

$merge writes pipeline output into an existing (or auto-created) target collection using on as the match key (default _id) and whenMatched/whenNotMatched to decide between update, replace, insert, keepExisting, or fail. It is not a bulk load: internally it is a stream of find-and-modify-style operations, applied incrementally as documents arrive.

Consequences: it takes document-level write locks and can produce write conflicts (WiredTiger write conflict errors) when the target is concurrently written by the application or by another $merge on overlapping keys. It cannot target the collection being aggregated in the same pipeline, which prevents self-merge but means you need an intermediate collection for in-place enrichment. And it is not atomic as a whole — a failure halfway leaves the target partially updated, so the pipeline must be idempotent and re-runnable.

Design implication: run $merge against a target nobody else writes to — a summary collection consumed only by readers — and prefer $merge over $out when readers must keep seeing valid data throughout.

$out is a replace-the-whole-collection operation. It builds the result in a temporary collection, then renames it into place. That gives a strong property: readers see the old collection completely, then the new one completely — never a half-built state. The cost is that any concurrent writer to the target is in for a surprise, the rename needs an exclusive moment, and the whole result must be materialized before it becomes visible.

$merge is incremental: it upserts as documents flow, so readers can see a partially updated collection during the run, and concurrent writers contend at the document level, risking write conflicts. In exchange it can update a subset of documents (whenMatched with a $set), preserve documents it did not touch, and run against a collection that other pipelines are also feeding.

So: $out for full rebuilds of a derived collection refreshed on a schedule, where atomicity of publication matters more than availability of the target; $merge for continuous incremental maintenance, and whenever dropping untouched documents would be wrong. Neither is safe as a substitute for a transaction.

$densify inserts documents into gaps along a numeric, date, or partitionByFields sequence at a given granularity. $fill then populates missing values using value, locf (last observation carried forward), or linear interpolation. Together they turn sparse time-series-shaped data into a dense grid suitable for charting.

The cost is that output size is chosen by the granularity, not by the data. A year of sparse events densified at one-minute granularity is 525,600 buckets per partition — the stage generates those documents whether or not any data exists. With partitionByFields, multiply by the number of partitions. Both stages require sorted input per partition, so they add a sort, and $fill is window-like and buffers.

Backfire conditions: fine granularity over a long range, many partitions, or being placed before a $match that would have filtered the range. The pattern that works is to filter the time range first, densify at the coarsest granularity the chart can use, and push the densification to the reporting layer when the consumer can tolerate gaps.

Aggregation explain is shaped differently from a find. You get two things: a $cursor stage that shows the query plan for the indexable prefix (with its own IXSCAN/FETCH, indexBounds, and examined counts), and a stages array listing every subsequent pipeline stage with per-stage metrics.

{ "stages": [
    { "$cursor": { "executionStats": { "nReturned": 1200,
                  "totalKeysExamined": 1200, "totalDocsExamined": 1200 } } },
    { "$group": { "nReturned": 12, "executionTimeMillisEstimate": 340 } },
    { "$sort":  { "nReturned": 12, "spills": 0 } } ] }

Read it top-down looking for the first stage where nReturned collapses or where time concentrates. A $cursor returning 1.2M rows into a $group that outputs 12 is a story about missing index coverage at the start, not about the group. spills, usedDisk, and executionTimeMillisEstimate per stage are the three fields that turn "the aggregation is slow" into a named stage.

$documents is a source stage that produces documents from a literal expression — either an array of document literals or an expression that evaluates to one. It must be the first stage, and it lets a pipeline run without touching a collection at all:

db.aggregate([
  { $documents: [ { _id: 1, amount: 10, status: "A" },
                  { _id: 2, amount: 5,  status: "B" } ] },
  { $group: { _id: "$status", total: { $sum: "$amount" } } }
])

Its value in an interview answer is as a testing discipline: you can unit-test the transformation semantics of a $group, $setWindowFields, or $densify chain in isolation, with deterministic fixtures and no seed data, and without a server-side $lookup into a real collection. It is also useful for generating parameter rows to $lookup against, and for reproducing a production bug with three documents instead of a dump.

Limits worth knowing: results flow back to the client as a normal cursor, the stage cannot appear anywhere but first, and it is a debugging/utility tool rather than a production data source.

Because at that point the documents do not exist in any collection. $group emits newly computed documents whose _id is the group key and whose fields are accumulator results. There is no B-tree mapping those values to anything, so the planner has nothing to seek into — the $match is a filter on a derived stream and is executed as a plain stage.

This is often a correctness-of-performance issue rather than a query error, because the query looks selective:

[ { $group: { _id: "$customerId", spend: { $sum: "$amount" } } },
  { $match: { spend: { $gt: 10000 } } } ]   // no index possible

The fix is to move selectivity to where the data is stored: filter on stored fields before the group ($match on date range, region, status), so the group sees a smaller input and produces a smaller output. Where the post-group filter is essential (a HAVING-style condition), accept it as an in-memory filter over a small result set — that is cheap. The failure mode is a large group output filtered to a handful of rows, where the work was already done.

The idea is to compute expensive aggregations once, on a schedule or on change, into a small collection shaped exactly like the queries that need it — then serve dashboards from a covered index scan on a few thousand documents instead of a $group over millions.

The design decisions:

• Grain. Pick the coarsest grain that answers the query: per tenant per hour, not per event. Anything you cannot derive from the summary must be answered elsewhere.
• Key. Make the summary's _id the natural grain key ({tenantId, bucket}), so the refresh is an idempotent $merge on _id rather than a delete-and-insert.
• Refresh. Either a scheduled $merge over a bounded window, or incremental maintenance via change streams updating counters.
• Late data. Recompute the last N buckets each run, because events arrive out of order and a bucket is never final on first write.

The cost is staleness and a second source of truth. Mitigate with a documented freshness contract and a rebuild path that reconstructs the summary from raw data, so a bad refresh is recoverable rather than terminal.

PV0 was a variant of the Raft family with leader leases and a looser notion of consensus; elections could produce a new primary before the old one had definitively stepped aside, and the set relied on wall-clock-based leases to limit that window. PV1 (default since 4.0, and the only protocol in modern releases) replaced that with terms in the Raft sense: every election increments a monotonically increasing term, votes are granted at most once per term, and the old primary learns it has been superseded the moment it sees a higher term.

PV1 also introduced a well-defined majority commit point maintained through replication, catch-up takeover (a more up-to-date secondary can pre-empt the incumbent), and stricter oplog chaining. Practically, PV1 makes failover deterministic and eliminates the class of bugs where two members briefly both believe they are primary. You can still see the protocol with rs.conf().protocolVersion, and rs.status() reports the current term and electionId.

A term is a logical epoch counter shared by the set: every election attempt gets a strictly higher number than any term a member has seen. When the primary disappears, eligible secondaries wait out electionTimeoutMillis (default 10 s) with randomised jitter, then one of them calls for an election via replSetRequestVotes, advertising its term, its last applied opTime, and its priority.

Other voting members grant the vote only if that candidate's term is at least theirs, their oplog is at least as up to date, and they have not already voted in that term. The candidate needs a strict majority of voting members — two of three, three of five — and then runs replSetElection followed by replSetUpdatePosition to publish itself. On success it writes a no-op oplog entry, which is how the new term becomes visible, and clients discover it through SDAM. A split vote simply means the next term starts and the jitter makes a repeat unlikely.

Majority arithmetic is what makes elections safe, so what matters is that the voting population is odd and small. Three voters tolerate one failure; four voters still need three votes and tolerate only one failure while costing an extra node — strictly worse than three. Five tolerate two. Adding voters without adding failure tolerance only lengthens elections and increases the chance of a split vote.

An arbiter fixes the vote count but not the durability. It holds no data, cannot become primary and cannot serve reads, so in a two-data-node-plus-arbiter set a majority (primary + arbiter) can acknowledge a write that exists on exactly one data-bearing node. That is a real data-loss window on failover, not a theoretical one. For genuine high availability you want an odd number of data-bearing voting members, ideally across failure domains, and an arbiter only as a deliberate cost compromise.

An arbiter is a mongod started with replSetName that joins the set with arbiterOnly: true. It participates in heartbeats, stores the replica set config, votes in elections, and helps form a majority so an even number of data nodes can still elect a primary.

It does not store data, does not keep an oplog, cannot become primary, cannot serve reads or writes, and holds no copy from which to rebuild. It also cannot be a sync source, so a set of two data nodes plus an arbiter has exactly one usable replica of the data. Consequences worth stating in an interview: arbiters add latency to elections without adding durability, they still consume a host, and they must run the same major version as the data-bearing members. In modern deployments the honest advice is to prefer three data-bearing members; arbiters survive mainly in cost-constrained two-node setups and in DR configurations where the third site only needs a vote.

priority is a member setting from 0 to 1000 (default 1). It does not add votes; it biases who calls the election first and who wins. A member with higher priority triggers an election sooner after the primary is lost, and votes are only granted to a lower-priority candidate if no higher-priority eligible member is reasonably current. That is why a set with a preferred primary in the main region and a lower-priority DR member consistently comes back to the main region.

When priorities tie — the common case, all members at 1 — the tie is broken by oplog freshness and then by election jitter, so the winner is effectively arbitrary. Do not design around a specific winner. If you need determinism, set distinct priorities such as 5, 4, 3 rather than 1, 1, 1, keep the highest-priority nodes in the same low-latency failure domain, and remember that priority: 0 makes a member ineligible for election while it still votes and serves reads.

rs.stepDown(seconds) asks the current primary to relinquish leadership: it stops accepting writes, waits briefly for an electable secondary to catch up (bounded by secondaryCatchUpPeriodSecs), then closes client connections and campaigns for no election for seconds (default 60). During that window it is a normal secondary.

rs.stepDown(120)\n// or explicitly: db.adminCommand({ replSetStepDown: 120, secondaryCatchUpPeriodSecs: 15, force: false })

replSetFreeze is the same cooldown applied to a member that is not primary: db.adminCommand({ replSetFreeze: 300 }) prevents it from seeking election for 300 seconds, and { replSetFreeze: 0 } unfreezes. Use freeze to keep a node out of the race during maintenance, and stepdown to hand leadership to a specific node. Note force: true steps down even without a caught-up secondary — that trades a short write outage for a faster handover, so never use it casually in production.

When a primary is asked to step down, it does not want to hand leadership to a node that is minutes behind, because that node would then serve state that is missing recent writes. secondaryCatchUpPeriodSecs (member-level, default 10) is the maximum time the primary waits for an electable secondary to reach its own opTime before giving up and stepping down anyway.

db.adminCommand({ replSetStepDown: 120, secondaryCatchUpPeriodSecs: 30 })

Raise it when the set is large, the oplog application is behind, or a replica is across a slow link and you would rather wait than lose freshness. Lower it when you care more about a fast handover than about the new primary being perfectly current. If no secondary reaches the threshold within the window, the primary still steps down unless force: false and it decides to stay — which is why an explicit stepdown during maintenance should always be followed by checking rs.status() for the new primary and its optimeDate.

Three distinct outcomes, and the distinction is what interviewers probe. A write that reached the primary and was applied to the oplog locally but not yet replicated may be rolled back if the new primary never received it: the old primary truncates that suffix into a rollback/ directory. A write that got a majority commit point is safe. A write that was still in the network or in the storage engine when leadership was lost is simply not acknowledged to the client, and the client sees a network error or NotWritablePrimary.

The primary does not wait for in-flight operations to finish: it closes the listener, kills in-progress operations after a grace period, and lets the driver retry. This is why w: 1 plus a retry loop can produce an acknowledged-then-vanished document, and why writes must be idempotent (deterministic _id, upsert with $setOnInsert) for retries to be safe. With w: "majority" a successful response means the write survived the stepdown.

Two things. First, a member becomes primary only with votes from a strict majority of voting members, and each voter grants at most one vote per term — so two candidates cannot both collect a majority in the same term, because the two majorities would have to intersect in at least one voter. Second, an existing primary that cannot reach a majority of voting members for longer than the heartbeat timeout steps down voluntarily, so the minority side of a partition has no writable primary for long.

That intersection property is the whole guarantee: any two majorities of an odd set overlap. It also means the minority side is simply unavailable rather than divergent — availability is sacrificed, consistency is preserved. What can look like split brain is usually a stale driver view or a member that was reconfigured with force: true, which is the one operational way to manufacture two primaries. Duplicate primaries also appear when the same data directory is started twice.

A three-member set needs 2 votes: it survives one failure and loses writability if two members are down. A five-member set needs 3 votes: it survives two failures. In both cases you can lose floor(n/2) voters before writes stop, which is why the extra two nodes in a five-member set buy exactly one more tolerated failure at roughly 66% more cost.

The arithmetic also drives write latency: w: "majority" means the write is acknowledged once a majority has applied it, so in a five-member set spread across regions you need three acknowledgements, not two. And it drives election resilience: with five voters a single partition that isolates two members still leaves a majority, whereas in a three-member set isolating two members leaves one node that can never be primary. Choose the size from your failure-domain model and your budget, not by reflex — and remember that adding voters without adding data-bearing members changes nothing about durability.

By default, a secondary whose oplog is demonstrably ahead of the primary (it has applied writes the primary has not seen, typically after a former primary rejoined) can trigger a catch-up takeover: it forces an election rather than waiting, precisely because it is the better candidate. catchUpTakeoverDelayMillis controls how long it waits before doing so.

Setting settings.catchUpTakeoverDisabled: true in the replica set configuration turns that behaviour off, so a more current secondary waits for the normal election timeout and competes on priority instead of pre-empting. The practical use is environments where you want leadership to stay where you put it — for example a set with a strong regional preference — and importantly in some arbiter configurations, where catch-up takeover could otherwise push leadership onto an unexpected member. The price is a longer window after failover in which the best candidate is not yet primary. It is a rare, deliberate setting, not a default you should flip for convenience.

If the surviving members are fewer than a majority, the set has no primary at all — not a slow primary, no primary. Writes fail with NotWritablePrimary or a server-selection timeout, reads with readPreference: primary fail too, and secondaries keep serving only if the client explicitly allows secondary reads. Nothing recovers automatically: this is by design, because promoting a minority would risk losing committed data.

Recovery requires restoring enough voters to form a majority. That usually means bringing the failed region back, or performing a forced reconfiguration from the surviving side with rs.reconfig(cfg, {force: true}) to remove the unreachable members and lower the majority requirement. Forced reconfig is an operator decision with real consequences: the removed members still believe they are members, so when they return they must be wiped and resynced, and any writes they had that were not replicated are lost. This is the scenario to rehearse in a game day rather than discover at 3 a.m.

Members exchange heartbeats every settings.heartbeatIntervalMillis (default 2000 ms). A member marks a peer as unreachable after roughly heartbeatTimeoutSecs (default 10) of silence. When the primary is unreachable, an eligible secondary waits settings.electionTimeoutMillis (default 10000 ms) after detecting loss of the primary before calling for votes — and the wait is jittered by the protocol so candidates do not collide.

rs.conf().settings\n// { heartbeatIntervalMillis: 2000, electionTimeoutMillis: 10000, ... }

So the practical failover window is heartbeat detection plus election plus driver discovery, typically 10–30 seconds end to end. The numbers are a trade-off: shorter timeouts detect real failures faster but mistake GC pauses, snapshot stalls or transient packet loss for death, producing unnecessary elections. Longer timeouts trade availability for stability. Both are replica set settings and changing them requires a reconfig that all voting members accept.

Because the cost of a spurious election in a multi-region set is much higher than the cost of a slightly slower real failover. Cross-region round trips can be 100–250 ms, and a node under heavy write load may briefly stop responding to heartbeats because of cache eviction, a checkpoint stall or a long index build. With a 10-second timeout that pause becomes an election; the new primary is in another region, latency for your users triples, and when the old node returns it may trigger catch-up takeover and another election.

Raising electionTimeoutMillis to, say, 20–30 seconds means a real failure takes an extra 10–20 seconds to detect, but you avoid election churn from transient stalls. Pair it with a driver serverSelectionTimeoutMS comfortably above the new detection window, otherwise clients give up before the new primary is known. The setting must be accepted by all voting members, and it does not shorten the rollback risk window, so it is not free.

rs.status() is the wrapper around db.adminCommand({ replSetGetStatus: 1 }), and it is the single most informative command in a failover. Read it in this order: top-level myState and term; members[].stateStr to see who is PRIMARY, SECONDARY, RECOVERING or (ARBITER); members[].health, where 1 means reachable; members[].optimeDate and members[].replicationLag to quantify staleness; and members[].lastHeartbeatMessage, which often names the actual problem.

rs.status().members.map(m => ({ name: m.name, state: m.stateStr, health: m.health, lag: m.optimeDate }))

Then look at optimes.appliedOpTime, durableOpTime and lastCommittedOpTime to see whether the majority commit point is advancing, plus electionDate and setVersion. If setVersion differs between members you have a partially applied reconfiguration, which changes the whole diagnostic path.

Connect to a surviving member, take the current config, edit it, and reconfig with force: true, which skips the requirement that a majority of the old configuration acknowledge the change:

cfg = rs.conf()\ncfg.members = cfg.members.filter(m => m.host !== \"m3.example.com:27017\")\nrs.reconfig(cfg, { force: true })

The risk is that you are unilaterally rewriting history: any writes the isolated members had that never replicated are gone, the set version increases so those members cannot simply rejoin, and if the "unreachable" primary is actually alive and serving clients on a partition, you have two primaries writing divergent data until the old one is fenced off. Treat force: true as a last resort after confirming the missing members are truly down, wipe and resync them afterwards, and record the exact config you applied. In Atlas you cannot do this at all; support-driven recovery replaces it.

An oplog entry is a BSON document in local.oplog.rs with the fields ts (the BSON timestamp that defines order), t (term), h (hash), v (version), op (i, u, d, c, n), ns, ui (collection UUID), o (the payload) and, for updates, o2 (the query that selected the document). Transaction operations add lsid, txnNumber, stmtId and prevOpTime.

Idempotency comes from expressing changes as absolute state rather than as intent. An insert carries the whole document, a delete carries the _id, and an update records the specific fields it changed in o.$set or o.$unset rather than replaying the original operator tree, with o2 letting the applier find the target if the document moved. Applying the same entry twice therefore converges to the same state. That is what allows a secondary to retry a batch after a crash without corrupting data — It is also why $inc replays safely here but not in your own change stream consumer.

ts is a BSON timestamp: 4 bytes of seconds since the epoch plus a 4-byte ordinal that increments within that second. The primary assigns it when it appends the entry, and because appends are serialised under a lock the resulting sequence is a strict total order — there are no ties, and no two entries share a ts.

That single ordered log is the backbone of the whole system: secondaries apply entries in ts order, the majority commit point is a position in that order, change stream resume tokens are derived from it, and point-in-time recovery replays it up to a chosen ts. Comparisons are cheap and lexicographic, which is why ranges like { ts: { $gt: lastTs } } are the natural way to tail the oplog. It also explains why you cannot reorder or hand-craft oplog entries safely, and why the ordering is logical rather than wall-clock: wall exists as a separate field for human debugging, and clock skew never reorders replication.

The secondary's oplog fetcher pulls batches from its sync source with a query on local.oplog.rs filtered by ts greater than its last applied position, typically using an awaitable tailing cursor so new entries arrive without polling. The batch is applied by writer threads that perform the actual document changes in parallel for independent namespaces, then a barrier ensures the entries are inserted into the secondary's own oplog in exactly the original order. That is why secondaries keep an oplog of their own: it is the history they have applied, and it is what other members chain from.

Application is idempotent, so after a crash the node re-reads from its last durable position and reapplies, possibly duplicating work but not data. Two positions matter: appliedOpTime (what has been applied, possibly not yet durable) and durableOpTime (journaled). Monitoring serverStatus().metrics.repl.apply exposes batch counts and total apply time, which is where you look when applying rather than fetching is the bottleneck.

Do not use the configured size as a proxy — measure the window. rs.printReplicationInfo() reports the first and last oplog entry timestamps and the resulting time span; that span is the true window, and it shrinks as write volume grows. To project forward, measure how much oplog the primary generates per hour and divide:

// window_hours = oplogSizeMB / oplog_MB_per_hour\n// e.g. 51200 MB / (1200 MB/hour) = 42.6 hours

Get the rate from db.serverStatus().oplog (logSizeBytes divided across a time range) or by differencing the ts of the first and last entries over a known period while watching metrics.repl.buffer and actual throughput. Then account for peak, not average: a nightly bulk load can consume a day's window in an hour. A comfortable target is at least 24–72 hours at peak, so a weekend outage or a broken sync source does not force a full resync.

Both features read the same log. A change stream resumes from a resume token that encodes an oplog position; if the consumer was down long enough for that position to roll off the end of the oplog, the server cannot reconstruct what happened in between and returns an error such as ChangeStreamHistoryLost (or a resume error naming the missing token). There is no backfill — the events are simply gone, and the consumer must resynchronise from a full read of the collection and reconcile.

Point-in-time recovery has the same constraint from the other direction: restoring to a moment requires a base backup plus every oplog entry from that backup's position forward. If the oplog no longer covers the interval, the restore target is unreachable and you are left with the base snapshot plus whatever later full backup exists. The design rule is therefore blunt: size the oplog for your worst consumer downtime and your backup cadence, not for the average write rate, and alert on window as a first-class metric.

replSetResizeOplog changes the oplog's maximum size without restarting the member, and it must be run on every member individually — it is not a replica set setting:

db.adminCommand({ replSetResizeOplog: 1, size: 102400 })  // size in MB

Growing is safe and cheap: the collection's maximum size is raised and WiredTiger allocates as entries accumulate. Shrinking is the dangerous direction — MongoDB must truncate the oldest entries immediately to fit, which can cut the window from days to hours in one command and break change streams or PITR coverage that was relying on it. There is also a minimum size (roughly 990 MB) below which the command is rejected.

Because the oplog is a capped collection, space is not returned to the filesystem after a shrink; it is reused later. Run the change during low write volume, verify the new window with rs.printReplicationInfo() on that member, and remember that a resize does not propagate — a forgotten member keeps the old size and eventually has a different window than its peers.

appliedOpTime is how far a member has applied the oplog locally. lastCommittedOpTime — the majority commit point — is the newest entry that a majority of voting members have durably written, as tracked by the primary. They are different clocks: a secondary can be fully applied and yet the commit point has not moved, because some other member is behind.

That gap is exactly the window in which w: "majority" writes wait, so a single lagging member delays every majority write on the primary even though the primary itself is current. It also bounds what reads with readConcern: "majority" can see: they see only committed data, so they can appear to go backwards relative to a local read. And it bounds rollback: only entries above the commit point can be rolled back. When latency spikes with no CPU or IO pressure, compare rs.status().optimes.lastCommittedOpTime against each member's optime — that comparison usually names the culprit.

A rollback happens when a member's oplog contains entries that the new primary's history does not: the member was primary, accepted writes that never reached a majority, lost leadership, and must now discard that suffix to converge. The node transitions to ROLLBACK, finds the common point with its sync source, and writes the discarded documents to <dbPath>/rollback/ as BSON files named after the namespace and timestamp.

ls /var/lib/mongodb/rollback/\nbsondump <db>.<coll>.<timestamp>.bson | head

Those files are the only record; MongoDB does not merge them back. You inspect them, decide which documents are legitimate business writes, and reapply manually or with a script. Rollbacks are bounded by the oplog window and by rollbackTimeLimitSecs; beyond that the node cannot roll back and requires a full resync. Preventing them is a write concern decision: w: "majority" writes cannot be rolled back, which is the honest answer to "how do I avoid this".

With w: 1 the primary acknowledges as soon as it has applied the write locally. Nothing has been replicated yet. If the primary dies in the next few milliseconds, the surviving members elect a new primary from the data they have, the old primary's unsent suffix is discarded on rejoin, and the client's successful write never existed — a rollback, visible as a missing document.

The failure is worse than "we lost the last write": the client has already emitted a receipt, an email, a payment confirmation. Recovery means reconciling application state against a database that disagrees with it, which is why rollback files matter.

The fix is not exotic: use w: "majority" for anything a user was told succeeded. If the latency is unattractive, the honest engineering work is on topology and network placement, not on lowering the guarantee. w: 1 combined with an aggressive retry policy is a data-loss generator, because the retry can duplicate a write that was in fact rolled back.

It is a field in the replica set configuration (settings.writeConcernMajorityJournalDefault, default true) that decides whether w: "majority" also implies journaling on each acknowledging member. With the default, a majority write is acknowledged only after those members have written the entry to their on-disk journal, so it survives an OS crash or power loss, not just a process crash.

rs.conf().settings.writeConcernMajorityJournalDefault  // true

Setting it to false makes majority writes in-memory (j: false semantics): faster, and still safe against a mongod crash because WiredTiger can recover from its own log, but not safe against machine-level failure. The per-operation writeConcern: { w: "majority", j: true } still overrides it upward. Change this only with a clear durability story, and note that it is a set-wide setting applied by reconfiguration, so it affects every client at once — including ones that never asked for the looser behaviour.

Flow control is a mechanism that throttles the primary's write rate when secondaries fall too far behind. The primary continually measures the replication lag implied by its majority commit point; if that lag exceeds flowControlTargetLagSeconds (default 10 s), it starts injecting small delays into write tickets, and it logs a warning once lag passes the warning threshold.

db.adminCommand({ setParameter: 1, enableFlowControl: true })\ndb.serverStatus().metrics.operationCounters  // plus metrics.repl for lag context

It is enabled by default in recent versions and it is a back-pressure mechanism, not a fix: it buys the secondaries time to catch up by slowing the source, which shows up as increased write latency while lag stays flat. That pattern — rising latency, stable lag, low CPU — is the signature. The right response is to find why the secondaries are slow (IO, cache, network, missing indexes on the primary's hot path) rather than disabling flow control, which just lets the oplog window burn faster.

A transaction is not one entry. Each statement inside it produces an oplog entry carrying the session and transaction identifiers — lsid, txnNumber — plus a stmtId, and consecutive entries are chained with prevOpTime so a secondary can tell whether it has the whole chain. Entries are written as partial transaction operations; only the commit (or abort) entry makes the group visible.

Consequences you should be able to state. First, a transaction of 500 writes adds roughly 500 entries and consumes oplog space in proportion, so large transactions shorten the effective window. Second, replication of a transaction is atomic from the reader's point of view: a change stream does not emit half a transaction. Third, oplog size and majority commit position are evaluated across the whole chain, so an uncommitted transaction's entries cannot be part of the commit point. Fourth, the 16 MB per-entry limit still applies, but the transaction total is bounded by the transaction lifetime and resource limits, not by one entry.

A normal rs.reconfig does not rewrite history — raising setVersion changes membership, not the log. Truncation happens through three real paths. A member that must roll back truncates the divergent suffix at the common point with its sync source. Shrinking the oplog with replSetResizeOplog deletes the oldest entries immediately and can cut the window to hours. A member that is removed and re-added starts with an empty oplog and resyncs from scratch, so its window restarts at zero.

What breaks is anything anchored to an absolute oplog position: change stream resume tokens whose position is gone fail with a history-lost error, point-in-time restore targets before the new first entry become unreachable, and a replica trying to catch up from a point that no longer exists on any peer is forced into a full initial sync. This is why reconfigurations that touch votes or membership should be done one change at a time, with the oplog window and every change stream consumer's checkpoint verified before and after.

Yes. A non-voting member is a full replica: it keeps an oplog, applies writes, can be a sync source for other members, and serves reads. What it cannot do is vote in elections or count toward the majority, so it neither helps nor hurts the set's ability to elect a primary, and it does not count toward the majority required by w: "majority".

cfg = rs.conf()\ncfg.members[2].votes = 0\ncfg.members[2].priority = 0\nrs.reconfig(cfg)

The typical use is a disaster-recovery copy in a third region: it costs replication bandwidth and a host, but you deliberately do not want it participating in elections, because a slow link would add latency to every vote and every majority write. Limits to remember: a set allows up to 50 members and at most 7 voting members, changing votes changes the majority arithmetic and should be a separate reconfiguration from changes to priority or host names, and a votes:0 node with priority above 0 is rejected.

Historically initial sync was logical: the new member builds indexes from the config, copies every collection by scanning it, then applies the oplog tail accumulated during the copy. It works everywhere but is slow and hammers the sync source with reads.

Since 5.0 there is also file-copy-based initial sync: the source streams its data files while writes continue, then the new member applies the oplog from the captured start position. It is dramatically faster on large datasets because it moves files instead of documents and does not rebuild indexes one document at a time. Conditions matter: both nodes must use WiredTiger, the source must be a healthy member of the same set, and if the copy cannot be validated or the engines differ, MongoDB falls back to the logical path. Either way the new member must not serve reads during sync and must not be electable — set priority: 0 and hidden: true before rs.add(), so a half-synced node cannot win an election.

Nothing blocks writes — the primary keeps serving — but a syncing member can hurt in four ways. It performs full collection scans on its sync source, which by default may be the primary, consuming IO and cache. If it builds indexes logically, that is index-build load on the new node, not the primary, but the read pressure is real. It also requires the primary to retain oplog entries for the entire sync: if the sync takes longer than the oplog window, the process restarts from zero, potentially forever. And if the new member is eligible, it may be elected while still syncing.

Mitigations: pre-create the member as hidden with priority: 0; point it at a dedicated sync source if you have one (initialSyncSourceReadPreference, or by placing the source deliberately); grow the oplog before large syncs; and run the sync in a low-traffic window for large collections. Watch rs.status() for the STARTUP2 to RECOVERING to SECONDARY transition and confirm the lag reaches zero before promoting.

Stage it as a passive node and promote it only after it is caught up. Create the config entry with the safe flags rather than accepting defaults:

rs.add({ host: \"m4.example.com:27017\", priority: 0, hidden: true, votes: 0 })\n// wait for stateStr: SECONDARY and lag to settle, then:\ncfg = rs.conf(); cfg.members[3].priority = 1; cfg.members[3].hidden = false\ncfg.members[3].votes = 1; rs.reconfig(cfg)

hidden: true keeps drivers from routing reads to it, priority: 0 keeps it out of elections, and votes: 0 means the initial sync cannot change the majority arithmetic mid-flight. Note buildIndexes: false is another option for pure DR nodes, but such members can never become primary and must stay at priority 0 and votes 0. Change one dimension at a time: promoting votes and priority in the same reconfiguration as the addition is how people accidentally create a set where the new, still-warm node is one of the three voters.

Start with the secondary's ability to apply, not with the network. The usual order: IO saturation on the secondary (slow disks, competing backup or snapshot jobs) shows as high apply batch time; CPU or cache pressure, where the working set no longer fits and WiredTiger evicts constantly; missing or badly designed indexes, so each applied update triggers a collection scan; large write bursts on the primary, including bulk loads and big transactions; and network throughput or latency between members, which shows as a slow fetcher while apply time stays low.

The diagnostic split is fetch versus apply. db.serverStatus().metrics.repl.buffer tells you whether entries are queued waiting to be applied; metrics.repl.apply.batches.totalMillis tells you how long applying takes. If the buffer is empty and lag persists, the fetcher or network is the problem. If the buffer grows, the applier is the bottleneck — that is IO, CPU or index work, and it will not be fixed by adding bandwidth.

rs.status() gives you the operational view: per-member optimeDate and a computed replicationLag in seconds, plus syncingTo so you can see the replication chain. rs.printSecondaryReplicationInfo() formats it readably. The caveat is that these are sampled at the moment you call it and derived from timestamps, so a momentarily idle primary can make lag look like zero.

rs.printSecondaryReplicationInfo()\ndb.serverStatus().metrics.repl\n// apply.batches.num, apply.batches.totalMillis, apply.ops,\n// buffer.count, buffer.sizeBytes, preload

metrics.repl is the counter view for monitoring systems: buffer depth tells you backlog, apply batch timings tell you applier health, and deltas over time reveal a trend rather than a snapshot. Alert on sustained lag — five minutes above threshold — not on a single spike, and alert separately on buffer.count growth, which precedes visible lag and is the earlier warning.

secondaryDelaySecs (member setting, the replacement for slaveDelay as of 5.0) makes a member apply oplog entries only after they are that many seconds old — typically 3600 for an hour of protection. The member must be priority: 0 and hidden: true, because it is intentionally stale and must never serve as primary or receive normal reads.

cfg = rs.conf()\ncfg.members[2].secondaryDelaySecs = 3600\ncfg.members[2].priority = 0\ncfg.members[2].hidden = true\nrs.reconfig(cfg)

What it protects against is a logical error that replicates fast: a bad migration, an errant deleteMany, a bug that overwrites a field. You notice within the hour, stop the delayed member before the damage reaches it, and use it as the recovery source. What it does not protect against is hardware loss of the primary plus the other replicas, or an error nobody notices for longer than the delay.

Because the delay is a scheduling device, not a veto. secondaryDelaySecs: 3600 means "apply each entry one hour after its timestamp"; the node still receives the oplog continuously and will apply the drop or deleteMany as soon as its clock passes the threshold. If nobody intervenes, the delayed member becomes identical to the others — including the damage.

The protection only exists if you act inside the window: detect the incident, then stop the delayed member (shut it down or block replication) before the entry is applied, verify its data is pre-incident, and treat it as the restore source. That requires monitoring and a documented runbook, because the window is measured in the delay you configured. The other real gap is that a delayed member cannot help with a failure that happens before the delay elapses: if the primary and the other secondaries are lost, the delayed node is an hour behind with no source to catch up from.

replSetFreeze tells a member not to seek election for a number of seconds. It is the tool for keeping a node out of the election race — during maintenance, while you are validating a rebuild, or when you want a specific region to hold leadership:

db.adminCommand({ replSetFreeze: 600 })   // stay secondary for 10 minutes\ndb.adminCommand({ replSetFreeze: 0 })     // unfreeze immediately

A stepdown's cooldown is the same idea applied to the member that just gave up leadership: rs.stepDown(120) both hands over and prevents that node from re-winning for 120 seconds. The difference is the starting state and the intent: stepdown moves leadership away from a primary that is currently serving; freeze suppresses a candidacy that has not started. Both are temporary by design — they do not persist across restarts and they do not survive a reconfig. Freezing every member by mistake leaves the set with no eligible candidate, so apply it to the nodes you want to exclude, and record when it expires.

Apply backlog is the gap between what the secondary has fetched and what it has applied. It is visible as metrics.repl.buffer.count and buffer.sizeBytes, and it is the leading indicator of lag: the fetcher can be keeping up perfectly while the applier falls behind.

The applier takes locks on the namespaces it writes, and it yields those locks so that user reads on the secondary are not starved. That politeness is exactly why heavy read traffic on a secondary slows replication: each yield is progress lost, and with a large read workload the applier can spend most of its time waiting to reacquire locks. Long-running reads, unindexed queries and background analytics on the same node all make it worse. Countermeasures are structural — keep reporting off members that must stay current, use read preference tags so real-time reads go elsewhere, ensure the secondary's queries are indexed, and check apply.batches timing after any change in read traffic. You cannot tune yielding away; you can only stop competing with the applier.

Usually because the secondary's workload is different from the primary's, not because replication is heavier. A secondary that also serves reads carries the WiredTiger cache plus query working sets. Applying a large burst — a bulk load, a big transaction, an index build — forces cache eviction, and eviction pressure plus concurrent reader allocations can push the process past the container limit. In containers the classic failure is that the cgroup limit is lower than what cacheSizeGB assumes, so the kernel OOM-kills mongod before WiredTiger evicts.

Fixes: size storage.wiredTiger.engineConfig.cacheSizeGB for the container, not the host; leave headroom for query memory and connections; remove analytics from members that must stay current; avoid unbounded $in and huge batch reads; and watch cache evicted pages and cache bytes read into cache in FTDC. A node killed repeatedly will resync, which loads it even more — a loop worth breaking early.

It adds one replication round trip plus the journal write on the acknowledging members, so the floor is roughly the latency to the majority position, not to the farthest node. In a three-member set with two members in the same datacentre, that is a local network hop plus fsync — single-digit milliseconds. Spread three members across three regions and the majority includes a cross-region hop, so 80–200 ms per write becomes normal.

The second component is the tail: w: "majority" waits for the slowest member that completes the majority, so one lagging replica can hold up every write even though the others are current — the commit point is a minimum, not an average. This is why latency spikes with flat CPU usually mean a replication problem, not a query problem. Measure with serverStatus().metrics.getLastError.wtime and compare lastCommittedOpTime to each member's optime before blaming the network.

nearest means "lowest network latency among members that are eligible", and it includes the primary. It says nothing about freshness, load or role. The traps follow directly: in a low-latency local network the "nearest" member may be a secondary that is 30 seconds behind, or it may be the primary, so your read distribution is not what you assumed; and because latency is measured by the driver's own round-trip samples, the choice can flip between members and give you inconsistent results across consecutive reads.

If what you want is "spread reads across secondaries", say so with secondaryPreferred. If you want "read from a nearby secondary but not a stale one", combine it with maxStalenessSeconds, which filters out members whose estimated lag exceeds the limit. If you need read-your-writes, nearest is the wrong tool entirely — use primary or causal consistency with afterClusterTime. And always test with a realistic topology, because nearest behaves very differently in one datacentre than across three regions.

The driver's SDAM layer keeps a topology description built from its own monitoring connections: each server has a type (RSPrimary, RSSecondary, RSArbiter, Unknown) and a round-trip time. After a partition heals, monitoring resumes, servers transition from Unknown back to their real types, and server selection then picks a member matching the read preference — preferring lower RTT among eligible candidates, and requiring a primary for writes. Reconnection is therefore driven by the monitor's heartbeatFrequencyMS (default 10 s, minimum 500 ms), not by the application's retry rate.

What makes it slow: the driver must wait for a server to be rediscovered and for a primary to exist; if the partition caused an election, that is a second wait. A large maxPoolSize then opens many connections at once on the newly discovered primary, which can look like a second outage. That is why minHeartbeatFrequencyMS, pool sizing and a serverSelectionTimeoutMS longer than the expected failover window are all part of the same design decision.

A heartbeat that arrives late is indistinguishable from a member that is gone, so anything that pauses a process produces the same signal as a crash: a long GC pause in a client-side proxy, a checkpoint or eviction stall, a hypervisor snapshot, a saturated NIC, a CPU steal event on a shared host. If the pause exceeds the heartbeat timeout, peers mark the member unreachable and may elect a new primary; when it returns, the set churns again.

Reduce them by removing the causes rather than by shortening timeouts: keep mongod off oversubscribed hosts, avoid storage snapshots during peak, keep the working set inside cache, and monitor serverStatus().metrics.repl alongside host metrics so you can correlate a lag spike with a disk stall. If pauses are inherent to your platform, raising electionTimeoutMillis is the correct lever. And do not run members across links where packet loss is routine: heartbeats are small and cheap, but they are also the only evidence of liveness you have.

Server Discovery and Monitoring maintains a topology description and a monitor per known server. When the primary's monitoring connection fails, its type becomes Unknown, the topology emits a description change, and any in-progress selection fails or blocks. The driver keeps monitoring all seeds and all discovered members; when a secondary is promoted, the next heartbeat identifies it as RSPrimary, the topology becomes usable for writes again, and the connection pool for that server is created and warmed on demand.

Two behaviours matter operationally. First, operations already sent to the old primary fail with a network error or a "not primary" error; the driver does not silently move them. Second, retryable writes (with retryWrites=true) get exactly one retry after a fresh server selection, which usually lands on the new primary. Everything else — non-retryable commands, transactions mid-flight, reads outside a session — is your application's problem, which is why serverSelectionTimeoutMS must exceed the election window and why applications need their own retry policy with idempotent operations.

You see MongoServerSelectionError (Node.js and others) or a Timeout naming the topology, with a message listing each server and why it was rejected — "connection refused", "server is in state RSSecondary", "not primary". That detail is the diagnostic: it tells you whether no primary existed, whether the hosts were unreachable, or whether your read preference could not be satisfied.

// connection string\nmongodb://h1,h2,h3/db?replicaSet=rs0&serverSelectionTimeoutMS=20000

The default is 30 seconds, which is generous for elections but not for a region-level outage. If timeouts cluster exactly at the timeout value, clients are waiting the full window — that is an availability problem, not a slow query. If they cluster earlier, you are hitting a pool or socket timeout instead. Read the full error string before changing anything: raising serverSelectionTimeoutMS hides the symptom while requests pile up, and lowering it turns a 12-second election into a burst of application errors.

An operation is safe to retry when applying it twice has the same effect as applying it once, and when the retry cannot be confused with a different logical request. In MongoDB terms that means: a client-supplied deterministic _id on inserts so a duplicate-key error tells you the first attempt succeeded; upserts with $setOnInsert rather than plain inserts; updates that set absolute values rather than incrementing; and a natural idempotency key stored in the document so a replayed request is detectable.

db.payments.updateOne(\n  { _id: requestId },\n  { $setOnInsert: { amount, currency, status: \"pending\" } },\n  { upsert: true })

Note the asymmetry: retryable writes protect the database from double-applying, but they cannot know whether your side effect — an email, a charge — already happened. And a rolled-back w: 1 write is the worst case, because the retry legitimately re-creates something the client already treated as done.

Make the drill a normal, reversible operation and run it during business hours with observers. Preconditions: w: "majority" on critical writes, retryable writes enabled, idempotent handlers, dashboards on lag and election counts, and a rollback plan that is just "step down again". Then, in ascending order of realism, run rs.stepDown(60) on a secondary first as a no-op sanity check, then on the primary while measuring client error rate and p99 latency through the handover. Escalate in staging to kill -9 of the primary process, and to a blocked port (iptables) to simulate a partition rather than a clean exit.

db.adminCommand({ replSetStepDown: 60, secondaryCatchUpPeriodSecs: 15 })

Record time-to-new-primary, time-to-first-successful-write, connection storm size, and whether any write was lost. A drill that only verifies "a new primary appeared" teaches nothing; the value is the client-side timeline. Announce it, keep it short, and never drill during a backup or index build.

Never touch two members at once, and always leave the primary for last. The loop is: pick a secondary, confirm rs.status() shows it healthy and caught up, confirm the other two are healthy, stop it, do the work, restart it, watch it return to SECONDARY and reach zero lag, then move on. When only the primary remains, either step it down so the work happens on a secondary-turned-primary, or restart it and accept one election.

rs.status().members.map(m => [m.name, m.stateStr, m.health])\nrs.stepDown(120)

Guardrails that prevent most incidents: check the oplog window before starting, because a node offline longer than the window forces a full resync; verify the majority still exists at every step (in a three-member set, one node down is your entire budget); pause backup jobs and index builds; and keep a written abort criterion — for example "if lag exceeds 60 s or a second member becomes unhealthy, stop". Rolling maintenance is a sequencing discipline, not a technical trick.

rs.stepDown() is a cooperative handover. The primary stops accepting writes, waits for an electable secondary to catch up, then relinquishes leadership; the set elects a new primary in seconds, usually with no rollback because the successor is current. Connections close cleanly and the old primary becomes a secondary that immediately starts applying.

Killing the process (kill -9, OOM kill, host failure) is an uncooperative failure. No handover, no catch-up wait: peers detect the loss by heartbeat timeout, the election starts later, in-flight writes are simply lost, and the primary's unjournalled suffix may be rolled back when it restarts and recovers through WiredTiger. It is also the only way to test crash recovery and the driver's real behaviour.

So: use stepDown for planned maintenance — it is faster and cleaner. Use process kills deliberately in drills and chaos tests, never as a shortcut during maintenance. And remember stepDown does not simulate the case where the primary is unreachable rather than polite, which is the failure mode that actually hurts.

Do the arithmetic before the command. A set needs a strict majority of voting members to elect and to commit w: "majority" writes; in a three-voter set that is two, so one member can be down. During any change, count how many voters would be unavailable if the change goes wrong.

Adding: add the member with votes: 0 and priority: 0 first, let it sync, then raise votes in a separate reconfiguration, one change at a time. Removing: remove non-voting members first, and never remove a member whose departure leaves the set with an even, undersized voting population mid-flight. Use rs.remove() only from the primary, and if the primary is unavailable use rs.reconfig(cfg, {force: true}) knowing it is a last resort.

rs.add({ host: \"m4:27017\", votes: 0, priority: 0 })\nrs.remove(\"m4:27017\")

Remember the limits: at most 50 members and 7 voting members. And verify after each step that rs.status() still has a primary.

Host names live in the replica set configuration, not in the data, so the operation is a reconfiguration — but the sequencing matters because members address each other by the names in the config. Prepare DNS first: every new name must resolve, from every member and from every client, to the right host, and it must be the name the member itself will use as its bindIp/advertised identity.

cfg = rs.conf()\ncfg.members[1].host = \"m2.newzone.example.com:27017\"\nrs.reconfig(cfg)

Change one member at a time, connect to the primary through a name that is not changing, and expect a brief window where heartbeats to that member fail until DNS and the member agree. You cannot change the _id of a member, and you cannot set a host that another member already uses. If you are changing the primary's name, step it down first so you are not reconfiguring the node you are connected to. Finally, update connection strings and any monitoring that keys on host names in the same change window.

Hidden members have priority: 0 and hidden: true: they replicate and vote, but cannot become primary and are not selected by normal read preference modes. That makes them a reasonable home for reporting only if you accept the coupling: every analytics query competes with the oplog applier for cache and IO on that node, and a saturated reporting member falls behind.

Practical setup: give the node more memory than the others, keep its queries indexed and bounded, put a hard maxTimeMS on reports, and pin reporting clients to it deliberately — most drivers will not send secondary reads to a hidden member, so either connect with directConnection=true or use a tag set that matches only that member and confirm your driver's selection rules.

cfg.members[2].tags = { role: \"reporting\" }\n// client: readPreferenceTags=role:reporting

The alternative worth mentioning in an interview is a dedicated analytics path (Atlas Analytics node, or a change-stream-fed store) so reporting never shares fate with replication.

mongosh connects with readPreference: primary unless the URI says otherwise, so rs.secondaryOk() — the legacy shell helper — or the modern equivalent is needed to read from a secondary:

db.getMongo().setReadPref(\"secondary\")\ndb.getMongo().setReadPref(\"secondary\", [{ region: \"us-east\" }])\ndb.getMongo().getReadPref()

Two things surprise people. First, the setting is per connection, so a new shell or a new URI resets it. Second, some operations simply refuse to run on a secondary regardless of preference: writes, $out/$merge stages, and anything requiring a primary. If you get "not primary and secondaryOk=false", you are on a secondary with the default preference — not a permissions problem. For diagnostics it is often better to connect directly to a specific member (directConnection=true) so you know exactly which node answered, rather than relying on preference-based routing.

Lag alone is a late signal. A defensible set: sustained lag above threshold for several minutes; oplog window (as a duration, from rs.printReplicationInfo()) trending below your recovery objective; election frequency, because more than a couple per day means instability, from db.serverStatus().metrics.repl.electionMetrics (elections called, successful elections, average election time) and the stepdown counter; member state flapping, counted as transitions per hour rather than a snapshot; apply backlog (metrics.repl.buffer.count) growing; and write concern latency (metrics.getLastError.wtime) p99, which is the client-visible symptom of a lagging majority.

Also alert on "no primary" as a distinct page, and on a member in RECOVERING or ROLLBACK for more than a few minutes. Every one of these should be a rate or a duration, not an instantaneous value — a single sample of lag is noise, and paging on noise trains people to ignore pages.

The mechanics are the same — majority vote, election, new primary — but the operational surface is different. Atlas runs a three-node replica set (or a sharded cluster) on managed infrastructure, keeps an odd number of voters, performs rolling maintenance and version upgrades for you, and exposes failover through a documented "Test Failover" operation rather than rs.stepDown(). You do not get rs.reconfig or replSetStepDown privileges, so forced reconfiguration and topology surgery become a support-assisted recovery.

What Atlas gives you: automated backups and PITR, alerts on replication lag and elections, multi-region and global clusters with zone-based placement, and a connection string that abstracts the member list. What it does not give you: control over election timing, arbitrary member types, or the ability to inspect rollback files. Self-managed gives you all of that plus the obligation to rehearse it. The interview point is that failover design is about the client and the write concern, not about who runs the nodes — a badly designed client fails over badly on either platform.

Do not try to repair it in place on a live set. The reliable path is replacement: stop the member, confirm the other members are healthy and hold a majority, then remove it from the configuration (or leave it there while it is down if a majority still exists), wipe its dbPath, and restart it so it performs a fresh initial sync.

// on the primary, only if a majority remains\nrs.remove(\"bad:27017\")\n// on the failed host: stop mongod, move the old dbPath aside, start clean\nrs.add({ host: \"bad:27017\", priority: 0, hidden: true, votes: 0 })

Before wiping, preserve the old files: if the corruption is logical, they may be the only copy of a write that never replicated, and mongod --repair or a bsondump salvage can recover documents — but only offline and only as a last resort, never as a step towards rejoining. Validate afterwards with db.collection.validate() if you suspect storage-layer corruption, and check the host's disk and RAM, because a node that corrupts once will do it again.

Model the surviving state explicitly: with a 3-2-2 or 2-2-1 layout, losing a region leaves fewer members than before, and if the lost region held a majority of voters there is no primary at all until a forced reconfig. Capacity planning therefore has two parts — votes and resources.

Votes: ensure the remaining regions can still form a majority, which usually means the two data-bearing regions must each hold enough voters, or you accept an operator-driven forced reconfig with a documented time budget. Resources: the surviving members must absorb the full read and write load, so check headroom for 100% traffic: RAM for the working set, IOPS, and connections. Also re-evaluate w: "majority" latency, because the majority now spans a longer path, and re-check oplog sizing since the surviving primary writes everything.

The honest test is a game day: take a region out, measure time-to-writable and throughput, and write down what broke. Numbers from a spreadsheet are not a capacity plan.

Because it looks redundant and is not. You have two copies of the data, so a single host failure leaves you with one copy and no redundancy; the arbiter supplies the third vote but no third copy. Elections work, which is what makes the design seductive, but the durability story is unchanged from a single node.

The specific hazard is w: "majority". In a PSA (primary-secondary-arbiter) set, the majority is two votes, and the primary plus the arbiter satisfy it — so a write can be acknowledged as majority-committed while existing on exactly one data-bearing member. If that member then fails before replicating, the write is rolled back, which contradicts what "majority" is supposed to mean. Add the practical costs: arbiters add election latency, cannot serve reads, cannot be a sync source, and must be upgraded in lockstep, so they are operational overhead with no data benefit.

The credible alternatives are three data-bearing members, or accepting a two-node design explicitly as non-HA with a documented RPO above zero.

It protects against one failure mode — a mongod process or host dying — and against nothing else. An AZ failure, a rack power event, or a control-plane incident takes all three members at once, so you have the cost of a replica set and the availability of a single node.

The mismatch is between the failure domain you replicate across and the failure domain your SLA names. If the business cares about AZ-level events, members must be in distinct AZs; if it cares about region-level events, you need a cross-region member, which then changes latency and w: "majority" cost. A common compromise: two members in the primary AZ for low-latency majority writes, one in a second AZ with priority: 0, and an explicit statement that a primary-AZ loss means a manual or scripted recovery. Whatever you choose, verify that the placement is enforced and not just documented — a node that drifts back into the same AZ during a rebuild silently breaks the design.

The obvious damage is local: reporting queries evict the working set from the secondary's WiredTiger cache, the oplog applier yields its locks to those reads, and the node falls behind. The non-obvious damage is on the primary, through the majority commit point. w: "majority" waits for a majority of voting members to durably apply each write, so a secondary that is 90 seconds behind reporting load becomes the slowest member of every majority — and every write on the primary inherits that latency.

So a reporting job can present as "the primary is slow" with the primary's own CPU and IO flat. The diagnostic is to compare lastCommittedOpTime with each member's optime; the offender is the one holding the commit point back. Mitigations: keep heavy analytics off voting members, use a dedicated hidden member with more RAM, bound queries with maxTimeMS and indexes, or feed analytics from change streams into a separate store. Do not solve it by lowering write concern — that trades latency for data loss.

It makes "confirmed" a lie. A payment recorded with w: 1 can be rolled back after the client has been told it succeeded, leaving an external charge with no matching record. The failure is silent and asymmetric: the ledger says it never happened, the provider says it did, and reconciliation is manual.

The correct write concern for money movement is { w: "majority", j: true }, on the same operation that changes the balance, ideally with a unique index or transaction ensuring the invariant. If the latency is unacceptable, the design changes, not the guarantee: keep the majority in one low-latency region, write a local intent record with w: 1 plus a domain event confirmed durably by another process, and let the state machine tolerate a pending state. What you must never do is tell the user "paid" on the basis of an uncommitted write.

Also, retryWrites will happily retry a rolled-back write and double-charge unless the operation carries an idempotency token: the retry layer is not a substitute for durability.

Because secondary says nothing about how current the data is. Replication is asynchronous, so a secondary can be seconds or minutes behind, and the driver will happily serve a read from the most available secondary. The user writes a profile, reloads, and sees the old value — the classic read-your-writes violation, which looks like a caching bug and is not.

Three fixes, in increasing strength. Use primary for anything that must reflect a just-completed write. Use readConcern: "majority" so at least the data is committed, though that still permits staleness relative to your own write. Or use causal consistency: a session that carries afterClusterTime guarantees the secondary waits until it has applied everything the primary had when your write was acknowledged.

const session = client.startSession({ causalConsistency: true });\nawait coll.updateOne({ _id }, { $set: { name } }, { session });\nawait coll.findOne({ _id }, { session, readPreference: \"secondary\" });

Also remember maxStalenessSeconds is a bound, not a guarantee of freshness.

Every restart is a full data loss for that member. A pod rescheduled onto another node starts with an empty dbPath, so it performs an initial sync — expensive, and impossible if the rest of the set is unavailable. Worse, if all three members restart at once, they come up empty, have no oplog and no data, and there is nothing to sync from: the deployment is simply gone.

The related failure is identity. Replica set membership is keyed on host names, so with ephemeral pod names every restart requires a reconfiguration, which is exactly the operational burden a StatefulSet with stable network identities and a PersistentVolumeClaim per replica exists to remove. Checklist: a PVC per member, stable DNS names, podManagementPolicy: Parallel only when you understand election timing, resource limits above the WiredTiger cache, and a readiness probe that checks replication health rather than just port availability. And back up independently of the volumes — a storage class that loses its backing disk is not a backup strategy.

As a server-selection problem that looks like a database outage. Symptoms: MongoServerSelectionError or a topology timeout, application startup failures, and a message listing hosts that could not be reached — while the cluster is perfectly healthy and reachable from another network.

Frequent causes: the URI lists a hostname only resolvable inside the VPC; a mongodb+srv record whose SRV targets are private; a member's configured host name pointing at an internal name that clients cannot resolve; certificate names not matching the name used; or split-horizon DNS where the container resolver differs from the host's. Remember that a replica set's members address each other using the names in rs.conf(), so a client can connect but still fail to discover the primary if the discovered names are unresolvable.

nslookup m2.example.com\nmongosh \"mongodb://h1:27017/?replicaSet=rs0\" --eval \"rs.status().members.map(m => m.name)\"

Fix by using names that are resolvable from every client and every member, and verify from inside the same network namespace the app runs in.

retryWrites=true (default in modern drivers) retries a single write once, after a fresh server selection, when the error is retryable and the deployment supports it. That covers the common blip — a primary stepping down between the write and its acknowledgement. It does not cover a down primary for 60 seconds, a write that failed for a validation or duplicate-key reason, a non-retryable command, or anything outside a session-aware write path.

Design implications: the write must still be idempotent for the cases retryable writes cannot see, such as a w: 1 write that was rolled back, or a client crash after the server applied the write but before the response arrived. Give inserts a deterministic _id, use $setOnInsert upserts keyed by an idempotency token, and never build business logic that assumes exactly-once execution. Retryable writes reduce error rates; they do not make an operation idempotent, and they cannot protect a side effect like an email or a card charge that lives outside the database.

First separate the two meanings: it is not primary, or it is primary and refusing. Check rs.status().myState and db.hello().isWritablePrimary. If it is not primary, you have an election problem — look at term, electionDate and whether a majority is reachable. If it says primary but writes fail, the errors name the cause: NotWritablePrimary after a stepdown, a WriteConcernFailed timeout from a lagging majority, or a disk-full error.

db.hello()\ndb.serverStatus().wiredTiger.cache\ndb.currentOp({ active: true })\nrs.printReplicationInfo()

Common culprits: the disk is full or the volume is read-only, so WiredTiger cannot write; oplog or journal space is exhausted; w: "majority" is timing out because one member is stuck, so writes wait rather than fail fast; flow control is throttling; or the node is in a stepdown cooldown and has not re-won. Check lastHeartbeatMessage for the other members, and look at filesystem free space before anything else — it is the most common and most boring answer.

Repeated elections mean the set keeps changing its mind about who should lead. Typical causes: an unstable network where heartbeats fail intermittently; electionTimeoutMillis set too low for the platform's pause behaviour; nodes too overloaded to respond within the timeout; members whose oplog is far behind, so a candidate can never win and the vote repeats; and priority configurations that make two members compete evenly while the third keeps abstaining.

Diagnose by reading metrics.repl.electionMetrics and the server log for "election succeeded/failed" reasons, then correlating with host metrics. Breaking the loop: raise electionTimeoutMillis so transient stalls stop triggering votes, fix the underlying IO or CPU saturation, give one member a clearly higher priority so leadership is deterministic, and make sure all voting members can actually reach each other. In a multi-region set, also confirm the two low-latency members are the likely candidates, since electing the far member adds latency to every subsequent write.

You discover behaviour changes in production. Between majors, MongoDB changes defaults, removes deprecated commands and options, bumps index versions, and — most importantly — ties new features to the feature compatibility version. If you raise setFeatureCompatibilityVersion before every member and every driver path is ready, you lose the ability to downgrade, because the on-disk format is already the new one.

The disciplined sequence: read the release notes for removed and deprecated behaviour, check driver and tool versions, upgrade secondaries one at a time, step down the primary, upgrade it last, then raise FCV as a separate deliberate step after the set has been stable. Watch for silent changes — default write concern, slaveDelay becoming secondaryDelaySecs in 5.0, telemetry and logging additions. Test the upgrade on a restored copy of production, not on an empty cluster, because the failures that matter are data-shape and query-planner failures. Treat a downgrade path as a requirement, not a luxury.

Because the database's redundancy stops at the cluster boundary. A sharded cluster with three-member replica sets per shard is still unavailable if the only mongos process is down, or if the single application host is down, or if the load balancer in front of them has one target. The replica set protects data and provides failover; none of that helps when there is no router to send the request through.

The fix is mundane and often skipped: run at least two mongos routers per application, list multiple hosts in the connection string (host1,host2 using topology discovery), put them behind a load balancer with health checks, and don't let the routers share a host or AZ. On the application side, run more than one instance and make the readiness probe depend on a database round trip, so a broken instance leaves the pool instead of receiving traffic. Any place where a single process is the only path to data is a SPOF, regardless of how many replicas exist behind it.

The cost is one cross-region round trip per write, plus journaling, with the majority defined by member placement. Three regions with one member each means every write needs an acknowledgement from another continent: 100–250 ms p50 and much worse at p99. The same guarantee with two members in the primary region costs a local hop, because the majority is satisfied locally.

Alternatives, in order of preference. Place two of three voting members in the primary region and keep the third as a priority: 0 DR member, so majority latency is local while durability still spans regions. Reduce the number of operations needing majority: batch writes, use transactions for multi-document invariants instead of per-document round trips, and reserve strict concern for state transitions rather than telemetry. Consider w: "majority" with j: false only if you understand the difference between process and host failure. What you should not do is quietly drop to w: 1 on financial data — that moves the cost from latency to reconciliation.

Start from the business, not from the tool. RPO is how much data you may lose; RTO is how long you may be down. An RPO of fifteen minutes rules out nightly dumps entirely — you need continuous oplog-based recovery, such as Atlas continuous cloud backups or volume snapshots plus oplog replay. An RPO of twenty-four hours tolerates one snapshot a day, but only if the business truly accepts losing a day of writes.

Then convert RTO into restore throughput. Measure a full restore of your real data size, add cluster provisioning, DNS or connection-string cutover, TLS validation and application smoke tests. If a 400 GB restore takes three hours, an RTO of one hour is fiction unless you keep a warm standby cluster.

Finally tier your datasets: a payments collection and a log archive should not share the same policy, and the tiering decision belongs in writing, signed by the data owner.

Each answers a different question.

• Logical dump (mongodump): portable, selective, survives version and platform changes, restores one collection if needed. Slow on large datasets, heavy on the source, and not point-in-time consistent without oplog capture.
• Volume snapshot: fast and cheap at scale, taken outside the database, restored by attaching disks. It is tied to the storage provider and to the same major version, and it captures a whole deployment, not a collection.
• Atlas continuous cloud backup: snapshots plus oplog, giving point-in-time restore to a new or existing cluster with almost no load on the source. The trade-off is that you inherit Atlas retention limits, region choices and pricing.

Mature deployments usually run snapshots as the primary recovery mechanism and periodic logical dumps as the portability and compliance layer.

WiredTiger writes to the journal continuously but only checkpoints the data files periodically — every 60 seconds by default. Between checkpoints, committed writes live in memory and in the journal, so the data files on disk are a torn mix of old and new pages: half-written index entries, half-applied updates.

Copy those files while mongod runs and you get a database where pages disagree with each other. Include the journal in the same snapshot and recovery can replay from the last checkpoint to a consistent point, because the journal is the source of truth for everything after the checkpoint.

This is why "snapshot the volume" is only a backup when the whole data directory, journal included, is captured atomically. Splitting the journal onto a different volume that is snapshotted separately is a classic way to produce a backup that will not open.

db.fsyncLock()   // optional: freeze writes for the copy window
db.fsyncUnlock()

Back up a secondary, never the primary, so the dump does not evict the working set the application depends on. Prefer a member that exists for this purpose: a hidden, priority-0 secondary in the same region, or a dedicated backup node.

Consistency comes from the read position, not the member. Use --oplog with mongodump so the dump records the oplog position when it started and captures the oplog entries written during the dump; mongorestore --oplogReplay then applies them, giving a point-in-time consistent restore across collections.

mongodump --host rs0/backup1:27017 --readPreference=secondary \
          --oplog --gzip --archive=/backup/rs0-$(date +%F).gz

The write concern that matters here is the one your application uses: if it writes with w:1, a secondary backup can legitimately miss acknowledged writes that were rolled back after a failover, so the backup can be "consistent" and still be behind what clients were told was committed.

Because chunk metadata and the data it describes must come from the same instant. The config database records which ranges of the shard key live on which shard; if you dump shard A, then the balancer migrates a chunk from A to B, then you dump shard B, your backup describes a cluster that never existed. Restoring it produces missing ranges or duplicate ranges, and the cluster refuses to start cleanly.

The safe sequence is to stop the balancer, capture the config database and each shard at effectively the same point, then re-enable balancing.

sh.stopBalancer()
// dump config servers, then each shard, recording timestamps
sh.startBalancer()

For restore the order is reversed: config first, so mongos knows the topology, then the shards. This is also why a single mongodump through a mongos is not a valid sharded backup — it reads collections sequentially while the cluster keeps moving chunks underneath it.

Index definitions are part of the dump: mongodump writes them alongside each collection, and mongorestore recreates them unless you pass --noIndexRestore. They are also the cheapest thing to lose, because they can be rebuilt from the data — at a cost. On a multi-terabyte collection an index build can take hours and saturate I/O, so a restore that rebuilds every index is a slower restore.

Configuration data is different, and people forget it:

• The config database in a sharded cluster — without it the dump is unrestorable.
• admin.system.users and admin.system.roles — your users and custom roles.
• Collection options: validators, collation, time series options, clustered index settings.

A useful trick for large restores: restore data with --noIndexRestore, then build indexes afterwards, and let secondaries build in parallel rather than serially during the load.

mongodump reads every document of every collection through normal queries. On the primary that means a long stream of full collection scans, each pulling pages into the WiredTiger cache and evicting the hot working set. The visible symptoms are a latency spike, more disk reads and a cache that suddenly has to refill — during business hours, on the node serving writes.

Practical mitigations, in order of preference:

1. Dump from a dedicated hidden or priority-0 secondary, ideally in the same region to avoid cross-region read cost.
2. Use managed snapshots so the source sees no query load at all.
3. Throttle: --numParallelCollections 1, a projection of only what you need, and a maintenance window.
4. Prefer --gzip --archive to a local fast disk, not to a network share that back-pressures the read.

mongodump --host rs0/backup1:27017 --readPreference=secondary \
          --numParallelCollections=1 --gzip --archive=/backup/dump.gz

Note that a dump is not a snapshot: without --oplog different collections are read at different times, which is fine for reference data and wrong for money.

Compression is the easy half: mongodump --gzip --archive=... produces a single compressed stream, and mongorestore --gzip --archive=... reads it back. A single archive is much easier to ship, checksum and encrypt than a directory tree.

Encryption is the half people get wrong. gzip is not encryption. Encrypt the artifact after dumping, or encrypt the destination:

mongodump --gzip --archive=/tmp/d.gz
age -r $BACKUP_PUBKEY -o /backup/d.gz.age /tmp/d.gz
sha256sum /backup/d.gz.age > /backup/d.gz.age.sha256

Keep the private key outside the backup account and outside the database account — a key stored beside the ciphertext means a single compromised credential loses both. Object storage with server-side encryption and a bucket policy that denies deletion is a good second layer, and writing the checksum next to the artifact lets you prove the file you are about to restore is the file you wrote.

Retention exists to satisfy two different clocks: operational recovery (hours to days, where you need dense restore points) and compliance or investigation (months to years, where you need sparse but immutable copies). A grandfather-father-son scheme serves both: keep 7 daily, 4 weekly, 12 monthly, and one or two yearly snapshots.

Three details decide whether the policy survives an audit:

• Immutability. If a compromised administrator can delete snapshots, retention is a suggestion. Atlas backup compliance policies make retention unchangeable, block early deletion and require extra confirmation; object storage has equivalent lock modes.
• Legal hold. A hold must override expiry, so it needs an owner, a ticket reference and a documented release process.
• Deletion semantics. Know whether terminating a cluster deletes its backups — on some managed tiers it does.

Also align retention with your oplog window: a snapshot you cannot roll forward is worth less than it looks.

They buy survival of a regional failure and, more importantly, isolation from your own account: a copy in a different region and a different account is the only thing that helps when the primary region is down or the credentials are compromised.

The cost is threefold: storage duplicated per region, inter-region data transfer on every copy, and slower restores because the bytes are far from the compute you will restore onto. On Atlas this is delivered by additional snapshot regions or by snapshot distribution to another region; the restore target must then be created there.

Design questions worth asking in the interview:

• Is the backup region in the same cloud and the same legal jurisdiction? Data-residency rules may forbid the copy you just made.
• Do you have the keys there too? An encrypted backup in a region where the KMS key is unavailable is not a backup.
• Have you restored from the remote copy at least once?

A restore test is a rehearsal with a clock and a checklist, not a mongorestore that exits zero.

1. Pick a real snapshot or dump from production, not a synthetic dataset.
2. Provision a disposable cluster at the same major version, with the same TLS and auth configuration as production.
3. Restore and time it, including provisioning and index builds.
4. Verify: document counts per collection against the source at that timestamp, validate({full: true}) on the critical collections, index count and types, users and roles present.
5. Point a real application build at it and run smoke tests — reads, writes, an aggregation, a login.
6. Record the measured RTO and the failures found, then destroy the cluster.

Monthly for the tier-1 dataset, quarterly for the rest, and always after a topology change, a major upgrade or a backup policy change. Automate it so it runs without a human remembering, and alert when the drill fails — a silent drill is theatre.

Two different mechanics, and only one of them is native.

From a logical dump you have full control: restore one namespace into a side database and copy across, or restore directly with a namespace filter.

mongorestore --gzip --archive=/backup/dump.gz \
  --nsInclude "shop.orders" --nsFrom "shop.*" --nsTo "recovery.*"
mongorestore --gzip --archive=/backup/dump.gz --nsInclude "shop.orders" \
  --queryFile ./range.json

// range.json
{ "createdAt": { "$gte": { "$date": "2026-01-01T00:00:00Z" },
                 "$lt":  { "$date": "2026-02-01T00:00:00Z" } } }

From continuous backup point-in-time restore is cluster-wide: you cannot ask Atlas to rewind one collection. The pattern is to restore to a temporary cluster at the chosen timestamp, export the collection, and load it into production.

Watch the consequences of partial data: documents that reference the missing ones now dangle, unique indexes can collide with rows that already exist, and any derived or aggregated state is now wrong. Partial restore is a surgical tool, and surgery needs a plan for referential integrity.

You get operation granularity, not wall-clock granularity. The restore replays oplog entries up to your target timestamp, and the last entry applied is the last one whose timestamp is less than or equal to the target. So the finest real resolution is "the state after the last write before T" — if nothing happened for ten minutes, no timestamp in that window produces a different result.

Three constraints bound what you can promise:

• Atomic units. A transaction or an index build appears in the oplog as a single unit, so you cannot restore to a point inside it.
• Oplog window. The rewindable range starts at the oldest oplog entry, not at the oldest snapshot.
• Scope. It is cluster-wide. There is no per-collection PITR, which is why "just rewind the orders collection" is usually a two-step restore.

In an interview, answer with the window you can prove — "we can rewind 48 hours with per-operation precision" — rather than an unqualified "we have PITR".

Your promise silently becomes a snapshot restore, and every write between the snapshot and the incident is gone. This is the most common false guarantee in MongoDB operations, because the oplog window shrinks precisely when you need it: a bulk import, a backfill, or a re-index generates far more oplog than the steady state, and the oldest entries roll off.

Detection is a metric, not a hope. Track the window as the timestamp difference between the first and last oplog entry and alert when it drops below your committed RPO horizon:

db.getReplicationInfo()
// or
db.getSiblingDB("local").oplog.rs.find().sort({$natural:1}).limit(1)

Mitigations: size the oplog for peak write volume rather than average, set a minimum retention where the platform offers one (Atlas exposes an oplog minimum retention setting), and cap bulk jobs so they cannot outrun retention. Then write the honest fallback into the runbook: if the target predates the oplog, restore the newest snapshot and reconcile from application logs.

You keep production serving while the restore happens somewhere else, then move the pointer. The restore target is a separate cluster, built to the same major version, with the same auth, TLS and index configuration; you restore the dump or the snapshot there and validate while users keep writing to the old system.

Cutover is where the discipline is:

1. Restore to time T, then keep replaying or dual-writing so the gap stays small.
2. Freeze writes with a short maintenance flag, or flip the application to a write-pause mode.
3. Apply the final delta, verify counts on the few critical collections.
4. Switch the connection string — DNS with a low TTL, or a secrets-manager value the app re-reads.
5. Keep the old cluster readable for a defined rollback window, then decommission.

Watch for cached connections: drivers hold sockets to the old host for minutes, so force a restart or accept a mixed-write period you must reconcile.

Time a real drill end to end and break it into phases with timestamps: detection and decision, cluster provisioning, artifact transfer, data load, index builds, validation, cutover, and cache warm-up. Almost always the documented number was computed from the load only, or from a dataset a tenth of the current size.

Typical findings and their fixes:

• Transfer dominates because the artifact is in another region → co-locate backup and restore target.
• Index builds are serial → restore with --noIndexRestore, load, then build in parallel.
• One monolithic mongorestore → raise parallel collections, use a bigger target host, restore independent collections concurrently.
• Human latency → pre-authorised runbook, two named decision makers, credentials already escrowed.

Then publish the measured number and either invest to hit the target or renegotiate it with the business. An RTO nobody has tested is a rumour.

Declaring a disaster is a decision with a named owner, not a consensus. Typically the on-call engineer detects, the incident commander declares, and the data owner approves any restore that will discard writes — because rewinding to T-10 minutes destroys ten minutes of real customer transactions. That approval is a two-person rule, logged with a timestamp.

The first page should be readable under stress:

• Declaration criteria, as falsifiable conditions: region unavailable, corruption on two members, confirmed data loss beyond RPO.
• Named roles: incident commander, database lead, comms.
• Decision tree: restore in place versus restore to a new cluster versus fail over to the DR region, with the trade-off in one line each.
• Exact commands and identifiers: cluster names, snapshot IDs, the KMS key alias, the escalation phone numbers.

Bias to declaring early: a false alarm costs a meeting, a late declaration costs the RPO. And rehearse the runbook — a document nobody has executed is fiction with good formatting.

Two separate problems that people merge and then lose half of.

Inside the database, users and roles live in the admin database — system.users and system.roles. Dumping admin captures them, including the stored SCRAM credentials, so a restore keeps existing passwords. Many teams prefer to treat users as code instead: keep role definitions in version control and recreate users from a script, so the restore is reproducible and reviewable rather than a binary blob.

mongodump --db admin --collection system.roles --out /backup/admin
db.getSiblingDB("admin").getRoles()

For Atlas, database users and most security settings are not in the cluster at all — they live in the control plane. Back them up through the Admin API or, better, Terraform: project, cluster tier, IP access list, private endpoints, custom roles, alert configuration, backup compliance policy. A cluster restored from snapshot comes back without any of that context, so an infrastructure-as-code repository is the real configuration backup.

Detection comes from three places. First, WiredTiger checksums: a bad page surfaces as a checksum mismatch in the log and often prevents startup or fails reads on the affected pages. Second, validation: db.collection.validate({full: true}) walks the collection and its indexes and reports inconsistencies — expensive, so run it on a secondary, one collection at a time. Third, symptom-level signals: documents that fail to decode, indexes whose counts drift, aggregation results that stop reconciling.

What you can salvage depends on the damage. Index corruption is cheap: drop and rebuild. Document damage on one member is fixed by resyncing it from a healthy peer. Logical corruption — bad writes, a buggy migration — is not storage damage at all, and only a point-in-time restore fixes it, because every member faithfully holds the same wrong data.

Keep the damaged files before you repair anything; forensics after an overwrite is guesswork.

You let the replica set heal itself rather than restoring a backup. The clean version is initial sync: stop mongod on the bad member, move or delete its data directory, start it again with the same configuration, and it copies the full dataset from a healthy member and then tails the oplog. The cluster keeps a primary and at least one other secondary the whole time, so availability is unaffected — provided you never take two members down at once.

systemctl stop mongod
mv /var/lib/mongo /var/lib/mongo.corrupt.$(date +%s)
systemctl start mongod
rs.status()   // watch stateStr: STARTUP2 -> SECONDARY

Practical cautions: initial sync is I/O- and network-heavy, so throttle it or do it off-peak; a member with a huge dataset can take hours; and if the member is also a backup source, pause that job. If the data directory is intact and only the process failed, restarting is enough — resync is for corruption, not for a crash.