A statistical model that learned the joint distribution of data and can sample from it. Everything else — chat, code, images, agents — is engineering around that one primitive.
A generative model learns P(x) — the probability distribution of the training data — and then draws new samples from it. For language models the distribution is factorised one token at a time:
P(x1…xn) = ∏t=1..n P(xt | x1…xt-1)
That single equation explains most behaviour you will observe: fluent output, no source of truth, sensitivity to how the prefix is phrased, and the fact that a wrong answer is generated with exactly the same machinery as a right one.
It is
It is not
A compressor of statistical regularities in its training data
A database of facts with citations
Deterministic given a seed, temperature and prompt
Reproducible across model versions
Excellent at transformation, summarisation, drafting, classification
Reliable at exact arithmetic, current events, or counting
A component inside a system that supplies context and checks
A replacement for retrieval, rules, or a business process
Mental model. Treat the model as a very well-read intern with no memory of yesterday, no access to your systems, and full confidence. The engineering job is to give it context, constrain its output, and verify what it returns.
2. Anatomy of a generative AI system
The model is one box in five. Most quality gains in production come from the other four.
Each stage owns a different failure mode: retrieval owns relevance, prompt assembly owns instruction drift, the model owns fluency and reasoning, validation owns correctness of contract, and output owns user trust.
3. The transformer in one page
Text is split into tokens (sub-word units). Each token becomes an embedding vector; the sequence passes through N identical blocks of self-attention plus a feed-forward network; a final linear layer scores every vocabulary item for the next position.
Self-attention
Attention(Q, K, V) = softmax( Q·KT / √dk ) · V
Each token builds a query, compares it against the keys of all previous tokens, normalises the scores with softmax, and mixes the corresponding values. The 1/√dk factor keeps the dot products from saturating the softmax as dimensionality grows.
Concept
What it buys you
Where you feel it
Context window
How much text the attention can look at
Long documents, chat history, agent transcripts
Causal mask
Token t only sees tokens before it
Why generation is left-to-right and streaming works
KV cache
Reuse of previous keys/values
Latency and memory per concurrent request
Positional encoding
Order information
Degradation on very long or shuffled inputs
Attention is not memory. Everything the model "knows" about your session lives in the tokens you resend. Truncate the transcript and the knowledge disappears with it.
4. How a model is trained
Stage
Objective
Data
Effect
Pre-training
Next-token prediction on trillions of tokens
Web, books, code
Knowledge, syntax, reasoning traces
Supervised fine-tuning (SFT)
Imitate curated instruction→answer pairs
Tens of thousands of examples
Follows instructions, adopts format
Preference tuning (RLHF / DPO)
Prefer chosen over rejected answers
Pairwise human or model judgements
Helpfulness, tone, refusal behaviour
Distillation
Small model mimics large model outputs
Teacher generations
Cheaper, faster, slightly less capable
LoRA / adapters
Fine-tune low-rank deltas
Domain corpus
Style and vocabulary without full retraining
Fine-tuning vs retrieval. Use retrieval when the problem is missing knowledge. Use fine-tuning when the problem is format, vocabulary or task shape. Fine-tuning facts into a model is expensive and drifts; retrieval stays auditable.
5. Decoding: turning probabilities into text
At each step the model produces a score for every token in the vocabulary. A softmax turns scores into probabilities and a decoding rule picks one.
pi = exp(zi/T) / ∑j exp(zj/T)
Parameter
What it does
Practical setting
temperature
Flattens (high) or sharpens (low) the distribution
0–0.3 for extraction and code; 0.7–1.0 for ideation
top_p
Keeps the smallest set of tokens whose mass reaches p
0.9–0.95; do not tune together with temperature
top_k
Keeps only the k highest-probability tokens
Rarely needed when top_p is set
max_tokens
Hard stop on generation length
Always set it; truncation must be handled
stop sequences
End generation on a marker
Cheap way to bound structured output
Determinism. Temperature 0 makes sampling greedy, not reproducible: floating-point non-associativity, batching and provider-side version changes still move the output. Design for it — validate the contract, not the exact string.
6. Beyond text: images, audio, video
Modality
Core mechanism
Controls that matter
Images
Diffusion: learn to reverse a noising process, then sample from noise
Prompt, negative prompt, steps, guidance scale, seed, aspect ratio
Speech
Neural codecs plus autoregressive or flow models
Voice, speaking rate, prosody, language
Video
Temporal diffusion or latent sequence models
Duration, motion strength, keyframe conditioning
Embeddings
Encoder mapping text to a vector space
Dimension, normalisation, distance metric
Diffusion is worth one formula because it explains the main knob, guidance: the model learns εθ(xt, t), the noise present at step t, and sampling walks backwards. Classifier-free guidance extrapolates between the unconditional and conditional predictions, trading diversity for prompt adherence.
7. Grounding with retrieval (RAG)
Retrieval-augmented generation replaces the model's memory with a search step whose results are pasted into the prompt. It converts an unsolvable problem (make the model know your data) into a solvable one (make search return the right chunk).
Retrieval cannot fix a bad question. Query rewriting, synonym expansion and filters usually move relevance more than swapping the embedding model.
8. Prompting patterns that survive production
Pattern
Use when
Sketch
Role + contract
Any task with a fixed output shape
"You extract fields. Reply only with JSON matching this schema."
Few-shot
Format or judgement is hard to describe
2–5 representative input/output pairs, including a hard negative
Chain of thought
Multi-step arithmetic or logic
"Work through the steps, then give the final answer on its own line."
Decomposition
Task mixes independent sub-problems
One call per sub-task, deterministic assembly
Self-check
Output is verifiable
Second pass: "verify each claim against the sources; list failures."
Tool use
Data or actions live outside the model
Typed functions with validation on the way in and out
Prompt hygiene. Version prompts in the repository, pin the model version, and treat every prompt change as a code change with an evaluation run attached. A prompt nobody can diff is a configuration bug waiting to happen.
9. Evaluation: what to measure
Layer
Metric
Notes
Task accuracy
Exact match, F1, pass@k
Only valid with a stable, versioned gold set
Grounding
Faithfulness, citation precision
Check that quoted spans exist in the source
Retrieval
Recall@k, MRR, nDCG
Measure separately from generation
Safety
Refusal rate on adversarial set, leakage rate
Needs a red-team corpus that grows over time
Operations
p95 latency, cost per request, error rate
Product metrics, not model metrics
Preference
Win rate against a baseline
LLM-as-judge is fast but biased; calibrate on human labels
Judge bias. An LLM judge prefers longer answers, its own family of models, and the first option it sees. Randomise order, pin the judge version, and keep a human-labelled slice as ground truth.