THE DESCENT

Here is the one fact that reorganizes everything below it. Every time you press Enter, your app mails the entire conversation so far — hidden instructions, every earlier turn from both sides, and your new sentence — to a server. The server reads all of it, writes the next chunk, sends it back, and keeps nothing. There is no "session" living somewhere with your name on it. (Providers do cache and log as optimizations and policy — but correctness never depends on the server remembering you.) Each turn is one stateless HTTPS POST. The client serializes the system prompt plus every prior turn into a JSON messages array; the server tokenizes it, runs the model, streams the reply back, and retains no conversational state for the next call. "Memory" is client-side replay. Caching and logging exist server-side as optimizations and policy — but correctness never depends on the server remembering you.

That single fact explains most of what people find mysterious: why the model knows nothing about your other chats (they were never in the envelope), why long conversations get slower and pricier (the envelope grows every turn), and why "it remembered me" always means "software on the client side put that back into the envelope." Statelessness is a deliberate architecture choice, the same one behind every scalable REST service: a request that carries all its own context can be served by any replica, so the provider gets horizontal scale and the client inherits the memory problem — transcripts, summaries, retrieval are all client-side responsibilities.

C++ The conversation is a std::vector<Message> — roughly struct Message { Role role; std::string content; } — serialized to JSON and re-sent whole on every call: push_back, never mutate.

This is not a metaphor — below is essentially the whole thing your chat app sends. An ordinary web request, the same kind your browser makes for any page, carrying one JSON body. Step through it. The wire shape below follows the published Anthropic Messages API — every field shown is in the public docs; other providers differ in field names, not in kind. Step through it.

One honest caveat about that hidden field: the system prompt mechanism is public science, but any given product's exact production text is only known if its vendor chooses to publish it not public. Roles are just special marker tokens in one linear sequence — there is no hard channel separating "instructions" from "input" at the wire level. The model is trained to weight system-role text more heavily, which is why prompt injection (untrusted text impersonating instructions) is a structural risk to engineer around, not a bug to patch. The mechanism is public; a specific product's deployed system-prompt text is the vendor's to publish or not not public.

The reply "typing itself" across your screen is not theater. The model genuinely produces one token at a time, and the server forwards each one the instant it exists — waiting for the whole answer before showing anything would be the same computation with worse manners. With stream: true the response arrives as SSE deltas — in the Anthropic shape, content_block_delta events — which the client concatenates into the visible reply. Same tokens, same order; the win is time-to-first-token, not total time.

The envelope has a hard size limit. Hidden instructions, tool descriptions, every turn either of you has ever sent in this chat, plus room reserved for the reply — all of it shares one fixed budget. When the budget fills, something must be thrown away or compressed, and that is the moment a long chat starts "forgetting" its own beginning. The context window is one budget with four tenants: system prompt, tool definitions, full history, reserved output. Published frontier limits run from 128K to roughly 1M tokens public science — the numbers are in each vendor's API docs. Overflow means truncation, compaction (summarize old turns), or a hard error; the model cannot attend to what was never sent.

C++ The context window is a fixed stack-frame budget, not a heap — everything the call needs must fit in one frame. Overflow errors at the API; the silent fall-off of the oldest data happens in the client's truncation policy, not on the server.

Run the numbers once and long-chat behavior stops being mysterious. Take a 200,000-token budget, subtract fixed overhead and reply room, and divide by the cost of a typical turn: Worked with round numbers — a 200,000-token budget, 3,000 tokens of fixed system + tool overhead, 4,000 reserved for output, and an average of 1,200 tokens per full turn (user message + assistant reply):

But the sharper insight is the total bill. Because the whole conversation is re-mailed every turn, you pay for the early turns again and again — over that 160-turn chat you are billed for about eighty times more tokens than the final transcript actually contains. Append-only replay has a quadratic cost curve: turn k re-sends all k−1 prior turns, so a T-turn chat bills O(T²) input tokens in total, not O(T). Summing the arithmetic series for T = 160:

That one calculation is why every serious AI system summarizes old turns, retrieves only what is relevant, or starts fresh — and why "just make the window bigger" changes what is affordable, never the mechanism. Hold that thought: the descent now follows the envelope inside the machine, starting with what a token actually is. Quadratic replay is the root cause behind three optimizations you will meet on the way down and back up: KV caching inside one generation (Level 3), prompt caching across turns (published, priced API features public science), and context compaction in agent loops (Level 5). Next: what a "token" actually is.

The model never sees letters. A tokenizer chops your text into reusable fragments — whole common words, pieces of rare ones — and hands the model only the fragment numbers. Everything the model will ever know about your sentence has to survive that conversion. Input is segmented by BPE (or a close relative): a merge table learned offline by greedily fusing the most frequent adjacent pair, applied at runtime as a deterministic merge-replay encode. Output: a sequence of integer ids.

A rough napkin figure for English prose: ~4 characters ≈ 1 token — it drifts for code, dense JSON, and non-Latin scripts, which tokenize less efficiently. Vocabularies typically run tens of thousands to a few hundred thousand entries architecture class · publicly inferred

The vocab table — the first data structure

The tokenizer's whole worldview fits in one two-column table: a fragment of text on one side, its id number on the other. That file ships with the model and never changes after training. Concretely, a string→int map serialized to disk (open models ship it as a JSON vocab plus an ordered merges list), inverted at decode time to an id-indexed array of strings.

{ "the": 12, "str": 901, "aw": 33, "berry": 17, "␣tastes": 2044, … }  // vocab: string → id
[ "t h", "th e", "s t", "st r", "a w", … ]                            // merges: ordered rules

C++ Tokens ≈ indices into a vocab array: decode is const char* s = vocab[id]; — encode is the only interesting direction, and it is a replay of the learned merges over the merge table.

From id to vector — the embedding lookup

An id number carries no meaning by itself — 17 is not "berry-ish". So the model keeps a giant learned table with one row per vocabulary entry, and a token's meaning is its entire row: a long list of numbers positioning it in a space where words used alike sit close together. The embedding matrix W_E has shape V × d_model; lookup is a bare array index, no hashing, no search. Directions in that space are discovered during training, not designed — nothing labels them.

Internal width d_model is commonly a few thousand for frontier-class models architecture class · publicly inferred — worked examples on this page use toy sizes (d_model = 4, V = 8) so every number fits on screen.

C++ Embedding lookup ≈ &weights[token_id * d_model] — a float* lookup table: one strided read per token, then billions of multiply-accumulates; tokenization cost is noise next to the forward pass.

The dot product — similarity as alignment

Once words are points, one question drives everything downstream: how alike are two of them? The machine's answer is a single number — big when two directions agree, zero when they have nothing in common. Every "which words matter here?" decision at Level 2 is built from exactly this. The dot product is the primitive under attention scores: multiply component-wise, sum. Level 2's Q·Kᵀ is this operation done for every token pair at once.

C++ A dot product is the tightest loop in computing: float s = 0; for (int i = 0; i < d; i++) s += a[i] * b[i]; — attention is billions of these, batched into GEMM calls.

One honest caveat before descending: this whole stage is order-blind. "The dog bit the man" and "the man bit the dog" produce the same set of points — so position has to be injected explicitly, which is part of what the next level's machinery handles. Embedding lookup is per-token and attention is permutation-equivariant, so positional information is added to the embeddings or folded into attention itself; several schemes exist, with rotary embeddings widely used in open models — which one a given frontier product uses is unpublished not public

Carry two things down to Level 2: your sentence is now a grid of numbers, and similarity between any two words is one cheap multiplication away. X ∈ ℝ^{positions×d_model} is the residual stream's initial state, and the dot product you just computed by hand is the exact operation inside softmax(QKᵀ/√d_k)V.

If you have ever written a neural network's forward pass in C++ — allocate the buffers, loop over layers, multiply, add, apply a nonlinearity — nothing on this level is foreign. A transformer block is that same forward pass, industrialised: This level is where the reading actually happens — and it is not magic, it is arithmetic on an industrial scale: bigger matrices, one genuinely new operation, and a strict discipline about how information flows. The new operation is self-attention, and it is the machinery behind every moment a model seems to "remember" what you said.

Attention is how the model reads. When an answer depends on something said three pages earlier, attention is what reaches back and pulls exactly that forward — not by keyword search, but by learned relevance. It is the single idea that made this generation of AI work. Self-attention computes, for each position, a similarity-weighted average over the value vectors of all visible positions — weights recomputed from the data on every forward pass. Content-addressed, not address-addressed: nothing is looked up in a table.

Three roles for every token

Each token's vector is projected — by three learned weight matrices — into three new vectors. Same input, three different lenses:

C++ A hash map is address-addressed: one key, one bucket, one value. Attention is a content-addressed gather-reduce — score q against every stored k (in practice, GEMM calls), softmax the scores, return the blend Σ wᵢ·vᵢ. That is also the cost story: a map lookup is O(1); attention is O(n) per position, O(n²) per sequence.

The formula

The whole mechanism is one line of math: compare, weigh, blend. Everything else on this level is that line, unpacked. Dot every query against every key, scale by √d_k so the softmax stays in a well-behaved range, normalize each row into weights, and take the weighted sum of the values. One expression — the engine of the whole architecture.

The load-bearing word is softmax. It turns raw scores — any real numbers — into weights that are all positive and sum to exactly 1:

Softmax, by hand

Numbers are the proof. Here is one attention step small enough to do on paper: one question, three candidates, and watch how the votes get cast and counted. One query against three 2-D keys, no scaling — every arithmetic step shown. This is the worked example to reproduce on a whiteboard from memory.

Two guards on the scores

Before the votes are counted, two rules apply: keep the scores in a calm range, and never let a word see the future. A model that could peek ahead while learning to predict would be cheating on its own exam. Scale by 1/√d_k — in high dimensions raw dot products grow with √d, and large scores push softmax into near-one-hot saturation where gradients vanish. Then apply the causal mask before the softmax.

That second rule is the causal mask: position n may attend to positions 1…n, never to n+1. Future scores are set to −∞ before the softmax, so e^−∞ = 0 and the future gets exactly zero weight — the only exact zeros in the whole mechanism.

The full computation, with real numbers

Now the same story at full width: four words go in as four rows of numbers, and seven moves later they come out understanding each other. Every number on this screen is genuinely computed — nothing is decorative. Toy dimensions — 4 tokens, d_model = 4, single head — but real arithmetic end to end: X → Q,K,V → QKᵀ → ÷√d_k + mask → row softmax → ×V → residual add → MLP. Production models run exactly this with d in the thousands and many heads.

The same thing in twelve lines of C

Strip away the scale and the entire mechanism fits on an index card. The magic is not in the code — it is in the learned numbers the code multiplies. One head, one query position, no batching — the shape of every real implementation. Production engines fuse the loops into GEMMs and fused kernels, but compute the same thing.

C++ Attention at production scale ≈ GEMM calls: three projections, one QKᵀ, one weights-times-V — five big matrix multiplies per head per layer, all against preallocated buffers.

The block around the attention

Attention is half of a layer. Each block wraps two sub-blocks, and the wrapping is not plumbing — it is what makes depth possible. First, every sub-block sits on a residual connection: the output is added to the input, never replacing it, so information can ride an untouched highway from the first layer to the last. Second, each sub-block is paired with layer normalization — one line's worth of idea: rescale each vector to a calm mean and variance so dozens of stacked layers stay numerically sane.

The second sub-block is a small private network that each word passes through alone — no talking between words. Attention moves information between words; this refines what each word now knows. Much of what the model has memorized about the world appears to live here. The feed-forward network is two linear maps with a nonlinearity between, applied per position: expand to ~4×d_model, squash back. Empirically, much of the model's stored factual and procedural knowledge is localized in these weights.

Many heads, then many floors

The attention trick runs many times in parallel, each copy with its own learned lens — one may track grammar, another track who "she" refers to, another track facts. Then the whole two-part block is stacked, floor on floor, and the tower is the model. Multi-head: h independent sets of Q/K/V projections at d_k = d_model/h each, run in parallel, concatenated, projected back to d_model. Then the block stacks: output of layer i is input of layer i+1, shape n×d_model preserved the whole way down.

That parallel trick is multi-head attention; one detail worth saying aloud: attention itself is order-blind — shuffle the tokens and the same weights appear, shuffled. Word order only exists because position was injected into the vectors back on Level 1.

How deep do the towers go? Open models publish their configurations — GPT-2 small, published: 12 layers, d_model = 768, 12 heads. Frontier models of this architecture class are typically dozens of blocks deep with model widths in the thousands architecture class · publicly inferred. The exact layer count, width, and head count of Claude or GPT-class production models not public — treating that boundary precisely is the difference between knowing the field and guessing.

C++ A block ≈ a pure function float[n][d] → float[n][d]; the network is that one function composed N times — you could typedef it.

The forward pass ends in a bet sheet

After all the matrix math of Level 2, the model's output is surprisingly plain: a raw score for every token it knows. Every word-piece in the vocabulary gets a number saying "how strongly I'd bet the next token is you." Nothing has been chosen yet — the model hands back the whole bet sheet. The final hidden state of the last position is multiplied by an unembedding matrix W_U ∈ ℝ^d×V, yielding one raw score per vocabulary entry. These are unbounded reals — no probabilities yet, and only the last position's row matters for the next token.

That raw score vector has a name: logits — and everything sampling does downstream is a transformation of this one array.

C++ The logit vector is literally float logits[V] — one flat array, V = vocab size; the model's entire "decision" is an argmax-or-sample over one array of floats.

The bet sheet is long. GPT-2's published vocabulary is 50,257 entries; current open models publish vocabularies around 128,000. Every entry gets scored — every single token. Published tokenizers put V between ~5·10⁴ and ~2·10⁵ (GPT-2: 50,257; Llama 3: ~128k — both published). The unembedding is a [1×d]·[d×V] GEMM per decode step, so vocabulary size is a real serving cost, not a footnote. public science

Same softmax, new knob

The same softmax from Level 2 turns scores into a distribution — the formula doesn't change, the audience does: four attention scores there, the entire vocabulary here. The knob that reshapes it is temperature, and its cold limit has a name too: greedy decoding at T → 0.

Temperature is the model's risk dial. Turn it down and the favorite wins almost every time — crisp, repeatable, a little boring. Turn it up and the long shots get real chances — varied, creative, occasionally weird. Same scores underneath; different appetite. Dividing logits by T before exponentiating scales the gaps between scores: T → 0 sends the max toward probability 1 (greedy in the limit); T > 1 compresses differences toward uniform. Crucially it is monotone — it reshapes the distribution without ever reordering the candidates.

The API's sampling knobs — temperature, top_p — are published request parameters; the default a consumer chat product runs behind the curtain is not public, and this page won't guess. Saying "that isn't published" out loud is what separates knowing the field from guessing.

Two published cousins, top-p (nucleus) and top-k, trim the tail instead of rescaling it.

They exist because a 100,000-entry bet sheet has a very long tail of nonsense: keep only the most probable tokens whose bets add to, say, 90% (top-p), or keep a fixed top handful (top-k), and roll the dice inside that set. top-p keeps the smallest prefix of the sorted distribution with cumulative mass ≥ p, then renormalizes; top-k truncates to the k largest. Both apply to the post-temperature distribution, before the draw.

Watch one token get born

The whole cycle — predict, draw, append, repeat — is autoregressive generation.

Here is the pipeline on one screen, with real numbers: five candidate words fighting over what follows "The cat sat on the". Toy V = 5, real arithmetic end to end: logits → softmax(T) → CDF → multinomial draw → append. Every number matches the table above.

The loop that writes everything

This loop is also the honest answer to "why did it say something different this time?" One line of it is a weighted dice roll — on purpose. Turn the temperature to zero and the dice go away, along with most of the spark. The multinomial draw is the designed-in nondeterminism of generation (greedy T → 0 removes it at this layer). It is also why fan-out — running the same prompt several times and reconciling — is a rational verification strategy, not superstition.

Never do the same work twice

There is an obvious scandal in that loop: to write token 501, the model conditions on tokens 1 through 500. Does it redo all that work every single time? No — it files the work away, and only ever reads the files. Under the causal mask, position i's key and value depend only on the fixed prefix — once computed they are immutable. So cache them, per layer, per head, per position, and each decode step computes Q, K, V for the new position only.

That filing cabinet is the KV cache.

C++ A preallocated append buffer (high-water mark): two big float tensors indexed [layer][head][position] plus a write cursor — append, never realloc, exactly like any high-water-mark buffer you'd carve out at startup.

Frontier-class models are far wider and deeper than GPT-2 small; the same arithmetic at class-typical dimensions lands in gigabytes of cache for one long conversation architecture class · publicly inferred — this page won't attach a number to any specific product. Published techniques like grouped-query attention (Ainslie et al., 2023) exist precisely to shrink this cache — evidence of how binding the constraint is. public science

So a long chat gets slower and pricier as it grows for two plain reasons: every new word must consult the entire filing cabinet, and the cabinet occupies the same premium memory the model itself lives in — one cabinet per open conversation. Per-token attention reads the full cached K,V — O(n) memory traffic per step, growing linearly — and cache residency is per-sequence, in the same HBM as the weights. Context length therefore sets latency, the concurrency ceiling, and the price, mechanically.

Two kinds of processor

A CPU is a few brilliant workers with a huge planning department: most of its silicon is spent deciding what to do next — predicting, reordering, caching. A GPU inverts that: thousands of simple workers, almost no planning, nearly all the silicon spent actually multiplying. A transformer is matrix multiplication all the way down, so the GPU's trade is exactly the right one. A CPU spends its transistor budget on latency-hiding for ONE instruction stream — branch prediction, out-of-order windows, deep caches. A GPU spends it on throughput: wide SIMD lanes executing in lockstep, latency hidden by oversubscription (swap in another warp while one waits on memory). Dense matmul has no branches and total data parallelism — the ideal tenant.

C++ A GPU is your SIMD intuition made physical: the inner loop you would vectorize with AVX becomes thousands of hardware lanes running the same instruction on different data — warp after warp, no branches, pure throughput.

The bottleneck is the pipe, not the pump

Here is the single most useful hardware fact in AI, and it surprises almost everyone: to write ONE word, the chip must read essentially the entire model out of memory. Not the relevant part — all of it, every word. So the speed limit is not how fast the chip can think; it is how fast memory can pour. Generating one token at batch 1 touches every weight once: each parameter is loaded from HBM, used in one multiply-add, and discarded. That is an arithmetic intensity of ~1 FLOP/byte on machines built to sustain hundreds — the textbook memory-bound corner of the roofline model.

Numbers used above are published: NVIDIA H100 SXM datasheet (3.35 TB/s HBM3, ~989 TFLOP/s dense BF16, 700 W TDP) and Meta's Llama-3.1-70B weight release (70.6B parameters). Ceilings on this level are quoted per H100-worth of bandwidth — a 141 GB model does not fit one 80 GB card, and sharding across k GPUs scales bandwidth and ceiling by k. The reasoning is general to any autoregressive transformer. public science

The same fact, in ten lines of C

C++ Batching ≈ amortizing one weight-load across many requests — the same reason you restructure AoS to SoA: touch memory once, use it many times.

Shrinking the bytes: quantization

If the pipe is the limit, make the water denser. Store each number in half the space and the same pipe delivers words twice as fast — and the answers barely change. Squeeze too far, though, and the model's numbers get visibly wrong. Compression is a dial, not a free lunch. Quantization stores each weight as a small integer q with a shared scale s, reconstructing w ≈ s·q at compute time. Halving the bits halves the bytes streamed per token — which, in a bandwidth-bound regime, roughly doubles the throughput ceiling.

C++ int8 quantization is fixed-point DSP: store an int8_t q plus one float scale per block, reconstruct w ≈ s·q on the fly — precision traded for bytes, exactly like audio samples.

The triangle you will be asked about

Every serving decision is a three-way pull between how fast one user sees an answer, how many answers per second the fleet produces, and what each answer costs. You cannot win all three; batch size is the dial that slides you around the triangle. Latency, throughput, cost per token. Growing the batch raises intensity (throughput up, cost down) but adds queueing delay and, once compute-bound, longer passes — latency up. Chat products buy latency; batch APIs sell it back at a discount. Same iron, different corner.

Watts, honestly

One AI chip draws about 700 watts — roughly half of what an average U.S. home draws around the clock. Stack chips into servers, servers into racks, racks into halls, and the electric bill becomes the second law of AI economics, right after memory bandwidth. H100 SXM TDP: 700 W (published). A DGX H100 node — 8 GPUs plus CPUs, NVLink, fans — peaks near 10.2 kW (published). Above that level the public record thins: rack densities and hall capacities below are class-typical figures, not any provider's spec sheet.

And here the honest map runs out: how many chips any given AI provider runs, where, and at what cost is simply not published. Anyone who quotes you those numbers as fact is guessing. Frontier weights exceed one accelerator's HBM, so models are sharded — tensor parallelism within layers, pipeline parallelism across them — making interconnect part of the performance envelope. That much is class-standard. Fleet sizes, chip mixes per request, and unit economics of specific providers are unpublished.

Anthropic has publicly stated it uses a mix of AWS Trainium, Google TPUs, and NVIDIA GPUs public science — but which hardware served any particular request, or in what proportion, is unknowable from outside not public.

C++ Sharding a matmul across chips is your threaded blocked-GEMM with the sync points made of network: partial results reduced across devices inside every forward pass.

This is the bottom of the shaft: sand, electricity, and one stubborn ratio of math to memory. Everything below your chat box is now accounted for. The climb back up starts with a question — what can a machine that predicts tokens actually DO? Bottom of the descent: every layer above the iron — sampling, attention, tokens — is software riding on FLOP/byte. Next level up, the output loop stops being text and starts being actions: the agent loop.

An agent is a model wrapped in a loop by a harness — plus three more ingredients: tools it can request, accumulating context as memory, and a stopping condition.

Remove any one ingredient and the whole thing collapses back into something less. No tools — it's just chat. No memory — it re-decides from scratch every step, with no sense of progress. No stopping condition — it either runs forever or quits at random. The intern needs a phone, a notepad, and a quitting time. Degenerate cases: tools = ∅ → a chat model; memory dropped → a stateless policy re-deciding per step; no stop condition → unbounded iteration or arbitrary halting. The loop body is two calls: the model proposes an action from state; the harness executes it and folds the result back into state.

The stopping condition deserves more respect than it gets. The loop needs a reason to end: the model saying it's done, a cap on steps, a budget, or a human hand on the switch. “Just let it run until it's finished” is how you get a runaway intern — burning budget in a silent loop, taking twenty actions nobody asked for. In practice the stop check is a disjunction: the model returns no tool call (self-signaled completion) OR iteration cap OR token/cost budget exhausted OR explicit human interrupt. Production harnesses enforce all four; relying on self-signaled completion alone is the classic runaway failure.

C++ The agent loop is a while loop around an RPC: while (!stop(state)) { action = model(state); state += harness.execute(action); } — memory is just the state object growing each iteration, and the stop check is the sentinel that keeps it from becoming an infinite loop.

Tool calling (also called function calling) is the single mechanism that turns a chat model into something that does things. Each tool is declared to the model as a name, a description of when to use it, and a JSON Schema for its input.

The critical fact: the model never runs anything. It emits a request to act, shaped like a function call — and something else, code you wrote, decides whether and how to actually run it. That pause between wanting and doing is the entire safety story: at that boundary you can validate, sandbox, log, or demand a human's approval. The model emits a tool_use block mid-generation — tokens like any other tokens. The harness parses it, validates the arguments against the schema with exactly the discipline you'd apply to any untrusted RPC caller, executes the real action, and appends a tool_result message. Everything downstream of the model is ordinary engineering: input validation, sandboxing, logging, approval gates.

C++ A tool call is a marshalled RPC: the model serializes a struct, the harness deserializes and dispatches through a vtable-style lookup — handlers[name](input) — and the JSON Schema is the interface contract, doing the same job a .proto file does.

The Model Context Protocol (MCP) exists to kill an integration tax.

MCP is USB for tools. Before USB, every device needed its own port and its own cable; after, one connector fits any machine. MCP does that for AI: any tool source can plug into any AI app, because both sides agreed on the plug — build the integration once, and everyone can use it. An open protocol — RPC-style JSON messages over two transports: stdio for a locally spawned child process, streamable HTTP for remote servers with their own auth — with standardized capability discovery: a client connects, asks “what do you offer?”, and receives typed lists of tools, resources, and prompts. The host runs one dedicated client per server: a stateful 1:1 session, never shared, never fanned out.

C++ An MCP server is a plugin that exports a typed interface table the host queries at load time — dlopen plus a registry of function descriptors — capability discovery at run time instead of hardcoded linking at build time.

An agent's working memory is the conversation itself — sharp, complete, and temporary. Close the session and the agent wakes up tomorrow with no idea what it did today. So good agents do exactly what good engineers do: they write things down. The context window is finite, re-sent whole on every call, and volatile across sessions. Durable memory is artifacts on disk the agent deliberately writes — via the same tool-call mechanism as everything else — and re-reads at boot. When context nears its ceiling, compaction summarizes old turns to their essentials — exact wording traded for a bounded footprint.

This is not theory — my own fleet runs on it. Two Claude sessions share one repository and never speak directly: one designs, one implements. They pass the baton through a file called HANDOFF.md, keep distilled lessons in a MEMORY.md that outlives every session, and settle any disagreement by checking the git history — because a note can go stale, but the commit log can't. Real coordination pattern from my own multi-agent setup: shared, versioned artifacts instead of a live channel. HANDOFF.md is a baton with a one-writer-at-a-time rule; MEMORY.md persists doctrine across context resets; and on every session resume the agent reconciles against git log as ground truth, because either side's last written state may be stale.

C++ The context window ≈ a fixed-size arena allocator torn down at process exit; anything that must survive gets fwrite'd to disk — an agent's files are its serialization format.

One intern with a phone is useful. The next level down the return climb is a staffed office: many agents, divided work, and someone whose whole job is checking the work — because the checking, it turns out, is the hard part. Next: orchestration — planner/worker/verifier topologies, isolated contexts per worker, and reliability bought with independent verification, trading time, memory, and compute against each other.

Level 3 showed that a model's output is a draw from a probability distribution — which means a fluent, confident, wrong answer is not a malfunction; it is the expected failure mode. You do not fix that with a better prompt. You fix it with architecture: more independent looks, and an adversary whose only job is to break the answer before you see it. Sampling means errors are stochastic. Stochastic errors across separately-seeded, separately-prompted contexts are (partially) uncorrelated — so fan-out buys coverage a single reroll cannot, and a refuter in a fresh context does not inherit the priors that produced the error.

A subagent is not a thread and not a function call. It is a separate conversation: its own system prompt, its own fresh context, its own loop. It cannot see its siblings, and the parent sees only the final structured text it returns. That informational isolation is not a limitation — it is the entire mechanism. It is what makes a second opinion a second opinion.

C++ Fan-out and join ≈ std::async futures: launch N workers, .get() every future before the reduce — Promise.all is the same join with different spelling.

If one reviewer catches a given bug six times out of ten, three reviewers working separately catch it nearly nineteen times out of twenty. The gain does not come from anyone being smarter — it comes from their misses not lining up. Model each finder as an independent Bernoulli trial with hit probability p. The only way the bug survives is if every finder misses:

public science Basic probability of independent trials — and the honesty clause: the independence assumption is the weak joint. Agents sharing one base model share failure tendencies, so their errors correlate; different prompts and different lanes reduce the correlation, they do not erase it. That gap is exactly why the fleet does not vote — it verifies. Voting averages correlated guesses; adversarial verification demands each claim survive an attack from a context that never saw the reasoning behind it.

The manager of this fleet is not an AI. It is a short, boring, deterministic program: it hands out the jobs, waits for everyone, sends the results to a professional skeptic, and files what survives. Judgment lives in the workers; control lives in code you can read. The orchestrator is plain script: real loops, real conditionals, no model in the control path. Each agent() call is a concurrent request into an isolated context; returns are validated against a JSON Schema at the tool layer, so a non-conforming object is rejected and retried instead of parsed by hope.

Self-review is weak for a structural reason, not a moral one: the same context that produced the error is still sitting in front of the model when it checks itself, quietly arguing for the error. The fix is a reviewer who never saw the reasoning — and who is told to attack, not to approve. In self-review, attention still weights the tokens of the flawed reasoning; the model’s own prior output conditions its next output. A refuter in a fresh context inherits none of those priors, and the imperative “refute” shifts the likely continuations from agreement toward attack. Findings that cannot survive refutation die before a human reads them.

C++ The orchestrator ≈ plain main() calling remote workers over RPC — deterministic control flow with the model as a callee, the way you would never let a heuristic drive your scheduler.

There are two ways to chain minds. An assembly line, where each station starts the moment a part arrives — fastest when the parts are independent. And a gate, where nobody moves until everyone reports — mandatory when the next decision needs the full picture. Verification is always a gate. A pipeline overlaps stages — stage k+1 consumes lane i’s return as it lands — maximizing throughput on independent items. A barrier synchronizes so the reduce sees the complete set. Use pipelines for per-item transforms; use barriers wherever correctness depends on completeness: dedupe, ranking, verification verdicts.

And shared documents die of too many authors. In my fleet every file has exactly one writer at any moment; everyone else reads. When two roles must share one workspace, they pass a baton — and the project history, not anyone’s memory, is the referee of who holds it. The one-writer rule: parallel lanes get disjoint working directories; the two roles that share one repo are serialized by a baton file, and git log — not the baton’s prose — is ground truth, because prose about repo state drifts and commits do not.

Why a hub with one manager instead of a team that all talk to each other? Because conversations multiply faster than people do. Peer-to-peer coordination scales quadratically; hub-and-spoke scales linearly. That asymmetry is the entire argument for an orchestrator:

C++ The baton ≈ std::unique_ptr ownership — exactly one owner may mutate, a handoff is an explicit std::move, and shared mutation becomes an error you are never allowed to make.

Fanning out is not free. Every extra mind costs money, and every extra mind adds coordination. You are always trading between three pressures: the clock, the bill, and the size of the room where the results get combined. Three axes: wall-clock time, total compute (tokens ≈ dollars), and memory — each lane’s context budget plus, critically, the merge payload that must fit the parent’s context. Watch the marker: every fleet design is a point inside this triangle. Worked numbers below use a 200K-token context, a published API shape. public science

When not to orchestrate — the checklist I actually apply before spawning anything:

• The job fits comfortably in one worker’s head. Task + evidence < one context budget → one agent, no ceremony.
• The output must speak with one voice — parallel writers produce a committee, not an author. Globally-coherent artifacts (one essay, one API design) resist partitioning; fan out research, keep authorship single.
• The work can’t be split into pieces that don’t touch — coordination will eat everything the extra hands produce. No clean partition of shared mutable state → the n(n−1)/2 coordination term dominates the coverage gain.
• You can’t say precisely what a finished piece looks like — then you’ll be merging vibes, not results. No return schema and no acceptance check per lane → the merge degenerates to a model reading prose, and the determinism advantage is gone.

The honest capability map

The fastest way to lose money on AI is to ask it to be something it isn't. Everything on this page tells you exactly what it is: a machine that continues text extremely well. Build on what that's genuinely good at, and put hard checks around everything else. The architecture (L2–L3) predicts the failure modes: it optimizes plausible-next-token, not verified-true-statement. So reliability clusters around transformation of supplied context, and falls off wherever ground truth must come from frozen weights instead of what's in the context.

Verification is architecture, not hope

So the honest answer to "can we trust it?" is: you don't trust it — you verify it, the same way you don't trust a bridge, you load-test it. My builds ship with the checking built in: automated tests, an independent reviewer whose job is to prove the work wrong, and a replay that pins the model’s recorded answers in place and confirms everything built around them still produces the same result. Three gates, in order of cost: deterministic tests (exit codes, skipped counts as failed), adversarial review by a fresh context briefed to refute (self-review has self-preferential bias), and a determinism trace — replay the pipeline around the model against its recorded inputs and outputs and compare digests. This is the gate I actually run; here it is as code.

Three build shapes — cheapest first, always

Most "AI projects" fail by starting too big. The discipline is a ladder: one model call if that solves it; a scripted workflow if the process needs steps; a self-directing agent only when the task is genuinely open-ended — and each step up has to earn its complexity. Tier ladder: single call → workflow (deterministic code sequences the model calls) → agent (the model picks its next step, L5). Going up a tier is justified against four checks — complexity, value, viability, cost-of-error — and a "no" on any check means stay a tier down.

Shape 1, opened up: RAG in five steps

The model's knowledge is frozen the day training ends, and your business changes daily. Retrieval fixes that without retraining anything: find the few passages that matter, put them in front of the model, and let it answer from what it can actually see. RAG composes three things this page already built: embeddings (L1) for both chunks and query, dot-product similarity (L2's math) for retrieval, and the messages array (L0) as the delivery vehicle. Nothing new is invented — it's the same machine, aimed at your corpus.

Why retrieval and not the two obvious alternatives? Pasting everything into every request re-pays for the whole library on every question, and retraining the model to memorize your documents makes it stale the day a policy changes. Retrieval updates like a database and bills like a pamphlet. The three-way tradeoff, compressed: long context re-pays the full corpus token cost per call and caps at the window size; fine-tuning bakes facts into weights, so updates cost a training run and lag reality; RAG pays only for retrieved chunks and updates incrementally — and it's the only one of the three that hands you citations.

The three questions I ask a company first

Before any architecture gets drawn, three questions decide the whole project — and none of them are about AI. These three answers pick the tier, the grounding strategy, and the verification budget respectively — the entire design falls out of them.

Tokens are metered compute public science

AI compute is billed like electricity: you pay per unit moved through the machine, and Level 4 showed you exactly which machine. That means the cost of a feature is a five-line calculation you can do before building anything — here's a real one. Published API price sheets bill per million tokens, input and output priced separately (output costs more — it's generated one memory-bound forward pass at a time, L3–L4). Rates below are an example mid-tier published rate; prices change, the method doesn't. Worked example: a support assistant reading 1,000 tickets a day, each call ~2,000 input tokens (system prompt + retrieved chunks + the ticket) and ~300 output tokens.

Three hundred dollars a month to pre-read every support ticket — against what that reading costs in salaried hours today. When the arithmetic is that visible, the build decision stops being a leap of faith and becomes a line item. And the same arithmetic exposes the failure modes early: context length multiplies input cost linearly, naive full-history chat replays make per-turn cost grow with conversation length, and retrieval caps input at top-k chunks instead of the whole corpus — cost control and architecture are the same decision.

The risks, named — and where each one is stopped

The honest pitch names its own risks. There are three that matter, and none of them are handled by hoping the model behaves — each one is stopped by a piece of architecture. Threat model, in one pass: injection at the context boundary, exfiltration at the data boundary, and irreversible actions at the effect boundary. One enforcement mechanism per boundary, all outside the model.

Text that gives orders. A model reads instructions and documents through the same channel — so a document can try to impersonate an instruction. A retrieval assistant reads your files; a malicious file can say "ignore your rules." The defense isn't a smarter prompt — it's a harness that treats everything retrieved as evidence, never as authority. Prompt injection is structural (L0 showed why: roles are token markers, and the system-vs-user distinction is trained, not enforced). So enforcement lives outside the model: retrieved content is data, tool permissions are allowlisted in the harness, and nothing observed in content can grant a capability.

Data leaving the building. Every model call is data going somewhere. Before the first call, we decide exactly what may go — and everything else can't, by construction rather than by policy-memo. Data boundaries are enforced at the call site: an explicit allowlist of fields that may enter a prompt, permission-filtered retrieval so the index never serves a chunk the requesting user couldn't open, per-tenant index isolation, and secrets that never transit a context at all.

Actions you can't take back. Sending, deleting, paying, deploying — those wait for a human, every time. Not because the system usually fails, but because "usually" is not a governance policy. The HITL gate from the code above, as standing infrastructure: the irreversible set is enumerated, each member is behind an approval, and approvals are logged next to the determinism trace. Autonomy is granted per action class, never blanket.

How an engagement runs

Four phases, no mystery — and every phase maps to levels you've already descended through on this page. The level chips are literal: each phase is the practical application of the machinery those levels explained.

The whole descent, on one whiteboard

Everything on this page fits in one drawing and ninety seconds of talking. Here is the drawing; below it, the ninety seconds. One shaft down, one shaft up: L0→L4 is the mechanism, L5→L7 is the leverage. Every arrow in it was earned by a level above.

The ninety seconds, spoken

1. L0When you press Enter, nothing mystical happens: the whole conversation is packed into one JSON request and replayed to a stateless server. The model sees text — nothing else.
2. L1Text can't be multiplied, so it becomes numbers: the sentence splits into tokens, each token indexes a row of coordinates, and meaning becomes distance.
3. L2The heart is one repeated block: every token scores every earlier token — a dot product — softmax turns the scores into weights, and the weights mix information forward. Stack that block many times.
4. L3The output isn't an answer — it's a probability for every possible next token. Sample one, append it, run again: a while loop, one word per lap.
5. L4That loop runs on iron, and moving the weights through memory — not the math — is the bottleneck. That's why batching, and the serving economics behind the price sheet, look the way they do.
6. L5Wrap the loop in tools and it becomes an agent: observe, think, act — a while loop around an RPC.
7. L6One agent can be wrong quietly. A fleet with an independent verifier is wrong loudly — and loudly is fixable. Verification is architecture, not hope.
8. L7And this level is the point: find the toil, find the data, decide what must never be wrong — then build the smallest shape that works, with the gates built in. That's what I deliver.