What are LLMs and how they actually work
Next-token prediction, sampling, and the context window, built by hand rather than described.
Two ideas carry the whole course. A language model predicts one token at a time. The application decides what it gets to see. Almost every attack in the next ninety-odd modules lives in the gap between those two sentences.
1 · It predicts the next token
Given a sequence, a model returns a probability distribution over what comes next. A sampler picks one, appends it, and asks again. Nothing is looked up. Nothing is decided.
Click a token to choose it, or let the sampler pick. Then drag the temperature to 0.1 and to 2.0.
Predict the next token
0 tokens generatedThe sequence so far
The support ticket says the invoice
What the model returns. A distribution, not an answer
↓ tap one to add it
Three things you just saw:
- It predicts, it does not retrieve. A correct answer and a confident fabrication come out of the same machinery.
- You get a distribution, not an answer. The same input can produce different results. That is why this course reports attack results as
n/Nacross repeated trials. One success is a data point, not a finding. - Nothing in there is an instruction. The model continues a sequence. Text that reads like a rule is still just text.
2 · The app builds what it sees
The model has no memory. Between calls it retains nothing, so anything resembling memory is the application reassembling a context window and resending the whole thing.
Toggle the sources below. Watch the budget, then tick show what the model actually receives.
Build the context window
5,690 / 8,000 tokensSources, toggle what the app puts in the window
Budget
Asked for 8,090 tokens. 1 source dropped silently to fit.
Who wrote this window
3 of 5 sources can be written to by someone outside the organisation.
Most of what lands in a production context window was written by neither the developer nor the user.
Retrieved documents, tool results, ticket bodies, persisted memory. Each is content a third party can influence, arriving in the same stream as the system’s instructions. That is what an LLM application is.
Where this goes
You have the mechanism: a next-token predictor, and a window the application packs with content of mixed provenance.
Module 0.2 runs the same loop for longer before answering, which adds text to that window that neither the developer nor the user wrote. Then Module 1.1 asks the obvious question about the flat stream you just assembled: if it all arrives as one sequence, what stops the ticket body from being read as an instruction? The answer is more useful than “nothing”.
Sources