Assume compromise

Two modules in, you know the boundary inside the model is coarse and unenforced. The obvious response is to try harder at the model. This module covers why that fails, and the single reframe that turns an open research problem into ordinary engineering work.

The wrong objective and the right one

Almost every team starts here:

Make the model impossible to fool.

This one is not yours to work on. Whether a frontier model can be talked out of its safety training is decided by a research programme at a handful of labs, on a release cadence you do not control, with results that change under you. You cannot specify it, test it to completion, or promise it to anyone.

Replace it with this:

Make fooling the model insufficient to compromise the system.

The second objective is about authorization, scoping, validation and blast radius. They live in code the builder owns, are testable deterministically, and hand them to an auditor. Ordinary engineering, and unglamorous, which is part of why it gets skipped.

It is also the opening line of the OWASP 2026 Top 10:

LLM01's prevention section says the same thing as design guidance: build the surrounding system on the explicit assumption that the model's instruction boundary will eventually be bypassed, and constrain both what the model may do and what its outputs may reach, so a successful injection does not become a successful exploit.

The same attack, two architectures

An employee, or a poisoned document, convinces the model that an emergency authorization applies. The model is fully persuaded. It emits the tool call.

// The tool trusts the model's judgement, and its arguments.
async function approveExpense({ expenseId, actorId }) {
  // actorId came from the model. So did the decision to call this.
  return db.expenses.update(expenseId, {
    state: "approved",
    approvedBy: actorId,
  });
}

// Model was persuaded  ->  money moves.

// The tool re-derives authority from trusted state, ignoring
// everything the model asserted about who is asking.
async function approveExpense({ expenseId }, ctx) {
  const actor = ctx.authenticatedUser;          // not from the model
  const expense = await db.expenses.load(expenseId);

  if (!policy.canApprove(actor, expense))       throw new Forbidden();
  if (!workflow.isCurrentApprover(actor, expense)) throw new Forbidden();
  if (expense.state !== "AwaitingApproval")     throw new Conflict();

  return expenses.approve(expense, { actor });
}

// Model was persuaded  ->  403, alert, nothing moves.

In the second architecture the injection still worked, and the model was compromised just as thoroughly. No security invariant depended on the model behaving correctly, so the compromise had nowhere to go.

Where the leverage is

A way to direct attention, not something to compute. This framing is ours rather than published:

Risk  ≈  P(model compromise)  ×  available capability
                              ×  reachable assets
                              ×  autonomy

You cannot drive the first factor near zero. That is the empirical finding of the last two years, and module 6.6 covers the evidence. The other three are ordinary engineering variables you can drive down this quarter, in code you control.

The question to ask about any system

Worth memorising. It cuts through almost any architecture discussion:

The answer you want: very little, because every important operation is independently authorized, scoped, validated, monitored, and where necessary approved by a human. A system that can honestly say that has moved from prompt engineering into security engineering.

The answer you usually get the first time anyone asks is a long pause followed by someone opening the tool definitions. The pause is the finding.

What this does to the metrics

Under the old objective a successful jailbreak is a failure. Under the new one it can be a non-event:

Attack succeeded against the model:      YES
Model requested an unauthorized payout: YES
Backend rejected it:                    YES
Sensitive data exposed:                 NO
Money moved:                            NO
Alert generated:                        YES

From the model's perspective the jailbreak worked. From the system's perspective the architecture worked as designed. A mature build should make that outcome routine, which is why this course insists you report attack success rate and unauthorized action rate, never the first alone. Module 6.8 covers the measurement discipline.

The line to remember. Ours, but it follows directly from OWASP's:

Check yourself

1. A guard model blocks 99.5% of injection attempts in testing.
   Which of the four risk factors did you just improve, and are you
   now allowed to say the system is safe?

2. A red-team report says "we achieved a jailbreak." What is the
   very next question you ask?

3. Why is "make the model impossible to fool" a bad objective even
   if it were achievable?

4. Rewrite this as something you can build:
   "harden the assistant against prompt injection."