AI Security Audit: a practical guide for LLM agents

The four planes of an AI audit

Every AI security audit we run decomposes the system into four planes. Skip one and the audit is incomplete.

How to audit AI agent memory for security risk

Agent memory — short-term scratchpads, long-term vector stores, and shared knowledge bases — is the highest-leverage target in a modern AI system. A single poisoned document can persist into thousands of future sessions. Audit memory against these five failure modes:

• Cross-session leakage — one user's memory surfacing in another user's context window.
• Indirect prompt injection persisted into long-term memory from a single poisoned document.
• PII accumulation — embeddings retaining identifiers long after the source row was deleted.
• Tool-call replay — stored 'successful plans' that re-execute privileged actions on a new task.
• Memory-rank attacks — adversarial content engineered to score high in vector recall.

Concretely: enumerate every write path to the memory store, confirm tenant isolation at the index level (not just the query filter), require provenance metadata on every chunk, and run a recall-poisoning evaluation that injects a benign canary and verifies it can be detected, retrieved, and purged.

The 10-step audit checklist

1. Inventory every model, agent, prompt, tool, and memory store. Unknown surfaces cannot be audited.
2. Threat-model with STRIDE plus AI-specific categories: prompt injection, model extraction, data poisoning, jailbreaks, tool abuse.
3. Classify data crossing the model boundary. Tag PII, regulated, and secret-bearing fields.
4. Lock down system prompts and tool schemas in version control. Treat prompt changes as code changes.
5. Isolate tool credentials per agent. No shared service accounts. Scope to least privilege at the API level, not in the prompt.
6. Wrap every tool call in a policy engine (OPA, Cedar, or a typed allow-list). The model proposes; the policy decides.
7. Quarantine untrusted text before it reaches the model: strip HTML, label provenance, and segregate from instructions.
8. Stand up an evaluation harness with adversarial suites — Garak, PyRIT, promptfoo — and run on every prompt or model change.
9. Log full traces (input, retrieved context, tool calls, outputs) to an append-only store. You cannot investigate what you did not record.
10. Define memory TTL, deletion, and right-to-be-forgotten flows before launch — not after the first incident.

Mapping to compliance frameworks

A defensible AI audit traces every control back to a recognized framework. The four below cover the common ground for regulated industries operating LLM systems today.

Evaluation harnesses, not vibes

The single biggest gap we see is teams shipping AI features without a regression suite. An AI evaluation harness is the audit equivalent of a test pipeline: a versioned set of adversarial prompts, jailbreak payloads, indirect injection documents, and tool-abuse scenarios that run on every model upgrade, prompt change, or new tool. Pass/fail thresholds become part of your change-management evidence — the same way unit tests do for traditional software.

When to bring in an outside auditor

Internal teams are the right owners for continuous evaluation. An outside auditor is the right call before a regulated launch, after a material model or architecture change, or when board-level assurance is on the line. Templebit's AI and cybersecurity practices run these audits jointly so the findings are technically grounded and audit-defensible in the same report.