Skip to content

1. Framing Secure GenAI and ML

Overview

Generative AI (GenAI) and Machine Learning (ML) systems introduce a different security model compared to traditional applications. Even though they are often built using familiar components—such as APIs, microservices, and cloud platforms—their core behavior is different. Traditional systems follow fixed logic defined by code, while GenAI systems rely on data, probability, and learned patterns.

This difference changes how we think about security. The way we analyze risks, design controls, and operate systems must also evolve.

This document is a structured set of notes based on training (SANS SEC545: GenAI and LLM Application Security) and additional research. The points below are the ideas this chapter uses to frame the rest of the minibook.

Core concepts

  1. GenAI security needs a different model than traditional AppSec, even when the stack looks familiar.
  2. We start with threat modeling and attack surface, then place specific attacks in context—not exploit lists first.
  3. Traditional assumptions (code defines behavior, inputs are data, failures are repeatable) break for GenAI.
  4. The guiding question becomes: who can influence the system, through which inputs, and with what impact?
  5. Models, prompts, and data are core security elements; we monitor behavior and keep human oversight where decisions matter.

After reading this chapter, we should be able to say why GenAI security differs, how this minibook is organized, and which principles apply across later chapters.

Foundation Comparison

The table below contrasts traditional AppSec with AI security across development, build-time, and runtime:

Area Traditional AppSec AI Security
Development Model Test-driven, deterministic Data-driven, probabilistic
Build-Time Controls SCA, SAST, fuzzing AI supply chain checks, model validation, red teaming
Build-Time Risks Vulnerabilities, insecure code, misconfigurations Poisoned data, rogue models, data leakage
Runtime Controls WAF, API security, DAST Monitoring, guardrails, data integrity checks
Runtime Risks DDoS, data breaches, compromised apps Jailbreaking, information disclosure, resource abuse

From Exploit-First to Threat Model–First

A lot of GenAI security content focuses on specific attacks—for example, prompt injection or jailbreak techniques. These examples are useful, but they often focus on symptoms rather than root causes.

In these notes, we take a different approach. The sequence below is how the rest of the minibook is ordered:

  • Start with threat modeling (identify assets, boundaries, and who can influence what)

  • Then validate the attack surface (where inputs can change system behavior)

  • Understand architectural differences (how GenAI systems behave differently from traditional systems)

  • Finally, look at specific attacks in context

This approach helps reason about classes of problems, not just individual issues.

Why GenAI Security Is Different

Traditional application security is built on a few assumptions. GenAI systems break those assumptions, so the security question changes.

Traditional assumptions include:

  • Code defines system behavior
  • Inputs are treated as data
  • Execution flow is predictable
  • Failures are repeatable

GenAI systems break these assumptions in the following ways:

  • Prompts can act like instructions, not just input
  • Data can influence behavior in unexpected ways
  • Outputs are generated, not explicitly coded
  • The same input may produce different results

Because of this, the key question changes.

Instead of asking “Where is the bug?”, we ask: “Who can influence the system, through which inputs, and with what impact?”

What This Document Covers

This document is organized into three main areas. Each area answers a different part of the framing problem.

1. Securing GenAI Applications

Focus on system design. The topics below show how common patterns introduce risk:

  • Common patterns (e.g., retrieval-based systems, agent-style workflows, fine-tuning)
  • How these systems work internally
  • Where risks are introduced

2. Securing the GenAI Lifecycle

Focus on the full lifecycle. The stages below extend the traditional software lifecycle and introduce new risks:

  • Data collection and preparation
  • Model training and updates
  • Prompt design and feedback loops

3. MLSecOps

Focus on operations. The points below describe how production ML security relates to DevSecOps:

  • How to secure models and ML pipelines in production
  • How this builds on, but differs from, DevSecOps

Guiding Principles

A few principles apply across all sections. We carry these forward through the minibook:

  • Treat models, prompts, and data as core security elements
  • Assume misuse and manipulation—not just coding mistakes
  • Keep human oversight where decisions matter
  • Monitor behavior and outputs, not just system logs
  • Treat non-deterministic behavior as a design characteristic, not a flaw

Intended Audience

These notes are written for the roles below. Basic knowledge of application security is helpful. Prior experience with machine learning is not required.

  • Security architects and application security engineers
  • ML and platform engineers
  • Cloud and infrastructure security teams
  • Governance, risk, and compliance roles