Writing Secure Software: Classroom Bootcamp

An eight-hour classroom bootcamp on securing web applications and AI systems, using Smarter as the case study. Students attack an AI application with prompt injection, defend it with deterministic controls, map its defenses to the OWASP Top 10 lists, and study how it handles secrets and outbound requests. See Course Formats and Grading for the standard bootcamp day, the grading scale and the course policies, and Writing Secure Software for the Internet for an overview of the subject.

At a Glance

Students

Developers and security students who want to understand both classic web security and the new attacks on AI systems.

Deployment

A shared Smarter instance run by the instructor for the attack and defense labs, with an account per team. Students who want to read and run the code clone the repository.

Cost

Expect less than $3 USD of model charges per student, capped by a per-student Budget.

Warning

Students attack only the class’s own Smarter instance, within the rules set by the instructor. Any real vulnerability found in Smarter is reported privately, as described in the project’s SECURITY.md, never in a public issue or an assignment.

Learning Objectives

By the end of the bootcamp, students can:

  1. Explain the OWASP Top 10 and the OWASP Top 10 for LLM Applications, with an example of each category.

  2. Carry out and explain direct and indirect prompt injection against an AI application.

  3. Explain why deterministic controls (guardrails, allow lists, least privilege) are stronger than instructions to a model.

  4. Explain how an application keeps credentials out of prompts, code and logs.

  5. Explain server-side request forgery, and how an application that fetches URLs defends against it.

Technical Prerequisites

  • Required: structured programming in Python or JavaScript, enough to read a function and follow its logic.

  • Required: HTTP: methods, headers, cookies, status codes, and the browser’s developer tools.

  • Required: the command line, curl, Git and a GitHub account.

  • Required: basic generative AI concepts: system and user prompts, and tool calls.

  • Helpful: Linux, Docker, DNS and IP addressing, and YAML.

Schedule

Block

Min

Content

1

45

Lecture 1: The threat landscape. Attackers, assets and attack surfaces. The OWASP Top 10 and the OWASP Top 10 for LLM Applications. Why AI applications add new attack surfaces: the prompt, retrieved content and tools.

2

45

Lab 1: Red team. Each team builds an LLMClient that guards a secret phrase with system prompt instructions only, then attacks another team’s. The journal records every attempt. Produces Assignment 1.

3

45

Lecture 2: Prompt injection. Direct and indirect injection, jailbreaks, data exfiltration through links and images, and why no instruction can make a model reliably ignore instructions. Guardrails as deterministic controls, and their limits.

4

45

Lab 2: Blue team. Teams add guardrails (prompt injection, secrets leak, script injection) to their LLMClient, and the attackers try again. Produces Assignment 2.

5

45

Lecture 3: Secrets. How Smarter stores provider and plugin credentials as encrypted Secrets, why an API plugin’s credentials never appear in a prompt, and the .env file, detect-private-key and detect-aws-credentials hooks.

6

45

Lecture 4: Outbound requests and SSRF. Server-side request forgery, cloud metadata addresses, DNS rebinding, and a reading of smarter/smarter/apps/plugin/plugin/safe_http.py: public https only, fresh DNS resolution, and why that check must never be cached.

7

45

Lab 3: Map the defenses. In groups, map each Smarter defense to a category of the two OWASP lists, with the file or document that implements it, and find the gaps. Produces Assignment 3.

8

45

Lecture 5: Secure development. Static analysis with bandit, dependency scanning, pre-commit hooks, code review, and Smarter’s ADR 016 on security.

9

45

Lab 4: Read the code like an attacker. Run bandit on one Smarter app, triage its findings, and decide which are real. Ungraded.

10

45

Lecture 6: Disclosure and response. Responsible disclosure, security policies, CVEs, how open source projects handle reports, and incident response.

11

30

Capstone kickoff. Students choose a capstone and scope it with the instructor.

Graded Assignments

Assignment 1: Red Team Log

Teams record their attacks on another team’s prompt-only LLMClient: at least ten attempts with different techniques, and which succeeded.

Turn in: the attack log, taken from the journal, with each technique named, and a half-page summary.

Criterion

Pts

Exemplary

Developing

Variety

6

Ten or more attempts across several techniques (role play, encoding, translation, instruction override).

Repeats of one technique.

Evidence

5

From the journal.

Paraphrased.

Classification

5

Each attempt mapped to an OWASP LLM category.

Unclassified.

Summary

4

Explains why the successful attacks worked.

Lists results only.

Assignment 2: Blue Team Report

Teams add guardrails to their LLMClient, are attacked again, and report the before-and-after results, including legitimate prompts that were wrongly blocked.

Turn in: the guardrail manifests, the second attack log, and a one-page report.

Criterion

Pts

Exemplary

Developing

Controls

6

Guardrails chosen for the observed attacks, and configured correctly.

Guardrails added at random.

Effectiveness

5

Most attacks now fail, shown by the log.

No measured improvement.

False positives

5

Legitimate prompts tested; wrongful blocks reported.

Not tested.

Honesty

4

What still gets through is reported.

Claims complete protection.

Assignment 3: Defense Map

Groups map Smarter’s defenses to the OWASP Top 10 and the OWASP Top 10 for LLM Applications, citing the file or document that implements each, and identify three categories where defense is weak or depends on the operator.

Turn in: a table and a one-page gap analysis.

Criterion

Pts

Exemplary

Developing

Coverage

5

Every category of both lists considered.

Several skipped.

Accuracy

7

Each defense correctly described and cited.

Defenses assumed, not found.

Gap analysis

6

Three well-reasoned gaps with suggested mitigations.

Vague gaps.

Clarity

2

Well organized.

Hard to follow.

Capstone Projects

Students choose one. Capstones are due one week after the bootcamp.

Capstone A: A Threat Model of an AI Application

Learning objective. Produce a structured threat model of a real AI application, and verify its most important mitigations by testing.

Students choose a built-in LLMClient with tools (for example research_analyst or data_analyst) and threat model it with STRIDE and the OWASP LLM list: a data flow diagram with trust boundaries, threats, existing mitigations, residual risk, and tests of the three most important mitigations on the class instance.

What students turn in:

  • A threat model document (four to six pages) with a data flow diagram.

  • Evidence of the three mitigation tests, from the journal.

  • A prioritized list of recommendations.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Data flow and trust boundaries

20

Accurate, with every boundary marked.

Boundaries missing.

Threat identification

25

Systematic, covering classic and AI-specific threats.

A few obvious threats.

Mitigations

20

Existing defenses correctly identified in code or docs.

Assumed.

Verification

20

Three mitigations tested, with evidence.

Untested.

Recommendations

15

Prioritized by risk, realistic.

Unprioritized.

Capstone B: A Hardened Assistant

Learning objective. Build an AI application with defense in depth, and prove its security properties with an adversarial test suite.

Students build an LLMClient with at least one tool plugin and access to sensitive information (a fictional employee directory, for example), then harden it with least-privilege configuration, guardrails on input and output, a budget, and a tight system prompt. They write and run an adversarial test suite of at least twenty-five attacks and ten legitimate requests.

What students turn in:

  • The manifests, with comments explaining each security decision.

  • The test suite and its results from the journal, with attack success rate and false positive rate.

  • A two-page security design note.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Defense in depth

25

Several independent layers, each justified.

Relies on the system prompt.

Least privilege

15

Tools can do only what the job needs.

Broad access.

Test suite

25

Twenty-five varied attacks, including indirect injection, plus legitimate requests.

Few or repetitive attacks.

Results

20

Rates measured and honestly reported.

Cherry-picked results.

Design note

15

Explains residual risk.

Claims the system is secure.

Further Study