Writing Secure Software: Two-Week Online Course

A two-week online course on securing web applications and AI systems, using Smarter as the case study, with six 30-minute recorded lectures and a 90-minute live session each week. Week 1 covers attacks on AI applications and their defenses, and week 2 covers secrets, outbound requests and secure development. See Course Formats and Grading for the format, the grading scale and the course policies, and Writing Secure Software for the Internet for an overview of the subject.

At a Glance

Students

Developers and security students.

Deployment

A shared Smarter instance run by the instructor for attack and defense exercises, with an account per team. Students clone the repository to read and scan the code.

Workload

3 hours of instruction and 4 to 6 hours of assignments each week.

Cost

Expect less than $5 USD of model charges per student, capped by a per-student Budget.

Warning

Students attack only the class’s own Smarter instance. Any real vulnerability in Smarter is reported privately, as described in the project’s SECURITY.md.

Learning Objectives

By the end of the course, students can:

  1. Explain both OWASP Top 10 lists with examples.

  2. Attack an AI application with direct and indirect prompt injection, and defend it with deterministic controls.

  3. Explain how an application protects credentials throughout their life.

  4. Explain and defend against server-side request forgery.

  5. Use static analysis and code review to find vulnerabilities, and handle them responsibly.

Technical Prerequisites

  • Required: structured programming in Python or JavaScript.

  • Required: HTTP, cookies and the browser’s developer tools.

  • Required: the command line, curl, Git and a GitHub account.

  • Required: basic generative AI concepts: system and user prompts, and tool calls.

  • Helpful: Linux, Docker, DNS and IP addressing, YAML, and Python virtual environments for running bandit.

Schedule

Week

Item

Content

1

Lecture 1 (30 min)

The threat landscape. Attackers, assets, attack surfaces, and both OWASP Top 10 lists.

1

Lecture 2 (30 min)

Prompt injection. Direct and indirect injection, jailbreaks and data exfiltration.

1

Lecture 3 (30 min)

Deterministic controls. Guardrails, allow lists, least privilege for tools, and measuring false positives.

1

Live session 1 (90 min)

Red team exercise: teams attack each other’s prompt-only LLMClients, live. Start Assignment 1.

2

Lecture 4 (30 min)

Secrets. Encrypted Secrets, credentials kept out of prompts, and the detect-private-key and detect-aws-credentials hooks.

2

Lecture 5 (30 min)

SSRF and outbound requests. Metadata addresses, DNS rebinding, and smarter/smarter/apps/plugin/plugin/safe_http.py.

2

Lecture 6 (30 min)

Secure development and disclosure. bandit, dependency scanning, code review, ADR 016, responsible disclosure and incident response.

2

Live session 2 (90 min)

Capstone presentations (about 5 minutes each, with 2 minutes of questions).

Assignment 1 is due at the end of week 1, Assignment 2 in the middle of week 2, and Assignment 3 and the capstone at the start of Live session 2.

Graded Assignments

Assignment 1: Red Team, Blue Team

Teams submit the log of their attacks from Live session 1, then add guardrails to their own LLMClient, invite the attackers back, and report the before-and-after results.

Turn in: both attack logs from the journal, the guardrail manifests and a one-page report.

Criterion

Pts

Exemplary

Developing

Attacks

5

Ten or more, varied and classified by OWASP LLM category.

Repetitive or unclassified.

Defenses

6

Guardrails chosen for the observed attacks.

Added at random.

Measurement

5

Before and after, with false positives tested.

No measurement.

Honesty

4

Residual risk reported.

Claims complete protection.

Assignment 2: SSRF Analysis

Students explain, with references to the code, how safe_http.py stops a plugin from reaching private and metadata addresses, then list five SSRF techniques (for example redirects, DNS rebinding, IPv6 and decimal IP notation) and say whether each would be stopped and where.

Turn in: a two-page analysis with code references.

Criterion

Pts

Exemplary

Developing

Code understanding

7

Each check explained with line references.

General description.

Techniques

6

Five techniques, each correctly assessed.

Fewer, or incorrect assessments.

Reasoning

5

Explains why DNS must be resolved fresh before every request.

Not explained.

Responsible handling

2

Any suspected weakness reported privately.

Published.

Assignment 3: Static Analysis Triage

Students run bandit on one Smarter app, triage every finding as a true positive, false positive or accepted risk, and justify each decision.

Turn in: the bandit output and a triage table.

Criterion

Pts

Exemplary

Developing

Complete triage

6

Every finding classified.

Some skipped.

Justification

8

Each decision supported by reading the code.

Decisions without reasons.

Insight

4

Notes what static analysis cannot find.

Treats a clean scan as proof of security.

Clarity

2

Well organized.

Disorganized.

Capstone Projects

Students choose one, and present it in Live session 2.

Capstone A: A Threat Model of an AI Application

Learning objective. Produce a structured threat model of a real AI application, and verify its key mitigations by testing.

Students threat model a built-in LLMClient with tools, using STRIDE and the OWASP LLM list: data flow diagram, trust boundaries, threats, mitigations found in the code, residual risk, and tests of the three most important mitigations.

What students turn in:

  • A threat model document with a data flow diagram.

  • Evidence of mitigation tests.

  • Prioritized recommendations, and a five-minute presentation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Data flow and boundaries

20

Accurate and complete.

Boundaries missing.

Threats

25

Systematic, classic and AI-specific.

A few obvious ones.

Mitigations in code

20

Found and cited.

Assumed.

Verification

20

Three tested, with evidence.

Untested.

Recommendations and presentation

15

Prioritized; clear presentation.

Unprioritized.

Capstone B: A Hardened Assistant with an Adversarial Test Suite

Learning objective. Build an AI application with defense in depth, and prove its security properties with tests.

Students build an LLMClient with a tool and access to fictional sensitive data, harden it, and run an adversarial suite of at least thirty attacks (including indirect injection through a web page or document it reads) and fifteen legitimate requests. Classmates are invited to attack it during the week before presentations.

What students turn in:

  • Commented manifests.

  • The test suite, results, attack success rate and false positive rate.

  • A two-page design note on residual risk, and a five-minute presentation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Defense in depth

25

Independent layers, each justified.

System prompt only.

Least privilege

15

Tools limited to the job.

Broad access.

Test suite

25

Thirty varied attacks, including indirect, plus legitimate requests.

Repetitive.

Results and honesty

20

Rates measured; classmates’ successful attacks reported.

Cherry-picked.

Design note and presentation

15

Residual risk explained.

Claims to be secure.

Further Study