AI Features in Software: Four-Week Online Course
A four-week online course on putting LLMs inside a piece of software, safely, taught on the Doodle of Smarter’s web console: the sign-in page’s illustration, which a Smarter LLMClient, the Doodle Editor, chooses each day, and another, the Doodle Artist, draws. The whole feature is open source, in apps/doodle/, with its history in three commits that grow it from a calendar. There are twelve 30-minute recorded lectures and a 90-minute live session each week. Week 1 covers the feature and its AI parts, week 2 prompting from code and the contract with the model, week 3 judges, sanitizers and tests, and week 4 people, operations and shipping. See Course Formats and Grading for the format, the grading scale and the course policies, and AI Features in Software for an overview of the subject.
At a Glance
Students |
Python developers who want to add LLM features to software they build. |
Deployment |
Each student runs Smarter from source ( |
Models |
The models that the Doodle’s manifests name, through the instructor’s OpenAI key, and OpenAI’s moderation model. |
Workload |
3 hours of instruction and 4 to 6 hours of assignments each week. |
Cost |
Under $10 per student in model charges, capped with a Budget. Assignment 4 and Capstone B prompt the most. |
Learning Objectives
By the end of the course, students can:
Decide which parts of a feature a model should do and which code should do, and build the feature’s safe answer first.
Declare a feature’s AI parts as Smarter LLMClients and Guardrails, and prompt them from Python.
Design a model’s request and answer as a validated contract with a menu of choices, and check every answer, and every input to the prompt, in code.
Evaluate a judge model’s accuracy on labeled examples, and decide whether a feature fails open or closed.
Defend a feature against generated markup and a model’s mistakes with an allowlist sanitizer and tests in which the model is patched.
Design the human review, audit trail, switches, disclosure and cost controls that a feature needs before its AI output reaches users.
Technical Prerequisites
Required: Python: functions, classes, exceptions and type hints, and JSON.
Required: Git, GitHub and pull requests, with a fork of smarter.
Required by the end of week 1: Docker, and Smarter installed from source with a superuser login, showing the sign-in page’s doodle. See the Smarter Instructor Guide.
Helpful: Pydantic models, Python’s
unittest.mock, reading a Django app, YAML, and having called an LLM API.Not needed: JavaScript or React, machine learning, or SVG. The course reads the Doodle’s React app and SVG, but students don’t write either.
Schedule
The files that the lectures refer to are listed in full under Smarter in the Source Code.
Week |
Item |
Content |
|---|---|---|
1 |
Lecture 1 |
The newsroom. A feature as desks: the calendar, the editor, the platform’s checks, the tone judge, the artist, the sanitizer and a superuser. Which jobs suit a model (taste, judgment, drawing), which suit code (rules, limits, security), and which need a person. The blog post. |
1 |
Lecture 2 |
The safe answer first. Phase 1 (commit 0df9280ff): the catalog and its schedules, regions from a time zone and nothing else, the planner, the models and the switches. What visitors see when every AI part fails. |
1 |
Lecture 3 |
AI resources as configuration. The editor’s and artist’s manifests: model,
temperature, system prompt, example prompts, |
1 |
Live session 1 |
Setup triage. The editor in the workbench, live, with its example prompts and a request the class writes together. Start Assignment 1. |
2 |
Lecture 4 |
Prompting from code. |
2 |
Lecture 5 |
A menu, not a blank page. |
2 |
Lecture 6 |
Checks in code, on the way out and the way in. |
2 |
Live session 2 |
Review of Assignment 1. Writing a management command that prompts the editor, together, and reading its prompt in the journal. Start Assignment 2. |
3 |
Lecture 7 |
A second model as judge. The |
3 |
Lecture 8 |
Generated markup is untrusted. What an SVG can carry. |
3 |
Lecture 9 |
Testing a feature whose model is patched out. |
3 |
Live session 3 |
Review of Assignment 2. A red-team workshop: the class attacks the editor and the artist, and traces each attempt to the layer that stopped it. Capstone proposals discussed. Start Assignment 3. |
4 |
Lecture 10 |
People and paper trails. Proposed drawings, Preview the week, Approve and Reject in
|
4 |
Lecture 11 |
Running it. Celery Beat every six hours, |
4 |
Lecture 12 |
Cost and design review. Reading a feature’s cost per day from the journal, capping it with a Budget, and choosing cheaper models for judges. The blog post’s takeaways as a checklist for reviewing any AI feature, applied to a feature the students use every day. |
4 |
Live session 4 |
Capstone presentations (about 6 minutes each, with 2 minutes of questions), and a course retrospective. |
Assignments 1 to 3 are due before the next live session. Assignment 4 is due in the middle of week 4. The capstone proposal is due before Live session 3, and the capstone at the start of Live session 4.
Smarter in the Source Code
Every lecture, lab and assignment in this course rests on the files and commits below. Every
reference is to the main branch of github.com/smarter-sh/smarter, unless another repository is named, so that students
can open each one during class. Python paths are under smarter/smarter/. The case that runs
through the course is the web console’s Doodle, the illustration on the sign-in page that two
Smarter LLMClients choose and draw each day, described in How Two Smarter LLMClients Create the
Sign-in Page’s Daily Doodle
and in Doodle React Component.
The Feature Without the AI
Topic |
What Smarter does |
Where to look |
|---|---|---|
The app’s map |
The Doodle is an ordinary Django app. Its package docstring lists each module: regions, schedules, the catalog, the planner, the editor, the sanitizer, caching and the api. |
|
The catalog |
Nineteen occasions with artwork drawn and reviewed ahead of time, each a short YAML file
and an SVG of its decorations. A date is fixed, the nth weekday of a month, or a country’s
holiday by name, worked out by the |
|
Regions |
The browser sends its time zone and nothing else. Smarter maps it to one of 12 regions and the visitor’s local date. No IP address is looked up, and nothing about the visitor is stored. |
|
The planner |
The calendar alone chooses each region’s doodle a week ahead: the due theme of the highest priority, or the default. A doodle that a person chose is never replanned. This is the safe answer that the AI parts improve on, and fall back to. |
|
The models |
|
|
The schedule |
A Celery task plans the doodles every six hours from Celery Beat, then asks the editor about
new days. |
AI Resources as Configuration
Topic |
What Smarter does |
Where to look |
|---|---|---|
The editor |
An LLMClient manifest: a model, a temperature of 0.2, a long system prompt that describes the request and the answer, example prompts that are real requests, and two output Guardrails. Its taste is configuration, changed by editing YAML, not by a release. Smarter LLM Client. |
llmclient-smarter-doodle-editor.yaml (example-manifests) |
The artist |
A second LLMClient, at temperature 0.6, whose system prompt describes the base scene in coordinates, the palette, a limit on shapes, and what it must never draw. |
llmclient-smarter-doodle-artist.yaml (example-manifests), base.svg |
Output Guardrails |
The editor’s answer passes OpenAI’s moderation model and a Guardrail that strips a stray code fence so the JSON parses. Smarter Guardrail. |
json-code-fence-output.yaml, moderation-output.yaml (example-manifests) |
Staff only |
|
llmclient-smarter-doodle-editor.yaml (example-manifests) |
Prompting from code |
The platform prompts its own LLMClients through the chat api, as any client would.
|
|
Extracted, not copied |
|
|
Finding the resources |
|
Trust Nothing the Model Says
Topic |
What Smarter does |
Where to look |
|---|---|---|
A menu, not a blank page |
|
|
The contract with the model |
Pydantic models describe the request ( |
|
Every choice checked again |
|
|
Blocked topics |
|
|
Inputs filtered before the model |
Trends come from Wikipedia’s most viewed articles. |
Judges, Sanitizers and People
Topic |
What Smarter does |
Where to look |
|---|---|---|
A judge of tone |
Word lists can’t tell sad news from a festival. |
upbeat-occasion-input.yaml (example-manifests), llm_judge_strategy.py |
Failing closed |
|
|
Asking a Guardrail directly |
|
|
Models that differ |
A model that rejects a parameter, as one rejects the judge’s temperature, is retried once without it. The chat provider and the judge share one list of such quirks. |
|
Generated markup |
The artist’s SVG is untrusted. The sanitizer keeps an allowlist of shapes and attributes,
strips handlers, links, ids, |
|
Two locks on one door |
The page shows artwork with |
|
A person decides |
A drawing is saved as proposed. In the Django admin, Preview the week shows every region’s doodle for every planned day, with who chose it and the editor’s reason, and Approve and Reject buttons. Doodle Editor Runs are the paper trail. |
|
Switches |
The editor, the artwork, the trends and the credit each have a waffle switch.
|
Into the Product
Topic |
What Smarter does |
Where to look |
|---|---|---|
The api |
|
|
The page |
The single-page console’s sign-in pages show the doodle, its caption, and “Chosen by” or “Drawn by a Smarter AI application” with a link that explains how; until release v0.42.10, a React app that Django hosted showed it, with a template tag. Django-React Integration. |
|
Testing with the models patched |
The editor’s and artist’s answers are patched in tests, with a test for every refusal and every failure. The app has well over a hundred tests. |
|
Dangerous input on file |
The sanitizer is tested against saved drawings, one safe, one crowded, one empty and one dangerous; trends against saved Wikimedia responses, so no test needs the network. |
The Doodle in the History
The Doodle was built in three phases, each a commit whose body explains what it adds, how it was tested and what it ran into, so students can watch an AI feature grow from a calendar.
Commit |
Why it is worth studying |
Where to look |
|---|---|---|
Phase 1: a calendar |
The whole feature with no LLM at all: regions, schedules, the catalog, the planner, the api, the React app and the switches. The safe answer comes first. |
|
Phase 2: an editor and an artist |
Two LLMClients, the prompting helpers extracted from |
|
Phase 3: trends and a judge |
Wikipedia trends, the tone check that fails closed, and a fix to every |
|
Credit where it’s due |
The “Learn how” link under an AI-chosen doodle points to the article that explains it. |
Graded Assignments
Assignment 1: The Editor’s Taste
Students apply a copy of the Doodle Editor’s manifest with an editorial policy of their own, such
as science and education only, and run the same five requests through both editors, including two
that tempt the model off the menu. They compare the choices and reasons, and say which of them
check_decision() would refuse.
Turn in: the manifest, and a three-page Markdown report with each request, both answers and the analysis.
Criterion |
Pts |
Exemplary |
Developing |
|---|---|---|---|
A working manifest |
4 |
Applies cleanly; the policy is stated in the system prompt, not scattered. |
Edits that contradict the rest of the prompt. |
Evidence |
6 |
Real answers from both editors, pasted as they came. |
Paraphrased, or one editor only. |
Analysis |
6 |
Explains each difference by a line of the prompts, and reads the reasons. |
Lists differences without explaining them. |
The off-menu attempts |
4 |
Shows what the model did, and names the check that would catch it. |
Assumes the prompt is enough. |
Assignment 2: Prompting From Code
Students write a management command in their fork that prompts their copy of the editor for a
day, with editor_request() and the helpers in prompting.py, and prints each decision with
the verdict of check_decision(). The api key it uses must be deleted even when the prompt
fails, and the command must be tested with the model patched.
Turn in: a pull request against their own fork, the command’s output for two days, and its test output from the container.
Criterion |
Pts |
Exemplary |
Developing |
|---|---|---|---|
Uses the platform |
5 |
Prompts through |
Calls a provider’s SDK directly. |
Parses and checks |
5 |
|
Prints the raw answer. |
Cleans up |
4 |
The api key is deleted on success and failure, and a test proves it. |
A key left behind when the prompt fails. |
Tested |
6 |
Tests with the model patched, in the style of |
Untested, or tested by prompting a model. |
Assignment 3: Trust Nothing
Students write at least eight tests that feed crafted answers through parse_answer() and
check_decision(), and crafted trends through region_trends(), with the model and the
network patched. They look for one input that passes but shouldn’t, and propose a fix.
Turn in: a pull request against their own fork with the tests, their output from the container, and a paragraph on the gap they found, or why they found none.
Criterion |
Pts |
Exemplary |
Developing |
|---|---|---|---|
Cases covered |
7 |
Answers and trends, eight or more distinct ways to fail, each asserting the reason given. |
Only malformed JSON and off-menu. |
Isolation |
4 |
No real prompts or requests; saved responses, as in |
Tests that call the LLM or Wikipedia. |
The gap |
6 |
A real gap with a failing test and a fix, or a convincing search that found none. |
“The checks look fine.” |
Runs as shown |
3 |
Output from the container matches the pull request. |
Took the tests’ passing on trust; output not included. |
Assignment 4: A Judge, Measured
Students write an llm_judge Guardrail for a criterion that code can’t check, label twenty
inputs by hand, and run each through the Guardrail with evaluate(), as tone_check()
does. They report its false positives and false negatives, try one change to its prompt or
threshold, and say whether a feature that uses it should fail open or closed.
Turn in: the Guardrail’s manifest, the labeled inputs and results as a table, and a one-page analysis.
Criterion |
Pts |
Exemplary |
Developing |
|---|---|---|---|
A fair test set |
5 |
Labeled before running, with hard cases on both sides of the line. |
Twenty easy cases, or labels chosen after the results. |
Results |
5 |
Every input’s verdict and score, recorded as they came. |
A summary without the data. |
The change |
4 |
One change, measured on the same set, kept or reverted on the evidence. |
Several changes at once, or none measured. |
Fail open or closed |
6 |
A decision argued from what each kind of error costs the feature. |
“Closed, to be safe,” with no costs weighed. |
Capstone Projects
Students choose one, propose it before Live session 3, and present it in Live session 4.
Capstone A: An AI Feature of Your Own
Learning objective. Build an AI feature into software, with its AI parts as Smarter resources, its safety in code, and a person in the loop.
In their fork of Smarter, or in a Django project of their own that prompts a Smarter platform, students build a feature in which an LLMClient makes a choice or writes something that users see: a tip of the day picked from the documentation, a suggested description for a new resource, or a summary of an account’s week. It must work without the model, give the model a menu or a schema, check every answer in code, judge what code can’t check with a Guardrail, record what was asked and answered, let a person approve or override what is new, have a switch, and say on the page when an AI made it. A feature meant for Smarter itself must be agreed with the maintainers in a GitHub issue first.
What students turn in:
A pull request against their own fork, with the LLMClient’s and Guardrail’s manifests.
Tests with the model patched, and their output, with at least 90% coverage of the feature’s code.
A two-page design note: the feature’s desks, what each check guards against, whether each one fails open or closed and why, and the cost per day from the journal.
A six-minute presentation, including one answer the code refuses and one a person rejects.
How it is graded (100 points):
Criterion |
Pts |
Exemplary |
Developing |
|---|---|---|---|
A safe answer first |
15 |
The feature works, and says nothing wrong, with the model off. |
Breaks or shows an error without the model. |
Contract and checks |
20 |
A validated request and answer, and every rule of the prompt enforced in code. |
Trusts the prompt. |
Judge and review |
20 |
A judge for what code can’t check, and a person approves what is new. |
The model’s output goes straight to users. |
Record, control, disclosure |
15 |
What was asked and answered is kept; a switch turns the AI off; the page credits it. |
No record, no switch, or no disclosure. |
Tests |
20 |
Refusals and failures tested, with the model patched, at 90% coverage or more. |
Only the happy path, or tests that prompt a model. |
Design note |
10 |
Each check tied to the failure it prevents and its cost. |
A description of the code. |
Capstone B: Red Team the Newsroom
Learning objective. Evaluate an AI feature’s defenses with evidence, layer by layer, and strengthen the weakest.
On their own platform, students attack the Doodle with at least forty cases: requests that tempt
the editor off the menu, trend titles written to inject instructions, occasions that are sad but
contain no blocked word, and briefs that ask the artist for scripts, links, external images or a
drawing over the base scene. For each case they record which layer stopped it: the prompt, an
output Guardrail, check_decision(), the trends filter, the judge, the sanitizer, or the person
in the admin. They fix the two weakest points they find, each with a test that fails before the
fix.
What students turn in:
A table of the cases, with the input, what each model answered, and the layer that stopped it, or none.
A four-page analysis of where the defenses overlap and where they are thin, citing the code and the commits.
A pull request against their own fork with the two fixes and their tests.
How it is graded (100 points):
Criterion |
Pts |
Exemplary |
Developing |
|---|---|---|---|
Range of attacks |
20 |
Every layer is tested, with cases a careful attacker would try. |
Forty variants of one attack. |
Evidence |
20 |
Real answers and editor runs, recorded as they came. |
Results from memory, or assumed from the code. |
Attribution to layers |
20 |
Each case traced to the code or resource that stopped it. |
“It was blocked.” |
The fixes |
25 |
Failing tests, then fixes that pass without breaking the app’s other tests. |
Suggestions without tests. |
Writing |
15 |
Concise, organized and specific. |
Long or general. |
Recommended Texts
Required: How Two Smarter LLMClients Create the Sign-in Page’s Daily Doodle and Doodle React Component, in week 1.
Required: Smarter LLM Client, in week 1.
Required: Smarter Guardrail and LLM Guardrails in YAML: How Smarter Declares, Orders and Audits Them, in week 3.
Required: the OWASP Top 10 for LLM Applications, on prompt injection and improper output handling, in week 3.
Reference: Chip Huyen, AI Engineering (O’Reilly, 2025), on evaluation, guardrails and the architecture of AI applications.
Reference: John Berryman and Albert Ziegler, Prompt Engineering for LLMs (O’Reilly, 2024), on designing the prompts and answers that software consumes.
In the repository: editor.py, prompting.py, sanitize.py and the three Doodle commits, in order.
Further Study
Official documentation: OpenAI’s structured outputs guide, on making a model answer in a JSON Schema, and the Pydantic documentation, on validating what comes back.
Security: the OWASP Top 10 for LLM Applications, especially prompt injection and improper output handling, and OWASP’s Cross Site Scripting Prevention Cheat Sheet, for the reasoning behind the Doodle’s sanitizer.
Human-AI interaction: Google’s People + AI Guidebook, on designing review, explanation and disclosure, and Saleema Amershi et al., Guidelines for Human-AI Interaction (CHI 2019), eighteen guidelines tested on real products.
Risk management: the NIST AI Risk Management Framework, for the vocabulary that governance teams will use when they review an AI feature.
Standards: the SVG 2 specification, for what an SVG can carry besides shapes, and JSON Schema.
YouTube: the AI Engineer channel, for talks on evaluations, guardrails and LLMs in production, and the PyCon US channel, for talks on Pydantic and testing.
Podcasts: Latent Space, on building with LLMs, and Practical AI, on putting AI into products.
Smarter: articles on blog.smarter.sh related to this course: How Two Smarter LLMClients Create the Sign-in Page’s Daily Doodle, the course’s case; LLM Guardrails in YAML: How Smarter Declares, Orders and Audits Them, on the Guardrails around the editor and its judge; DOMPurify and Markdown: Rendering LLM Output as Untrusted HTML in an Embeddable Chat Widget, on the same problem as the artist’s SVG, in the chat; Build a Slack Bot With a Smarter LLMClient, on another piece of software that leaves the AI parts on the platform; and Django-React Integration, on how the console hosts the Doodle’s React app. The case study Guardrails is a lesson with labs and discussion questions.