Prompt Engineering: Four-Week Online Course

A four-week online course in prompt engineering, for students who need no programming experience. Students build, evaluate and brand an AI assistant by editing a YAML manifest. Everything students build is a Smarter LLMClient manifest, the same YAML that runs in production, and they start from the nineteen built-in LLMClients that ship with every installation: the one-line simple assistant, and customer_support, a complete support agent with a persona, refusals, a skill, a tool and four guardrails. The prompt engineering workbench shows every request sent to the model, every tool call and every plugin selected, beside the chat, so students see the effect of each edit at once. There are twelve 30-minute recorded lectures and a 90-minute live session each week. The extra time, compared with the two-week course, goes to evaluation, retrieval and multi-step workflows. See Course Formats and Grading for the format, the grading scale and the course policies, and Prompt Engineering for an overview of the subject.

At a Glance

Students	Anyone: business, communications, education, and the humanities as well as computing.
Deployment	A shared Smarter instance run by the instructor. Students need only a web browser. Students who want to may install their own with the Quick Start in week 3.
Workload	3 hours of instruction and 4 to 6 hours of assignments each week.
Cost	Expect less than $10 USD of model charges per student, capped by a per-student Budget.

Learning Objectives

By the end of the course, students can:

1. Explain how a model turns a prompt into an answer, including tokens, context, sampling and tool calls.

2. Write and maintain system prompts with a persona, a task, constraints, an output format and refusals.

3. Evaluate an assistant with a test set and a scoring rubric, including an LLM-as-judge, and improve it from the evidence.

4. Choose a model and settings by comparing quality, latency, tokens and cost.

5. Ground an assistant with knowledge, retrieval and tool plugins, and protect it with guardrails.

6. Design, document and present an AI assistant for a real audience.

Technical Prerequisites

• Required: a computer with a current web browser and a reliable connection for video calls, and experience of using a chat assistant such as ChatGPT, Claude or Gemini.

• Required: basic YAML: key-value pairs, indentation, lists and multi-line strings. Lecture 1 reviews it, and Learn YAML in Y minutes covers it.

• Required by week 2: a GitHub account, and enough Git to commit and push a file through the GitHub web interface or GitHub Desktop.

• Helpful: Visual Studio Code with a YAML extension, a spreadsheet program, and reading JSON.

• Not needed: programming.

Schedule

The files that the lectures refer to are listed in full under Smarter in the Source Code.

Week	Item	Content
1	Lecture 1	How a language model reads a prompt. Tokens, next-token prediction, the context window, and system and user messages. A YAML refresher, and a tour of the web console and the workbench.
1	Lecture 2	Anatomy of a good system prompt. Persona, audience, task, constraints, output format and refusals. A line-by-line reading of the built-in customer_support LLMClient.
1	Lecture 3	Techniques. Examples (few-shot prompting), delimiters, step-by-step instructions, and asking for structured output. When each helps, and when it only costs tokens.
1	Live session 1	Live build of an assistant from the simple LLMClient. Pairs critique each other’s system prompts. Start Assignment 1.
2	Lecture 4	Testing an assistant. Test sets with expected behavior, edge cases, adversarial prompts, and regression testing after every change.
2	Lecture 5	Scoring answers. Writing a rubric, scoring by hand, and using a second model as a judge. How judges are biased, and how to check them against human scores.
2	Lecture 6	Settings, models and cost. defaultTemperature, defaultMaxTokens, provider and defaultModel. Comparing providers fairly. Reading Smarter’s token and cost records, and the dashboard’s cost charts.
2	Live session 2	Review of Assignment 1. Live comparison of three models on a shared test set, scored by the class. Start Assignment 2.
3	Lecture 7	Knowledge. Why models make things up. Grounding with Static and Skill plugins.
3	Lecture 8	Retrieval. Embeddings and similarity search, explained without mathematics. Loading documents into a Vectorstore and answering with citations.
3	Lecture 9	Tools. Websearch and API plugins. Smarter offers a plugin only when the prompt matches one of its searchTerms (exactly, with a typo, or as a synonym), and the model then decides, from the plugin’s description, whether to call it. Reading plugin selection and tool calls in the Console.
3	Live session 3	Review of Assignment 2. Students debug tool calls that do not happen, using the Console. Capstone proposals are discussed in small groups. Start Assignment 3.
4	Lecture 10	Safety. Prompt injection, jailbreaks and data leakage, shown live. Guardrails and why they are enforced while a system prompt is only a request.
4	Lecture 11	Workflows. Breaking a job into steps with an Orchestrator, and when one prompt is better than several.
4	Lecture 12	Designing for a real audience. Interviewing a client, writing a welcome message and examples, branding with styles, and deploying.
4	Live session 4	Capstone presentations (about 6 minutes each, with 2 minutes of questions), and a course retrospective.

Assignments 1 to 3 are due before the next live session. Assignment 4 is due in the middle of week 4, three days before Live session 4. The capstone proposal (one paragraph) is due before Live session 3, and the capstone at the start of Live session 4.

Smarter in the Source Code

Every lecture, lab and assignment in this course works with the manifests below. They are the built-in examples that ship with every Smarter installation, so students can open them in the web console, or read them on the main branch of github.com/smarter-sh/smarter. No programming is needed to read them. The Python files are listed for instructors and curious students. Python paths are under smarter/smarter/.

Prompts and Settings

Topic	What Smarter does	Where to look
A minimal assistant	The simple LLMClient: a one-line system prompt, a small model, temperature 1.0 and a 4,096-token limit. Every setting a prompt engineer changes sits under spec.config.prompt.	llmclient-simple.yaml
A production-quality system prompt	The customer_support LLMClient: a persona (Sam), an audience, a narrow job, an output format, refusals, a skill for on-brand replies, a plugin that reads web pages, and four input guardrails.	llmclient-customer-support.yaml
Seventeen more examples	Built-in LLMClients for weather, mathematics, dates, data analysis, research, ticket triage, workplace writing, programming and others, each a worked example of a different prompting job.	llmclient/data/llm-clients/
Every setting, defined	The Pydantic model that defines and validates provider, defaultModel, defaultSystemRole, defaultTemperature, defaultMaxTokens and the app... settings, each with a description. A misspelled key is rejected, with an error that says why.	llmclient/manifest/models/llmclient/spec.py
Branding	spec.config.styles: logo, background, avatars, fonts and colors for each part of the chat, validated as plain CSS. Styling Your LLMClient’s Chat.	llmclient/manifest/models/llmclient/styles.py

Knowledge and Tools

Topic	What Smarter does	Where to look
Two decisions, not one	A plugin is offered to the model only when Smarter selects it, because the prompt contains one of its searchTerms exactly, with a typo, or as a synonym, or because its selector directive is ALWAYS. Only then does the model decide, from the plugin’s description, whether to call it. Natural Language Processing (NLP).	plugin/plugin/base.py (selected()), plugin/nlp.py
Static knowledge	A Static Plugin returns fixed facts, here about a fictional Willy Wonka product, so the model quotes them instead of making them up.	everlasting-gobstopper.yaml
Skills as knowledge	SkillPlugins for brand voice, meeting notes, code review and SQL analysis, and three of Anthropic’s published skills.	skill-brand-voice.yaml, sample-plugins/
Tools that act	Websearch Plugins for research, news, local search and reading a page, and an ImageSearchPlugin that lets the model show real pictures instead of inventing urls.	websearch-research-assistant.yaml, image-search.yaml
Built-in functions	A calculator, a date calculator and a weather function that any LLMClient can list under spec.functions.	prompt/functions/

Safety, Cost and Evidence

Topic	What Smarter does	Where to look
Guardrails	Eighteen built-in guardrails, each a short manifest. off_topic_input asks a small model whether a message is in scope, and pii_redaction_input removes personal data before the model sees it.	off-topic-input.yaml, guardrail/data/guardrails/
The workbench Console	Beside the chat, the Console shows the LLMClient’s configuration, every request sent to the model, every tool call, which plugins were selected, and the server’s logs.	smarter-chat: src/components/Console/
Tokens and cost	Every request is recorded as a charge, with its tokens, provider and model, and priced per model. Cost Accounting.	account/models/charge.py, account/models/llm_prices.py
A record of every prompt	Each prompt, answer and tool call is kept, so an assignment can ask for the conversation that produced a result. Smarter Journal.	prompt/models/
A per-student allowance	A Budget that caps what each student can spend in a month. Smarter Budget.	student-monthly-allowance.yaml

Graded Assignments

Assignment 1: A Focused Assistant

Students turn the simple LLMClient into an assistant with one narrow job, using at least two of the techniques from Lecture 3, and explain each design choice.

Turn in: the manifest in a GitHub repository, three example conversations, and a half-page design note.

Criterion	Pts	Exemplary	Developing
Manifest	4	Valid, applies cleanly, and its welcome message and examples suit the job.	Applies only with help, or keeps defaults.
System prompt	6	Clear persona, audience, job, constraints and refusals.	The job or refusals are vague.
Techniques	6	Two techniques used where they help, visible in the conversations.	Techniques named but not used, or used where they do not help.
Design note	4	Explains why, not only what.	Restates the system prompt.

Assignment 2: An Evaluation Harness

Students write a test set of twenty prompts and a five-criterion scoring rubric, score the answers by hand, then build a judge LLMClient that scores them with the same rubric, and compare the judge’s scores with their own.

Turn in: the test set, the rubric, a spreadsheet of human and judge scores, the judge’s manifest, and a one-page analysis of where the judge agrees and disagrees.

Criterion	Pts	Exemplary	Developing
Test set	5	Varied: typical, edge, ambiguous and adversarial prompts.	Variations of one easy question.
Rubric	5	Observable criteria that two people would apply the same way.	Vague criteria such as “helpful”.
Judge	5	Returns scores in a consistent format, using the rubric.	Returns free text, or ignores the rubric.
Agreement analysis	5	Quantifies agreement, and explains the disagreements.	Says only that the judge “mostly agrees”.

Assignment 3: Grounded Assistant

Students give their assistant knowledge with a Vectorstore built from at least five real documents (for example a course handbook, a club’s rules or a product manual) and one tool plugin, then measure with their evaluation harness whether answers improved.

Turn in: the manifests, the documents or links to them, before-and-after scores, and Console screenshots of a retrieval and a tool call.

Criterion	Pts	Exemplary	Developing
Retrieval	6	Answers cite the right documents, and the assistant says when it does not know.	Retrieval works, but answers ignore it or invent sources.
Tool plugin	4	Called when it should be, and its result is used.	Never called, without explanation.
Measured improvement	6	Same test set and rubric before and after, with the change reported honestly.	No before-and-after comparison.
Explanation	4	Explains which questions improved and which did not, and why.	Reports scores without interpretation.

Assignment 4: Red Team Report

Students attack a classmate’s assistant (assigned by the instructor) with at least ten attempts at prompt injection, jailbreaking, going off topic or extracting its instructions, then write up the results for the owner, who adds guardrails in response.

Turn in: an attack log from the journal, a two-page report with findings ranked by severity, and, as the owner, the guardrail manifests that answer the report.

Criterion	Pts	Exemplary	Developing
Attack variety	5	Ten or more attempts, across several techniques.	Repeats of the same attack.
Report quality	5	Each finding has the prompt, the result, a severity and a suggested fix.	Findings without evidence or fixes.
Response as owner	6	Guardrails stop the reported attacks without blocking normal use, with evidence.	Guardrails added but not tested, or block normal prompts.
Professionalism	4	Constructive and specific, as a security report should be.	Mocking or vague.

Capstone Projects

Students choose one, propose it before Live session 3, and present it in Live session 4.

Capstone A: A Deployed Assistant for a Real Client

Learning objective. Take an AI assistant from a client’s needs to a deployed, evaluated and governed application, and hand it over.

Students interview a real client, build an LLMClient with retrieval, at least one tool plugin, guardrails and custom styles, evaluate it with their harness, and have the instructor deploy it so that the client can try it.

What students turn in:

• A Git repository with every manifest, the documents used for retrieval (or links), and a README.md that explains how to apply and maintain the assistant.

• A three-page design brief: the client, the audience, the job, the risks, and the reason for each plugin, guardrail and setting.

• The evaluation: test set of at least thirty prompts, rubric, scores and an analysis.

• Written feedback from the client, of a paragraph or more.

• A six-minute presentation with a live demonstration.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
Fit to the client	20	Design follows from the interview, and the client’s feedback is addressed.	Generic assistant with a client’s name.
Prompt and grounding	20	Answers are accurate, cited and on topic, and say when they do not know.	Accurate on easy questions, invents answers on hard ones.
Governance	15	Guardrails and a budget match the risks named in the brief.	Guardrails unrelated to the stated risks.
Evaluation	25	Thirty or more varied prompts, a clear rubric, and honest results that drove changes.	Few tests, or results that did not influence the design.
Handover and presentation	20	README.md lets someone else maintain it. Presentation is clear and on time.	No maintenance notes, or the presentation overruns.

Capstone B: Choosing a Model for an Organization

Learning objective. Run a rigorous, reproducible evaluation that answers a real business question about models and prompts, and communicate it to decision makers.

A fictional organization (or a real one, with permission) wants one assistant for a defined job, such as answering HR policy questions or drafting first replies to customer email. Students build the same assistant on at least three models from at least two providers, each with its best system prompt, and evaluate them on quality, latency, cost and safety.

What students turn in:

• A Git repository with every manifest, the test set of at least forty prompts, the rubric and the raw scores.

• A judge LLMClient, validated against human scores on at least ten answers.

• A four-page recommendation memo, with charts, for a non-technical manager: which model, which prompt, the expected monthly cost for a stated number of users, and the risks.

• A six-minute presentation of the recommendation.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
Evaluation design	25	Fair comparison: each model’s prompt tuned with the same effort, and the same tests.	One prompt tuned for one model and reused for the others.
Validated judge	15	Agreement with human scores measured and reported.	Judge used without checking.
Cost and latency	20	Measured from Smarter’s records and projected to a monthly figure, with assumptions stated.	Taken from price lists, not measured.
Safety	15	Adversarial prompts included, and each model’s failures reported.	Safety not tested.
Memo and presentation	25	A clear recommendation that a manager could act on, with the trade-offs stated.	Data without a recommendation.

Recommended Texts

• Required: John Berryman and Albert Ziegler, Prompt Engineering for LLMs (O’Reilly, 2024). Week 1: chapters 1 to 4. Week 2: chapters 5, 6 and 10. Week 3: chapters 7 to 9. Week 4: chapter 11.

• Required: Chip Huyen, AI Engineering: Building Applications with Foundation Models (O’Reilly, 2025). Week 2: chapters 3 and 4, on evaluation. Week 3: chapter 6, on retrieval and agents.

• Background: Ethan Mollick, Co-Intelligence: Living and Working with AI (Portfolio, 2024).

• Smarter documentation: Smarter LLM Client, Smarter Plugin, Smarter Vectorstore, Smarter Guardrail, Smarter Orchestrator and Cost Accounting.

Further Study

• Online guides: Anthropic’s prompt engineering documentation, OpenAI’s prompt engineering guide, and the community Prompt Engineering Guide from DAIR.AI, which catalogs techniques with links to the research papers behind them.

• YouTube: Andrej Karpathy’s Intro to Large Language Models and Deep Dive into LLMs like ChatGPT on his channel, the clearest explanations of what a model actually does with a prompt. The neural networks series on 3Blue1Brown, for the mathematics, visually.

• Podcasts: Latent Space, for practitioners building with models, and Practical AI, for a broader audience.

• Writing: Simon Willison’s weblog, which tests new models and prompting techniques as they appear, and Ethan Mollick’s One Useful Thing, on using AI well at work and in teaching.

• Practice: Gandalf, a game in which you prompt a model into revealing a password, is a memorable lesson in why instructions alone do not control a model.