Building AI Applications: Four-Week Online Course

A four-week online course on the architecture of AI applications: retrieval, tool calling, structured data, MCP servers and multi-agent workflows, each a resource declared in a YAML manifest. Students work on the resources that ship with every Smarter installation: nineteen built-in LLMClients, from the bare simple assistant to research_analyst, which searches the web through an Exa MCP server, reads pages and PDFs, and checks its citations with guardrails; twenty-five MCP clients for services such as GitHub, Notion and Stripe; Vectorstores on Qdrant and Pinecone; and an Orchestrator with six coordination strategies. There are twelve 30-minute recorded lectures and a 90-minute live session each week. Week 1 covers the building blocks, week 2 knowledge, week 3 action, and week 4 orchestration and production. See Course Formats and Grading for the format, the grading scale and the course policies, and Building AI Applications for an overview of the subject.

At a Glance

Students	Product managers, analysts, designers, consultants and developers.
Deployment	A shared Smarter instance run by the instructor, with a Vectorstore backend, a sample database behind a Connection, and two or three MCP clients set up in advance.
Workload	3 hours of instruction and 4 to 6 hours of assignments each week.
Cost	Expect less than $12 USD of model charges per student, capped by a per-student Budget.

Learning Objectives

By the end of the course, students can:

1. Describe the parts of an AI application, and choose the right ones for a job.

2. Build and tune retrieval-augmented assistants, and measure their accuracy.

3. Let a model query structured data safely, and check its answers.

4. Connect an assistant to external systems with plugins and MCP servers, and limit what they can do.

5. Design orchestrated workflows, and decide on evidence when they are worth their cost.

6. Evaluate, govern and present an AI application as a product.

Technical Prerequisites

• Required: a computer with a current web browser and a reliable connection for video calls.

• Required: reading and editing YAML (Learn YAML in Y minutes) and reading JSON.

• Required: a general idea of what a database table and a web API are.

• Required by week 2: a GitHub account, and enough Git to commit files through the web interface or GitHub Desktop.

• Helpful: experience writing system prompts, basic SQL, and a spreadsheet program.

• Not needed: programming.

Schedule

The files that the lectures refer to are listed in full under Smarter in the Source Code.

Week	Item	Content
1	Lecture 1	Anatomy of an AI application. Model, instructions, knowledge, tools, guardrails and orchestration as Smarter resources.
1	Lecture 2	Models and prompts for applications. Choosing a model, system prompts that stay on task, and structured output that other systems can use.
1	Lecture 3	Reading manifests. A guided tour of customer_support, research_analyst and data_analyst: what each resource adds, confirmed in the Console.
1	Live session 1	Architecture teardown in groups, then a short live build. Start Assignment 1.
2	Lecture 4	Retrieval, part 1. Embeddings, similarity search and citations, without mathematics. Building a Vectorstore.
2	Lecture 5	Retrieval, part 2. Chunking, top-k, metadata, stale documents, and measuring retrieval quality separately from answer quality.
2	Lecture 6	Structured data. Two safe patterns, both shipped with Smarter: a SQL Plugin runs a fixed, parameterized query that the model can only fill in (stackademy_sql), and a SkillPlugin teaches the model a schema so that it writes SQL for a person to review (the data_analyst LLMClient). A Connection with a read-only database user, its password in a Secret, and how wrong answers still happen.
2	Live session 2	Review of Assignment 1. Diagnose retrieval failures as a class. Start Assignment 2.
3	Lecture 7	Tools. Function calling, API and Websearch plugins. Plugin selection by search terms decides which tools Smarter offers, and their descriptions steer which ones the model calls (Natural Language Processing (NLP)).
3	Lecture 8	MCP. The Model Context Protocol and MCP clients: authentication, tool selection, and trusting a server.
3	Lecture 9	Tool safety. What a tool can do if the model is tricked. Prompt injection through retrieved content, least privilege, and Guardrails.
3	Live session 3	Review of Assignment 2. Students attack each other’s tool-using assistants with injected content. Capstone proposals discussed. Start Assignment 3.
4	Lecture 10	Orchestration. Workflows, routing, agents and the Orchestrator.
4	Lecture 11	Evaluation. Test sets, rubrics, LLM judges, and regression testing an application.
4	Lecture 12	Production. Budgets, cost accounting, the journal, deployment, and what changes when real users arrive.
4	Live session 4	Capstone presentations (about 6 minutes each, with 2 minutes of questions), and a course retrospective.

Assignments 1 to 3 are due before the next live session. Assignment 4 is due in the middle of week 4. The capstone proposal is due before Live session 3, and the capstone at the start of Live session 4.

Smarter in the Source Code

Every lecture, lab and assignment in this course works with the manifests and files below. The manifests ship with every Smarter installation, so students can open them in the web console, or read them on the main branch of github.com/smarter-sh/smarter. No programming is needed to read them. The Python files are listed for instructors and curious students. Python paths are under smarter/smarter/.

Applications to Take Apart

Application	What it shows	Where to look
simple	A bare model: a one-line system prompt and nothing else. The baseline for every comparison.	llmclient-simple.yaml
customer_support	A complete application: a persona, a skill for on-brand replies, a plugin that reads web pages, and four input guardrails for abuse, self-harm, prompt injection and jailbreaks.	llmclient-customer-support.yaml
research_analyst	Web search through the Exa MCP server, page and PDF readers, an image search, and output guardrails that catch fabricated citations and regulated advice.	llmclient-research-analyst.yaml
data_analyst	A SkillPlugin that teaches the model a data warehouse’s schema and house rules, so that it writes SQL for a person to review, with DeepWiki through MCP for SQL documentation.	llmclient-data-analyst.yaml, skill-sql-analyst.yaml
Stackademy	A fictional university’s course catalog, served two ways: by a SQL Plugin over a database, and by an API Plugin over a REST API, each with its own LLMClient.	plugin/data/stackademy/

Knowledge and Data

Topic	What Smarter does	Where to look
Vector databases	Qdrant, self-hosted on Smarter’s own Kubernetes cluster or in Qdrant Cloud, and Pinecone, each declared in a manifest. Smarter creates the database, extracts text from PDFs, Markdown, CSV, HTML and other formats, splits it into chunks, embeds and loads it, and takes snapshots on a schedule. Smarter Vectorstore.	vectorstore/data/vectorstores/, vectorstore/backends/, vectorstore/extract.py
Retrieval strategies	A Vectorsearch names a Vectorstore and how to query it: similarity, similarity with a score threshold, or maximal marginal relevance, how many results, and metadata filters. Smarter Vectorsearch.	vectorstore/service.py (search())
A parameterized query	A SQL Plugin runs a fixed query written by a person. The model can only supply its parameters, with types, descriptions and allowed values, so it never writes SQL that runs. SQL Plugin.	stackademy-plugin-sql.yaml, plugin/plugin/sql.py
Connections and credentials	A Connection names the database or API, and its password is the name of a Secret, so neither the model nor the plugin’s author ever sees it. Smarter Connection.	connection/data/sample-connections/

Action and Orchestration

Topic	What Smarter does	Where to look
Plugin selection	Smarter offers a plugin’s tool to the model only when the prompt matches one of its search terms, exactly, with a typo or as a synonym. The model then decides, from the tool’s description, whether to call it. Natural Language Processing (NLP).	plugin/plugin/base.py (selected())
Twenty sample plugins	Static, Skill, Websearch and ImageSearch plugins, ready to attach to an LLMClient.	plugin/data/sample-plugins/
MCP clients	Twenty-five ready-made MCP clients, for GitHub, Notion, Linear, Stripe, Hugging Face, Microsoft Learn and others. Each states its transport, its credentials (a Secret) and the tools the model may use, such as only the read-only get_*, list_* and search_* tools of GitHub’s server. Smarter MCP Client.	mcpclient/data/mcpclients/, github.yaml
Safe connections to MCP servers	Only the HTTP and SSE transports are allowed, never stdio, which would run a command on the server, and every endpoint must resolve to a public address.	mcpclient/connection.py
Orchestration	An Orchestrator coordinates several LLMClients with one of six strategies (sequential, parallel, supervisor, router, voting and debate), gives each a role (planner, executor, critic, router, summarizer or tool caller), and records every run and step. Smarter Orchestrator.	orchestrator/enum.py, orchestrator/models/

Production

Topic	What Smarter does	Where to look
Guardrails	Eighteen built-in guardrails, run as a pipeline on the prompt before the model sees it and on the reply before the user does. Smarter Guardrail.	guardrail/data/guardrails/, guardrail/services/pipeline.py
Cost and budgets	Every request is charged, with its tokens, to the user, the application and the account, and Budgets cap the spending. Cost Accounting.	account/models/charge.py, budgets/
The record of every conversation	Each prompt, answer, tool call and plugin selection is stored, and shown in the workbench’s Console. Smarter Journal.	prompt/models/

Graded Assignments

Assignment 1: Architecture Teardown

Students document two built-in LLMClients: every resource each one uses, its role, and the path of one question, confirmed in the Console. They finish with a paragraph on why the two designs differ.

Turn in: two diagrams and a one-page explanation.

Criterion	Pts	Exemplary	Developing
Completeness	6	Every resource identified with its role.	Resources missing or confused.
Data flow	6	Order of model calls, tool calls and results shown, and confirmed in the Console.	Boxes without flow, or inferred only.
Comparison	4	Explains the design differences by the jobs the assistants do.	Lists differences without explanation.
Clarity	4	A newcomer could follow it.	Cluttered or unlabeled.

Assignment 2: Measured Retrieval

Students build a Vectorstore over at least ten documents, write twenty questions with the passage that answers each, and measure two things separately: whether the right passage was retrieved, and whether the answer was right. They try one change to improve each.

Turn in: manifests, the question set with expected passages, before-and-after measurements, and a one-page analysis.

Criterion	Pts	Exemplary	Developing
Question set	5	Varied questions, each linked to the passage that answers it.	Questions without expected passages.
Two measurements	6	Retrieval and answer quality measured separately.	Only answers checked.
Improvements	5	One change per measurement, with its effect.	Changes without measured effect.
Analysis	4	Explains which failures were retrieval and which were generation.	Does not distinguish them.

Assignment 3: Data Assistant with a Risk Review

Students build an assistant over the sample database, check ten answers against the truth, and write a risk review of what could go wrong if managers relied on it, with mitigations they implement.

Turn in: manifests, the questions, answers, the SQL behind each one and checks, and a two-page risk review.

Criterion	Pts	Exemplary	Developing
Configuration	4	Read-only connection, credentials in a Secret.	Unnecessary access, or credentials in the manifest.
Accuracy	6	All ten checked; failures explained from the SQL.	Answers accepted unchecked.
Risk review	6	Specific risks, ranked by likelihood and impact.	Generic risks of AI.
Mitigations	4	At least two mitigations implemented and shown working.	Mitigations proposed but not implemented.

Assignment 4: Tools Under Attack

Students build an assistant with a Websearch or API plugin and an MCP client, then try to make it misuse a tool through prompt injection, including instructions hidden in a web page or document that it reads. They add defenses and repeat the attacks.

Turn in: manifests, an attack log from the journal before and after the defenses, and a one-page report.

Criterion	Pts	Exemplary	Developing
Working tools	4	Both tools called when appropriate.	One tool never used.
Attacks	6	Direct and indirect injection attempts, at least eight in total.	Direct attacks only.
Defenses	6	Guardrails and narrower tool access stop most attacks, shown in the second log.	Defenses added but not re-tested.
Report	4	Explains what still gets through, honestly.	Claims complete protection.

Capstone Projects

Students choose one, propose it before Live session 3, and present it in Live session 4.

Capstone A: An AI Product for a Real User

Learning objective. Take an AI application from a user’s need to an evaluated, governed and deployed product, and present it as a product manager would.

Students interview a real user and build an application with retrieval, at least one tool, at least two guardrails, a budget, and an Orchestrator if the job has distinct steps. The instructor deploys it so that the user can try it.

What students turn in:

• A Git repository with every manifest and a README.md for maintainers.

• A three-page product brief: the user, the job, the architecture with a diagram, the risks and their controls, and the expected monthly cost.

• An evaluation with at least thirty test cases, a rubric and results.

• Written feedback from the user.

• A six-minute presentation with a live demonstration.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
User fit	15	The design follows from the interview; feedback is addressed.	Generic application.
Architecture	20	Each component justified; orchestration used only where it helps.	Components added because they exist.
Accuracy and evaluation	25	Thirty varied tests, honest results, failures addressed.	Few tests, or results that did not change the design.
Governance and cost	20	Guardrails and budget match the risks; cost projected from measurements.	Controls unrelated to risks; cost guessed.
Presentation	20	Clear, on time, persuasive to a non-technical audience.	Technical walkthrough with no product story.

Capstone B: An Agentic Workflow with an Evaluation Suite

Learning objective. Design a multi-step workflow, prove with an evaluation suite that each step and the whole work, and decide where a single assistant would do.

Students choose a job with at least three distinct steps, such as reading a support ticket, looking up the customer’s records and drafting a reply, or researching, outlining and writing a briefing. They build it as an Orchestrator of specialized LLMClients with tools, and as a single LLMClient, and evaluate both.

What students turn in:

• A Git repository with both designs.

• An evaluation suite: test cases for each step and end-to-end, a rubric, and a validated judge LLMClient.

• Results for both designs, with cost and latency.

• A three-page report and a six-minute presentation.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
Workflow design	20	Steps are well bounded, with clear inputs and outputs.	Steps overlap, or one step does everything.
Evaluation suite	25	Tests per step and end-to-end; judge validated against human scores.	End-to-end tests only; judge unchecked.
Comparison	20	Fair comparison with the single assistant, on quality, cost and latency.	Single assistant neglected.
Robustness	15	Handles bad inputs and tool failures gracefully.	Fails silently when a tool fails.
Report and presentation	20	Clear recommendation with trade-offs.	Results without a recommendation.

Recommended Texts

• Required: Chip Huyen, AI Engineering: Building Applications with Foundation Models (O’Reilly, 2025). Week 1: chapters 1 and 2. Week 2: chapter 6 (retrieval). Week 3: chapter 6 (agents). Week 4: chapters 3, 4 and 10.

• Required: Anthropic, Building effective agents (2024), before Lecture 10.

• Optional: Jay Alammar and Maarten Grootendorst, Hands-On Large Language Models (O’Reilly, 2024), chapters 2 and 8, for the mechanics of embeddings and retrieval.

• Optional: Paul Iusztin and Maxime Labonne, LLM Engineer’s Handbook (Packt, 2024), for students who go on to build applications in code.

• Smarter documentation: Smarter Vectorstore, Smarter MCP Client, Smarter Connection, SQL Plugin, Smarter Orchestrator, Smarter Guardrail and Cost Accounting.

Further Study

• Essays: Anthropic’s Building effective agents, the clearest short guide to when a workflow, a single agent or several agents is the right design. Eugene Yan’s writing on patterns for building LLM systems and products.

• Standards: the Model Context Protocol specification and its directory of reference servers.

• Courses: the free short courses at DeepLearning.AI on retrieval, agents, MCP and evaluation, and the Hugging Face courses on agents and open models.

• YouTube: Andrej Karpathy’s channel, especially Intro to Large Language Models, and the AI Engineer World’s Fair talks on the AI Engineer channel.

• Podcasts: Latent Space, the podcast of the AI engineering community, and Practical AI.

• Writing: Simon Willison’s weblog, on tools, agents and prompt injection as they evolve.