Building AI Applications: Two-Week Online Course

A two-week online course on the architecture of AI applications: retrieval, tool calling, structured data, MCP servers and multi-agent workflows, each a resource declared in a YAML manifest. Students work on the resources that ship with every Smarter installation: nineteen built-in LLMClients, from the bare simple assistant to research_analyst, which searches the web through an Exa MCP server, reads pages and PDFs, and checks its citations with guardrails; twenty-five MCP clients for services such as GitHub, Notion and Stripe; Vectorstores on Qdrant and Pinecone; and an Orchestrator with six coordination strategies. There are six 30-minute recorded lectures and a 90-minute live session each week. Week 1 covers knowledge (retrieval and structured data), and week 2 covers action (tools, MCP and orchestration). See Course Formats and Grading for the format, the grading scale and the course policies, and Building AI Applications for an overview of the subject.

At a Glance

Students

Product managers, analysts, designers, consultants and developers.

Deployment

A shared Smarter instance run by the instructor, with a Vectorstore backend, a sample database behind a Connection, and one or two MCP clients set up in advance.

Workload

3 hours of instruction and 4 to 6 hours of assignments each week.

Cost

Expect less than $6 USD of model charges per student, capped by a per-student Budget.

Learning Objectives

By the end of the course, students can:

  1. Describe the parts of an AI application and how they fit together.

  2. Build retrieval-augmented assistants that answer from documents, with citations.

  3. Let a model query structured data safely, and check its answers.

  4. Connect an assistant to external systems with plugins and MCP servers.

  5. Choose between a single assistant and an orchestrated workflow on evidence.

Technical Prerequisites

  • Required: a computer with a current web browser and a reliable connection for video calls.

  • Required: reading and editing YAML (Learn YAML in Y minutes) and reading JSON.

  • Required: a general idea of what a database table and a web API are.

  • Helpful: experience writing system prompts, from the Prompt Engineering course or elsewhere, a GitHub account, and basic SQL for reading queries.

  • Not needed: programming.

Schedule

The files that the lectures refer to are listed in full under Smarter in the Source Code.

Week

Item

Content

1

Lecture 1 (30 min)

Anatomy of an AI application. Model, instructions, knowledge, tools, guardrails and orchestration as Smarter resources. A tour from simple to research_analyst.

1

Lecture 2 (30 min)

Retrieval. Embeddings, similarity search, chunking and citations, without mathematics. Building a Vectorstore.

1

Lecture 3 (30 min)

Structured data. Two safe patterns, both shipped with Smarter: a SQL Plugin runs a fixed, parameterized query that the model can only fill in (stackademy_sql), and a SkillPlugin teaches the model a schema so that it writes SQL for a person to review (the data_analyst LLMClient). A Connection with a read-only database user, its password in a Secret, and how wrong answers still happen.

1

Live session 1 (90 min)

Live build of a teaching assistant over a syllabus. In groups, students find questions that break it, and diagnose why. Start Assignment 1.

2

Lecture 4 (30 min)

Tools. Function calling, API and Websearch plugins. Plugin selection by search terms decides which tools Smarter offers, and their descriptions steer which ones the model calls (Natural Language Processing (NLP)).

2

Lecture 5 (30 min)

MCP. The Model Context Protocol, its servers and tools, and MCP clients in Smarter.

2

Lecture 6 (30 min)

Orchestration and production. Workflows versus agents, the Orchestrator, guardrails, budgets and evaluation.

2

Live session 2 (90 min)

Capstone presentations (about 5 minutes each, with 2 minutes of questions), then a discussion: which architectures appeared, and why.

Assignment 1 is due at the end of week 1, Assignment 2 in the middle of week 2, and Assignment 3 and the capstone at the start of Live session 2.

Smarter in the Source Code

Every lecture, lab and assignment in this course works with the manifests and files below. The manifests ship with every Smarter installation, so students can open them in the web console, or read them on the main branch of github.com/smarter-sh/smarter. No programming is needed to read them. The Python files are listed for instructors and curious students. Python paths are under smarter/smarter/.

Applications to Take Apart

Application

What it shows

Where to look

simple

A bare model: a one-line system prompt and nothing else. The baseline for every comparison.

llmclient-simple.yaml

customer_support

A complete application: a persona, a skill for on-brand replies, a plugin that reads web pages, and four input guardrails for abuse, self-harm, prompt injection and jailbreaks.

llmclient-customer-support.yaml

research_analyst

Web search through the Exa MCP server, page and PDF readers, an image search, and output guardrails that catch fabricated citations and regulated advice.

llmclient-research-analyst.yaml

data_analyst

A SkillPlugin that teaches the model a data warehouse’s schema and house rules, so that it writes SQL for a person to review, with DeepWiki through MCP for SQL documentation.

llmclient-data-analyst.yaml, skill-sql-analyst.yaml

Stackademy

A fictional university’s course catalog, served two ways: by a SQL Plugin over a database, and by an API Plugin over a REST API, each with its own LLMClient.

plugin/data/stackademy/

Knowledge and Data

Topic

What Smarter does

Where to look

Vector databases

Qdrant, self-hosted on Smarter’s own Kubernetes cluster or in Qdrant Cloud, and Pinecone, each declared in a manifest. Smarter creates the database, extracts text from PDFs, Markdown, CSV, HTML and other formats, splits it into chunks, embeds and loads it, and takes snapshots on a schedule. Smarter Vectorstore.

vectorstore/data/vectorstores/, vectorstore/backends/, vectorstore/extract.py

Retrieval strategies

A Vectorsearch names a Vectorstore and how to query it: similarity, similarity with a score threshold, or maximal marginal relevance, how many results, and metadata filters. Smarter Vectorsearch.

vectorstore/service.py (search())

A parameterized query

A SQL Plugin runs a fixed query written by a person. The model can only supply its parameters, with types, descriptions and allowed values, so it never writes SQL that runs. SQL Plugin.

stackademy-plugin-sql.yaml, plugin/plugin/sql.py

Connections and credentials

A Connection names the database or API, and its password is the name of a Secret, so neither the model nor the plugin’s author ever sees it. Smarter Connection.

connection/data/sample-connections/

Action and Orchestration

Topic

What Smarter does

Where to look

Plugin selection

Smarter offers a plugin’s tool to the model only when the prompt matches one of its search terms, exactly, with a typo or as a synonym. The model then decides, from the tool’s description, whether to call it. Natural Language Processing (NLP).

plugin/plugin/base.py (selected())

Twenty sample plugins

Static, Skill, Websearch and ImageSearch plugins, ready to attach to an LLMClient.

plugin/data/sample-plugins/

MCP clients

Twenty-five ready-made MCP clients, for GitHub, Notion, Linear, Stripe, Hugging Face, Microsoft Learn and others. Each states its transport, its credentials (a Secret) and the tools the model may use, such as only the read-only get_*, list_* and search_* tools of GitHub’s server. Smarter MCP Client.

mcpclient/data/mcpclients/, github.yaml

Safe connections to MCP servers

Only the HTTP and SSE transports are allowed, never stdio, which would run a command on the server, and every endpoint must resolve to a public address.

mcpclient/connection.py

Orchestration

An Orchestrator coordinates several LLMClients with one of six strategies (sequential, parallel, supervisor, router, voting and debate), gives each a role (planner, executor, critic, router, summarizer or tool caller), and records every run and step. Smarter Orchestrator.

orchestrator/enum.py, orchestrator/models/

Production

Topic

What Smarter does

Where to look

Guardrails

Eighteen built-in guardrails, run as a pipeline on the prompt before the model sees it and on the reply before the user does. Smarter Guardrail.

guardrail/data/guardrails/, guardrail/services/pipeline.py

Cost and budgets

Every request is charged, with its tokens, to the user, the application and the account, and Budgets cap the spending. Cost Accounting.

account/models/charge.py, budgets/

The record of every conversation

Each prompt, answer, tool call and plugin selection is stored, and shown in the workbench’s Console. Smarter Journal.

prompt/models/

Graded Assignments

Assignment 1: Retrieval Assistant with Evaluation

Students build an assistant over at least five documents of their choice, test it with fifteen questions (twelve answerable, three not), and adjust the system prompt once to improve the results.

Turn in: the manifests, the documents or links, and before-and-after results with citations.

Criterion

Pts

Exemplary

Developing

Retrieval and citations

6

Answers draw on, and name, the right documents.

Answers ignore retrieval, or invent sources.

Unanswerable questions

4

The assistant admits when it does not know.

The assistant invents answers.

Improvement

6

One targeted change, with its measured effect.

Changes without before-and-after results.

Reporting

4

Honest and complete.

Incomplete or inflated.

Assignment 2: Data Assistant

Students build an assistant over the sample database, ask eight business questions, and check each answer, and the SQL behind it, against the true figure.

Turn in: the manifests, the questions, answers, the SQL behind each one, and a short failure analysis.

Criterion

Pts

Exemplary

Developing

Configuration

4

Read-only connection; credentials kept in a Secret.

Works, but with unnecessary access.

Accuracy check

6

Every answer checked against the truth.

Answers accepted without checking.

Failure analysis

6

Wrong answers explained from the SQL, with a fix tried.

Wrong answers noted, not explained.

Risk note

4

Names what could go wrong if real managers relied on it, and a mitigation.

No discussion of risk.

Assignment 3: A Tool-Using Assistant

Students give an assistant two tools: one plugin (API or Websearch) and one MCP client provided by the instructor. They show conversations in which the model uses each tool, one in which it correctly uses neither, and explain what decided the model’s choice.

Turn in: the manifests, three Console screenshots, and a one-page explanation.

Criterion

Pts

Exemplary

Developing

Both tools work

6

Each tool is called and its result used.

One tool never called.

Restraint

4

A conversation where no tool is needed and none is called.

Tools called for every message.

Explanation

6

Links the model’s choices to the tool descriptions and system prompt.

Describes what happened without explaining why.

Safety

4

Notes what each tool could do if misused, and how it is limited.

No discussion of safety.

Capstone Projects

Students choose one, and present it in Live session 2.

Capstone A: A Knowledge and Action Assistant

Learning objective. Combine retrieval, structured data or tools into one coherent assistant for a defined user, and show that it is accurate and safe.

Students design an assistant for a real or realistic user, such as a new employee onboarding assistant, a club treasurer’s helper, or a course advisor. It must use retrieval and at least one of a SQL Plugin, an API Plugin or an MCP client, and at least one guardrail.

What students turn in:

  • A Git repository with every manifest, and a README.md.

  • An architecture diagram and a two-page design note that justifies each component.

  • A test set of twenty-five questions with results.

  • A five-minute presentation with a live demonstration.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Architecture

25

Each component justified by the user’s needs; the diagram matches the build.

Components added because they were available.

Accuracy

25

Grounded, cited and correct answers; honest about gaps.

Correct on easy questions only.

Safety

15

Guardrails and access limits match the risks.

A guardrail unrelated to the risks.

Testing

20

Twenty-five varied questions, failures reported and explained.

Few or easy questions.

Presentation

15

Clear, on time, shows design decisions.

Shows chat only.

Capstone B: Architecture Comparison Report

Learning objective. Compare alternative architectures for the same job, and recommend one on evidence of cost, latency and quality.

Students choose a job, such as answering questions about a product catalog. They build it three ways: a single LLMClient with documents in a Static Plugin, a retrieval-based LLMClient with a Vectorstore, and an Orchestrator of specialized LLMClients. They evaluate all three on fifteen inputs.

What students turn in:

  • A Git repository with all three designs.

  • The inputs, rubric and a table of scores, tokens, cost and response time.

  • A three-page report with charts and a recommendation.

  • A five-minute presentation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Three working designs

25

All three complete every input.

One or more incomplete.

Fair evaluation

20

Same inputs, model and rubric for all three.

Inconsistent conditions.

Measurements

20

From Smarter’s records; repeated where results vary.

Estimated or partial.

Analysis and recommendation

25

Explains the trade-offs and when each design is right.

A winner without reasons.

Presentation

10

Clear and on time.

Overruns or unclear.

Further Study

  • Essays: Anthropic’s Building effective agents, the clearest short guide to when a workflow, a single agent or several agents is the right design. Eugene Yan’s writing on patterns for building LLM systems and products.

  • Standards: the Model Context Protocol specification and its directory of reference servers.

  • Courses: the free short courses at DeepLearning.AI on retrieval, agents, MCP and evaluation, and the Hugging Face courses on agents and open models.

  • YouTube: Andrej Karpathy’s channel, especially Intro to Large Language Models, and the AI Engineer World’s Fair talks on the AI Engineer channel.

  • Podcasts: Latent Space, the podcast of the AI engineering community, and Practical AI.

  • Writing: Simon Willison’s weblog, on tools, agents and prompt injection as they evolve.