Building AI Applications: Classroom Bootcamp

An eight-hour classroom bootcamp on the architecture of AI applications: retrieval, tool calling, structured data, MCP servers and multi-agent workflows. Every component is a resource declared in a YAML manifest, so the bootcamp covers the whole landscape without teaching a programming language. Students work on the resources that ship with every Smarter installation: nineteen built-in LLMClients, from the bare simple assistant to research_analyst, which searches the web through an Exa MCP server, reads pages and PDFs, and checks its citations with guardrails; twenty-five MCP clients for services such as GitHub, Notion and Stripe; Vectorstores on Qdrant and Pinecone; and an Orchestrator with six coordination strategies. See Course Formats and Grading for the standard bootcamp day, the grading scale and the course policies, and Building AI Applications for an overview of the subject.

At a Glance

Students

Product managers, analysts, designers, consultants and developers who want to understand how AI applications are put together.

Deployment

A shared Smarter instance run by the instructor, with a Vectorstore backend, a sample database behind a Connection, and one or two MCP clients set up in advance.

Cost

Expect less than $3 USD of model charges per student, plus any vector database charges, capped by a per-student Budget.

Learning Objectives

By the end of the bootcamp, students can:

  1. Describe the parts of an AI application: model, instructions, knowledge, tools, guardrails and orchestration.

  2. Build a retrieval-augmented assistant that answers from documents, with citations.

  3. Let a model answer questions about structured data, and explain the risks of doing so.

  4. Connect an assistant to external systems through plugins and MCP servers.

  5. Compare a single assistant with an orchestrated workflow on cost, latency and quality.

Technical Prerequisites

  • Required: a laptop with a current web browser, and experience of using a chat assistant.

  • Required: reading and editing YAML (Learn YAML in Y minutes) and reading JSON.

  • Required: basic understanding of what a database table and a web API are. No SQL needed.

  • Helpful: the Prompt Engineering course or equivalent experience writing system prompts.

  • Not needed: programming.

Schedule

The files that the lectures refer to are listed in full under Smarter in the Source Code.

Block

Min

Content

1

45

Lecture 1: Anatomy of an AI application. The model, the system prompt, knowledge, tools, guardrails and orchestration, and how each appears as a Smarter resource. A tour of the built-in simple, customer_support and research_analyst LLMClients, from a bare model to a complete application.

2

45

Lab 1: Read an application. In pairs, take the research_analyst manifest apart: list every resource it uses, what each does, and draw how a question flows through it. Produces Assignment 1.

3

45

Lecture 2: Retrieval. Why models make things up. Embeddings and similarity search explained without mathematics, chunking, and citations. Vectorstores in Smarter.

4

45

Lab 2: A teaching assistant. Load a course syllabus and handbook into a Vectorstore, and build an LLMClient that answers questions about them with citations. Produces Assignment 2.

5

45

Lecture 3: Structured data. Two safe patterns, both shipped with Smarter: a SQL Plugin runs a fixed, parameterized query that the model can only fill in (stackademy_sql), and a SkillPlugin teaches the model a schema so that it writes SQL for a person to review (the data_analyst LLMClient). A Connection with a read-only database user, its password in a Secret, and how wrong answers still happen.

6

45

Lecture 4: Tools and MCP. Function calling, API and Websearch plugins, and the Model Context Protocol. MCP clients in Smarter.

7

45

Lab 3: A data assistant. Build an assistant that answers five business questions in plain English over a sample database, and check each answer against the true figure. Produces Assignment 3.

8

45

Lecture 5: Orchestration. Workflows and agents, routing, and when several specialized assistants beat one general one. The Orchestrator resource.

9

45

Lab 4: One assistant or several? Build a two-step workflow (for example research, then write) as one LLMClient and as an Orchestrator, and compare tokens, time and output. Ungraded.

10

45

Lecture 6: Production concerns. Guardrails, budgets, the journal, evaluation and deployment. What changes when real users arrive.

11

30

Capstone kickoff. Students choose a capstone, sketch its architecture diagram and get feedback.

Smarter in the Source Code

Every lecture, lab and assignment in this course works with the manifests and files below. The manifests ship with every Smarter installation, so students can open them in the web console, or read them on the main branch of github.com/smarter-sh/smarter. No programming is needed to read them. The Python files are listed for instructors and curious students. Python paths are under smarter/smarter/.

Applications to Take Apart

Application

What it shows

Where to look

simple

A bare model: a one-line system prompt and nothing else. The baseline for every comparison.

llmclient-simple.yaml

customer_support

A complete application: a persona, a skill for on-brand replies, a plugin that reads web pages, and four input guardrails for abuse, self-harm, prompt injection and jailbreaks.

llmclient-customer-support.yaml

research_analyst

Web search through the Exa MCP server, page and PDF readers, an image search, and output guardrails that catch fabricated citations and regulated advice.

llmclient-research-analyst.yaml

data_analyst

A SkillPlugin that teaches the model a data warehouse’s schema and house rules, so that it writes SQL for a person to review, with DeepWiki through MCP for SQL documentation.

llmclient-data-analyst.yaml, skill-sql-analyst.yaml

Stackademy

A fictional university’s course catalog, served two ways: by a SQL Plugin over a database, and by an API Plugin over a REST API, each with its own LLMClient.

plugin/data/stackademy/

Knowledge and Data

Topic

What Smarter does

Where to look

Vector databases

Qdrant, self-hosted on Smarter’s own Kubernetes cluster or in Qdrant Cloud, and Pinecone, each declared in a manifest. Smarter creates the database, extracts text from PDFs, Markdown, CSV, HTML and other formats, splits it into chunks, embeds and loads it, and takes snapshots on a schedule. Smarter Vectorstore.

vectorstore/data/vectorstores/, vectorstore/backends/, vectorstore/extract.py

Retrieval strategies

A Vectorsearch names a Vectorstore and how to query it: similarity, similarity with a score threshold, or maximal marginal relevance, how many results, and metadata filters. Smarter Vectorsearch.

vectorstore/service.py (search())

A parameterized query

A SQL Plugin runs a fixed query written by a person. The model can only supply its parameters, with types, descriptions and allowed values, so it never writes SQL that runs. SQL Plugin.

stackademy-plugin-sql.yaml, plugin/plugin/sql.py

Connections and credentials

A Connection names the database or API, and its password is the name of a Secret, so neither the model nor the plugin’s author ever sees it. Smarter Connection.

connection/data/sample-connections/

Action and Orchestration

Topic

What Smarter does

Where to look

Plugin selection

Smarter offers a plugin’s tool to the model only when the prompt matches one of its search terms, exactly, with a typo or as a synonym. The model then decides, from the tool’s description, whether to call it. Natural Language Processing (NLP).

plugin/plugin/base.py (selected())

Twenty sample plugins

Static, Skill, Websearch and ImageSearch plugins, ready to attach to an LLMClient.

plugin/data/sample-plugins/

MCP clients

Twenty-five ready-made MCP clients, for GitHub, Notion, Linear, Stripe, Hugging Face, Microsoft Learn and others. Each states its transport, its credentials (a Secret) and the tools the model may use, such as only the read-only get_*, list_* and search_* tools of GitHub’s server. Smarter MCP Client.

mcpclient/data/mcpclients/, github.yaml

Safe connections to MCP servers

Only the HTTP and SSE transports are allowed, never stdio, which would run a command on the server, and every endpoint must resolve to a public address.

mcpclient/connection.py

Orchestration

An Orchestrator coordinates several LLMClients with one of six strategies (sequential, parallel, supervisor, router, voting and debate), gives each a role (planner, executor, critic, router, summarizer or tool caller), and records every run and step. Smarter Orchestrator.

orchestrator/enum.py, orchestrator/models/

Production

Topic

What Smarter does

Where to look

Guardrails

Eighteen built-in guardrails, run as a pipeline on the prompt before the model sees it and on the reply before the user does. Smarter Guardrail.

guardrail/data/guardrails/, guardrail/services/pipeline.py

Cost and budgets

Every request is charged, with its tokens, to the user, the application and the account, and Budgets cap the spending. Cost Accounting.

account/models/charge.py, budgets/

The record of every conversation

Each prompt, answer, tool call and plugin selection is stored, and shown in the workbench’s Console. Smarter Journal.

prompt/models/

Graded Assignments

Assignment 1: Architecture Teardown

Students document how the built-in research_analyst LLMClient works: every resource it references, what each contributes, and the path of one question through the system, confirmed by the Console’s view of the tool calls.

Turn in: a one-page diagram and a half-page explanation.

Criterion

Pts

Exemplary

Developing

Completeness

6

Every resource in the manifest is identified, with its role.

Some resources missing, or roles confused.

Data flow

6

Diagram shows the order of model calls, tool calls and results.

A list of boxes with no flow.

Evidence

4

The flow is confirmed against real tool calls in the Console.

Inferred from the manifest only.

Clarity

4

A newcomer could follow the diagram.

Unlabeled or cluttered.

Assignment 2: Retrieval Assistant

Students build an assistant over a set of documents, and test it with ten questions that the documents answer and three that they do not.

Turn in: the manifests, the list of documents, and the test results with citations.

Criterion

Pts

Exemplary

Developing

Working retrieval

6

Answers draw on the right passages.

Retrieval runs, but answers ignore it.

Citations

5

Each answer names its source document.

Citations missing or invented.

Unanswerable questions

5

The assistant says that it does not know.

The assistant invents answers.

Testing

4

All thirteen results reported honestly.

Results incomplete.

Assignment 3: Data Assistant

Students build an assistant over the sample database, ask it five business questions in plain English, and check each answer against the true figure, which the instructor provides.

Turn in: the manifests, the questions, the assistant’s answers and the tool calls from the Console, and the check against the true figures.

Criterion

Pts

Exemplary

Developing

Configuration

5

Plugin and connection configured with read-only access.

Works, but with unnecessary access.

Accuracy check

6

Every answer checked against the true figure.

Answers accepted without checking.

Failure analysis

5

Wrong answers are explained from the SQL and the tool call’s parameters.

Wrong answers noted but not explained.

Improvement

4

System prompt or plugin description changed to fix one failure.

No attempt to fix failures.

Capstone Projects

Students choose one. Capstones are due one week after the bootcamp.

Capstone A: A Knowledge Assistant for an Organization

Learning objective. Design and build a grounded assistant that combines retrieval with at least one tool, and show with evidence that it is accurate.

Students pick an organization (real, with permission, or fictional with realistic documents) and build an assistant that answers questions from its documents and takes one action, such as searching the web for current information or calling a public API.

What students turn in:

  • The manifests, and the documents or links to them.

  • An architecture diagram and a one-page design note.

  • A test set of twenty questions with results, including questions that the documents do not answer.

  • A five-minute recorded or live demonstration.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Architecture

20

Each component is chosen for a stated reason, and the diagram matches the build.

Components added without reason.

Grounding

25

Accurate, cited answers, and honest “I don’t know” answers.

Invents answers when documents are silent.

Tool use

15

The tool is called when needed, and its result is used.

Tool configured but unused.

Testing

25

Twenty varied questions, with failures reported and explained.

Few or easy questions.

Demonstration

15

Shows a retrieval, a tool call and a refusal, clearly and on time.

Shows only easy chat.

Capstone B: Single Assistant versus Workflow

Learning objective. Decide on evidence whether a job needs a multi-step workflow or a single assistant.

Students choose a job with clear steps, such as screening job applications against criteria and drafting replies, or researching a topic and writing a briefing. They build it as one LLMClient and as an Orchestrator of two or three specialized ones, and compare the two on ten inputs.

What students turn in:

  • The manifests for both designs.

  • The ten inputs, a scoring rubric, and a table of quality scores, tokens, cost and response time.

  • A two-page recommendation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Both designs work

25

Both produce complete outputs on all ten inputs.

One design is incomplete.

Fair comparison

20

Same inputs, same model, comparable effort on each design.

One design clearly neglected.

Measurements

20

Quality scored with a rubric, cost and time from Smarter’s records.

Impressions instead of measurements.

Analysis

20

Explains where each design wins and why.

Names a winner without explanation.

Recommendation

15

A clear, conditional recommendation (“use the workflow when…”).

No recommendation.

Further Study

  • Essays: Anthropic’s Building effective agents, the clearest short guide to when a workflow, a single agent or several agents is the right design. Eugene Yan’s writing on patterns for building LLM systems and products.

  • Standards: the Model Context Protocol specification and its directory of reference servers.

  • Courses: the free short courses at DeepLearning.AI on retrieval, agents, MCP and evaluation, and the Hugging Face courses on agents and open models.

  • YouTube: Andrej Karpathy’s channel, especially Intro to Large Language Models, and the AI Engineer World’s Fair talks on the AI Engineer channel.

  • Podcasts: Latent Space, the podcast of the AI engineering community, and Practical AI.

  • Writing: Simon Willison’s weblog, on tools, agents and prompt injection as they evolve.