Extension Development: Four-Week Online Course

A four-week online course on building tools for LLMs as Smarter Extensions: Python packages that a platform administrator installs at run time, and that every LLMClient on the platform can then use. Students learn from the official weather Extension, which implements the get_weather tool of OpenAI’s function calling guide for real, from the open source framework that runs it, apps/llmtool/extension/, and from the signed extensions-registry that publishes it. There are twelve 30-minute recorded lectures and a 90-minute live session each week. Week 1 covers tools and the framework, week 2 writing tools, week 3 settings, the lifecycle and testing, and week 4 loading, publishing and trust. See Course Formats and Grading for the format, the grading scale and the course policies, and Extension Development for an overview of the subject.

At a Glance

Students	Python developers who want to give LLMs new abilities.
Deployment	Each student runs Smarter from source (make init), as its superuser. Extensions are installed by a superuser, so a shared instance doesn’t suit this course. Week 1 includes time for setup.
Models	Any OpenAI-compatible model that supports tool calling, through the instructor’s provider key.
Workload	3 hours of instruction and 4 to 6 hours of assignments each week.
Cost	Under $10 per student in model charges, capped with a Budget. The weather Extension needs a Google Maps Geocoding API key, whose free monthly allowance covers the course. A GitHub account’s free Actions minutes cover the release workflow.

Learning Objectives

By the end of the course, students can:

1. Explain how Smarter offers a tool to an LLM, runs its call and returns the result, and choose between a Plugin, an Extension and an MCP server for a given need.

2. Write an Extension’s pyproject.toml and tools whose Pydantic arguments models are their definitions.

3. Design tools’ results and error messages for an LLM to read, and treat its arguments as untrusted input.

4. Use settings, secrets, waffle switches, SAM resources and lifecycle hooks to manage what an Extension needs on a platform.

5. Test tools without the platform, the database or an LLM, and run the tests in CI.

6. Package, version, release and publish an Extension, and evaluate the risks of installing someone else’s.

Technical Prerequisites

• Required: Python: classes and inheritance, type hints, exceptions, packages and relative imports.

• Required: Git, GitHub and pull requests.

• Required by the end of week 1: Docker, and Smarter installed from source with a superuser login. See the Smarter Instructor Guide.

• Helpful: Pydantic models, Python’s unittest and unittest.mock, GitHub Actions, and having called an LLM API.

• Not needed: Django or JavaScript. The framework hides both.

Schedule

The files that the lectures refer to are listed in full under Smarter in the Source Code.

Week	Item	Content
1	Lecture 1	How LLMs call tools. Tool definitions, tool calls and results, and the loop that runs them in openai_compatible_chat_provider.py: handle_extensions() and process_tool_call(). How LLM Tool Calling Works.
1	Lecture 2	Plugins, Extensions and MCP. Configuration or code, account or platform, selected by search terms or offered on every prompt: Extensibility. Why Extensions replaced three functions hardcoded in the chat provider, read from the body of commit fda234a37.
1	Lecture 3	A tour of the weather Extension. Its pyproject.toml, docstring, models, services and tool, and how its result changed in 0.2.0 (commit 51ec052a5).
1	Live session 1	Setup triage. Enabling the weather Extension in the web console, and tracing a tool call live. Start Assignment 1.
2	Lecture 4	The pyproject.toml. [project] and [tool.smarter], validated by manifest.py: the permanent slug, requires-smarter, dependencies that are checked but never installed, and compatibility_problems(). The Extension’s pyproject.toml.
2	Lecture 5	The tool protocol. SmarterExtensionTool and ExtensionContext in protocol.py; the docstring as the description and the arguments model as the JSON Schema; problems(); and what validate_extension prints. ADR 031.
2	Lecture 6	Results and errors for an LLM. Results that need no guessing, ExtensionToolError messages written for the LLM, and what toolkit.py does with invalid arguments, exceptions, timeouts and non-JSON. The LLM’s arguments as untrusted input: the calculator’s evaluator.py.
2	Live session 2	Review of Assignment 1. Designing an arguments model together, and watching the LLM use it. Start Assignment 2.
3	Lecture 7	Settings and secrets. Typed settings and SecretStr in settings.py, secrets stored as Smarter Secrets, and the naming convention that lets add_builtin_extensions configure them.
3	Lecture 8	The lifecycle. Waffle switches and SAM manifests that an Extension creates and its uninstall removes (resources.py), the eight hooks (lifecycle.py), and the installer’s order of work. The Extension Lifecycle.
3	Lecture 9	Testing. call_tool() and tool_context() in testing.py; mocking services; manage.py test_extension; the framework’s own fixtures (echo, broken, incompatible); and CI that tests against a checkout of Smarter (test.yml in the weather repository).
3	Live session 3	Review of Assignment 2. A test-writing workshop on the planted bugs of an instructor’s Extension. Capstone proposals discussed. Start Assignment 3.
4	Lecture 10	Loading code at run time. loader.py: one archive per version in the database, atomic extraction, a module namespace per version and sha256, and why that needs relative imports and no work at import time.
4	Lecture 11	Packaging and publishing. The archive’s limits and reproducible builds (archive.py), semantic-release (commit 45f3690c2), and the registry: entries, validation, the signed index and trust badges. Publishing an Extension.
4	Lecture 12	Trust and operations. Extensions run with the platform’s own access, unsandboxed. The registry’s signature and sha256 checks, and what they do and don’t protect. Updates, rollbacks and errors on the web console’s Extensions pages (Managing Extensions), and the signals that log every tool call.
4	Live session 4	Capstone presentations (about 6 minutes each, with 2 minutes of questions), and a course retrospective.

Assignments 1 to 3 are due before the next live session. Assignment 4 is due in the middle of week 4. The capstone proposal is due before Live session 3, and the capstone at the start of Live session 4.

Smarter in the Source Code

Every lecture, lab and assignment in this course rests on the files and commits below. Every reference is to the main branch of github.com/smarter-sh/smarter, unless another repository is named, so that students can open each one during class. Python paths are under smarter/smarter/.

Tools and Extensibility

Topic	What Smarter does	Where to look
The tool call loop	Smarter offers the LLM the tools that its LLMClient lists, the LLM asks to call one, Smarter runs it and returns the result, and the LLM answers from it. Extension tools are loaded by handle_extensions() beside the MCP clients’ tools, and dispatched by process_tool_call(). How LLM Tool Calling Works.	openai_compatible_chat_provider.py
Plugins and Extensions	Two ways to add a tool, for two kinds of people. A Plugin is a SAM manifest that configures one of six built-in tool patterns; an Extension is a Python package that a platform superuser installs at run time. Extensibility compares them line by line.	apps/llmtool/plugin/, apps/llmtool/extension/
The framework’s map	The package’s module docstring lists each module and what it is for: the manifest, the archive, the protocol, settings, the loader, lifecycle, the installer, the toolkit and the registry.	extension/__init__.py
The models	Extension, ExtensionVersion (each version’s archive is stored in the database), ExtensionSetting and ExtensionResource. An Extension is available to every account, or to listed ones.	models/extension.py
Listing an Extension	An LLMClient lists Extensions by slug in spec.extensions. The built-in weather LLMClient’s system prompt names the tool exactly as the LLM sees it, ext_weather_get_weather, and a test keeps prompts and manifests from drifting apart.	llmclient-weather.yaml

The Official Extensions

Topic	What it shows	Where to look
The weather Extension	The developer guide’s worked example: the get_weather tool from the OpenAI function calling guide, implemented for real, with a secret setting, a setting with a default, two third-party services that may fail, and a result modeled with Pydantic.	smarter-extension-weather: pyproject.toml, weather/tools.py, weather/models.py, weather/services.py
The documentation	The package’s module docstring, in reStructuredText, is the Extension’s documentation, written for the administrator who decides whether to install it. The platform renders it with raw HTML and file inclusion disabled.	weather/__init__.py, extension/docs.py
Untrusted input	The calculator never calls eval(). It parses the LLM’s expression with ast and evaluates only the nodes it allows, with a cap on exponents so that 9**9**9 cannot exhaust the platform’s memory.	calculator/evaluator.py, calculator/tools.py
The smallest Extension	One tool and no settings, for date arithmetic: a template to start from.	smarter-extension-date-calculator
The test fixtures	The framework’s own test Extensions: echo, whose tools succeed, fail, refuse, time out and return non-JSON, with a secret setting, a waffle switch, a SAM manifest and lifecycle hooks; broken, which does not import; and incompatible.	tests/data/extensions/

Writing an Extension

Topic	What Smarter does	Where to look
The pyproject.toml	A standard Python project file, not a SAM manifest. ExtensionManifest validates its [project] and [tool.smarter] tables. Dependencies are checked against the platform’s installed libraries and never installed, and compatibility_problems() explains every unmet Python, Smarter or library requirement. The Extension’s pyproject.toml.	extension/manifest.py
The tool protocol	A tool is a class with a name, a Pydantic arguments_model and a call() method. Its docstring is its description for the LLM, and its arguments model’s JSON Schema is its parameters, so the definition offered to the LLM can never disagree with the validation. problems() explains why a class is not a valid tool. ADR-031: Pydantic Models Over Raw JSON.	extension/protocol.py
Running a tool	ExtensionToolkit names each tool ext_<slug>_<tool>, validates the LLM’s arguments, runs the call in a thread with the manifest’s timeout, and checks that the result is JSON. Invalid arguments, exceptions, timeouts and ExtensionToolError are returned to the LLM as {"error": ...} rather than failing the prompt, and a broken Extension is skipped and reported.	extension/toolkit.py
Settings and secrets	Settings are declared in pyproject.toml and configured by a superuser. Values are coerced to their declared types, and a secret setting is a Smarter Secret, encrypted at rest, that reaches the tool as a SecretStr, read at call time and never cached.	extension/settings.py
Lifecycle hooks	Eight hooks, from pre_install to post_uninstall, all doing nothing by default. A pre_* hook that raises cancels the action; a post_* hook that raises leaves the Extension installed but disabled. The Extension Lifecycle.	extension/lifecycle.py
Switches and resources	Waffle switches declared in pyproject.toml are created as ext_<slug>_<name>, and SAM manifests in the package’s manifests/ folder are applied through the real brokers. Each is recorded, so that uninstalling deletes exactly what the Extension created.	extension/resources.py
Testing helpers	tool_context() and call_tool() let an Extension’s tests call its tools as the platform does, without installing the Extension, the database or an LLM, while letting unexpected exceptions reach the test. Testing an Extension.	extension/testing.py, tests/test_tools.py (weather)
Management commands	validate_extension prints each tool’s definition exactly as the LLM will see it; package_extension, test_extension, install_extension, uninstall_extension and add_builtin_extensions cover the rest of a developer’s loop.	management/commands/

Installing, Loading and Publishing

Topic	What Smarter does	Where to look
The archive	A zip file, validated without importing it: at most 10 MB, 50 MB expanded and 2,000 files, with no links or paths that leave its directory. build_archive() fixes every timestamp, so the same source always builds the same sha256.	extension/archive.py
The installer	The one place where an Extension’s state changes: install, update, roll back, configure, enable, disable and uninstall. The web console, the management commands and the registry client all go through it.	extension/installer.py
Loading code at run time	Each process extracts a version’s archive once, atomically by rename, and imports it under a name unique to its version and sha256, so an update loads beside the old version instead of reloading modules. This is why an Extension’s modules use relative imports.	extension/loader.py
The registry	A public GitHub repository whose CI validates each submission, builds an index, and signs it with an Ed25519 key. Platforms verify the signature on every read, through the hardened HTTP client, and check every archive against the signed sha256. Publishing an Extension.	extension/registry.py, extensions-registry: scripts/validate.py, scripts/build_index.py, publishers.yaml
Releasing a version	semantic-release on each push to main: a fix releases a patch and a feat a minor version. The workflow builds the archive and prints the archive_url and sha256 that the registry entry needs.	release.yml, test.yml, scripts/package.py (weather)
The web console	Superusers browse the registry, install, upload, configure, enable, update, roll back and uninstall on two React pages, whose api is typed by Pydantic models exported as a JSON Schema. Managing Extensions.	extension/views.py, extension/contract.py, smarter-extension-console/
Observability	Eight signals, from extension_installed to extension_tool_failed, feed Smarter’s logs, progress messages and history. The log_extension waffle switch turns on the framework’s logging.	apps/llmtool/signals.py

Extensions in the History

Extensions arrived in one large commit, with two follow-ups, and the official Extensions have histories of their own.

Commit	Why it is worth studying	Where to look
Extensions arrive	Adding a Python tool used to mean changing Smarter itself: three functions hardcoded in the chat provider, a choices enum, and a release. The commit body explains every module of the new framework, the migration from spec.functions to spec.extensions, and how it was verified against the live registry.	commit fda234a37
Drift after a big change	Built-in system prompts still named tools that the LLM was no longer offered. The fix renames them to ext_<slug>_<tool> and adds a test that every tool a prompt names belongs to an Extension that its manifest lists.	commit 1aa73c2a0
From admin forms to the console	Extension management moves from the Django admin to two React pages, with a typed contract, and the admin becomes read-only.	commit e353a68a7
A result the LLM can read	get_weather 0.2.0 returns current conditions, starts the forecast at the current hour, gives times in the location’s timezone, and names the unit of every measurement.	commit 51ec052a5 (weather)
Automating releases	Hand-made tags replaced by semantic-release, so that a commit’s type decides the version.	commit 45f3690c2 (weather)
A registry release	The commit that publishes two Extensions’ 0.2.0: one YAML file per version.	commit a42a93ae9 (extensions-registry)

Graded Assignments

Assignment 1: A Tool Call, Traced

Students follow two weather questions, one that succeeds and one that fails, from the prompt to the answer, naming the code that handles each step, and compare the tool definition that validate_extension prints with the source that produced it.

Turn in: a two-page Markdown report with the conversation, the tool calls, their arguments and results, and the code that handled each step.

Criterion	Pts	Exemplary	Developing
Evidence	6	Real tool calls, arguments and results, from the chat and the logs.	Paraphrased, or from the docs.
Code located	6	Each step tied to the file and method that handles it.	Steps named, not located.
Definition explained	5	Each part of the definition traced to the docstring, model or slug.	Pasted without comment.
Failure path	3	Explains how the error reached the LLM, and what it did with it.	Only the success.

Assignment 2: Two Tools

Students write an Extension with two related tools that do something an LLM can’t do reliably on its own, with described Pydantic arguments and result models, and show an LLMClient choosing between them correctly.

Turn in: a Git repository with the Extension, the output of validate_extension, and a conversation that uses both tools.

Criterion	Pts	Exemplary	Developing
Models	6	Arguments and results typed, constrained and described.	Bare fields and hand-built dicts.
Tool choice	5	Descriptions distinct enough that the LLM picks the right tool.	The LLM confuses the two.
Untrusted input	5	Every argument checked before use; nothing evaluated or executed.	Trusts the LLM.
Valid Extension	4	validate_extension passes; relative imports; nothing done at import time.	Validates only after help.

Assignment 3: Settings, a Hook and Tests

Students add a secret setting and a waffle switch to their Extension, and a lifecycle hook that checks something real, such as on_enable refusing to enable without a working API key. They test the tools and the hook with every service mocked.

Turn in: the updated repository, its test_extension output, and screenshots of the settings form and of the hook refusing to enable.

Criterion	Pts	Exemplary	Developing
Settings	5	Typed, described, and the secret never logged or returned.	A secret in a log or a result.
Hook	5	A real check, in the right hook, with a clear message.	A hook that does nothing useful.
Tests	6	Success, invalid arguments, tool errors and the hook, with no network.	Only the success.
Verified	4	The hook’s refusal shown on the platform, not only in a test.	Took the test’s word for it.

Assignment 4: A Release Pipeline

Students add CI to their Extension’s repository, modeled on the weather Extension’s test.yml and release.yml: tests on every pull request, and a release on each push to main that builds the archive and attaches it with its sha256. They write the registry entry that a release would need, without submitting it.

Turn in: the repository with two passing workflow runs and a GitHub release, and the draft extensions/<slug>/<version>.yaml.

Criterion	Pts	Exemplary	Developing
Tests in CI	5	Run against a checkout of Smarter, on every pull request.	Run only locally.
Release	6	Versioned from commit types, with the archive and sha256 attached.	A hand-made tag.
Reproducible	5	Two builds of the same commit have the same sha256, shown.	Not checked.
Registry entry	4	Slug, version, URL and sha256 all match the release.	A sha256 copied from elsewhere.

Capstone Projects

Students choose one, propose it before Live session 3, and present it in Live session 4.

Capstone A: A Published Extension

Learning objective. Build, test, release and publish an Extension that gives an LLM an ability it lacks, to the standard of the official Extensions.

Students build an Extension around a real service or computation, with at least two tools, settings including a secret, a SAM resource or waffle switch, and a lifecycle hook. It is tested in CI, released by its workflow, and documented for administrators. They install it from its GitHub release on a clean platform, and show LLMClients using it. A capstone that submits its Extension to the Smarter registry agrees this with the maintainers in an issue on extensions-registry first, and follows Publishing an Extension.

What students turn in:

• The Extension’s repository, with its workflows, releases and CHANGELOG.md.

• A conversation in which an LLMClient uses each tool, including one failure it recovers from.

• A two-page design note on the tools, settings, lifecycle and the trust an administrator must place in it.

• A six-minute presentation, with questions on the code.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
Tool design	25	Arguments and results modeled and described; the LLM calls each tool when it should.	The LLM ignores a tool, or guesses at its result.
Robustness	20	Input validated as untrusted; every failure returns a useful error; nothing runs at import.	Exceptions escape, or the Extension fails to load.
Settings and lifecycle	15	Each used for a real need; uninstall leaves nothing behind.	Declared but unused, or left behind on uninstall.
Tests and CI	20	Mocked, thorough, and passing in CI.	Tests need an API key, or CI is red.
Release and documentation	10	A reproducible release, installed from GitHub; documentation an administrator can act on.	Installed only from a local build.
Presentation	10	Clear, and answers questions on any line.	Can’t explain key parts.

Capstone B: A Security Review of the Extension Framework

Learning objective. Evaluate the trust model of a plugin system that runs other people’s code, and propose improvements that keep it usable.

Students threat-model the path from a developer’s repository to a tool call: the archive, the registry’s CI, the signed index, the download, the installer, the loader, the toolkit and the settings. For each threat they cite the code that addresses it, or show that nothing does, with a demonstration Extension installed only on their own platform. They propose two improvements as Architecture Decision Records, such as sandboxing tool calls or signing archives by publisher.

What students turn in:

• A threat model, as a table of assets, threats, existing controls (with file references) and gaps.

• The demonstration Extension’s repository, and evidence of each demonstration.

• Two draft ADRs in the format of ADR.

• A six-minute presentation.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
Coverage	20	Every step from repository to tool call.	The archive only.
Evidence	25	Each control cited in the code; each gap demonstrated.	Claims without references.
Demonstrations	15	Safe, on the student’s own platform, and convincing.	Hypothetical.
Proposals	25	Two ADRs with alternatives, costs and effect on developers.	“Add sandboxing”, with no design.
Presentation	15	Clear and proportionate.	Alarmist or vague.

Recommended Texts

• Required: Extensibility and Building an Extension, in week 1.

• Required: OpenAI’s function calling guide, in week 1.

• Required: The Extension Lifecycle and Testing an Extension, in week 3.

• Required: Publishing an Extension and Managing Extensions, in week 4.

• Reference: Luciano Ramalho, Fluent Python (2nd edition, O’Reilly, 2022), on classes, type hints and packages.

• Reference: Chip Huyen, AI Engineering (O’Reilly, 2025), on how tools and agents fit into AI applications.

• Reference: Harry Percival and Bob Gregory, Architecture Patterns with Python (O’Reilly, 2020), on keeping services apart from the code that calls them, as the weather Extension’s services.py does.

• In the repository: protocol.py, toolkit.py, loader.py, and the weather Extension.

Further Study

• Official documentation: OpenAI’s function calling guide, whose get_weather example the weather Extension implements; the Pydantic documentation, on models, validation and JSON Schema; and the Python Packaging User Guide, including the pyproject.toml specification.

• Standards: PEP 440 (version identifiers), PEP 508 (dependency specifiers), Semantic Versioning, Conventional Commits, and JSON Schema.

• Other plugin ecosystems: the WordPress Plugin Handbook, the model for Smarter’s run-time install, and the VS Code Extension API, for how another platform describes, packages and publishes extensions.

• Software supply chain: The Update Framework, the design behind signed package indexes, Sigstore and SLSA, to compare with the registry’s Ed25519 signature and sha256 checks.

• Other tool protocols: the Model Context Protocol specification, the third way to give a Smarter LLMClient tools.

• YouTube: the PyCon US channel, for talks on packaging, Pydantic and testing, and the AI Engineer channel, for talks on tool use and agents.

• Podcasts: Talk Python To Me and Python Bytes, which cover packaging, Pydantic and the Python ecosystem regularly, and Latent Space, on building with LLMs.

• Writing: blog.smarter.sh, on how Smarter was built.