Extension Development: Two-Week Online Course

A two-week online course on building tools for LLMs as Smarter Extensions: Python packages that a platform administrator installs at run time, and that every LLMClient on the platform can then use. Students learn from the official weather Extension, which implements the get_weather tool of OpenAI’s function calling guide for real, and from the open source framework that runs it, apps/llmtool/extension/. They trace a tool call through the platform, write and test an Extension of their own, and package and install it. There are six 30-minute recorded lectures and a 90-minute live session each week. See Course Formats and Grading for the format, the grading scale and the course policies, and Extension Development for an overview of the subject.

At a Glance

Students	Python developers who want to give LLMs new abilities.
Deployment	Each student runs Smarter from source (make init), as its superuser. Extensions are installed by a superuser, so a shared instance doesn’t suit this course.
Models	Any OpenAI-compatible model that supports tool calling, through the instructor’s provider key.
Workload	3 hours of instruction and 4 to 6 hours of assignments each week.
Cost	Under $5 per student in model charges, capped with a Budget. The weather Extension needs a Google Maps Geocoding API key, whose free monthly allowance covers the course.

Learning Objectives

By the end of the course, students can:

1. Explain how Smarter offers a tool to an LLM, runs its call and returns the result, and choose between a Plugin, an Extension and an MCP server for a given need.

2. Write an Extension’s pyproject.toml and a tool whose Pydantic arguments model is its definition.

3. Design a tool’s result and error messages for an LLM to read.

4. Test a tool without the platform, the database or an LLM.

5. Package, install, configure and use an Extension, and explain the risks of installing someone else’s.

Technical Prerequisites

• Required: Python: classes and inheritance, type hints, exceptions, packages and relative imports.

• Required: Git and a GitHub account.

• Required by the end of week 1: Docker, and Smarter installed from source with a superuser login. Install it before the course starts: two weeks leave little time for setup. See the Smarter Instructor Guide.

• Helpful: Pydantic models, Python’s unittest and unittest.mock, and having called an LLM API.

• Not needed: Django or JavaScript. The framework hides both.

Schedule

The files that the lectures refer to are listed in full under Smarter in the Source Code.

Week	Item	Content
1	Lecture 1 (30 min)	Tools and extensibility. The tool call loop, and where Smarter runs it (handle_extensions() and process_tool_call() in openai_compatible_chat_provider.py). Plugins, Extensions and MCP servers compared, with Extensibility, and why Extensions replaced three functions hardcoded in the chat provider (commit fda234a37).
1	Lecture 2 (30 min)	Anatomy of an Extension. The weather Extension’s files, its pyproject.toml (validated by manifest.py), its docstring as documentation, and relative imports. The Extension’s pyproject.toml.
1	Lecture 3 (30 min)	The tool protocol, results and errors. SmarterExtensionTool and ExtensionContext in protocol.py; what toolkit.py does with every kind of failure; results that name their units and timezone (commit 51ec052a5); and the LLM’s arguments as untrusted input, in the calculator’s evaluator.py.
1	Live session 1 (90 min)	Enabling the weather Extension in the web console and tracing a tool call live. Then each student starts an Extension of their own, from the date calculator, and runs validate_extension on it. Start Assignments 1 and 2.
2	Lecture 4 (30 min)	Settings, secrets and the lifecycle. Typed settings and SecretStr (settings.py), switches and SAM resources (resources.py), and the eight lifecycle hooks (lifecycle.py). The Extension Lifecycle.
2	Lecture 5 (30 min)	Testing. call_tool() and tool_context() in testing.py, mocking services, and manage.py test_extension. Testing an Extension.
2	Lecture 6 (30 min)	Publishing and trust. The archive (archive.py), loading versions side by side (loader.py), releasing with semantic-release, the signed registry and its trust badges, and why an Extension, which runs with the platform’s own access, should come only from a publisher you trust. Publishing an Extension.
2	Live session 2 (90 min)	Capstone presentations (about 5 minutes each, with 2 minutes of questions).

Assignment 1 is due at the end of week 1, Assignment 2 in the middle of week 2, and Assignment 3 and the capstone at the start of Live session 2.

Smarter in the Source Code

Every lecture, lab and assignment in this course rests on the files and commits below. Every reference is to the main branch of github.com/smarter-sh/smarter, unless another repository is named, so that students can open each one during class. Python paths are under smarter/smarter/.

Tools and Extensibility

Topic	What Smarter does	Where to look
The tool call loop	Smarter offers the LLM the tools that its LLMClient lists, the LLM asks to call one, Smarter runs it and returns the result, and the LLM answers from it. Extension tools are loaded by handle_extensions() beside the MCP clients’ tools, and dispatched by process_tool_call(). How LLM Tool Calling Works.	openai_compatible_chat_provider.py
Plugins and Extensions	Two ways to add a tool, for two kinds of people. A Plugin is a SAM manifest that configures one of six built-in tool patterns; an Extension is a Python package that a platform superuser installs at run time. Extensibility compares them line by line.	apps/llmtool/plugin/, apps/llmtool/extension/
The framework’s map	The package’s module docstring lists each module and what it is for: the manifest, the archive, the protocol, settings, the loader, lifecycle, the installer, the toolkit and the registry.	extension/__init__.py
The models	Extension, ExtensionVersion (each version’s archive is stored in the database), ExtensionSetting and ExtensionResource. An Extension is available to every account, or to listed ones.	models/extension.py
Listing an Extension	An LLMClient lists Extensions by slug in spec.extensions. The built-in weather LLMClient’s system prompt names the tool exactly as the LLM sees it, ext_weather_get_weather, and a test keeps prompts and manifests from drifting apart.	llmclient-weather.yaml

The Official Extensions

Topic	What it shows	Where to look
The weather Extension	The developer guide’s worked example: the get_weather tool from the OpenAI function calling guide, implemented for real, with a secret setting, a setting with a default, two third-party services that may fail, and a result modeled with Pydantic.	smarter-extension-weather: pyproject.toml, weather/tools.py, weather/models.py, weather/services.py
The documentation	The package’s module docstring, in reStructuredText, is the Extension’s documentation, written for the administrator who decides whether to install it. The platform renders it with raw HTML and file inclusion disabled.	weather/__init__.py, extension/docs.py
Untrusted input	The calculator never calls eval(). It parses the LLM’s expression with ast and evaluates only the nodes it allows, with a cap on exponents so that 9**9**9 cannot exhaust the platform’s memory.	calculator/evaluator.py, calculator/tools.py
The smallest Extension	One tool and no settings, for date arithmetic: a template to start from.	smarter-extension-date-calculator
The test fixtures	The framework’s own test Extensions: echo, whose tools succeed, fail, refuse, time out and return non-JSON, with a secret setting, a waffle switch, a SAM manifest and lifecycle hooks; broken, which does not import; and incompatible.	tests/data/extensions/

Writing an Extension

Topic	What Smarter does	Where to look
The pyproject.toml	A standard Python project file, not a SAM manifest. ExtensionManifest validates its [project] and [tool.smarter] tables. Dependencies are checked against the platform’s installed libraries and never installed, and compatibility_problems() explains every unmet Python, Smarter or library requirement. The Extension’s pyproject.toml.	extension/manifest.py
The tool protocol	A tool is a class with a name, a Pydantic arguments_model and a call() method. Its docstring is its description for the LLM, and its arguments model’s JSON Schema is its parameters, so the definition offered to the LLM can never disagree with the validation. problems() explains why a class is not a valid tool. ADR-031: Pydantic Models Over Raw JSON.	extension/protocol.py
Running a tool	ExtensionToolkit names each tool ext_<slug>_<tool>, validates the LLM’s arguments, runs the call in a thread with the manifest’s timeout, and checks that the result is JSON. Invalid arguments, exceptions, timeouts and ExtensionToolError are returned to the LLM as {"error": ...} rather than failing the prompt, and a broken Extension is skipped and reported.	extension/toolkit.py
Settings and secrets	Settings are declared in pyproject.toml and configured by a superuser. Values are coerced to their declared types, and a secret setting is a Smarter Secret, encrypted at rest, that reaches the tool as a SecretStr, read at call time and never cached.	extension/settings.py
Lifecycle hooks	Eight hooks, from pre_install to post_uninstall, all doing nothing by default. A pre_* hook that raises cancels the action; a post_* hook that raises leaves the Extension installed but disabled. The Extension Lifecycle.	extension/lifecycle.py
Switches and resources	Waffle switches declared in pyproject.toml are created as ext_<slug>_<name>, and SAM manifests in the package’s manifests/ folder are applied through the real brokers. Each is recorded, so that uninstalling deletes exactly what the Extension created.	extension/resources.py
Testing helpers	tool_context() and call_tool() let an Extension’s tests call its tools as the platform does, without installing the Extension, the database or an LLM, while letting unexpected exceptions reach the test. Testing an Extension.	extension/testing.py, tests/test_tools.py (weather)
Management commands	validate_extension prints each tool’s definition exactly as the LLM will see it; package_extension, test_extension, install_extension, uninstall_extension and add_builtin_extensions cover the rest of a developer’s loop.	management/commands/

Installing, Loading and Publishing

Topic	What Smarter does	Where to look
The archive	A zip file, validated without importing it: at most 10 MB, 50 MB expanded and 2,000 files, with no links or paths that leave its directory. build_archive() fixes every timestamp, so the same source always builds the same sha256.	extension/archive.py
The installer	The one place where an Extension’s state changes: install, update, roll back, configure, enable, disable and uninstall. The web console, the management commands and the registry client all go through it.	extension/installer.py
Loading code at run time	Each process extracts a version’s archive once, atomically by rename, and imports it under a name unique to its version and sha256, so an update loads beside the old version instead of reloading modules. This is why an Extension’s modules use relative imports.	extension/loader.py
The registry	A public GitHub repository whose CI validates each submission, builds an index, and signs it with an Ed25519 key. Platforms verify the signature on every read, through the hardened HTTP client, and check every archive against the signed sha256. Publishing an Extension.	extension/registry.py, extensions-registry: scripts/validate.py, scripts/build_index.py, publishers.yaml
Releasing a version	semantic-release on each push to main: a fix releases a patch and a feat a minor version. The workflow builds the archive and prints the archive_url and sha256 that the registry entry needs.	release.yml, test.yml, scripts/package.py (weather)
The web console	Superusers browse the registry, install, upload, configure, enable, update, roll back and uninstall on two React pages, whose api is typed by Pydantic models exported as a JSON Schema. Managing Extensions.	extension/views.py, extension/contract.py, smarter-extension-console/
Observability	Eight signals, from extension_installed to extension_tool_failed, feed Smarter’s logs, progress messages and history. The log_extension waffle switch turns on the framework’s logging.	apps/llmtool/signals.py

Extensions in the History

Extensions arrived in one large commit, with two follow-ups, and the official Extensions have histories of their own.

Commit	Why it is worth studying	Where to look
Extensions arrive	Adding a Python tool used to mean changing Smarter itself: three functions hardcoded in the chat provider, a choices enum, and a release. The commit body explains every module of the new framework, the migration from spec.functions to spec.extensions, and how it was verified against the live registry.	commit fda234a37
Drift after a big change	Built-in system prompts still named tools that the LLM was no longer offered. The fix renames them to ext_<slug>_<tool> and adds a test that every tool a prompt names belongs to an Extension that its manifest lists.	commit 1aa73c2a0
From admin forms to the console	Extension management moves from the Django admin to two React pages, with a typed contract, and the admin becomes read-only.	commit e353a68a7
A result the LLM can read	get_weather 0.2.0 returns current conditions, starts the forecast at the current hour, gives times in the location’s timezone, and names the unit of every measurement.	commit 51ec052a5 (weather)
Automating releases	Hand-made tags replaced by semantic-release, so that a commit’s type decides the version.	commit 45f3690c2 (weather)
A registry release	The commit that publishes two Extensions’ 0.2.0: one YAML file per version.	commit a42a93ae9 (extensions-registry)

Graded Assignments

Assignment 1: A Tool Call, Traced

Students follow two weather questions, one that succeeds and one that fails, from the prompt to the answer, naming the code that handles each step, and compare the tool definition that validate_extension prints with the source that produced it.

Turn in: a two-page Markdown report with the conversation, the tool calls, their arguments and results, and the code that handled each step.

Criterion	Pts	Exemplary	Developing
Evidence	6	Real tool calls, arguments and results, from the chat and the logs.	Paraphrased, or from the docs.
Code located	6	Each step tied to the file and method that handles it.	Steps named, not located.
Definition explained	5	Each part of the definition traced to the docstring, model or slug.	Pasted without comment.
Failure path	3	Explains how the error reached the LLM, and what it did with it.	Only the success.

Assignment 2: A First Tool

Students write an Extension with one tool that does something an LLM can’t do reliably on its own, such as a unit conversion, a checksum or a lookup in a bundled table, with a setting an administrator configures. validate_extension must pass, and print the definition they intended.

Turn in: a Git repository with the Extension, and the output of validate_extension.

Criterion	Pts	Exemplary	Developing
Valid Extension	5	validate_extension passes; relative imports; nothing done at import time.	Validates only after the instructor’s help.
Arguments model	6	Every field typed, constrained and described for the LLM.	Bare str fields with no descriptions.
Description and docs	5	The docstring says what the tool does and when to use it; the module docstring serves an administrator.	“Does the thing.”
Worth a tool	4	Does something the LLM gets wrong without it.	Something the LLM does well already.

Assignment 3: Tested and Installed

Students test their tool with call_tool() and tool_context(), covering a success, invalid arguments and an ExtensionToolError with every network call mocked. They build its archive with package_extension, install and configure it on their platform, and show an LLMClient using it.

Turn in: the tests in the Extension’s repository, their test_extension output, the archive’s sha256, and the conversation.

Criterion	Pts	Exemplary	Developing
Cases covered	6	Success, invalid arguments and a tool error, each asserted precisely.	Only the success.
Isolation	4	No network, no API keys, no LLM.	Calls the real service.
Installed and used	6	Installed from its archive, configured, and called by an LLMClient.	Shown only in tests.
Runs as shown	4	The output and sha256 match the repository.	Took the tests’ passing on trust; output not included.

Capstone Projects

Students choose one, and present it in Live session 2.

Capstone A: An Extension for a Real Need

Learning objective. Build, test and ship an Extension that gives an LLM an ability it lacks, with settings an administrator configures.

Students build an Extension around a real service or computation, such as a public transit API, a currency converter with a configurable source, or a domain calculator for their own field. It has at least two tools or one tool and a secret setting, a result the LLM can interpret without guessing, tests with every service mocked, and a module docstring for administrators. They install its archive on their platform and show an LLMClient using it.

What students turn in:

• The Extension’s Git repository, with tests and a README.md.

• The archive and its sha256, from package_extension.

• A conversation in which an LLMClient uses each tool, including one failure it recovers from.

• A five-minute presentation.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
Tool design	25	Arguments and results modeled and described; the LLM calls each tool when it should.	The LLM ignores the tool, or guesses at its result.
Errors and input	20	The LLM’s input is validated as untrusted, and every failure returns a useful error.	Exceptions escape, or messages say only “error”.
Tests	20	Success and failure paths, with no network.	Tests that need an API key.
Settings and docs	15	Typed settings, a secret never logged, documentation an administrator can act on.	A key in the code, or no documentation.
Working install	20	Installed from its archive; the conversation shows every tool working.	Shown only in tests.

Capstone B: Plugin or Extension?

Learning objective. Decide, with evidence, when a tool should be configuration and when it should be code.

Students choose an integration with a public REST API, such as a weather, books or public data service, and build it twice: as an API Plugin with a Connection, and as an Extension. They run the same ten prompts against an LLMClient with each, and compare the two on effort, behavior, error handling, security and who can maintain them.

What students turn in:

• The Plugin manifests and the Extension’s repository.

• A table of the ten prompts, with each version’s tool calls, answers and tokens.

• A three-page recommendation, citing llmtool/plugin/api.py and the Extension framework, with the cases where each choice is right.

• A five-minute presentation.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
Both versions work	20	The same integration, working both ways.	One version is a stub.
Fair experiment	25	Same prompts, model and LLMClient settings, with results recorded as they came.	Different prompts, or results from memory.
Analysis	30	Effort, errors, selection, security and maintenance, each with evidence from the code.	Opinions without evidence.
Recommendation	15	Clear rules for choosing, with their exceptions.	“It depends.”
Presentation	10	Concise and organized.	Long or disorganized.

Recommended Texts

• Required: Extensibility, Building an Extension and OpenAI’s function calling guide, in week 1.

• Required: Testing an Extension and Publishing an Extension, in week 2.

• Reference: Luciano Ramalho, Fluent Python (2nd edition, O’Reilly, 2022), on classes, type hints and packages.

• Reference: Chip Huyen, AI Engineering (O’Reilly, 2025), on how tools and agents fit into AI applications.

• In the repository: protocol.py, toolkit.py and the weather Extension.

Further Study

• Official documentation: OpenAI’s function calling guide, whose get_weather example the weather Extension implements; the Pydantic documentation, on models, validation and JSON Schema; and the Python Packaging User Guide, including the pyproject.toml specification.

• Standards: PEP 440 (version identifiers), PEP 508 (dependency specifiers), Semantic Versioning, Conventional Commits, and JSON Schema.

• Other plugin ecosystems: the WordPress Plugin Handbook, the model for Smarter’s run-time install, and the VS Code Extension API, for how another platform describes, packages and publishes extensions.

• Software supply chain: The Update Framework, the design behind signed package indexes, Sigstore and SLSA, to compare with the registry’s Ed25519 signature and sha256 checks.

• Other tool protocols: the Model Context Protocol specification, the third way to give a Smarter LLMClient tools.

• YouTube: the PyCon US channel, for talks on packaging, Pydantic and testing, and the AI Engineer channel, for talks on tool use and agents.

• Podcasts: Talk Python To Me and Python Bytes, which cover packaging, Pydantic and the Python ecosystem regularly, and Latent Space, on building with LLMs.

• Writing: blog.smarter.sh, on how Smarter was built.