Self-Hosting Open Models: Two-Week Online Course

A two-week online course on serving open models with Smarter’s LLMHost, which runs them on Kubernetes and adds the GPU or CPU nodes they need. Smarter ships with twenty-four ready-to-apply model manifests and seven compute profiles priced per hour, supports seven inference engines, renders each model’s Kubernetes resources with a pure function that can be previewed, and creates and removes EKS node groups of GPU nodes as models need them. A self-hosted model is registered as a Provider and used by an LLMClient exactly like a commercial one, and its hours are charged like any other cost, so the comparison is fair. There are six 30-minute recorded lectures and a 90-minute live session each week. Week 1 covers how inference works and serving a model, and week 2 covers performance, comparison with commercial models, and the economics of self-hosting. See Course Formats and Grading for the format, the grading scale and the course policies, and Self-Hosting Open Models for an overview of the subject.

At a Glance

Students	Machine learning, MLOps and platform engineers, and developers.
Deployment	A shared Smarter instance on a managed Kubernetes cluster with GPU node groups, run by the instructor, who schedules GPU time. Students also run Ollama or llama.cpp locally.
Workload	3 hours of instruction and 4 to 6 hours of assignments each week.
Cost	Around a dollar an hour for each L4 GPU node while it runs. Budget for about 20 GPU-hours for a class of twenty, with a llmhost-compute Budget to stop new launches at a limit. It does not remove nodes that are already running.

Warning

GPU nodes cost money until the last LLMHost on them is undeployed. Students undeploy at the end of every working session, and the instructor checks daily.

Learning Objectives

By the end of the course, students can:

1. Explain how inference works, and estimate the memory a model needs.

2. Choose a model, quantization and inference engine for a hardware budget.

3. Deploy, monitor and undeploy models with LLMHost manifests.

4. Measure and compare the performance of inference engines.

5. Compare self-hosted and commercial models, and make a reasoned recommendation.

Technical Prerequisites

• Required: the Linux command line, Git and YAML.

• Required: Docker, and basic Kubernetes concepts.

• Required: generative AI basics: tokens, context windows and chat completion APIs.

• Required: Python, enough to call an OpenAI-compatible API and time it.

• Required before Live session 1: Ollama installed locally, with a small model pulled.

• Helpful: machine learning basics, and GPU concepts.

Schedule

The files that the lectures refer to are listed in full under Smarter in the Source Code.

Week	Item	Content
1	Lecture 1 (30 min)	Why self-host. Privacy, cost, control and availability; open weights and licenses.
1	Lecture 2 (30 min)	How inference works. Prefill and decode, the KV cache, batching, and GPU memory arithmetic.
1	Lecture 3 (30 min)	LLMHost. The manifest, the built-in examples and compute node groups, the lifecycle, and calling the endpoint.
1	Live session 1 (90 min)	Live deployment of qwen3-0.6b, then local Ollama measurements in pairs. Start Assignment 1.
2	Lecture 4 (30 min)	Inference engines and quantization. vLLM, SGLang, TGI, llama.cpp and Ollama; FP8, 4-bit, GGUF, AWQ and GPTQ.
2	Lecture 5 (30 min)	Self-hosted beside commercial. Registering an LLMHost as a Provider, and comparing models fairly.
2	Lecture 6 (30 min)	Economics and operations. Utilization, idle time, cost per thousand requests, monitoring and upgrades.
2	Live session 2 (90 min)	Capstone presentations (about 5 minutes each, with 2 minutes of questions).

Assignment 1 is due at the end of week 1, Assignment 2 in the middle of week 2, and Assignment 3 and the capstone at the start of Live session 2.

Smarter in the Source Code

Every lecture, lab and assignment in this course rests on the manifests and files below. Every reference is to the main branch of github.com/smarter-sh/smarter, so that students can open each one during class. Python paths are under smarter/smarter/.

Models and Compute

Topic	What Smarter has	Where to look
Twenty-four built-in models	Ready-to-apply LLMHost manifests for Llama 3.1, 3.2 and 3.3, Qwen 2.5 and Qwen 3, Gemma 3, Mistral, Phi-4, DeepSeek R1 distills and gpt-oss, GGUF models for CPU, and embedding models such as BGE-M3 and nomic-embed-text, each with its engine, resources and license.	llmhost/data/llmhost/, qwen3-0.6b.yaml
Seven compute profiles	Node groups from small CPU nodes to one NVIDIA L4, A10G or L40S, four L40S, and eight H100s, each with its instance type, memory and price per hour.	llmhost/data/compute/
The manifest, defined	Pydantic models for every section of an LLMHost: model (source, repository, pinned revision, quantization, capabilities), engine, compute, resources, storage, network, scaling and healthCheck.	llmhost/manifest/models/llmhost/spec.py
Finding a model	Discovery from the built-in catalog or the Hugging Face Hub, which sizes the pods and drafts a manifest.	llmhost/services/discovery.py

Serving

Topic	What Smarter does	Where to look
Seven inference engines	vLLM, Hugging Face TGI, SGLang, llama.cpp, Ollama, Hugging Face TEI for embeddings, and a custom engine, each turning the manifest into the right command, arguments and port.	llmhost/services/engines.py
Kubernetes resources	A volume for the weights, a Secret, a Deployment of the inference server, a Service and an optional Ingress, rendered by a pure function so that they can be previewed and tested.	llmhost/services/renderer.py
Nodes on demand	An EKS managed node group per compute profile, created at zero nodes, scaled up when a model needs a node, and scaled down by terminating a chosen idle node.	llmhost/services/nodegroups.py
The lifecycle	provisioning while a node joins, then pending, downloading the weights, deploying, and active, or degraded, inactive or error.	llmhost/manifest/enum.py (SAMLLMHostStatusEnum)
Health, status and cost	Kubernetes’ view of the pods mapped to a status, health probes of the engine’s endpoint, and a report of status, GPUs and cost.	llmhost/services/observability.py

Using and Paying for a Model

Topic	What Smarter does	Where to look
Beside commercial models	A self-hosted endpoint, with its API key Secret, is registered as a Provider, and any LLMClient can then use it in place of a commercial model. Smarter Provider.	apps/provider/, llmclient-simple.yaml
Hourly charges	Celery Beat charges each compute profile’s ready nodes every hour, at its price.	workers/celerybeat.py (charge-llmhost-computes)
A GPU budget	An example Budget that, once a compute profile’s nodes have cost $500 in a month, launches no more LLMHosts on it. Nodes that are already running are not removed, so undeploying is still everyone’s job.	llmhost-compute.yaml

Graded Assignments

Assignment 1: Local and Hosted Inference

Students measure a small model locally with Ollama, then deploy an LLMHost on the class cluster, record its lifecycle, and measure it the same way.

Turn in: the script, the annotated manifest, describe output at each stage, and a results table.

Criterion	Pts	Exemplary	Developing
Measurements	6	Tokens per second, time to first token and memory, repeated.	One run.
Manifest annotation	5	Every field correct.	Errors.
Lifecycle	5	Every status, with timings explained.	Final status only.
Clean up	4	Undeployed, shown.	Left running.

Assignment 2: Engine Bake-off

Teams serve the same model with two engines, and measure throughput and latency at 1, 4 and 16 concurrent requests.

Turn in: manifests, the load-testing script, a chart and a one-page analysis.

Criterion	Pts	Exemplary	Developing
Fair comparison	5	Controlled conditions.	Uncontrolled.
Measurement	6	Three concurrency levels.	One.
Analysis	6	Explained by batching and memory.	Describes.
Recommendation	3	For a stated workload.	None.

Assignment 3: Quantization Trade-off

Students compare two precisions of the same model (for example FP16 and a 4-bit GGUF) on fifteen tasks for quality, and on speed and memory.

Turn in: the manifests or local configuration, the tasks and scores, and a one-page analysis.

Criterion	Pts	Exemplary	Developing
Quality evaluation	7	Stated rubric, consistent scoring.	Impressions.
Performance	5	Speed and memory measured.	Missing.
Analysis	5	Where quality loss matters and where it does not.	Generic.
Clarity	3	Clear.	Unclear.

Capstone Projects

Students choose one, and present it in Live session 2.

Capstone A: A Self-Hosting Recommendation

Learning objective. Decide, with measurements, whether an organization should self-host a model for a workload, and at what scale.

Students compare a self-hosted model on LLMHost with a commercial model for a stated workload, on quality (twenty-five tasks), latency under load, and cost per thousand requests at realistic utilization.

What students turn in:

• Manifests, tasks, scores, measurements and a cost model.

• A three-page recommendation with the break-even volume.

• A five-minute presentation.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
Quality comparison	25	Representative, rubric-based, blind where possible.	Impressions.
Performance	20	Under load.	Single requests.
Cost model	25	Utilization, idle time and people; break-even.	Hourly price only.
Non-financial factors	15	Weighed.	Ignored.
Recommendation and presentation	15	Clear, conditional, on time.	None.

Capstone B: A Private Assistant on Open Models

Learning objective. Build an AI application whose prompts never leave the organization’s infrastructure, and measure what that costs in quality.

Students build an assistant for a privacy-sensitive job on self-hosted chat and embedding models, with guardrails, and compare it with the same assistant on a commercial model.

What students turn in:

• Manifests and a data flow diagram.

• A comparison on twenty questions.

• A two-page trade-off note, and a five-minute presentation.

How it is graded (100 points):

Criterion	Pts	Exemplary	Developing
Works	20	Self-hosted models only.	Commercial fallback somewhere.
Privacy evidence	25	Backed by configuration and logs.	Asserted.
Comparison	25	Fair and rubric-based.	Impressions.
Configuration	15	Suited to the hardware.	Defaults copied.
Note and presentation	15	Honest; clear.	One-sided.

Recommended Texts

• Required: Chip Huyen, AI Engineering: Building Applications with Foundation Models (O’Reilly, 2025). Week 1: chapter 2 (foundation models). Week 2: chapter 9 (inference optimization).

• Required: Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023), in week 2.

• Reference: Jay Alammar and Maarten Grootendorst, Hands-On Large Language Models (O’Reilly, 2024), chapters 1 to 3.

• Smarter documentation: Smarter LLM Host, Smarter Provider and Smarter on Kubernetes.

Further Study

• Documentation: vLLM, SGLang, Hugging Face Text Generation Inference, llama.cpp and Ollama.

• Courses: the free Hugging Face courses, especially the LLM course, and Hugging Face’s model cards and licenses, which are worth reading closely.

• Papers: Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023), the paper behind vLLM, and the GPTQ and AWQ papers on quantization.

• Leaderboards: LMArena for human preference rankings, and Artificial Analysis for speed and price comparisons across open and commercial models.

• YouTube: Andrej Karpathy’s Let’s build GPT: from scratch, in code, spelled out and Let’s reproduce GPT-2 on his channel, and Umar Jamil’s channel on transformer internals.

• Podcasts: Latent Space, which regularly covers inference engines and open models, and Practical AI.

• Community: the r/LocalLLaMA community, the most active place to compare open models on real hardware.