Self-Hosting Open Models: Two-Week Online Course

A two-week online course on serving open models with Smarter’s LLMHost, with six 30-minute recorded lectures and a 90-minute live session each week. Week 1 covers how inference works and serving a model, and week 2 covers performance, comparison with commercial models, and the economics of self-hosting. See Course Formats and Grading for the format, the grading scale and the course policies, and Self-Hosting Open Models for an overview of the subject.

At a Glance

Students

Machine learning, MLOps and platform engineers, and developers.

Deployment

A shared Smarter instance on a managed Kubernetes cluster with GPU node groups, run by the instructor, who schedules GPU time. Students also run Ollama or llama.cpp locally.

Workload

3 hours of instruction and 4 to 6 hours of assignments each week.

Cost

Around a dollar an hour for each L4 GPU node while it runs. Budget for about 20 GPU-hours for a class of twenty, with a llmhost-compute Budget as a cap.

Warning

GPU nodes cost money until the last LLMHost on them is undeployed. Students undeploy at the end of every working session, and the instructor checks daily.

Learning Objectives

By the end of the course, students can:

  1. Explain how inference works, and estimate the memory a model needs.

  2. Choose a model, quantization and inference engine for a hardware budget.

  3. Deploy, monitor and undeploy models with LLMHost manifests.

  4. Measure and compare the performance of inference engines.

  5. Compare self-hosted and commercial models, and make a reasoned recommendation.

Technical Prerequisites

  • Required: the Linux command line, Git and YAML.

  • Required: Docker, and basic Kubernetes concepts.

  • Required: generative AI basics: tokens, context windows and chat completion APIs.

  • Required: Python, enough to call an OpenAI-compatible API and time it.

  • Required before Live session 1: Ollama installed locally, with a small model pulled.

  • Helpful: machine learning basics, and GPU concepts.

Schedule

Week

Item

Content

1

Lecture 1 (30 min)

Why self-host. Privacy, cost, control and availability; open weights and licenses.

1

Lecture 2 (30 min)

How inference works. Prefill and decode, the KV cache, batching, and GPU memory arithmetic.

1

Lecture 3 (30 min)

LLMHost. The manifest, the built-in examples and compute node groups, the lifecycle, and calling the endpoint.

1

Live session 1 (90 min)

Live deployment of qwen3-0.6b, then local Ollama measurements in pairs. Start Assignment 1.

2

Lecture 4 (30 min)

Inference engines and quantization. vLLM, SGLang, TGI, llama.cpp and Ollama; FP8, 4-bit, GGUF, AWQ and GPTQ.

2

Lecture 5 (30 min)

Self-hosted beside commercial. Registering an LLMHost as a Provider, and comparing models fairly.

2

Lecture 6 (30 min)

Economics and operations. Utilization, idle time, cost per thousand requests, monitoring and upgrades.

2

Live session 2 (90 min)

Capstone presentations (about 5 minutes each, with 2 minutes of questions).

Assignment 1 is due at the end of week 1, Assignment 2 in the middle of week 2, and Assignment 3 and the capstone at the start of Live session 2.

Graded Assignments

Assignment 1: Local and Hosted Inference

Students measure a small model locally with Ollama, then deploy an LLMHost on the class cluster, record its lifecycle, and measure it the same way.

Turn in: the script, the annotated manifest, describe output at each stage, and a results table.

Criterion

Pts

Exemplary

Developing

Measurements

6

Tokens per second, time to first token and memory, repeated.

One run.

Manifest annotation

5

Every field correct.

Errors.

Lifecycle

5

Every status, with timings explained.

Final status only.

Clean up

4

Undeployed, shown.

Left running.

Assignment 2: Engine Bake-off

Teams serve the same model with two engines, and measure throughput and latency at 1, 4 and 16 concurrent requests.

Turn in: manifests, the load-testing script, a chart and a one-page analysis.

Criterion

Pts

Exemplary

Developing

Fair comparison

5

Controlled conditions.

Uncontrolled.

Measurement

6

Three concurrency levels.

One.

Analysis

6

Explained by batching and memory.

Describes.

Recommendation

3

For a stated workload.

None.

Assignment 3: Quantization Trade-off

Students compare two precisions of the same model (for example FP16 and a 4-bit GGUF) on fifteen tasks for quality, and on speed and memory.

Turn in: the manifests or local configuration, the tasks and scores, and a one-page analysis.

Criterion

Pts

Exemplary

Developing

Quality evaluation

7

Stated rubric, consistent scoring.

Impressions.

Performance

5

Speed and memory measured.

Missing.

Analysis

5

Where quality loss matters and where it does not.

Generic.

Clarity

3

Clear.

Unclear.

Capstone Projects

Students choose one, and present it in Live session 2.

Capstone A: A Self-Hosting Recommendation

Learning objective. Decide, with measurements, whether an organization should self-host a model for a workload, and at what scale.

Students compare a self-hosted model on LLMHost with a commercial model for a stated workload, on quality (twenty-five tasks), latency under load, and cost per thousand requests at realistic utilization.

What students turn in:

  • Manifests, tasks, scores, measurements and a cost model.

  • A three-page recommendation with the break-even volume.

  • A five-minute presentation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Quality comparison

25

Representative, rubric-based, blind where possible.

Impressions.

Performance

20

Under load.

Single requests.

Cost model

25

Utilization, idle time and people; break-even.

Hourly price only.

Non-financial factors

15

Weighed.

Ignored.

Recommendation and presentation

15

Clear, conditional, on time.

None.

Capstone B: A Private Assistant on Open Models

Learning objective. Build an AI application whose prompts never leave the organization’s infrastructure, and measure what that costs in quality.

Students build an assistant for a privacy-sensitive job on self-hosted chat and embedding models, with guardrails, and compare it with the same assistant on a commercial model.

What students turn in:

  • Manifests and a data flow diagram.

  • A comparison on twenty questions.

  • A two-page trade-off note, and a five-minute presentation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Works

20

Self-hosted models only.

Commercial fallback somewhere.

Privacy evidence

25

Backed by configuration and logs.

Asserted.

Comparison

25

Fair and rubric-based.

Impressions.

Configuration

15

Suited to the hardware.

Defaults copied.

Note and presentation

15

Honest; clear.

One-sided.

Further Study

  • Documentation: vLLM, SGLang, Hugging Face Text Generation Inference, llama.cpp and Ollama.

  • Courses: the free Hugging Face courses, especially the LLM course, and Hugging Face’s model cards and licenses, which are worth reading closely.

  • Papers: Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023), the paper behind vLLM, and the GPTQ and AWQ papers on quantization.

  • Leaderboards: LMArena for human preference rankings, and Artificial Analysis for speed and price comparisons across open and commercial models.

  • YouTube: Andrej Karpathy’s Let’s build GPT: from scratch, in code, spelled out and Let’s reproduce GPT-2 on his channel, and Umar Jamil’s channel on transformer internals.

  • Podcasts: Latent Space, which regularly covers inference engines and open models, and Practical AI.

  • Community: the r/LocalLLaMA community, the most active place to compare open models on real hardware.