Self-Hosting Open Models: Four-Week Online Course

A four-week online course on serving open models with Smarter’s LLMHost, with twelve 30-minute recorded lectures and a 90-minute live session each week. Week 1 covers open models and how inference works, week 2 serving models with LLMHost, week 3 performance and quantization, and week 4 applications, operations and economics. See Course Formats and Grading for the format, the grading scale and the course policies, and Self-Hosting Open Models for an overview of the subject.

At a Glance

Students

Machine learning, MLOps and platform engineers, and developers.

Deployment

A shared Smarter instance on a managed Kubernetes cluster with GPU and CPU node groups, run by the instructor, who schedules GPU time. Students also run Ollama or llama.cpp locally.

Workload

3 hours of instruction and 4 to 6 hours of assignments each week.

Cost

Around a dollar an hour for each L4 GPU node while it runs. Budget for about 40 GPU-hours for a class of twenty, with a llmhost-compute Budget as a cap.

Warning

GPU nodes cost money until the last LLMHost on them is undeployed. Students undeploy at the end of every working session, and the instructor checks daily.

Learning Objectives

By the end of the course, students can:

  1. Explain the open model landscape, licenses, and how transformer inference works.

  2. Estimate the memory and hardware a model needs, and choose a model and quantization.

  3. Deploy, monitor and undeploy models with LLMHost, including gated and embedding models.

  4. Benchmark inference engines, and tune them for a workload.

  5. Build applications on self-hosted models, beside commercial ones.

  6. Model the cost of self-hosting, and operate self-hosted models responsibly.

Technical Prerequisites

  • Required: the Linux command line, Git and YAML.

  • Required: Docker, and basic Kubernetes concepts.

  • Required: generative AI basics: tokens, context windows and chat completion APIs.

  • Required: Python, enough to call an OpenAI-compatible API and write a small load test.

  • Required by the end of week 1: Ollama or llama.cpp installed locally.

  • Required by week 2: a Hugging Face account, with the licenses of the gated models used in class accepted.

  • Helpful: machine learning basics, linear algebra, and GPU concepts.

Schedule

Week

Item

Content

1

Lecture 1

Why self-host. Privacy, data residency, cost, control and availability.

1

Lecture 2

The open model landscape. Model families, sizes, licenses (open weights versus open source), and reading a model card.

1

Lecture 3

How inference works. Transformers at a high level, prefill and decode, the KV cache, and memory arithmetic.

1

Live session 1

Local inference with Ollama, measured live. Start Assignment 1.

2

Lecture 4

LLMHost. The manifest (model, engine, compute, resources, network, storage) and the built-in examples.

2

Lecture 5

Compute. LLMHostCompute node groups, how Smarter adds and removes nodes, GPUs, and CPU inference with GGUF models.

2

Lecture 6

Endpoints and access. OpenAI-compatible endpoints, API keys in Secrets, ingress, gated models and access tokens.

2

Live session 2

Review of Assignment 1. Live deployment and lifecycle of an LLMHost. Start Assignment 2.

3

Lecture 7

Inference engines. vLLM and PagedAttention, SGLang, TGI, llama.cpp and Ollama.

3

Lecture 8

Quantization. FP8, INT8, 4-bit, GGUF, AWQ and GPTQ, and measuring quality loss.

3

Lecture 9

Benchmarking. Throughput, latency, time to first token, concurrency and load testing.

3

Live session 3

Review of Assignment 2. Engine comparison, live. Capstone proposals discussed. Start Assignment 3.

4

Lecture 10

Applications on open models. Registering a Provider, LLMClients, embedding models for retrieval, and evaluation.

4

Lecture 11

Operations. Health checks, monitoring, upgrades, model versioning with pinned revisions, and incident response.

4

Lecture 12

Economics. Utilization, idle time, autoscaling to zero, cost per thousand requests, and the break-even with commercial providers.

4

Live session 4

Capstone presentations (about 6 minutes each, with 2 minutes of questions), and a course retrospective.

Assignments 1 to 3 are due before the next live session. Assignment 4 is due in the middle of week 4. The capstone proposal is due before Live session 3, and the capstone at the start of Live session 4.

Graded Assignments

Assignment 1: Model Selection and Memory Estimates

Students choose three open models for a stated job, read their model cards and licenses, estimate the GPU memory each needs at two precisions and a stated context length, and check one estimate by running the model locally.

Turn in: a table of models, licenses, estimates and the measurement, and a one-page recommendation.

Criterion

Pts

Exemplary

Developing

Model choice

4

Suited to the job, with reasons.

Arbitrary.

Licenses

4

Restrictions correctly summarized.

Not read.

Memory estimates

7

Weights and KV cache estimated correctly; one checked.

Weights only.

Recommendation

5

Clear and reasoned.

None.

Assignment 2: LLMHost Lifecycle and a CPU Model

Students deploy a GPU LLMHost and a CPU (GGUF) LLMHost from the built-in examples, record both lifecycles, call both endpoints, and compare them.

Turn in: annotated manifests, describe output, measurements and a half-page comparison.

Criterion

Pts

Exemplary

Developing

Annotation

5

Every field correct.

Errors.

Lifecycles

5

Every status, with timings explained.

Final status only.

Comparison

6

Speed, memory and cost compared.

Partial.

Clean up

4

Both undeployed, shown.

Left running.

Assignment 3: Engine Bake-off

Teams benchmark one model on two engines at four concurrency levels, and tune one engine argument (for example the maximum batch size or context length) to improve throughput.

Turn in: manifests, the load test, a chart, and a one-page analysis.

Criterion

Pts

Exemplary

Developing

Fair comparison

5

Controlled.

Uncontrolled.

Measurement

5

Four levels, repeated.

One level.

Tuning

5

Measured effect, explained.

Unmeasured.

Analysis

5

Explained by batching and memory.

Describes.

Assignment 4: Quantization and Quality

Students compare three precisions of one model on twenty tasks, scored by a rubric and checked by a second person or a validated judge, and on speed and memory.

Turn in: the tasks, scores, measurements and a one-page analysis.

Criterion

Pts

Exemplary

Developing

Evaluation

7

Rubric, double-checked scores.

Impressions.

Performance

5

Speed and memory for each precision.

Missing.

Analysis

5

Which tasks degrade, and why.

Generic.

Recommendation

3

A precision for a stated use.

None.

Capstone Projects

Students choose one, propose it before Live session 3, and present it in Live session 4.

Capstone A: A Self-Hosting Business Case with Benchmarks

Learning objective. Produce an evidence-based recommendation on self-hosting for an organization, from benchmarks to a three-year cost model.

Students choose a workload and an organization, select and tune a self-hosted model on LLMHost, compare it with two commercial models on quality (thirty tasks), latency under load and cost, and build a three-year cost model with utilization scenarios.

What students turn in:

  • Manifests, tasks, scores, benchmarks and the cost model.

  • A four-page recommendation with the break-even volume and an operations plan.

  • A six-minute presentation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Quality comparison

20

Rubric-based, validated scoring.

Impressions.

Benchmarks

20

Under load, after tuning.

Untuned single requests.

Cost model

25

Scenarios, utilization, people; break-even.

Hourly price only.

Operations plan

15

Monitoring, upgrades and incidents covered.

Missing.

Recommendation and presentation

20

Clear, conditional, on time.

None.

Capstone B: A Fully Self-Hosted AI Application

Learning objective. Build and operate an AI application that runs entirely on self-hosted models, with retrieval and guardrails, and document how to run it.

Students build an assistant with retrieval over fictional private documents, using a self-hosted chat model, a self-hosted embedding model, and guardrails that use a self-hosted judge model, then compare it with a commercial equivalent and write a runbook.

What students turn in:

  • Manifests and a data flow diagram showing that nothing leaves the cluster.

  • A comparison on twenty-five questions.

  • A runbook: deploy, scale, upgrade the model, monitor, and recover.

  • A six-minute presentation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Works

20

Self-hosted end to end.

Commercial fallback.

Privacy evidence

20

Configuration and logs.

Asserted.

Comparison

20

Fair and rubric-based.

Impressions.

Runbook

25

Tested by another student.

Untested.

Presentation

15

Clear and on time.

Overruns.

Further Study

  • Documentation: vLLM, SGLang, Hugging Face Text Generation Inference, llama.cpp and Ollama.

  • Courses: the free Hugging Face courses, especially the LLM course, and Hugging Face’s model cards and licenses, which are worth reading closely.

  • Papers: Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023), the paper behind vLLM, and the GPTQ and AWQ papers on quantization.

  • Leaderboards: LMArena for human preference rankings, and Artificial Analysis for speed and price comparisons across open and commercial models.

  • YouTube: Andrej Karpathy’s Let’s build GPT: from scratch, in code, spelled out and Let’s reproduce GPT-2 on his channel, and Umar Jamil’s channel on transformer internals.

  • Podcasts: Latent Space, which regularly covers inference engines and open models, and Practical AI.

  • Community: the r/LocalLLaMA community, the most active place to compare open models on real hardware.