Software Quality: Two-Week Online Course

A two-week online course on software quality and quality assurance, taught on Smarter’s complete quality process. The process is all in the repository: more than five thousand Python unit tests and about 150 React test files, SmarterTestBase and shared test mixins, an in-memory cloud provider for test doubles, a test runner that skips tests needing real infrastructure, pre-commit hooks as the lint standard, Codecov flags for each suite, commitlint, semantic-release, and a test workflow that reproduces the developer setup in CI. There are six 30-minute recorded lectures and a 90-minute live session each week. Week 1 covers testing and coverage, and week 2 covers automated checks, CI and releases. See Course Formats and Grading for the format, the grading scale and the course policies, and Software Quality and Quality Assurance for an overview of the subject.

At a Glance

Students

Software developers and QA engineers who write Python or JavaScript.

Deployment

Each student runs Smarter from source, installed before Live session 1, with pre-commit hooks set up and a fork on GitHub.

Workload

3 hours of instruction and 4 to 6 hours of assignments each week.

Cost

Free on GitHub Actions for public forks, plus a few cents of model calls when running tests.

Learning Objectives

By the end of the course, students can:

  1. Explain the layers of a quality process, and what each catches.

  2. Run tests and coverage, and identify the riskiest untested code.

  3. Write meaningful unit tests with fixtures and test doubles.

  4. Configure and explain pre-commit hooks and CI workflows.

  5. Explain conventional commits, semantic versioning and automated releases.

Technical Prerequisites

  • Required: Python or JavaScript, enough to write a unit test.

  • Required: Git and GitHub, including pull requests.

  • Required: the command line and Docker basics.

  • Required before Live session 1: Smarter installed from source, and pre-commit run --all-files completed once.

  • Helpful: a unit testing framework, and YAML.

Schedule

The files that the lectures refer to are listed in full under Smarter in the Source Code.

Week

Item

Content

1

Lecture 1 (30 min)

What quality means. Defects and their cost, the test pyramid, and the layers of Smarter’s process. ADR 028.

1

Lecture 2 (30 min)

Good unit tests. Arrange, act, assert; fixtures; cleanup; SmarterTestBase; ADR 013.

1

Lecture 3 (30 min)

Test doubles. Mocks, fakes and stubs, patching external services, and the in-memory infrastructure provider.

1

Live session 1 (90 min)

Live coverage run on one app, then pairs write a first test together. Start Assignment 1.

2

Lecture 4 (30 min)

Automated checks. Every hook in .pre-commit-config.yaml: what it catches and what it cannot.

2

Lecture 5 (30 min)

CI. The workflows in .github/workflows/, and CI/CD.

2

Lecture 6 (30 min)

Commits and releases. Conventional commits, commitlint, semantic versioning and semantic-release (Semantic Versioning Guide).

2

Live session 2 (90 min)

Capstone presentations (about 5 minutes each, with 2 minutes of questions).

Assignment 1 is due at the end of week 1, Assignment 2 in the middle of week 2, and Assignment 3 and the capstone at the start of Live session 2.

Smarter in the Source Code

Every lecture, lab and assignment in this course rests on the files below. Every reference is to the main branch of github.com/smarter-sh/smarter, so that students can open each one during class. Python paths are under smarter/smarter/.

Tests

Topic

What Smarter does

Where to look

The Python suite

More than five thousand unit tests in about 460 modules, one test module per source module, with test data as YAML. ADR 013.

guardrail/tests/, guardrail/tests/data/

Base classes

SmarterTestBase, which has no transaction rollback, so fixtures are built once in setUpClass and every test deletes what it creates, and mixins that test any resource’s broker, API and views the same way.

lib/unittest/base_classes.py, lib/unittest/resource_views.py, lib/unittest/broker_branches.py

Test doubles

An in-memory cloud provider that replaces Route53, Certificate Manager, Kubernetes and email in tests, behind the same interface as the real one.

infrastructure/providers/memory.py

Tests that are skipped on purpose

A custom test runner skips tests tagged infrastructure, which deploy to a real cluster and wait up to 40 minutes for a certificate, unless they are asked for.

lib/unittest/runner.py

The front end suite

About 150 Vitest and Testing Library test files, with every Storybook story rendered as a test, and MSW mocks of the API.

smarter/react/vitest.shared.ts, smarter/react/test/

The traps, written down

What goes wrong in Smarter’s tests and why, from cleanup without rollback to Celery tasks that are not eager.

smarter-python-testing/SKILL.md, smarter-react-testing/SKILL.md

Coverage and Checks

Topic

What Smarter does

Where to look

Coverage

make coverage runs the suite under coverage.py in the container. The target is at least 90% for each Django app, 95% where that takes moderate effort, and Codecov reports Python and React coverage under separate flags.

Makefile (coverage), codecov.yml

Pre-commit hooks

black, isort, flake8, autoflake, pyupgrade, pydocstringformatter, bandit, codespell, Prettier and ESLint, file checks, secret detection and commitlint, the authoritative lint standard. ADR 028.

.pre-commit-config.yaml, smarter-code-quality/SKILL.md

Types

Type annotations everywhere, typed aliases instead of Any, and Pydantic models for structured data.

lib/typing.py, mypy.ini

Documentation checks

Docstrings formatted by a hook and published by Sphinx autodoc, and a script that checks links and autodoc targets in seconds rather than building the documentation. ADR 029.

check-docs.sh, docs/source/lint.py

CI and Releases

Topic

What Smarter does

Where to look

The test workflow

Two jobs: react checks format, lint and types and runs Vitest, and python builds the development environment step by step, with caches, and runs the suite in the container. It deliberately has no cloud credentials. It is started by hand, so a fork that wants it as an autograder adds a push or pull_request trigger. Unit Testing.

test.yml

Build, release and deploy

Separate workflows build the image, release the Helm chart, deploy, and keep the pre-commit hooks up to date. CI/CD.

.github/workflows/

Conventional commits and releases

commitlint checks every message, and semantic-release turns them into versions on main, next, beta and alpha, with changelogs. Semantic Versioning Guide.

commitlint.config.js, release.config.js

Quality when agents write the code

The definition of done that every coding agent works to: a pattern followed, tests run in the container, hooks passing, and docs updated.

CLAUDE.md

Graded Assignments

Assignment 1: Coverage Analysis and First Tests

Students run one app’s coverage, identify the three riskiest gaps, and write tests for one of them.

Turn in: a one-page analysis, and a pull request against their fork with the tests.

Criterion

Pts

Exemplary

Developing

Gap analysis

6

Riskiest code, with reasons.

Percentages only.

Tests

8

Behavior and error paths asserted.

Execute code without asserting outcomes.

Conventions

4

Fixtures, cleanup, mocks.

Leaves data, or real calls.

Coverage gain

2

Shown.

Not shown.

Assignment 2: Mocks and Fakes

Students write tests for code that calls an external service (an LLM provider, DNS, or HTTP), once with unittest.mock and once with a fake, and compare the two approaches.

Turn in: a pull request against their fork and a half-page comparison.

Criterion

Pts

Exemplary

Developing

No real calls

6

Provably isolated from the network.

A real call slips through.

Mock test

4

Patches at the right place; asserts the call and the outcome.

Patches the wrong name.

Fake test

4

Uses or extends a fake correctly.

Not attempted.

Comparison

6

Weighs brittleness and realism.

Superficial.

Assignment 3: Bug Hunt Report

Students introduce five deliberate defects of different kinds, one at a time, and record which layer catches each, including CI on their fork.

Turn in: a table with evidence, and a half-page conclusion.

Criterion

Pts

Exemplary

Developing

Variety

5

Five kinds.

Similar.

Evidence

6

Hook output, failing tests, CI runs.

Claims.

Analysis

6

Why each was caught or missed.

Results only.

Improvement

3

A new check proposed.

None.

Capstone Projects

Students choose one, and present it in Live session 2.

Capstone A: Bring a Module to 95%

Learning objective. Raise real production code’s coverage with tests that catch real regressions.

Students bring a module below 90% to at least 95%, and prove the tests are meaningful by showing three deliberate bugs that they catch.

What students turn in:

  • A pull request with tests and coverage before and after.

  • A table of bugs and the tests that catch them.

  • A five-minute presentation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Coverage

20

95% or more.

Below 90%.

Meaningful assertions

30

Deliberate bugs caught.

Tests pass with bugs.

Conventions

20

Project conventions followed.

Inconsistent.

Edge cases

15

Errors and boundaries.

Happy path.

Presentation

15

Clear and on time.

Overruns.

Capstone B: A Quality Gate for Your Own Project

Learning objective. Build a complete quality process for a project of your own, modeled on a working one.

Students add pre-commit hooks, tests with a coverage threshold, a CI workflow, commitlint and automated semantic releases to a project of their own.

What students turn in:

  • A public repository with the configuration, tests and workflows.

  • A passing run, a failing run and an automated release.

  • A five-minute presentation.

How it is graded (100 points):

Criterion

Pts

Exemplary

Developing

Pre-commit

20

Format, lint, spelling, security, secrets.

Formatting only.

Tests and threshold

20

Enforced in CI.

No threshold.

CI

20

Every push and pull request.

Manual.

Commits and releases

20

Demonstrated.

Not demonstrated.

Presentation

20

Clear, with comparison to Smarter.

Unclear.

Further Study