Smarter on Kubernetes

Smarter runs in two kinds of environment, and the same Docker image serves both:

Docker, with docker-compose.yml, on a developer’s computer. Every service runs as one container on one machine, the web console is served on http://localhost:9357, and nothing is reachable from the internet.
Kubernetes, with the Smarter Helm chart, in production. The platform runs as several Deployments that scale horizontally, behind a load balancer, and every deployed LLMClient gets its own hostname, DNS record, Kubernetes Ingress and TLS certificate.

The Python code is the same in both. What differs is the infrastructure around it, and that difference matters most when an LLMClient is deployed: in Docker, deploying an LLMClient changes a row in the database; in Kubernetes, it also creates cloud and cluster resources that someone has to support.

This section is for the people who install, operate and support Smarter in production: platform engineers, SREs, and the administrators who answer “why isn’t my LLMClient reachable?”. ADR-002 and ADR-003 record why Smarter chose Docker and Kubernetes (AWS EKS) in the first place.

The Services

Both environments run the same five processes. Only how they are started, scaled and connected differs.

Process	Docker (`docker-compose.yml`)	Kubernetes (Helm chart)
Web application and REST API	`smarter-app`: one container, uvicorn with auto-reload, on port 9357.	Deployment `smarter`: uvicorn with 2 workers per pod, on port 8000, behind the ClusterIP Service `smarter`. A HorizontalPodAutoscaler scales it on CPU (3 to 25 pods by default).
Operational Celery worker	`smarter-worker`, queue `default_celery_task_queue`.	Deployment `smarter-worker`, same queue, with its own HorizontalPodAutoscaler.
Infrastructure Celery worker	`smarter-worker-infrastructure`, queue `infrastructure_celery_task_queue`.	Deployment `smarter-worker-infrastructure`, same queue, a fixed number of replicas (`workerInfrastructure.replicaCount`, 1 by default).
Celery Beat	`smarter-beat`.	Deployment `smarter-beat`. Run exactly one replica, or scheduled tasks run more than once.
Database and cache	`smarter-mariadb` and `smarter-redis` containers, with local volumes.	MySQL or MariaDB and Redis, inside the cluster or managed by the cloud (`database.type` and the `SMARTER_MYSQL_*` and `CACHES_LOCATION` values).

The two Celery queues are separate on purpose. The infrastructure queue carries the slow tasks that deploy, verify and destroy DNS records, ingresses, certificates, node groups and databases, so that they can never delay the operational tasks that record prompt history, charges and budgets. See ADR-030.

The Helm chart also runs a Job, smarter-job-init, on every install and upgrade. It migrates the database, initializes the platform, loads the example resources and deploys the built-in LLMClients. On a new cluster, that one Job queues a deployment for every built-in LLMClient at once.

Docker and Kubernetes Compared

Concern	Docker	Kubernetes
Configuration	A `.env` file read by docker-compose. `ENVIRONMENT` is `local`.	The `env` map of the chart’s `values.yaml`, rendered into every pod’s environment. `ENVIRONMENT` is `alpha`, `beta`, `next` or `prod`.
Kubernetes namespace	None is used. `smarter_settings.environment_namespace` is `smarter-local`.	`<platform name>-<platform subdomain>-<environment>`, e.g. `smarter-platform-prod`. Each environment of an installation has its own namespace.
Hostnames	`localhost:9357` for the web console, the API and sandbox LLMClients.	`platform.example.com` (or `<environment>.platform.example.com`) for the web console, `api.platform.example.com` for the API, and one hostname per deployed LLMClient. See URL Patterns.
TLS	None: plain HTTP on localhost.	cert-manager issues a certificate for every hostname, including one per deployed LLMClient. Custom Domains also get an ACM certificate.
Routing a request to an LLMClient	Django resolves the LLMClient from the URL path (sandbox URLs).	Traefik routes the hostname to the `smarter` Service through the LLMClient’s own Ingress. Django then resolves the LLMClient from the `Host` header.
Deploying an LLMClient	Sets `deployed` in the database. The deployment tasks only create cloud resources if the containers have cloud credentials and a kubeconfig (see the warning below).	Creates a DNS A record, an Ingress, a cert-manager Certificate and a TLS Secret, then verifies them. See LLMClient Deployment.
Scaling	One container per service.	HorizontalPodAutoscalers for the app and the operational worker. Pods are stateless; all state is in the database, Redis, and the cloud.
Health checks	None.	Liveness, readiness and startup probes on `/healthz/` and `/readiness/`, with `Host: localhost`, which the security middleware lets through.
Upgrades	Rebuild the image and restart the containers.	`helm upgrade`: a rolling update with `maxUnavailable: 0`, then the init Job.
Logs	`docker logs <container>`.	`kubectl logs` per pod. Smarter reads the logs of the pods it creates itself, such as an LLMHost’s or a self-hosted vector store’s, with the Kubernetes service.
Cloud resources	None, normally.	Route53 records, ACM certificates, EKS node groups (LLMHost) and StatefulSets with volumes (self-hosted vector stores), recorded in the infrastructure resource ledger.

Warning

The deployment tasks act on whatever cloud credentials and kubeconfig their container has. If you give a Docker environment real AWS credentials and a kubeconfig for a real cluster, deploying an LLMClient locally creates real Route53 records and applies a real Ingress to the namespace smarter-local of that cluster. To keep a Docker environment self-contained, leave the cloud credentials out, or set SMARTER_LLM_CLIENT_TASKS_CREATE_DNS_RECORD=false and SMARTER_LLM_CLIENT_TASKS_CREATE_INGRESS_MANIFEST=false.

How Smarter Talks to the Cluster

Smarter never calls the Kubernetes API from a client library. The infrastructure services run kubectl as a subprocess, with a kubeconfig that the cloud provider writes (aws eks update-kubeconfig for EKS), and with the cloud identity of the pod. Every pod that runs the image has kubectl and the AWS CLI installed, and KUBECONFIG points at /home/smarter_user/data/.kube/config.

In practice, the infrastructure worker is the pod that changes the cluster: it applies and deletes Ingresses, and reads Certificates and Secrets. The permissions that this requires, and how to keep them small, are described in Operating Smarter at Scale.

Kubernetes, DNS and certificate calls all go through one facade, smarter.apps.infrastructure.services.infrastructure, so the cloud is a configuration choice (SMARTER_CLOUD_PROVIDER), and tests replace it with an in-memory provider.

Cluster Prerequisites

The Helm chart installs Smarter itself. It does not install the cluster add-ons that LLMClient deployment depends on. Before the first LLMClient is deployed, the cluster needs:

Traefik as an ingress controller, with the traefik IngressClass and the websecure entry point, exposed by a cloud load balancer.
A Traefik Middleware named https-redirect in the environment’s namespace. Every LLMClient Ingress refers to it as <namespace>-https-redirect@kubernetescrd.
cert-manager, with a ClusterIssuer named after the environment’s API domain, e.g. api.platform.example.com in production or alpha.api.platform.example.com in alpha. Every LLMClient Ingress names it in its cert-manager.io/cluster-issuer annotation.
A DNS zone for the API domain, whose A record points at the load balancer. Every LLMClient’s A record is a copy of it.
An identity for the pods that can manage that DNS zone and the namespace’s Ingresses. See Operating Smarter at Scale.

The Smarter Terraform modules build this infrastructure on AWS. See Production Deployment.