Smarter on Kubernetes

Smarter runs in two kinds of environment, and the same Docker image serves both:

Docker, with docker-compose.yml, on a developer’s computer. Every service runs as one container on one machine, the web console is served on http://localhost:9357, and nothing is reachable from the internet.
Kubernetes, with the Smarter Helm chart, in production. The platform runs as several Deployments that scale horizontally, behind a load balancer, and every deployed LLMClient gets its own hostname, DNS record, Kubernetes Ingress and TLS certificate.

The Python code is the same in both. What differs is the infrastructure around it, and that difference matters most when an LLMClient is deployed: in Docker, deploying an LLMClient changes a row in the database; in Kubernetes, it also creates cloud and cluster resources that someone has to support.

This section is for the people who install, operate and support Smarter in production: platform engineers, SREs, and the administrators who answer “why isn’t my LLMClient reachable?”. ADR-002 and ADR-003 record why Smarter chose Docker and Kubernetes (AWS EKS) in the first place.

The Services

Both environments run the same five processes. Only how they are started, scaled and connected differs.

Process	Docker (`docker-compose.yml`)	Kubernetes (Helm chart)
Web application and REST API	`smarter-app`: one container, uvicorn with auto-reload, on port 9357.	Deployment `smarter`: uvicorn with 4 workers per pod (`app.workers`), on port 8000, behind the ClusterIP Service `smarter`. Each worker accepts at most 100 concurrent requests (`app.limitConcurrency`) and answers 503 above that, so an overloaded pod sheds load instead of running out of memory. A HorizontalPodAutoscaler scales it on CPU and memory (3 to 25 pods by default, `hpa.*`), and a PodDisruptionBudget lets node drains evict only one app pod at a time (`app.podDisruptionBudget`).
Operational Celery worker	`smarter-worker`, queue `default_celery_task_queue`.	Deployment `smarter-worker`, same queue, with its own HorizontalPodAutoscaler.
Infrastructure Celery worker	`smarter-worker-infrastructure`, queue `infrastructure_celery_task_queue`.	Deployment `smarter-worker-infrastructure`, same queue, a fixed number of replicas (`workerInfrastructure.replicaCount`, 1 by default).
Celery Beat	`smarter-beat`.	Deployment `smarter-beat`. Run exactly one replica, or scheduled tasks run more than once.
Database and cache	`smarter-mariadb` and `smarter-redis` containers, with local volumes.	MySQL or MariaDB and Redis, inside the cluster or managed by the cloud (`database.type` and the `SMARTER_MYSQL_*` and `CACHES_LOCATION` values).

The two Celery queues are separate on purpose. The infrastructure queue carries the slow tasks that deploy, verify and destroy DNS records, ingresses, certificates, node groups and databases, so that they can never delay the operational tasks that record prompt history, charges and budgets. See ADR-030.

The Helm chart also runs a Job, smarter-job-init, on every install and upgrade. It migrates the database, initializes the platform, loads the example resources and deploys the built-in LLMClients. On a new cluster, that one Job queues a deployment for every built-in LLMClient at once. When it completes, it emails its whole log, with credentials redacted, to the chart’s databaseInit.notifyEmail, or by default to the platform’s admin, SMARTER_EMAIL_ADMIN, with manage.py email_job_log. The email reports whether the Job succeeded, and carries the log as an attachment. A log larger than 5 MB would exceed the SMTP server’s message size limit, so it is saved instead to the platform’s public S3 bucket, under job-logs/ with an unguessable name, and the email links to its CDN URL. The email is sent only when SMTP is configured (see SMTP Email Support), and a Job that is killed, for example for running out of memory, sends none.

A CronJob, smarter-mysql-backup, backs up the database every night (databaseScheduledBackup), with mariadb-dump, and uploads the compressed dump to databaseScheduledBackup.s3Uri, by default s3://<SMARTER_S3_BUCKET>/backups/sql. Point s3Uri at a private bucket when the platform’s bucket serves public files. The dump runs as a read-only account, SMARTER_MYSQL_BACKUP_USERNAME (smarter_backup by default), which the init Job creates, or updates, whenever SMARTER_MYSQL_BACKUP_PASSWORD is set, with SELECT, SHOW VIEW, TRIGGER, EVENT, SHOW CREATE ROUTINE on the platform’s database (SHOW CREATE ROUTINE exists in MariaDB 11.3 and later; on an older server the account gets the rest). The deploy workflow reads that password from the GitHub secret SMARTER_MYSQL_BACKUP_PASSWORD. Until it is set, the backup dumps as the root account. The backup’s container receives only what it needs: the database’s host, port and name, the backup account, the S3 location and the AWS credentials, and the root password only when there is no backup account. The password reaches mariadb-dump in a temporary option file, not on its command line. Before uploading, the backup checks that the dump ran to the end and that the archive is intact, and after uploading, that the object in S3 has the archive’s size.

The backup emails its result the same way, to databaseScheduledBackup.notifyEmail or to SMARTER_EMAIL_ADMIN, with the backup’s log attached. After a successful backup, the email names the backup file, its size, and its location in S3. The backup runs on a plain Debian image, as an init container, which installs only the MariaDB client and a pinned AWS CLI (databaseScheduledBackup.awsCliVersion), whose download it verifies against the AWS CLI team’s PGP signature. The email is sent by a second container, from the Smarter image, which passes that summary to email_job_log --summary. A failed backup is retried in a new pod, up to twice, and each attempt sends its own email.

Docker and Kubernetes Compared

Concern	Docker	Kubernetes
Configuration	A `.env` file read by docker-compose. `ENVIRONMENT` is `local`.	The `env` map of the chart’s `values.yaml`, rendered into every pod’s environment. `ENVIRONMENT` is `alpha`, `beta`, `next` or `prod`.
Kubernetes namespace	None is used. `smarter_settings.environment_namespace` is `smarter-local`.	`<platform name>-<platform subdomain>-<environment>`, e.g. `smarter-platform-prod`. Each environment of an installation has its own namespace.
Hostnames	`localhost:9357` for the web console, the API and sandbox LLMClients.	`platform.example.com` (or `<environment>.platform.example.com`) for the web console, `api.platform.example.com` for the API, and one hostname per deployed LLMClient. See URL Patterns.
TLS	None: plain HTTP on localhost.	cert-manager issues a certificate for every hostname, including one per deployed LLMClient. Custom Domains also get an ACM certificate.
Routing a request to an LLMClient	Django resolves the LLMClient from the URL path (sandbox URLs).	Traefik routes the hostname to the `smarter` Service through the LLMClient’s own Ingress. Django then resolves the LLMClient from the `Host` header.
Deploying an LLMClient	Sets `deployed` in the database. The deployment tasks only create cloud resources if the containers have cloud credentials and a kubeconfig (see the warning below).	Creates a DNS A record, an Ingress, a cert-manager Certificate and a TLS Secret, then verifies them. See LLMClient Deployment.
Scaling	One container per service.	HorizontalPodAutoscalers for the app and the operational worker; the app’s also scales on memory, measured against `app.resources.requests.memory`, and follows `hpa.behavior`. Pods are stateless; all state is in the database, Redis, and the cloud.
Health checks	None.	Startup and liveness probes on `/healthz/`, which the ASGI application answers before Django (`smarter.lib.django.healthz`), so they say whether the process is alive, and a readiness probe on `/readiness/`, which goes through Django and its middleware, so a pod whose Redis or database stalls is taken out of the Service rather than restarted. The startup probe checks every 2 seconds, so a pod takes traffic as soon as it has loaded, typically within 15 seconds. The probes send `Host: localhost`, which the security middleware lets through. A terminating pod waits 5 seconds (`preStop`) for the Service to stop sending it requests, then finishes its requests in progress.
Timeouts	The same, except the database’s.	Connecting to the database times out after 5 seconds, and to Redis after 2. A Redis command times out after 5 seconds. The web server’s database queries time out after `app.databaseReadTimeout` seconds (`SMARTER_DATABASE_READ_TIMEOUT`, 60 by default), and the workers’, the scheduler’s and the init Job’s never do, since migrations can run long. A SqlConnection’s `timeout` is its driver’s connect and query timeout.
Upgrades	Rebuild the image and restart the containers.	`helm upgrade`: a rolling update with `maxUnavailable: 0`, then the init Job.
Logs	`docker logs <container>`.	`kubectl logs` per pod. Smarter reads the logs of the pods it creates itself, such as an LLMHost’s or a self-hosted vector store’s, with the Kubernetes service.
Cloud resources	None, normally.	Route53 records, ACM certificates, EKS node groups (LLMHost) and StatefulSets with volumes (self-hosted vector stores), recorded in the infrastructure resource ledger.

Warning

The deployment tasks act on whatever cloud credentials and kubeconfig their container has. If you give a Docker environment real AWS credentials and a kubeconfig for a real cluster, deploying an LLMClient locally creates real Route53 records and applies a real Ingress to the namespace smarter-local of that cluster. To keep a Docker environment self-contained, leave the cloud credentials out, or set SMARTER_LLM_CLIENT_TASKS_CREATE_DNS_RECORD=false and SMARTER_LLM_CLIENT_TASKS_CREATE_INGRESS_MANIFEST=false.

How Smarter Talks to the Cluster

Smarter never calls the Kubernetes API from a client library. The infrastructure services run kubectl as a subprocess, with a kubeconfig that the cloud provider writes (aws eks update-kubeconfig for EKS), and with the cloud identity of the pod. Every pod that runs the image has kubectl and the AWS CLI installed, and KUBECONFIG points at /home/smarter_user/data/.kube/config.

In practice, the infrastructure worker is the pod that changes the cluster: it applies and deletes Ingresses, and reads Certificates and Secrets. The permissions that this requires, and how to keep them small, are described in Operating Smarter at Scale.

Kubernetes, DNS and certificate calls all go through one facade, smarter.apps.infrastructure.services.infrastructure, so the cloud is a configuration choice (SMARTER_CLOUD_PROVIDER), and tests replace it with an in-memory provider.

Cluster Prerequisites

The Helm chart installs Smarter itself. It does not install the cluster add-ons that LLMClient deployment depends on. Before the first LLMClient is deployed, the cluster needs:

Traefik as an ingress controller, with the traefik IngressClass and the websecure entry point, exposed by a cloud load balancer.
A Traefik Middleware named https-redirect in the environment’s namespace. Every LLMClient Ingress refers to it as <namespace>-https-redirect@kubernetescrd.
cert-manager, with a ClusterIssuer named after the environment’s API domain, e.g. api.platform.example.com in production or alpha.api.platform.example.com in alpha. Every LLMClient Ingress names it in its cert-manager.io/cluster-issuer annotation.
A DNS zone for the API domain, whose A record points at the load balancer. Every LLMClient’s A record is a copy of it.
An identity for the pods that can manage that DNS zone and the namespace’s Ingresses. See Operating Smarter at Scale.

The Smarter Terraform modules build this infrastructure on AWS. See Production Deployment.