Kubernetes Service

The Kubernetes service: resources in the platform’s Kubernetes cluster.

Kubernetes does not depend on a cloud: only the cluster’s credentials do. So KubernetesService is implemented once, by KubectlKubernetesService, with kubectl, and the cloud provider contributes only the kubeconfig, with update_kubeconfig(), e.g. aws eks update-kubeconfig for AWS EKS.

The platform uses it through smarter.apps.infrastructure.services.infrastructure .kubernetes. Tests replace it with configure_kubernetes().

from smarter.apps.infrastructure.services import infrastructure

infrastructure.kubernetes.apply_manifest(manifest)
class smarter.apps.infrastructure.services.kubernetes.InMemoryKubernetesService(ready=True, provider_name='memory', **kwargs)[source]

Bases: KubernetesService

A Kubernetes cluster in memory, for tests and local development.

Its resources are items: lists of resources, as kubectl get -o json returns them, keyed by kind, e.g. {"pod": [...], "node": [...]}. A kind that is not a key cannot be listed: find_resources() returns None for it, as for a cluster that is unavailable, or a CustomResourceDefinition that is not installed.

Parameters:
  • ready (bool) – Whether the cluster is ready.

  • provider_name (str) – The provider’s name, in signals and logs.

__init__(ready=True, provider_name='memory', **kwargs)[source]
apply_manifest(manifest)[source]

Record a manifest in manifests, without creating its resources.

Parameters:

manifest (str) – The manifest.

Return type:

None

calls

The (kind, namespace) of every find_resources() and list_resources() call, in order.

delete_resource(kind, name, namespace)[source]

Remove a resource from items.

Parameters:
  • kind (str) – The kind.

  • name (str) – The resource’s name.

  • namespace (str) – The resource’s namespace.

Return type:

bool

Returns:

True, also for a resource that does not exist, as Kubernetes’ delete is idempotent.

delete_resources(kinds, namespace, selector)[source]

Remove the resources of several kinds that match a label selector from items.

Parameters:
  • kinds (list[str]) – The kinds, e.g. ["ingress", "service"].

  • namespace (str) – The namespace.

  • selector (str) – An equality label selector (see matches_selector()).

Return type:

bool

Returns:

True.

find_resources(kind, namespace=None)[source]

Return every resource of a kind, and record the call.

Parameters:
  • kind (str) – The kind, e.g. node.

  • namespace (str | None) – The namespace of a namespaced kind, or None for every namespace.

Return type:

list[dict[str, Any]] | None

Returns:

The resources, or None if the cluster is not ready, or the kind is not a key of items, as for a cluster that cannot be read.

get_pod_logs(namespace, selector, container=None, tail=200)[source]

Return no log lines: the in-memory cluster runs no pods.

Parameters:
  • namespace (str) – Ignored.

  • selector (str) – Ignored.

  • container (str | None) – Ignored.

  • tail (int) – Ignored.

Return type:

str | None

Returns:

An empty string.

get_resource(kind, name, namespace)[source]

Find one resource of items by its kind, name and namespace.

Parameters:
  • kind (str) – The kind, e.g. ingress.

  • name (str) – The resource’s name.

  • namespace (str) – The resource’s namespace.

Return type:

dict[str, Any] | None

Returns:

The resource, or None if there is none.

list_resources(kind, namespace, selector=None)[source]

Return the resources of a kind in a namespace that match a label selector, and record the call.

Parameters:
  • kind (str) – The kind, e.g. pod.

  • namespace (str) – The namespace.

  • selector (str | None) – An equality label selector, or None for every resource (see matches_selector()).

Return type:

list[dict[str, Any]]

Returns:

The resources, or an empty list if the cluster is not ready, or has none.

manifests

Every manifest that apply_manifest() received, in order.

property ready: bool

Whether the cluster is ready, as set in the constructor.

class smarter.apps.infrastructure.services.kubernetes.KubectlKubernetesService(provider=None, allow_in_tests=False, **kwargs)[source]

Bases: KubernetesService

The platform’s Kubernetes cluster, through kubectl.

The cluster is ready once kubectl is configured, with the provider’s update_kubeconfig(), and the environment’s namespace, smarter_settings.environment_namespace, exists.

In a pod (see in_cluster_server()), kubectl authenticates as the pod’s ServiceAccount instead, so Smarter has exactly the permissions of the Helm chart’s Role and ClusterRole, and needs neither a kubeconfig nor the provider’s token.

Parameters:
  • provider (CloudProvider | None) – The cloud provider that writes the kubeconfig. None for a cluster whose kubeconfig is already in place.

  • allow_in_tests (bool) – Allow kubectl in the unit tests, e.g. when subprocess is mocked.

__init__(provider=None, allow_in_tests=False, **kwargs)[source]
apply_manifest(manifest)[source]

Create or update the resources of a manifest, with kubectl apply.

Resources that provision billable cloud resources, see billable_resources(), are announced with the billable resource signals. Nothing is applied if the cluster is not ready.

Parameters:

manifest (str) – The manifest, which may have several YAML documents.

Raises:

KubernetesServiceError – If the cluster rejects it.

Return type:

None

property configured: bool

Whether kubectl is configured for the cluster.

delete_resource(kind, name, namespace)[source]

Delete a resource with kubectl delete.

A resource that does not exist counts as deleted.

Sends the destroying and destroyed resource signals, which record it in the ledger, billable if its kind is in BILLABLE_KUBERNETES_KINDS.

Parameters:
  • kind (str) – The kind, e.g. ingress.

  • name (str) – The resource’s name.

  • namespace (str) – The namespace.

Return type:

bool

Returns:

True if it was deleted, False if kubectl failed or the cluster is not ready.

delete_resources(kinds, namespace, selector)[source]

Delete the resources of several kinds that match a label selector, with one kubectl delete.

Sends the destroying and destroyed resource signals for each billable kind.

Parameters:
  • kinds (list[str]) – The kinds, e.g. ["statefulset", "service"].

  • namespace (str) – The namespace.

  • selector (str) – A label selector, which is required, so that a namespace is never emptied by mistake.

Return type:

bool

Returns:

True if they were deleted, False if kubectl failed or the cluster is not ready.

Raises:

KubernetesServiceError – If kinds or selector is empty.

find_resources(kind, namespace=None)[source]

Return every resource of a kind with kubectl get, for the inventory.

Parameters:
  • kind (str) – The kind, e.g. node.

  • namespace (str | None) – The namespace of a namespaced kind, or None for a cluster-scoped kind.

Return type:

list[dict[str, Any]] | None

Returns:

The resources, or None if they could not be listed, e.g. because kubectl failed, its output is not a list, or the cluster is not ready.

get_pod_logs(namespace, selector, container=None, tail=200)[source]

Return the most recent log lines of the pods that match a label selector, with kubectl logs.

Parameters:
  • namespace (str) – The namespace.

  • selector (str) – A label selector.

  • container (str | None) – The container, or None for the pod’s default.

  • tail (int) – How many lines of each pod.

Return type:

str | None

Returns:

The logs, each line prefixed with its pod. If kubectl fails, its output and error, which explain why, e.g. a pod that is still pulling its image. None if the cluster is not ready.

get_resource(kind, name, namespace)[source]

Return a resource with kubectl get.

Parameters:
  • kind (str) – The kind, e.g. ingress.

  • name (str) – The resource’s name.

  • namespace (str) – The namespace.

Return type:

dict[str, Any] | None

Returns:

The resource, or None if it does not exist, kubectl fails, or the cluster is not ready.

get_resources(resources, namespace)[source]

Return several resources with one kubectl get.

Each kubectl call fetches a cluster token, e.g. with aws eks get-token, which takes seconds, so one call rather than several frees a Celery worker sooner. If the call fails, e.g. because a kind’s CustomResourceDefinition is not installed, each resource is fetched in turn.

Return type:

dict[tuple[str, str], dict[str, Any]]

property kubeconfig: dict[str, Any]

The platform’s kubeconfig file.

property kubeconfig_path: str

The path of the platform’s kubeconfig file.

list_resources(kind, namespace, selector=None)[source]

Return the resources of a kind with kubectl get, optionally those that match a label selector.

Parameters:
  • kind (str) – The kind, e.g. pods.

  • namespace (str) – The namespace.

  • selector (str | None) – A label selector, e.g. app=qdrant, or None for all.

Return type:

list[dict[str, Any]]

Returns:

The resources, or an empty list if there are none, kubectl fails, or the cluster is not ready.

property namespace_verified: bool

Whether the environment’s namespace exists.

property ready: bool

Whether kubectl is configured for the cluster, and the environment’s namespace exists.

Records the connection state, which sends a signal when it changes.

verify_namespace(namespace)[source]

Whether a namespace exists.

Return type:

bool

class smarter.apps.infrastructure.services.kubernetes.KubernetesService(provider_name, *args, **kwargs)[source]

Bases: InfrastructureService

The platform’s Kubernetes cluster.

Implementations provide the primitives: apply_manifest(), get_resource(), list_resources(), delete_resource(), delete_resources() and get_pod_logs(), and optionally find_resources(), for the inventory. The ingress operations that LLMClient deployments use are built on them.

abstractmethod apply_manifest(manifest)[source]

Create or update the resources of a manifest.

Raises:

KubernetesServiceError – If the cluster rejects them.

Return type:

None

certificate_wait_seconds: float = 60

Seconds between the checks of a cert-manager certificate, in verify_ingress_resources().

delete_certificate(name, namespace)[source]

Delete a cert-manager Certificate.

Parameters:
  • name (str) – The Certificate’s name.

  • namespace (str) – The Certificate’s namespace.

Return type:

bool

Returns:

True if it was deleted, or did not exist.

delete_ingress(name, namespace)[source]

Delete an Ingress.

Parameters:
  • name (str) – The Ingress’s name.

  • namespace (str) – The Ingress’s namespace.

Return type:

bool

Returns:

True if it was deleted, or did not exist.

delete_ingress_resources(hostname, namespace)[source]

Delete a host’s Ingress, its cert-manager Certificate, and its TLS Secret.

Return type:

tuple[bool, bool, bool]

Returns:

Whether the Ingress, the Certificate, and the Secret were deleted.

abstractmethod delete_resource(kind, name, namespace)[source]

Delete a resource by name.

A resource that does not exist counts as deleted.

Return type:

bool

abstractmethod delete_resources(kinds, namespace, selector)[source]

Delete the resources of several kinds that match a label selector.

Idempotent. A selector is required, so that a namespace is never emptied by mistake.

Return type:

bool

delete_secret(name, namespace)[source]

Delete a Secret.

Parameters:
  • name (str) – The Secret’s name.

  • namespace (str) – The Secret’s namespace.

Return type:

bool

Returns:

True if it was deleted, or did not exist.

error_class

alias of KubernetesServiceError

find_resources(kind, namespace=None)[source]

Return every resource of a kind, for the inventory.

Unlike list_resources(), it tells a kind that has no resources from one that could not be listed, so that the inventory never mistakes an unavailable cluster for an empty one. This implementation cannot list, and returns None.

Parameters:
  • kind (str) – The kind, e.g. node.

  • namespace (str | None) – The namespace of a namespaced kind, or None for a cluster-scoped kind, e.g. a node.

Return type:

list[dict[str, Any]] | None

Returns:

The resources, or None if they could not be listed, e.g. because the cluster is unavailable, or the kind’s CustomResourceDefinition is not installed.

abstractmethod get_pod_logs(namespace, selector, container=None, tail=200)[source]

Return the most recent log lines of the pods that match a label selector.

Parameters:
  • namespace (str) – The namespace.

  • selector (str) – A label selector.

  • container (str | None) – The container, or None for the pod’s default.

  • tail (int) – How many lines of each pod.

Return type:

str | None

Returns:

The logs, or None if they could not be read.

abstractmethod get_resource(kind, name, namespace)[source]

Return a resource, or None if it does not exist or the cluster is unavailable.

Parameters:
  • kind (str) – The kind, e.g. ingress.

  • name (str) – The resource’s name.

  • namespace (str) – The namespace.

Return type:

dict[str, Any] | None

Returns:

The resource, as kubectl get -o json returns it, or None.

get_resources(resources, namespace)[source]

Return several resources at once.

Implementations may override it to fetch them in one request. This one gets each in turn.

Parameters:
  • resources (list[tuple[str, str]]) – The (kind, name) of each resource, with a lowercase kind, e.g. ("ingress", "app").

  • namespace (str) – The namespace.

Return type:

dict[tuple[str, str], dict[str, Any]]

Returns:

The resources that exist, keyed by their (kind, name).

abstractmethod list_resources(kind, namespace, selector=None)[source]

Return the resources of a kind, optionally those that match a label selector.

Parameters:
  • kind (str) – The kind, e.g. pods.

  • namespace (str) – The namespace.

  • selector (str | None) – A label selector, e.g. app=qdrant, or None for all.

Return type:

list[dict[str, Any]]

Returns:

The resources, or an empty list.

service_name: str = 'kubernetes'

The name of the service in signals and logs, see InfrastructureServiceNames.

verify_certificate(name, namespace)[source]

Whether a cert-manager Certificate exists, and is Ready, i.e. issued.

Parameters:
  • name (str) – The Certificate’s name.

  • namespace (str) – The Certificate’s namespace.

Return type:

bool

verify_ingress(name, namespace)[source]

Whether an Ingress exists.

Parameters:
  • name (str) – The Ingress’s name.

  • namespace (str) – The Ingress’s namespace.

Return type:

bool

Returns:

True if it exists.

verify_ingress_resources(hostname, namespace, max_attempts=30)[source]

Verify that a host’s Ingress, its cert-manager Certificate, and its TLS Secret exist.

The Ingress is named after the host, and the Certificate and Secret <host>-tls.

Parameters:
  • hostname (str) – The host, e.g. example.3141-5926-5359.api.example.com.

  • namespace (str) – The namespace.

  • max_attempts (int) – How many times to check the Certificate, a minute apart. A Celery task passes 1, and checks again later, so that it does not block its worker.

Return type:

tuple[bool, bool, bool]

Returns:

Whether the Ingress, the Certificate, and the Secret are verified.

verify_secret(name, namespace)[source]

Whether a Secret exists.

Parameters:
  • name (str) – The Secret’s name.

  • namespace (str) – The Secret’s namespace.

Return type:

bool

Returns:

True if it exists.

smarter.apps.infrastructure.services.kubernetes.billable_resources(manifest)[source]

Return the resources of a manifest that provision billable cloud resources.

  • a PersistentVolumeClaim provisions a block storage volume.

  • a StatefulSet’s volumeClaimTemplates provision one volume per replica.

  • a Service of type LoadBalancer provisions a cloud load balancer.

Parameters:

manifest (str) – A Kubernetes manifest, which may have several YAML documents.

Return type:

list[tuple[str, str]]

Returns:

The (kind, name) of each, e.g. ("persistentvolumeclaim", "data").

smarter.apps.infrastructure.services.kubernetes.configure_kubernetes(factory)[source]

Set the factory of the Kubernetes service that get_kubernetes() returns.

Parameters:

factory (Callable[[], KubernetesService] | None) – Returns the service, or None to restore the default, KubectlKubernetesService with the configured cloud provider.

Return type:

None

smarter.apps.infrastructure.services.kubernetes.get_kubernetes()[source]

Return the Kubernetes service.

It is created once, and again if the cloud provider is reconfigured, so that it keeps its readiness, rather than configuring kubectl for each call.

Return type:

KubernetesService

smarter.apps.infrastructure.services.kubernetes.in_cluster_server()[source]

Return the Kubernetes api server’s url, if Smarter runs in a pod that has a ServiceAccount token.

Kubernetes sets KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT in every pod, and mounts the token in SERVICE_ACCOUNT_DIRECTORY, unless automountServiceAccountToken is false.

Return type:

str | None

Returns:

The url, or None outside a cluster, or without a token.

smarter.apps.infrastructure.services.kubernetes.manifest_kinds(manifest)[source]

Return the kinds of a manifest’s resources, e.g. ["Ingress"].

Return type:

list[str]

smarter.apps.infrastructure.services.kubernetes.matches_selector(item, selector)[source]

Decide whether a resource’s labels match a label selector, as kubectl get -l does.

Only equality requirements are supported, e.g. app.kubernetes.io/name=smarter,tier=web, which are all that the platform uses. Every requirement must match.

Parameters:
  • item (dict[str, Any]) – The resource, as kubectl get -o json returns it.

  • selector (str | None) – The selector, or None or an empty string, which every resource matches.

Return type:

bool

Returns:

True if every requirement of the selector matches one of the resource’s labels.

smarter.apps.infrastructure.services.kubernetes.redacted_manifest(manifest)[source]

Return a manifest that can be logged: the values of its Secrets replaced with <redacted>.

A Secret’s data and stringData hold credentials, e.g. an LLMHost’s provider API key, in plain text or base64.

Parameters:

manifest (str) – A Kubernetes manifest, which may have several YAML documents.

Return type:

str

Returns:

The manifest, with each Secret value redacted, or a placeholder if it isn’t valid YAML.