Kubernetes Service
The Kubernetes service: resources in the platform’s Kubernetes cluster.
Kubernetes does not depend on a cloud: only the cluster’s credentials do. So
KubernetesService is implemented once, by KubectlKubernetesService, with
kubectl, and the cloud provider contributes
only the kubeconfig, with
update_kubeconfig(), e.g.
aws eks update-kubeconfig for AWS EKS.
The platform uses it through smarter.apps.infrastructure.services.infrastructure
.kubernetes. Tests replace it with configure_kubernetes().
from smarter.apps.infrastructure.services import infrastructure
infrastructure.kubernetes.apply_manifest(manifest)
- class smarter.apps.infrastructure.services.kubernetes.InMemoryKubernetesService(ready=True, provider_name='memory', **kwargs)[source]
Bases:
KubernetesServiceA Kubernetes cluster in memory, for tests and local development.
Its resources are
items: lists of resources, askubectl get -o jsonreturns them, keyed by kind, e.g.{"pod": [...], "node": [...]}. A kind that is not a key cannot be listed:find_resources()returns None for it, as for a cluster that is unavailable, or a CustomResourceDefinition that is not installed.- Parameters:
- calls
The (kind, namespace) of every
find_resources()andlist_resources()call, in order.
- delete_resources(kinds, namespace, selector)[source]
Remove the resources of several kinds that match a label selector from
items.
- find_resources(kind, namespace=None)[source]
Return every resource of a kind, and record the call.
- Parameters:
- Return type:
- Returns:
The resources, or None if the cluster is not ready, or the kind is not a key of
items, as for a cluster that cannot be read.
- get_pod_logs(namespace, selector, container=None, tail=200)[source]
Return no log lines: the in-memory cluster runs no pods.
- get_resource(kind, name, namespace)[source]
Find one resource of
itemsby its kind, name and namespace.
- list_resources(kind, namespace, selector=None)[source]
Return the resources of a kind in a namespace that match a label selector, and record the call.
- manifests
Every manifest that
apply_manifest()received, in order.
- class smarter.apps.infrastructure.services.kubernetes.KubectlKubernetesService(provider=None, allow_in_tests=False, **kwargs)[source]
Bases:
KubernetesServiceThe platform’s Kubernetes cluster, through kubectl.
The cluster is ready once kubectl is configured, with the provider’s
update_kubeconfig(), and the environment’s namespace,smarter_settings.environment_namespace, exists.In a pod (see
in_cluster_server()), kubectl authenticates as the pod’s ServiceAccount instead, so Smarter has exactly the permissions of the Helm chart’s Role and ClusterRole, and needs neither a kubeconfig nor the provider’s token.- Parameters:
provider (
CloudProvider|None) – The cloud provider that writes the kubeconfig. None for a cluster whose kubeconfig is already in place.allow_in_tests (
bool) – Allow kubectl in the unit tests, e.g. when subprocess is mocked.
- apply_manifest(manifest)[source]
Create or update the resources of a manifest, with
kubectl apply.Resources that provision billable cloud resources, see
billable_resources(), are announced with the billable resource signals. Nothing is applied if the cluster is not ready.- Parameters:
manifest (
str) – The manifest, which may have several YAML documents.- Raises:
KubernetesServiceError – If the cluster rejects it.
- Return type:
- delete_resource(kind, name, namespace)[source]
Delete a resource with
kubectl delete.A resource that does not exist counts as deleted.
Sends the destroying and destroyed resource signals, which record it in the ledger, billable if its kind is in
BILLABLE_KUBERNETES_KINDS.
- delete_resources(kinds, namespace, selector)[source]
Delete the resources of several kinds that match a label selector, with one
kubectl delete.Sends the destroying and destroyed resource signals for each billable kind.
- Parameters:
- Return type:
- Returns:
True if they were deleted, False if kubectl failed or the cluster is not ready.
- Raises:
KubernetesServiceError – If
kindsorselectoris empty.
- find_resources(kind, namespace=None)[source]
Return every resource of a kind with
kubectl get, for the inventory.- Parameters:
- Return type:
- Returns:
The resources, or None if they could not be listed, e.g. because kubectl failed, its output is not a list, or the cluster is not ready.
- get_pod_logs(namespace, selector, container=None, tail=200)[source]
Return the most recent log lines of the pods that match a label selector, with
kubectl logs.- Parameters:
- Return type:
- Returns:
The logs, each line prefixed with its pod. If kubectl fails, its output and error, which explain why, e.g. a pod that is still pulling its image. None if the cluster is not ready.
- get_resources(resources, namespace)[source]
Return several resources with one
kubectl get.Each kubectl call fetches a cluster token, e.g. with
aws eks get-token, which takes seconds, so one call rather than several frees a Celery worker sooner. If the call fails, e.g. because a kind’s CustomResourceDefinition is not installed, each resource is fetched in turn.
- list_resources(kind, namespace, selector=None)[source]
Return the resources of a kind with
kubectl get, optionally those that match a label selector.
- class smarter.apps.infrastructure.services.kubernetes.KubernetesService(provider_name, *args, **kwargs)[source]
Bases:
InfrastructureServiceThe platform’s Kubernetes cluster.
Implementations provide the primitives:
apply_manifest(),get_resource(),list_resources(),delete_resource(),delete_resources()andget_pod_logs(), and optionallyfind_resources(), for the inventory. The ingress operations that LLMClient deployments use are built on them.- abstractmethod apply_manifest(manifest)[source]
Create or update the resources of a manifest.
- Raises:
KubernetesServiceError – If the cluster rejects them.
- Return type:
- certificate_wait_seconds: float = 60
Seconds between the checks of a cert-manager certificate, in
verify_ingress_resources().
- delete_ingress_resources(hostname, namespace)[source]
Delete a host’s Ingress, its cert-manager Certificate, and its TLS Secret.
- abstractmethod delete_resource(kind, name, namespace)[source]
Delete a resource by name.
A resource that does not exist counts as deleted.
- Return type:
- abstractmethod delete_resources(kinds, namespace, selector)[source]
Delete the resources of several kinds that match a label selector.
Idempotent. A selector is required, so that a namespace is never emptied by mistake.
- Return type:
- error_class
alias of
KubernetesServiceError
- find_resources(kind, namespace=None)[source]
Return every resource of a kind, for the inventory.
Unlike
list_resources(), it tells a kind that has no resources from one that could not be listed, so that the inventory never mistakes an unavailable cluster for an empty one. This implementation cannot list, and returns None.- Parameters:
- Return type:
- Returns:
The resources, or None if they could not be listed, e.g. because the cluster is unavailable, or the kind’s CustomResourceDefinition is not installed.
- abstractmethod get_pod_logs(namespace, selector, container=None, tail=200)[source]
Return the most recent log lines of the pods that match a label selector.
- abstractmethod get_resource(kind, name, namespace)[source]
Return a resource, or None if it does not exist or the cluster is unavailable.
- get_resources(resources, namespace)[source]
Return several resources at once.
Implementations may override it to fetch them in one request. This one gets each in turn.
- abstractmethod list_resources(kind, namespace, selector=None)[source]
Return the resources of a kind, optionally those that match a label selector.
- service_name: str = 'kubernetes'
The name of the service in signals and logs, see
InfrastructureServiceNames.
- verify_certificate(name, namespace)[source]
Whether a cert-manager Certificate exists, and is Ready, i.e. issued.
- verify_ingress_resources(hostname, namespace, max_attempts=30)[source]
Verify that a host’s Ingress, its cert-manager Certificate, and its TLS Secret exist.
The Ingress is named after the host, and the Certificate and Secret
<host>-tls.- Parameters:
- Return type:
- Returns:
Whether the Ingress, the Certificate, and the Secret are verified.
- smarter.apps.infrastructure.services.kubernetes.billable_resources(manifest)[source]
Return the resources of a manifest that provision billable cloud resources.
a PersistentVolumeClaim provisions a block storage volume.
a StatefulSet’s volumeClaimTemplates provision one volume per replica.
a Service of type LoadBalancer provisions a cloud load balancer.
- smarter.apps.infrastructure.services.kubernetes.configure_kubernetes(factory)[source]
Set the factory of the Kubernetes service that
get_kubernetes()returns.- Parameters:
factory (
Callable[[],KubernetesService] |None) – Returns the service, or None to restore the default,KubectlKubernetesServicewith the configured cloud provider.- Return type:
- smarter.apps.infrastructure.services.kubernetes.get_kubernetes()[source]
Return the Kubernetes service.
It is created once, and again if the cloud provider is reconfigured, so that it keeps its readiness, rather than configuring kubectl for each call.
- Return type:
- smarter.apps.infrastructure.services.kubernetes.in_cluster_server()[source]
Return the Kubernetes api server’s url, if Smarter runs in a pod that has a ServiceAccount token.
Kubernetes sets
KUBERNETES_SERVICE_HOSTandKUBERNETES_SERVICE_PORTin every pod, and mounts the token inSERVICE_ACCOUNT_DIRECTORY, unlessautomountServiceAccountTokenis false.
- smarter.apps.infrastructure.services.kubernetes.manifest_kinds(manifest)[source]
Return the kinds of a manifest’s resources, e.g.
["Ingress"].
- smarter.apps.infrastructure.services.kubernetes.matches_selector(item, selector)[source]
Decide whether a resource’s labels match a label selector, as
kubectl get -ldoes.Only equality requirements are supported, e.g.
app.kubernetes.io/name=smarter,tier=web, which are all that the platform uses. Every requirement must match.