Inventory

The inventory: what exists in the platform’s infrastructure, whoever created it.

The ledger, InfrastructureResource, records what the services create and destroy, as they do it. Much of the platform’s infrastructure is not created that way: the Kubernetes cluster, its add-ons and node groups are provisioned outside the platform, e.g. with Terraform, nodes come and go with the cluster’s autoscaler, and cert-manager issues certificates. sync_inventory() discovers them, see InventorySource, and reconciles the ledger. Besides the cluster itself, i.e. the cluster, its add-ons, node groups and nodes, it discovers only the environment’s resources, those of smarter_settings.environment_namespace, because the cluster is shared by every environment:

a resource that the ledger does not know is recorded as created,
an active resource that no longer exists is recorded as destroyed,

with the resource signals of smarter.apps.infrastructure.signals, so that the ledger is still written only by their receivers. A resource type that cannot be listed, e.g. because the cluster is unavailable, is left as it is, rather than recorded as destroyed.

The Celery Beat task smarter.apps.infrastructure.tasks.sync_infrastructure_inventory() runs it every few minutes, and manage.py sync_infrastructure_inventory on demand.

class smarter.apps.infrastructure.services.inventory.InventorySource(resource_type, kind, namespaced, billable, resource_id, include=<function InventorySource.<lambda>>)[source]

Bases: object

A kind of Kubernetes resource that the inventory lists, and how it is recorded.

Only the environment’s resources are recorded, i.e. those of smarter_settings.environment_namespace: the cluster is shared by every environment.

Parameters:

resource_type (str) – The ledger’s resource type.
kind (str) – The Kubernetes kind, e.g. node.
namespaced (bool) – Whether to list the environment’s namespace, rather than the cluster. Resources are named as the Kubernetes service records them, e.g. when it deletes an Ingress, so that the two agree.
billable (bool) – Whether the cloud bills for the resource.
resource_id (Callable[[dict[str, Any]], str]) – The resource’s cloud id.
include (Callable[[dict[str, Any]], bool]) – Whether a resource is recorded, e.g. only Services of type LoadBalancer, or only the PersistentVolumes of the environment’s claims.

__init__(resource_type, kind, namespaced, billable, resource_id, include=<function InventorySource.<lambda>>)

billable: bool

find(kubernetes)[source]

The resources of this source that exist, or None if they could not be listed.

Return type:: list[DiscoveredResource] | None

include()

kind: str

namespaced: bool

resource_id: Callable[[dict[str, Any]], str]

resource_type: str

smarter.apps.infrastructure.services.inventory.KUBERNETES_SOURCES: list[InventorySource] = [InventorySource(resource_type='kubernetes.node', kind='node', namespaced=False, billable=True, resource_id=<function <lambda>>, include=<function InventorySource.<lambda>>), InventorySource(resource_type='kubernetes.persistentvolume', kind='persistentvolume', namespaced=False, billable=True, resource_id=<function _volume_id>, include=<function _claimed_by_environment>), InventorySource(resource_type='kubernetes.loadbalancer', kind='service', namespaced=True, billable=True, resource_id=<function _load_balancer_hostname>, include=<function <lambda>>), InventorySource(resource_type='kubernetes.ingress', kind='ingress', namespaced=True, billable=False, resource_id=<function _load_balancer_hostname>, include=<function InventorySource.<lambda>>), InventorySource(resource_type='kubernetes.certificate', kind='certificate', namespaced=True, billable=False, resource_id=<function <lambda>>, include=<function InventorySource.<lambda>>)]

The Kubernetes resources that the inventory lists.

[
    InventorySource(resource_type='kubernetes.node', kind='node', namespaced=False, billable=True, resource_id=<function <lambda> at 0x131fbdfe0>, include=<function InventorySource.<lambda> at 0x131fbe090>),
    InventorySource(resource_type='kubernetes.persistentvolume', kind='persistentvolume', namespaced=False, billable=True, resource_id=<function _volume_id at 0x131fbddd0>, include=<function _claimed_by_environment at 0x131fbdf30>),
    InventorySource(resource_type='kubernetes.loadbalancer', kind='service', namespaced=True, billable=True, resource_id=<function _load_balancer_hostname at 0x131fbdc70>, include=<function <lambda> at 0x131fbe400>),
    InventorySource(resource_type='kubernetes.ingress', kind='ingress', namespaced=True, billable=False, resource_id=<function _load_balancer_hostname at 0x131fbdc70>, include=<function InventorySource.<lambda> at 0x131fbe090>),
    InventorySource(resource_type='kubernetes.certificate', kind='certificate', namespaced=True, billable=False, resource_id=<function <lambda> at 0x131fbe770>, include=<function InventorySource.<lambda> at 0x131fbe090>),
]

smarter.apps.infrastructure.services.inventory.discover()[source]

Discover the platform’s Kubernetes cluster: its cloud resources, and its own.

Return type:: dict[str, list[DiscoveredResource]]
Returns:: The resources, by resource type. A type that could not be listed is left out.

smarter.apps.infrastructure.services.inventory.reconcile(service, resource_type, resources)[source]

Reconcile the ledger’s active resources of a type with those that exist.

Parameters:

service (KubernetesService) – The service that sends the resource signals, whose provider names the ledger’s resources.
resource_type (str) – The resource type, e.g. kubernetes.node.
resources (list[DiscoveredResource]) – The resources of the type that exist.

Return type:

dict[str, int]

Returns:

The number of resources that exist, and of those recorded as created and destroyed.

smarter.apps.infrastructure.services.inventory.sync_inventory()[source]

Discover the platform’s Kubernetes cluster, and reconcile the ledger with it.

Return type:: dict[str, dict[str, int]]
Returns:: For each resource type that could be listed, the number of resources that exist, and of those recorded as created and destroyed.

Tasks

Celery tasks of the infrastructure app.

sync_infrastructure_inventory(): Celery Beat, every few minutes. Discover the platform’s Kubernetes cluster, i.e. the cluster, its add-ons, node groups, nodes, volumes, load balancers, ingresses and certificates, and reconcile the ledger with it. See smarter.apps.infrastructure.services.inventory.