Cluster Snapshot

A snapshot of the platform’s Kubernetes workload: its pods, the cluster’s nodes, and recent warnings.

The MCPServer’s staff-only platform_status tool (PlatformStatusTool) uses this module to answer help desk questions such as “is Smarter down?” or “why is it slow?”. cluster_snapshot() reads, at the moment it is called:

• the pods of the environment’s namespace, e.g. smarter-platform-prod: whether each one is ready, how often it has restarted, and what is wrong with it, e.g. CrashLoopBackOff, or a last restart that was OOMKilled.

• the nodes of the cluster, which every environment shares: whether each one is ready, and whether it reports a pressure condition, such as MemoryPressure.

• the CPU and memory that each pod and node uses, from metrics-server, which every environment installs. These are the numbers that kubectl top shows. They are compared with the pods’ memory limits and the nodes’ allocatable capacity, to spot what is about to run out.

• the warning events of the namespace, e.g. BackOff or FailedScheduling, of the last minutes.

Everything is read through smarter.apps.infrastructure.services.infrastructure .kubernetes, with find_resources(), which tells a cluster that could not be read from one that has nothing to list. Each read is one kubectl get call, five in all, and nothing is cached: the snapshot is always current. Nothing is written to the cluster, and nothing reaches a cloud monitoring service, such as CloudWatch, which would add cost.

A part that cannot be read, e.g. because metrics-server is not installed, is named in ClusterSnapshot.errors, and the other parts are still returned, so that one failure never hides the rest of the picture.

Kubernetes writes quantities with suffixes: CPU in cores or millicores, e.g. 2 or 250m, and memory in bytes, e.g. 1280Mi or 2Gi. parse_cpu() and parse_memory() turn them into plain numbers, so that they can be added and compared.

Example usage:

from smarter.apps.mcpserver.services.status.cluster import cluster_snapshot
from smarter.common.conf import smarter_settings

snapshot = cluster_snapshot(smarter_settings.environment_namespace, minutes=60)
for pod in snapshot.pods:
    if not pod.ready:
        print(pod.name, pod.problem)

Note

Experimental. The MCPServer was designed and coded by Claude Code (Anthropic’s Claude Opus 5.5), with Lawrence McDaniel as co-author. It is experimental, and will be documented.

class smarter.apps.mcpserver.services.status.cluster.ClusterSnapshot(namespace, pods=<factory>, nodes=<factory>, events=<factory>, errors=<factory>)

Bases: object

The environment’s pods, the cluster’s nodes, and the namespace’s recent warning events, at one moment.

cluster_snapshot() returns it, and PlatformStatusTool reports it. Every list is empty until it is read, and stays empty if it could not be read, in which case errors says so.

__init__(namespace, pods=<factory>, nodes=<factory>, events=<factory>, errors=<factory>)

errors: list[str]

The parts that could not be read, and why, e.g. pod metrics: metrics-server could not be read.

A part that is missing here, and has an empty list, was read, and had nothing in it.

events: list[WarningEvent]

The namespace’s warning events of the period, newest first.

namespace: str

The environment’s Kubernetes namespace, e.g. smarter-platform-prod.

nodes: list[NodeStatus]

The cluster’s nodes, sorted by name.

pods: list[PodStatus]

The namespace’s pods, except finished Job pods, sorted by component, then name.

class smarter.apps.mcpserver.services.status.cluster.NodeStatus(name, ready=False, unschedulable=False, pressures=<factory>, instance_type='', capacity_type='', created=None, cpu=None, memory=None, cpu_allocatable=None, memory_allocatable=None)

Bases: object

The state of one node of the Kubernetes cluster, as platform_status reports it.

A node is one EC2 instance that runs pods. The cluster’s nodes are shared by every environment, so a node’s problem can affect every environment. It is built from the node’s Kubernetes resource by node_status(), and its use of CPU and memory is added from metrics-server by read_nodes().

__init__(name, ready=False, unschedulable=False, pressures=<factory>, instance_type='', capacity_type='', created=None, cpu=None, memory=None, cpu_allocatable=None, memory_allocatable=None)

capacity_type: str = ''

ON_DEMAND or SPOT, or an empty string if the node does not say.

A spot node can be taken back by AWS at two minutes’ notice, which restarts its pods.

Type:

How the node is bought

cpu: float | None = None

The cores that the node uses now, from metrics-server, or None if they could not be read.

cpu_allocatable: float | None = None

The cores that pods may use on the node, i.e. its capacity less what the system reserves.

created: datetime | None = None

When the node joined the cluster, in UTC.

A young node may have replaced one that was taken away.

instance_type: str = ''

The node’s EC2 instance type, e.g. t3.xlarge, from its node.kubernetes.io/instance-type label.

memory: int | None = None

The bytes of memory that the node uses now, from metrics-server, or None if they could not be read.

memory_allocatable: int | None = None

The bytes of memory that pods may use on the node, i.e. its capacity less what the system reserves.

name: str

The node’s name, e.g. ip-10-0-0-1.ec2.internal.

pressures: list[str]

The conditions of NODE_PRESSURES that the node reports as True, e.g. ["MemoryPressure"].

ready: bool = False

its kubelet is healthy, and it can run pods.

Type:

Whether the node’s Ready condition is True

unschedulable: bool = False

it keeps its pods, but no new pod is scheduled on it, e.g. during a drain.

Type:

Whether the node is cordoned

class smarter.apps.mcpserver.services.status.cluster.PodStatus(name, component, phase='', node='', ready_containers=0, containers=0, restarts=0, problem='', started=None, cpu=None, memory=None, cpu_request=None, memory_request=None, memory_limit=None)

Bases: object

The state of one pod of the environment, as platform_status reports it.

It is built from the pod’s Kubernetes resource by pod_status(), and its use of CPU and memory is added from metrics-server by read_pods(). A pod is the running copy of one of the platform’s components, e.g. one of the two smarter app servers.

__init__(name, component, phase='', node='', ready_containers=0, containers=0, restarts=0, problem='', started=None, cpu=None, memory=None, cpu_request=None, memory_request=None, memory_limit=None)

component: str

What the pod runs, e.g. smarter, smarter-worker or mariadb.

See component_of().

containers: int = 0

How many containers the pod has.

cpu: float | None = None

The cores that the pod uses now, from metrics-server, or None if they could not be read.

cpu_request: float | None = None

The cores that the pod’s containers request, which the scheduler reserves for it, or None.

memory: int | None = None

The bytes of memory that the pod uses now, from metrics-server, or None if they could not be read.

memory_limit: int | None = None

The most bytes of memory that the pod’s containers may use, or None if they set no limit.

A container that uses more is killed, and its last state is OOMKilled.

memory_request: int | None = None

The bytes of memory that the pod’s containers request, or None if they request none.

name: str

The pod’s name, e.g. smarter-6f9c7d-a1b2c.

It changes on every deploy and restart.

node: str = ''

The name of the node that the pod runs on, or empty while it is not scheduled.

phase: str = ''

Pending, Running, Succeeded, Failed or Unknown.

Type:

Kubernetes’ phase of the pod

problem: str = ''

What is wrong with the pod, or an empty string if nothing is.

e.g. CrashLoopBackOff, ImagePullBackOff, Pending, or last restart: OOMKilled for a pod that runs now, but was killed for lack of memory before. See container_problem().

property ready: bool

it has containers, and every one of them is ready.

Returns:

False for a pod that has no containers yet, e.g. one that is still being scheduled.

Type:

Whether the pod is ready to serve

ready_containers: int = 0

How many of the pod’s containers are ready to serve.

restarts: int = 0

How often the pod’s containers have restarted since the pod was created, all added up.

started: datetime | None = None

When the pod started, in UTC, or None if it has not started yet.

class smarter.apps.mcpserver.services.status.cluster.WarningEvent(last_seen, object, reason, message, count=1)

Bases: object

A Kubernetes event of type Warning in the environment’s namespace, as platform_status reports it.

Kubernetes records an event when something happens to a resource, and marks the ones that need attention as warnings, e.g. a container that keeps crashing (BackOff), or a pod that no node has room for (FailedScheduling). It is built from the event’s resource by warning_event().

__init__(last_seen, object, reason, message, count=1)

count: int = 1

How many times the event happened.

Kubernetes counts repeats rather than recording each one.

last_seen: datetime | None

When the event last happened, in UTC, or None if the event does not say.

message: str

Kubernetes’ explanation, e.g. Back-off restarting failed container smarter-worker.

object: str

The resource that the event is about, as kind/name, e.g. pod/smarter-worker-5b8d4c-x9y8z.

reason: str

Kubernetes’ short reason, e.g. BackOff, FailedScheduling or Unhealthy.

smarter.apps.mcpserver.services.status.cluster.cluster_snapshot(namespace, minutes=60)

Read the environment’s pods, the cluster’s nodes, and the namespace’s warning events, now.

Each part is read on its own: one that fails is named in the snapshot’s errors, and the others are still read. It never raises for a cluster that cannot be reached.

Side effects: five kubectl get calls through smarter.apps.infrastructure.services.infrastructure .kubernetes (see read_pods(), read_nodes() and read_events()), each of which may first fetch a cluster token from the cloud provider. Nothing is written, and nothing is cached.

Parameters:

• namespace (str) – The environment’s namespace, e.g. smarter_settings.environment_namespace.

• minutes (int) – How far back to read warning events, in minutes.

Return type:

ClusterSnapshot

Returns:

The snapshot.

smarter.apps.mcpserver.services.status.cluster.mebibytes(value)

Format bytes as mebibytes, the unit that the platform’s Helm chart uses for memory.

Parameters:

value (int | None) – The bytes, e.g. from parse_memory(), or None if they are unknown.

Return type:

str

Returns:

The mebibytes, rounded, with Kubernetes’ suffix, e.g. 512Mi, or ? if they are unknown, e.g. because metrics-server could not be read.

smarter.apps.mcpserver.services.status.cluster.parse_cpu(quantity)

Convert a Kubernetes CPU quantity into cores.

Parameters:

quantity (str | None) – The quantity, e.g. a pod’s request of 250m, a node’s allocatable 3920m, or metrics-server’s use of 123456789n.

Return type:

float | None

Returns:

The cores, e.g. 0.25 for 250m. None if the quantity is missing, or its suffix is not one of CPU_UNITS, e.g. a memory quantity such as 1Gi.

smarter.apps.mcpserver.services.status.cluster.parse_memory(quantity)

Convert a Kubernetes memory quantity into bytes.

Parameters:

quantity (str | None) – The quantity, e.g. a pod’s limit of 2Gi, a node’s allocatable 15000Mi, or metrics-server’s use of 123456Ki.

Return type:

int | None

Returns:

The bytes, rounded down, e.g. 2147483648 for 2Gi. None if the quantity is missing, or its suffix is not one of MEMORY_UNITS, e.g. a CPU quantity such as 250m.

smarter.apps.mcpserver.services.status.cluster.percent(used, capacity)

Express a use as a whole percentage of a capacity, e.g. a pod’s memory of its limit.

Parameters:

• used (float | None) – The amount in use, e.g. bytes of memory or cores of CPU.

• capacity (float | None) – The most that may be used, in the same unit, e.g. the pod’s memory limit.

Return type:

int | None

Returns:

The percentage, rounded, which may be above 100. None if either amount is unknown, or the capacity is zero, e.g. for a pod that sets no memory limit.