Enumerations Classes

Smarter API LLMHost Manifest - enumerated datatypes.

An LLMHost separates what is served from how it is served, so that no single model hub or inference server is baked into the manifest:

• SAMLLMHostModelSource: where the weights come from, e.g. Hugging Face, the Ollama library, S3, a URL, or a pre-populated volume.

• SAMLLMHostEngine: the inference server that loads the weights and exposes an API, e.g. vLLM or llama.cpp.

• SAMLLMHostApiFormat: the wire protocol that the inference server speaks.

class smarter.apps.llmhost.manifest.enum.SAMLLMHostApiFormat(*values)

Bases: SmarterEnumAbstract

The wire protocol that an LLMHost’s inference server speaks.

• openai_compatible: the OpenAI /v1/chat/completions, /v1/completions and /v1/embeddings API. Every built-in engine speaks it.

• huggingface: the Hugging Face /generate API, of TGI.

• ollama_native: Ollama’s /api/chat API.

• custom: anything else.

CUSTOM = 'custom'

HUGGINGFACE = 'huggingface'

OLLAMA_NATIVE = 'ollama_native'

OPENAI_COMPATIBLE = 'openai_compatible'

class smarter.apps.llmhost.manifest.enum.SAMLLMHostEngine(*values)

Bases: SmarterEnumAbstract

The inference server that loads an LLMHost’s weights and exposes an API.

• vllm: vLLM, high-throughput GPU serving.

• tgi: Hugging Face Text Generation Inference.

• sglang: SGLang.

• llama_cpp: llama.cpp’s llama-server, for GGUF models, on CPU or GPU.

• ollama: Ollama, for GGUF models, on CPU or GPU.

• tei: Hugging Face Text Embeddings Inference, for embedding models.

• custom: any other container image, configured with engine.image, engine.port and engine.args.

CUSTOM = 'custom'

LLAMA_CPP = 'llama_cpp'

OLLAMA = 'ollama'

SGLANG = 'sglang'

TEI = 'tei'

TGI = 'tgi'

VLLM = 'vllm'

class smarter.apps.llmhost.manifest.enum.SAMLLMHostEventType(*values)

Bases: SmarterEnumAbstract

The type of an LLMHost lifecycle event.

DESTROYED = 'destroyed'

ERROR = 'error'

LAUNCHED = 'launched'

NODES_SCALED = 'nodes_scaled'

STATUS_CHANGED = 'status_changed'

class smarter.apps.llmhost.manifest.enum.SAMLLMHostModelSource(*values)

Bases: SmarterEnumAbstract

Where an LLMHost’s model weights come from.

• huggingface: a Hugging Face Hub repository, e.g. meta-llama/Llama-3.1-8B-Instruct. The inference engine downloads it, with the optional tokenSecret for gated models.

• ollama: an Ollama library tag, e.g. llama3.1:8b. Requires the ollama engine.

• s3: an S3 prefix, e.g. s3://bucket/models/llama, copied to the model volume by an init container. Credentials come from the pod’s service account (IRSA).

• url: a single file, e.g. a GGUF file, downloaded to the model volume by an init container.

• pvc: weights already on a PersistentVolumeClaim, named by storage.existingClaim. repository is the path within the volume.

HUGGINGFACE = 'huggingface'

OLLAMA = 'ollama'

PVC = 'pvc'

S3 = 's3'

URL = 'url'

classmethod downloaded()

The sources that an init container copies to the model volume.

Return type:

list[str]

classmethod local()

The sources that the engine loads from a path on the model volume.

Return type:

list[str]

class smarter.apps.llmhost.manifest.enum.SAMLLMHostQuantization(*values)

Bases: SmarterEnumAbstract

The numeric precision or compression of an LLMHost’s weights.

It is informational, and is used to estimate the GPU memory that a model requires. It does not change how the engine loads the weights: use engine.dtype or engine.args.

AWQ = 'awq'

BF16 = 'bf16'

FP16 = 'fp16'

FP32 = 'fp32'

FP8 = 'fp8'

GGUF = 'gguf'

GPTQ = 'gptq'

INT4 = 'int4'

INT8 = 'int8'

MXFP4 = 'mxfp4'

NONE = 'none'

classmethod bytes_per_parameter(quantization)

The approximate bytes that each parameter occupies in GPU memory.

Return type:

float

class smarter.apps.llmhost.manifest.enum.SAMLLMHostStatusEnum(*values)

Bases: SmarterEnumAbstract

The lifecycle state of an LLMHost.

• provisioning: Smarter is adding a node to the LLMHost’s compute’s node group, and the pods wait for it to join the cluster.

• pending: deployed, and waiting for the pods to be scheduled.

• downloading: an init container is copying the weights to the model volume.

• deploying: the inference server is starting, downloading or loading the weights.

• active: every replica is ready.

• degraded: some, but not all, replicas are ready, or the health check fails.

• inactive: not deployed, e.g. after undeploy.

• error: the deployment failed, e.g. the image cannot be pulled, or the server crashes.

ACTIVE = 'active'

DEGRADED = 'degraded'

DEPLOYING = 'deploying'

DOWNLOADING = 'downloading'

ERROR = 'error'

INACTIVE = 'inactive'

PENDING = 'pending'

PROVISIONING = 'provisioning'

classmethod deployed()

The states in which the LLMHost’s Kubernetes resources may exist, and so may use nodes.

error is included: e.g. a pod whose image cannot be pulled still holds its node. Status checks correct it to inactive if the resources do not exist.

Return type:

list[str]

class smarter.apps.llmhost.manifest.enum.SAMLLMHostStorageAccessMode(*values)

Bases: SmarterEnumAbstract

The access mode of an LLMHost’s model volume.

READ_ONLY_MANY = 'ReadOnlyMany'

READ_WRITE_MANY = 'ReadWriteMany'

READ_WRITE_ONCE = 'ReadWriteOnce'

class smarter.apps.llmhost.manifest.enum.SAMLLMHostTask(*values)

Bases: SmarterEnumAbstract

What an LLMHost’s model does.

• text-generation: chat and completions.

• embedding: vector embeddings.

EMBEDDING = 'embedding'

TEXT_GENERATION = 'text-generation'