SAMLLMHost

smarter.models.sam.llmhost

SAM model of the LLMHost kind, generated from the Smarter platform’s JSON Schema.

Do not edit: run scripts/generate_sam_models.py. See smarter.models.sam.

class smarter.models.sam.llmhost.SAMDependency

Bases: smarter.models.sam.base.SAMModel

A resource that depends on the resource that a manifest describes.

kind: str

Kind[String]: The SAM kind of the resource that depends on this resource, e.g. LLMClient.

Read only.

name: str

Name[String]: The name of the resource that depends on this resource.

Read only.

class smarter.models.sam.llmhost.SAMLLMHostEngineConfig

Bases: smarter.models.sam.base.SAMModel

Spec.engine: the inference server.

The common settings, e.g. contextLength, are translated into each engine’s own arguments, e.g. vLLM’s --max-model-len and llama.cpp’s --ctx-size. args and env are passed to the engine as they are, for everything else.

apiFormat: str | None

LLMHost.spec.engine.apiFormat[str]: the wire protocol.

One of: [‘openai_compatible’, ‘huggingface’, ‘ollama_native’, ‘custom’].

args: list[str] | None

LLMHost.spec.engine.args[list]: additional arguments, passed to the engine as they are.

command: list[str] | None

LLMHost.spec.engine.command[list]: overrides the image’s entrypoint.

For custom engines.

contextLength: Optional

LLMHost.spec.engine.contextLength[int]: the maximum context the server allocates, in tokens.

dtype: str | None

LLMHost.spec.engine.dtype[str]: e.g. auto, bfloat16, float16.

env: dict[str, str] | None

LLMHost.spec.engine.env[dict]: additional environment variables.

gpuMemoryUtilization: Optional

LLMHost.spec.engine.gpuMemoryUtilization[float]: the fraction of GPU memory the server may use.

image: str | None

LLMHost.spec.engine.image[str]: the container image.

Defaults to the engine’s official image. Pin a version in production. Required for custom.

maxConcurrentRequests: Optional

LLMHost.spec.engine.maxConcurrentRequests[int]: the requests that one replica serves at once.

name: str

LLMHost.spec.engine.name[str]: the inference server.

One of: [‘vllm’, ‘tgi’, ‘sglang’, ‘llama_cpp’, ‘ollama’, ‘tei’, ‘custom’].

port: Optional

LLMHost.spec.engine.port[int]: the container port.

Defaults to the engine’s port. Required for custom.

tensorParallelSize: Optional

LLMHost.spec.engine.tensorParallelSize[int]: the GPUs that one replica shards the model across.

trustRemoteCode: bool | None

LLMHost.spec.engine.trustRemoteCode[bool]: run Python code from the model repository.

Only for repositories that you trust.

class smarter.models.sam.llmhost.SAMLLMHostHealthCheck

Bases: smarter.models.sam.base.SAMModel

Spec.healthCheck: the probes that decide when a replica is ready.

path: str | None

LLMHost.spec.healthCheck.path[str]: the HTTP health path.

Defaults to the engine’s, e.g. /health.

periodSeconds: Optional

LLMHost.spec.healthCheck.periodSeconds[int]: how often the probes run.

startupTimeoutSeconds: Optional

LLMHost.spec.healthCheck.startupTimeoutSeconds[int]: the time the server has to download and load the weights before it is restarted.

Large models need more.

class smarter.models.sam.llmhost.SAMLLMHostMetadata

Bases: smarter.models.sam.base.SAMModel

Smarter API LLMHost Manifest - Metadata class.

annotations: None | list[dict[str, str | int | float | bool | datetime.date | datetime.datetime | Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=None, strict=None, min_length=None, max_length=None, pattern='^(?!^[-+.]*$)[+-]?0*\\d*\\.?\\d*$')] | uuid.UUID | bytes | list | dict[str, Any]]]

The manifest annotations.

Used for storing arbitrary metadata as

key-value pairs. Example: [{‘smarter.sh/test-manifest/project-name’: ‘Scooby dooby do’}]. The key should be a valid url-friendly string. The value accepts multi-line string values (YAML block scalars) and various scalar types including str, int, float, bool, datetime.date, datetime.datetime, decimal.Decimal, uuid.UUID, bytes, list, dict.

description: str | None

The description for this resource.

Be brief. Keep it under 255 characters.

name: str

The camelCase name of the manifest resource.

tags: list[str] | None

The tags of the manifest.

Used for generic resource categorization and search. Example: [‘tag1’, ‘tag2’]

version: str | None

The semantic version of the manifest.

Example: 0.1.0

class smarter.models.sam.llmhost.SAMLLMHostModelCapabilities

Bases: smarter.models.sam.base.SAMModel

What the model can do.

Informational, for discovery and for the LLMClients that use it.

functionCalling: bool | None

LLMHost.spec.model.capabilities.functionCalling[bool]: supports tool calls.

Some engines also require args, e.g. vLLM’s –enable-auto-tool-choice and –tool-call-parser.

reasoning: bool | None

LLMHost.spec.model.capabilities.reasoning[bool]: a reasoning model.

streaming: bool | None

LLMHost.spec.model.capabilities.streaming[bool]: streams responses.

vision: bool | None

LLMHost.spec.model.capabilities.vision[bool]: accepts images.

class smarter.models.sam.llmhost.SAMLLMHostNetwork

Bases: smarter.models.sam.base.SAMModel

Spec.network: how the LLMHost is reached.

It is always reachable inside the cluster, at its Service. With ingress, it is also reachable at https://<hostname>, with a TLS certificate from cert-manager. A public endpoint requires an API key, unless allowUnauthenticated.

allowUnauthenticated: bool | None

LLMHost.spec.network.allowUnauthenticated[bool]: permit an Ingress for an engine that does not authenticate requests, e.g. tgi and ollama.

Anyone who knows the hostname can use the model.

apiKeySecret: str | None

LLMHost.spec.network.apiKeySecret[str]: the name of a Smarter Secret with the API key that clients must send as a Bearer token.

If omitted, Smarter generates one when the LLMHost is launched, and stores it in the Secret llmhost_<name>_api_key.

hostname: str | None

LLMHost.spec.network.hostname[str]: the Ingress hostname.

Defaults to <name>.llmhost.<account number>.<environment API domain>.

ingress: bool | None

LLMHost.spec.network.ingress[bool]: create an Ingress.

class smarter.models.sam.llmhost.SAMLLMHostResources

Bases: smarter.models.sam.base.SAMModel

Spec.resources: the compute that each replica, one pod, requests.

These are the pod’s resource requests, not a node’s capacity. They must fit a node of the LLMHost’s compute, which also determines the GPU model.

cpu: str | None

LLMHost.spec.resources.cpu[str]: the pod’s CPU request, e.g. 4 or 3500m.

gpuCount: Optional

LLMHost.spec.resources.gpuCount[int]: the GPUs per replica.

0 runs on CPU.

gpuResource: str | None

LLMHost.spec.resources.gpuResource[str]: the Kubernetes extended resource, e.g. nvidia.com/gpu.

memory: str | None

LLMHost.spec.resources.memory[str]: the pod’s memory request and limit, e.g. 24Gi.

nodeSelector: dict[str, str] | None

LLMHost.spec.resources.nodeSelector[dict]: additional node labels.

serviceAccountName: str | None

LLMHost.spec.resources.serviceAccountName[str]: e.g. for IRSA access to an S3 model source.

shmSize: str | None

LLMHost.spec.resources.shmSize[str]: the size of /dev/shm.

tolerations: list[dict[str, Any]] | None

LLMHost.spec.resources.tolerations[list]: Kubernetes tolerations.

When gpuCount > 0, a toleration of the gpuResource taint is added.

vramRequiredGb: Optional

LLMHost.spec.resources.vramRequiredGb[int]: the GPU memory the model requires.

Informational.

class smarter.models.sam.llmhost.SAMLLMHostScaling

Bases: smarter.models.sam.base.SAMModel

Spec.scaling: the replicas.

replicas: Optional

LLMHost.spec.scaling.replicas[int]: the replicas.

More than one requires storage.accessMode ReadWriteMany or ReadOnlyMany.

class smarter.models.sam.llmhost.SAMLLMHostStatus

Bases: smarter.models.sam.base.SAMModel

Smarter API LLMHost Manifest - Status class.

Read only. Besides ownership, it reports the LLMHost’s lifecycle state, where it can be reached, and its accumulated cost, as of its most recent status check.

accountNumber: str

LLMHost.status.account_number: The account owner of this LLMHost.

Read only.

apiKeySecret: str | None

LLMHost.status.apiKeySecret: the Smarter Secret with the API key, if any.

Read only.

created: datetime

The date in which this resource was created.

Read only.

dependencies: list[smarter.models.sam.llmhost.SAMDependency] | None

Dependencies[List]: The resources that depend on this resource, and that must be deleted, or stop referring to it, before it can be deleted.

Lists only the resources that you can read. Read only.

deployedAt: datetime.datetime | None

LLMHost.status.deployedAt: when it was last launched.

Read only.

endpoint: str | None

LLMHost.status.endpoint: the base URL inside the cluster, e.g. for LLMClients.

Read only.

estimatedCost: Union

LLMHost.status.estimatedCost: uptimeHours times replicas times each replica’s share of its compute’s node price. Read only.

healthy: bool | None

LLMHost.status.healthy: the result of the last health check.

Read only.

hostStatus: str | None

LLMHost.status.hostStatus: the lifecycle state, e.g. pending, active or error.

Read only.

lastCheckedAt: datetime.datetime | None

LLMHost.status.lastCheckedAt: when its status was last checked.

Read only.

message: str | None

LLMHost.status.message: why the LLMHost is in its state, e.g. a container error.

Read only.

modified: datetime

The date in which this resource was most recently changed.

Read only.

publicUrl: str | None

LLMHost.status.publicUrl: the base URL of the Ingress, if any.

Read only.

readyReplicas: int | None

LLMHost.status.readyReplicas: ready.

Read only.

recordLocator: str

RecordLocator[String]: An optional identifier used to locate the resource record associated with this manifest.

Read only.

replicas: int | None

LLMHost.status.replicas: desired.

Read only.

uptimeHours: float | None

LLMHost.status.uptimeHours: hours since deployedAt.

Read only.

username: str

LLMHost.status.username: The Smarter user who created this LLMHost.

Read only.

class smarter.models.sam.llmhost.SAMLLMHostStorage

Bases: smarter.models.sam.base.SAMModel

Spec.storage: the model volume, a PersistentVolumeClaim mounted at /models.

It caches the weights, so that a restarted server does not download them again. The cluster’s storage class provisions its PersistentVolume.

accessMode: str | None

LLMHost.spec.storage.accessMode[str]: One of: [‘ReadWriteOnce’, ‘ReadWriteMany’, ‘ReadOnlyMany’].

existingClaim: str | None

LLMHost.spec.storage.existingClaim[str]: an existing PersistentVolumeClaim to mount, instead of creating one.

Required when model.source is pvc.

retain: bool | None

LLMHost.spec.storage.retain[bool]: keep the volume, and its downloaded weights, on undeploy.

It is always deleted when the LLMHost is deleted.

size: str | None

LLMHost.spec.storage.size[str]: e.g. 50Gi.

storageClass: str | None

LLMHost.spec.storage.storageClass[str]: defaults to the cluster’s default class.

class smarter.models.sam.llmhost.SAMLLMHostModel

Bases: smarter.models.sam.base.SAMModel

Spec.model: what is served, and where its weights come from.

architecture: str | None

LLMHost.spec.model.architecture[str]: e.g. llama, qwen3, gemma3.

capabilities: smarter.models.sam.llmhost.SAMLLMHostModelCapabilities | None

LLMHost.spec.model.capabilities[object]: what the model can do.

contextWindow: Optional

LLMHost.spec.model.contextWindow[int]: the model’s maximum context, in tokens.

embeddingDimensions: Optional

LLMHost.spec.model.embeddingDimensions[int]: for embedding models.

file: str | None

LLMHost.spec.model.file[str]: a single weights file within the repository, e.g. a GGUF file for llama_cpp.

For ollama with a Hugging Face source, the quantization tag, e.g. Q4_K_M.

license: str | None

LLMHost.spec.model.license[str]: e.g. apache-2.0.

parameterCount: Optional

LLMHost.spec.model.parameterCount[int]: the number of parameters.

quantization: str | None

LLMHost.spec.model.quantization[str]: the weights’ precision.

One of: [‘none’, ‘fp32’, ‘fp16’, ‘bf16’, ‘fp8’, ‘mxfp4’, ‘int8’, ‘int4’, ‘gguf’, ‘awq’, ‘gptq’].

repository: str

LLMHost.spec.model.repository[str]: the model, in the source’s terms: a Hugging Face repository id, e.g. Qwen/Qwen3-8B; an Ollama tag, e.g. llama3.1:8b; an S3 prefix, e.g. s3://bucket/models/qwen3; a URL; or, for pvc, a path within the volume.

revision: str | None

LLMHost.spec.model.revision[str]: a Hugging Face branch, tag or commit SHA.

Pin a commit SHA for reproducible deployments. Defaults to main.

servedName: str | None

LLMHost.spec.model.servedName[str]: the model name that clients send in their requests.

Defaults to the LLMHost’s name. Ollama always uses repository.

source: str | None

LLMHost.spec.model.source[str]: where the weights come from.

One of: [‘huggingface’, ‘ollama’, ‘s3’, ‘url’, ‘pvc’].

task: str | None

LLMHost.spec.model.task[str]: what the model does.

One of: [‘text-generation’, ‘embedding’].

tokenSecret: str | None

LLMHost.spec.model.tokenSecret[str]: the name of a Smarter Secret with a Hugging Face access token, for gated models, e.g. Llama and Gemma.

You must also accept the model’s license on huggingface.co.

class smarter.models.sam.llmhost.SAMLLMHostSpec

Bases: smarter.models.sam.base.SAMModel

Smarter API LLMHost Manifest LLMHost.spec.

compute: str | None

LLMHost.spec.compute[str]: the LLMHostCompute that the LLMHost runs on, e.g. gpu_l4_1x.

Defaults to the cheapest one that its resources fit.

engine: SAMLLMHostEngineConfig

LLMHost.spec.engine[object]: the inference server.

healthCheck: smarter.models.sam.llmhost.SAMLLMHostHealthCheck | None

LLMHost.spec.healthCheck[object]: probes.

model: SAMLLMHostModel

LLMHost.spec.model[object]: what is served.

network: smarter.models.sam.llmhost.SAMLLMHostNetwork | None

LLMHost.spec.network[object]: reachability.

resources: smarter.models.sam.llmhost.SAMLLMHostResources | None

LLMHost.spec.resources[object]: compute.

scaling: smarter.models.sam.llmhost.SAMLLMHostScaling | None

LLMHost.spec.scaling[object]: replicas.

storage: smarter.models.sam.llmhost.SAMLLMHostStorage | None

LLMHost.spec.storage[object]: the model volume.

class smarter.models.sam.llmhost.SAMLLMHost

Bases: smarter.models.sam.base.SAMModel

Smarter API Manifest - LLMHost.

apiVersion: str

ApiVersion[String]: Required.

The API version of the AbstractSAMBase.

kind: str

Kind[String]: Required.

The kind of resource described by the manifest.

metadata: SAMLLMHostMetadata

LLMHost.metadata[obj]: Required, the LLMHost metadata.

spec: SAMLLMHostSpec

LLMHost.spec[obj]: Required, the LLMHost specification.

status: smarter.models.sam.llmhost.SAMLLMHostStatus | None

LLMHost.status[obj]: Optional, Read-only.

Stateful status information about the LLMHost.