Pydantic Models

Smarter API LLMHost Manifest Constants.

smarter.apps.llmhost.manifest.models.llmhost.const.DEFAULT_SHM_SIZE = '8Gi'

Default size of /dev/shm, which PyTorch-based engines use for inter-process communication.

smarter.apps.llmhost.manifest.models.llmhost.const.DEFAULT_STARTUP_TIMEOUT_SECONDS = 3600

Default time that the inference server has to download and load the weights before it is restarted.

smarter.apps.llmhost.manifest.models.llmhost.const.DEFAULT_STORAGE_SIZE = '50Gi'

Default size of the model volume, which caches the weights across restarts.

smarter.apps.llmhost.manifest.models.llmhost.const.K8S_QUANTITY_PATTERN = '^[0-9]+(\\.[0-9]+)?(m|k|Ki|M|Mi|G|Gi|T|Ti|P|Pi|E|Ei)?$'

A Kubernetes resource quantity, e.g. 500m, 4 or 24Gi.

Smarter API Manifest - LLMHost.metadata.

class smarter.apps.llmhost.manifest.models.llmhost.metadata.SAMLLMHostMetadata(*, name: str, description: str | None, version: str | None, tags: List[str] | None = <factory>, annotations: List[dict[str, str | int | float | bool | ~datetime.date | ~datetime.datetime | ~decimal.Decimal | ~uuid.UUID | bytes | list | dict]] | None=<factory>)[source]

Bases: AbstractSAMMetadataBase

Smarter API LLMHost Manifest - Metadata class.

class_identifier: ClassVar[str] = 'LLMHost.metadata'
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

Smarter API Manifest - LLMHost.spec.

An LLMHost is a large language model that Smarter hosts on Kubernetes. Applying the manifest stores it; smarter deploy launches it, creating a Deployment, Service, PersistentVolumeClaim, and optionally an Ingress and a Secret, in the environment’s namespace; smarter undeploy destroys them.

The spec separates what is served (model) from how it is served (engine), and from where it runs (compute, resources, storage, network), so that a model from any source can be served by any engine that supports that source.

compute names an LLMHostCompute: a kind of node, e.g. g6.2xlarge with one NVIDIA L4, and the node group of those nodes, which Smarter scales as LLMHosts are launched and destroyed. resources is what one replica’s pod requests of that node; it must fit the node.

spec:
  model:
    source: huggingface                     # huggingface, ollama, s3, url or pvc
    repository: meta-llama/Llama-3.1-8B-Instruct
    revision: main
    servedName: llama-3.1-8b-instruct       # the model name of the OpenAI API
    tokenSecret: huggingface_token          # a Smarter Secret, for gated models
    task: text-generation                   # text-generation or embedding
    parameterCount: 8030000000
    contextWindow: 131072
    quantization: bf16
    capabilities:
      functionCalling: true
  engine:
    name: vllm                              # vllm, tgi, sglang, llama_cpp, ollama, tei or custom
    contextLength: 16384
    args:
      - --enable-auto-tool-choice
      - --tool-call-parser=llama3_json
  compute: gpu_a10g_1x                      # an LLMHostCompute: the node, and its node group
  resources:                                # what one replica's pod requests of the node
    gpuCount: 1
    cpu: "4"
    memory: 24Gi
  storage:
    size: 50Gi
  network:
    ingress: true
smarter.apps.llmhost.manifest.models.llmhost.spec.ENGINE_RULES: dict[str, dict[str, Any]] = {'custom': {'apiKey': False, 'sources': frozenset({'huggingface', 'ollama', 'pvc', 's3', 'url'}), 'tasks': None}, 'llama_cpp': {'apiKey': True, 'requiresFile': True, 'sources': frozenset({'huggingface', 'pvc', 's3', 'url'}), 'tasks': None}, 'ollama': {'apiKey': False, 'sources': frozenset({'huggingface', 'ollama'}), 'tasks': None}, 'sglang': {'apiKey': True, 'requiresGpu': True, 'sources': frozenset({'huggingface', 'pvc', 's3', 'url'}), 'tasks': None}, 'tei': {'apiKey': True, 'sources': frozenset({'huggingface', 'pvc', 's3', 'url'}), 'tasks': {'embedding'}}, 'tgi': {'apiKey': False, 'sources': frozenset({'huggingface', 'pvc', 's3', 'url'}), 'tasks': {'text-generation'}}, 'vllm': {'apiKey': True, 'requiresGpu': True, 'sources': frozenset({'huggingface', 'pvc', 's3', 'url'}), 'tasks': None}}

What each engine supports, for validation.

tasks: None means any task.

{
    'custom': {
        'apiKey': False,
        'sources': frozenset(['huggingface', 'ollama', 'pvc', 's3', 'url']),
        'tasks': None,
    },
    'llama_cpp': {
        'apiKey': True,
        'requiresFile': True,
        'sources': frozenset(['huggingface', 'pvc', 's3', 'url']),
        'tasks': None,
    },
    'ollama': {'apiKey': False, 'sources': frozenset(['huggingface', 'ollama']), 'tasks': None},
    'sglang': {
        'apiKey': True,
        'requiresGpu': True,
        'sources': frozenset(['huggingface', 'pvc', 's3', 'url']),
        'tasks': None,
    },
    'tei': {
        'apiKey': True,
        'sources': frozenset(['huggingface', 'pvc', 's3', 'url']),
        'tasks': set(['embedding']),
    },
    'tgi': {
        'apiKey': False,
        'sources': frozenset(['huggingface', 'pvc', 's3', 'url']),
        'tasks': set(['text-generation']),
    },
    'vllm': {
        'apiKey': True,
        'requiresGpu': True,
        'sources': frozenset(['huggingface', 'pvc', 's3', 'url']),
        'tasks': None,
    },
}
class smarter.apps.llmhost.manifest.models.llmhost.spec.SAMLLMHostEngineConfig(*, name: ~typing.Literal['vllm', 'tgi', 'sglang', 'llama_cpp', 'ollama', 'tei', 'custom'], image: str | None = None, apiFormat: ~typing.Literal['openai_compatible', 'huggingface', 'ollama_native', 'custom'] = 'openai_compatible', port: ~typing.Annotated[int | None, ~annotated_types.Ge(ge=1), ~annotated_types.Le(le=65535)] = None, command: list[str] | None = None, contextLength: ~typing.Annotated[int | None, ~annotated_types.Ge(ge=1)] = None, tensorParallelSize: ~typing.Annotated[int | None, ~annotated_types.Ge(ge=1), ~annotated_types.Le(le=16)] = None, gpuMemoryUtilization: ~typing.Annotated[float | None, ~annotated_types.Gt(gt=0.0), ~annotated_types.Le(le=1.0)] = None, dtype: str | None = None, maxConcurrentRequests: ~typing.Annotated[int | None, ~annotated_types.Ge(ge=1)] = None, trustRemoteCode: bool = False, args: list[str] = <factory>, env: dict[str, str] = <factory>)[source]

Bases: LLMHostBaseModel

Spec.engine: the inference server.

The common settings, e.g. contextLength, are translated into each engine’s own arguments, e.g. vLLM’s --max-model-len and llama.cpp’s --ctx-size. args and env are passed to the engine as they are, for everything else.

apiFormat: Literal['openai_compatible', 'huggingface', 'ollama_native', 'custom']
args: list[str]
class_identifier: ClassVar[str] = 'LLMHost.spec.engine'
command: list[str] | None
contextLength: int | None
dtype: str | None
env: dict[str, str]
gpuMemoryUtilization: float | None
image: str | None
maxConcurrentRequests: int | None
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

name: Literal['vllm', 'tgi', 'sglang', 'llama_cpp', 'ollama', 'tei', 'custom']
port: int | None
tensorParallelSize: int | None
trustRemoteCode: bool
classmethod validate_api_format(v)[source]
Return type:

str

classmethod validate_args(v)[source]
Return type:

list[str]

validate_custom()[source]
Return type:

SAMLLMHostEngineConfig

classmethod validate_env(v)[source]
Return type:

dict[str, str]

classmethod validate_name(v)[source]
Return type:

str

class smarter.apps.llmhost.manifest.models.llmhost.spec.SAMLLMHostHealthCheck(*, path: str | None = None, startupTimeoutSeconds: Annotated[int, Ge(ge=60), Le(le=21600)] = 3600, periodSeconds: Annotated[int, Ge(ge=1), Le(le=300)] = 10)[source]

Bases: LLMHostBaseModel

Spec.healthCheck: the probes that decide when a replica is ready.

class_identifier: ClassVar[str] = 'LLMHost.spec.healthCheck'
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

path: str | None
periodSeconds: int
startupTimeoutSeconds: int
classmethod validate_path(v)[source]
Return type:

str | None

class smarter.apps.llmhost.manifest.models.llmhost.spec.SAMLLMHostModel(*, source: Literal['huggingface', 'ollama', 's3', 'url', 'pvc']='huggingface', repository: str, revision: str | None = None, file: str | None = None, servedName: str | None = None, task: Literal['text-generation', 'embedding']='text-generation', tokenSecret: str | None = None, license: str | None = None, architecture: str | None = None, parameterCount: Annotated[int | None, ~annotated_types.Ge(ge=0)] = None, contextWindow: Annotated[int | None, ~annotated_types.Ge(ge=0)] = None, embeddingDimensions: Annotated[int | None, ~annotated_types.Ge(ge=0)] = None, quantization: Literal['none', 'fp32', 'fp16', 'bf16', 'fp8', 'mxfp4', 'int8', 'int4', 'gguf', 'awq', 'gptq']='none', capabilities: SAMLLMHostModelCapabilities = <factory>)[source]

Bases: LLMHostBaseModel

Spec.model: what is served, and where its weights come from.

architecture: str | None
capabilities: SAMLLMHostModelCapabilities
class_identifier: ClassVar[str] = 'LLMHost.spec.model'
contextWindow: int | None
embeddingDimensions: int | None
file: str | None
license: str | None
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

parameterCount: int | None
quantization: Literal['none', 'fp32', 'fp16', 'bf16', 'fp8', 'mxfp4', 'int8', 'int4', 'gguf', 'awq', 'gptq']
repository: str
revision: str | None
servedName: str | None
source: Literal['huggingface', 'ollama', 's3', 'url', 'pvc']
task: Literal['text-generation', 'embedding']
tokenSecret: str | None
classmethod validate_quantization(v)[source]
Return type:

str

classmethod validate_repository(v)[source]
Return type:

str

classmethod validate_source(v)[source]
Return type:

str

validate_source_fields()[source]

Validate repository against the source.

Return type:

SAMLLMHostModel

classmethod validate_task(v)[source]
Return type:

str

classmethod validate_token(v)[source]
Return type:

str | None

class smarter.apps.llmhost.manifest.models.llmhost.spec.SAMLLMHostModelCapabilities(*, streaming: bool = True, functionCalling: bool = False, vision: bool = False, reasoning: bool = False)[source]

Bases: LLMHostBaseModel

What the model can do.

Informational, for discovery and for the LLMClients that use it.

class_identifier: ClassVar[str] = 'LLMHost.spec.model.capabilities'
functionCalling: bool
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

reasoning: bool
streaming: bool
vision: bool
class smarter.apps.llmhost.manifest.models.llmhost.spec.SAMLLMHostNetwork(*, ingress: bool = False, hostname: str | None = None, apiKeySecret: str | None = None, allowUnauthenticated: bool = False)[source]

Bases: LLMHostBaseModel

Spec.network: how the LLMHost is reached.

It is always reachable inside the cluster, at its Service. With ingress, it is also reachable at https://<hostname>, with a TLS certificate from cert-manager. A public endpoint requires an API key, unless allowUnauthenticated.

allowUnauthenticated: bool
apiKeySecret: str | None
class_identifier: ClassVar[str] = 'LLMHost.spec.network'
hostname: str | None
ingress: bool
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

classmethod validate_hostname(v)[source]
Return type:

str | None

class smarter.apps.llmhost.manifest.models.llmhost.spec.SAMLLMHostResources(*, gpuCount: Annotated[int, ~annotated_types.Ge(ge=0), ~annotated_types.Le(le=16)] = 0, gpuResource: str = 'nvidia.com/gpu', vramRequiredGb: Annotated[int | None, ~annotated_types.Ge(ge=0)] = None, cpu: str | None = None, memory: str | None = None, shmSize: str = '8Gi', nodeSelector: dict[str, str]=<factory>, tolerations: list[dict[str, ~typing.Any]]=<factory>, serviceAccountName: str | None = None)[source]

Bases: LLMHostBaseModel

Spec.resources: the compute that each replica, one pod, requests.

These are the pod’s resource requests, not a node’s capacity. They must fit a node of the LLMHost’s compute, which also determines the GPU model.

class_identifier: ClassVar[str] = 'LLMHost.spec.resources'
cpu: str | None
gpuCount: int
gpuResource: str
memory: str | None
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

nodeSelector: dict[str, str]
serviceAccountName: str | None
shmSize: str
tolerations: list[dict[str, Any]]
classmethod validate_quantity(v, info)[source]
Return type:

str | None

vramRequiredGb: int | None
class smarter.apps.llmhost.manifest.models.llmhost.spec.SAMLLMHostScaling(*, replicas: Annotated[int, Ge(ge=1), Le(le=16)] = 1)[source]

Bases: LLMHostBaseModel

Spec.scaling: the replicas.

class_identifier: ClassVar[str] = 'LLMHost.spec.scaling'
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

replicas: int
class smarter.apps.llmhost.manifest.models.llmhost.spec.SAMLLMHostSpec(*, model: SAMLLMHostModel, engine: SAMLLMHostEngineConfig, compute: str | None = None, resources: SAMLLMHostResources = <factory>, storage: SAMLLMHostStorage = <factory>, network: SAMLLMHostNetwork = <factory>, scaling: SAMLLMHostScaling = <factory>, healthCheck: SAMLLMHostHealthCheck = <factory>)[source]

Bases: LLMHostBaseModel

Smarter API LLMHost Manifest LLMHost.spec.

class_identifier: ClassVar[str] = 'LLMHost.spec'
compute: str | None
engine: SAMLLMHostEngineConfig
healthCheck: SAMLLMHostHealthCheck
model: SAMLLMHostModel
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

network: SAMLLMHostNetwork
resources: SAMLLMHostResources
scaling: SAMLLMHostScaling
storage: SAMLLMHostStorage
validate_compatibility()[source]

Validate that the model, engine, storage and network work together.

Return type:

SAMLLMHostSpec

class smarter.apps.llmhost.manifest.models.llmhost.spec.SAMLLMHostStorage(*, size: str = '50Gi', storageClass: str | None = None, accessMode: Literal['ReadWriteOnce', 'ReadWriteMany', 'ReadOnlyMany'] = 'ReadWriteOnce', existingClaim: str | None = None, retain: bool = True)[source]

Bases: LLMHostBaseModel

Spec.storage: the model volume, a PersistentVolumeClaim mounted at /models.

It caches the weights, so that a restarted server does not download them again. The cluster’s storage class provisions its PersistentVolume.

accessMode: Literal['ReadWriteOnce', 'ReadWriteMany', 'ReadOnlyMany']
class_identifier: ClassVar[str] = 'LLMHost.spec.storage'
existingClaim: str | None
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid', 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

retain: bool
size: str
storageClass: str | None
classmethod validate_access_mode(v)[source]
Return type:

str

classmethod validate_size(v)[source]
Return type:

str

smarter.apps.llmhost.manifest.models.llmhost.spec.engine_supports_api_key(engine)[source]

Return whether an engine authenticates requests with an API key.

Return type:

bool

smarter.apps.llmhost.manifest.models.llmhost.spec.engine_supports_task(engine, task)[source]

Return whether an engine can serve a task.

Return type:

bool

Smarter API Manifest - LLMHost.status.

class smarter.apps.llmhost.manifest.models.llmhost.status.SAMLLMHostStatus(*, recordLocator: str, created: datetime, modified: datetime, dependencies: List[SAMDependency] | None = None, accountNumber: str, username: str, hostStatus: str | None = None, message: str | None = None, replicas: int | None = None, readyReplicas: int | None = None, healthy: bool | None = None, endpoint: str | None = None, publicUrl: str | None = None, apiKeySecret: str | None = None, deployedAt: datetime | None = None, lastCheckedAt: datetime | None = None, uptimeHours: float | None = None, estimatedCost: Decimal | None = None)[source]

Bases: AbstractSAMStatusBase

Smarter API LLMHost Manifest - Status class.

Read only. Besides ownership, it reports the LLMHost’s lifecycle state, where it can be reached, and its accumulated cost, as of its most recent status check.

accountNumber: str
apiKeySecret: str | None
class_identifier: ClassVar[str] = 'LLMHost.status'
deployedAt: datetime | None
endpoint: str | None
estimatedCost: Decimal | None
healthy: bool | None
hostStatus: str | None
lastCheckedAt: datetime | None
message: str | None
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

publicUrl: str | None
readyReplicas: int | None
replicas: int | None
uptimeHours: float | None
username: str

Smarter API LLMHost Manifest.

class smarter.apps.llmhost.manifest.models.llmhost.model.SAMLLMHost(*, apiVersion: str, kind: str, metadata: SAMLLMHostMetadata, spec: SAMLLMHostSpec, status: SAMLLMHostStatus | None = None)[source]

Bases: AbstractSAMBase

Smarter API Manifest - LLMHost.

class_identifier: ClassVar[str] = 'LLMHost'
metadata: SAMLLMHostMetadata
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

spec: SAMLLMHostSpec
status: SAMLLMHostStatus | None