Const

Constants for the llmhost app.

smarter.apps.llmhost.const.API_KEY_SECRET_NAME = 'llmhost_{name}_api_key': The name of the Smarter Secret in which Smarter stores a generated API key.

smarter.apps.llmhost.const.BUILTIN_COMPUTE_PATH = '/Users/mcdaniel/Desktop/gh/smarter-sh/smarter/smarter/smarter/apps/llmhost/data/compute'

the kinds of node that the built-in LLMHosts fit.

Type:: The built-in LLMHostCompute

smarter.apps.llmhost.const.BUILTIN_MANIFESTS_PATH = '/Users/mcdaniel/Desktop/gh/smarter-sh/smarter/smarter/smarter/apps/llmhost/data/llmhost': The example LLMHost manifests of the most popular open-weight models.

smarter.apps.llmhost.const.CONTAINER_BIND_ADDRESS = '0.0.0.0'

The address that inference servers listen on, inside their container.

Every interface of the pod, so that its Service can reach it. What can reach the pod is decided by the Service, which is inside the cluster, and the optional Ingress, not by this address.

smarter.apps.llmhost.const.DEFAULT_CPU_IMAGES = {'llama_cpp': 'ghcr.io/ggml-org/llama.cpp:server', 'tei': 'ghcr.io/huggingface/text-embeddings-inference:cpu-latest'}

The images of engines that have a separate build for CPU-only nodes.

{
    'llama_cpp': 'ghcr.io/ggml-org/llama.cpp:server',
    'tei': 'ghcr.io/huggingface/text-embeddings-inference:cpu-latest',
}

smarter.apps.llmhost.const.LABEL_COMPUTE = 'smarter.sh/compute': The node label of an LLMHostCompute’s node group, and its LLMHosts’ pod label and node selector.

smarter.apps.llmhost.const.MANAGED_KINDS = ['deployment', 'service', 'ingress', 'secret']

The kinds of an LLMHost’s Kubernetes resources that undeploy deletes.

The model volume is kept, unless purged.

['deployment', 'service', 'ingress', 'secret']

smarter.apps.llmhost.const.MAX_RESOURCE_NAME_LENGTH = 50: Leaves room for the suffixes that Kubernetes adds to pod names, within 63 characters.

smarter.apps.llmhost.const.MODELS_MOUNT_PATH = '/models'

Where the model volume is mounted.

Engines cache downloaded weights here.

smarter.apps.llmhost.const.RECONCILE_INTERVAL_SECONDS = 30: How often a compute is reconciled while its node group is adding or removing nodes.

smarter.apps.llmhost.const.RECONCILE_MAX_ATTEMPTS = 60

30 minutes.

Type:: How many times a compute is reconciled before Celery Beat’s reconcile takes over

smarter.apps.llmhost.const.RESOURCE_PREFIX = 'llmhost'

llmhost-<id>-<name>.

Type:: The prefix of the names of an LLMHost’s Kubernetes resources

smarter.apps.llmhost.const.SHM_MOUNT_PATH = '/dev/shm'

Where the shared-memory volume, a memory-backed emptyDir, is mounted.

PyTorch-based engines exchange tensors between processes through /dev/shm, and a container’s default 64 MB is too small.

smarter.apps.llmhost.const.STATUS_REFRESH_MINUTES = 5: How often Celery Beat refreshes the status of deployed LLMHosts, and reconciles every compute.

smarter.apps.llmhost.const.TAINT_COMPUTE = 'smarter.sh/compute': The taint of a CPU node group’s nodes, so that only LLMHosts are scheduled on them.

smarter.apps.llmhost.const.TAINT_GPU = 'nvidia.com/gpu'

The taint of a GPU node group’s nodes.

The NVIDIA device plugin, which a GPU node needs to advertise its GPUs, tolerates it, and pods that do not request GPUs do not.