Vectorstore Manifest
Constants for Vectorstore manifest models.
Smarter API Manifest - Vectorstore.metadata
- class smarter.apps.vectorstore.manifest.models.vectorstore.metadata.SAMVectorstoreMetadata(*, name: str, description: str | None, version: str | None, tags: List[str] | None = <factory>, annotations: dict[str, str] | None=<factory>)[source]
Bases:
AbstractSAMMetadataBaseSmarter API Vectorstore Manifest - Metadata class.
- model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialize private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
Smarter API Manifest - Vectorstore.spec.
A Vectorstore is a vector database for retrieval-augmented generation (RAG): documents are split into chunks, embedded with a Provider’s embeddings model, and loaded into it, and searched by meaning. It is either self-hosted, by Smarter on its Kubernetes cluster, or a managed service reached through an ApiConnection.
spec:
backend: qdrant # qdrant or pinecone
hosting: self_hosted # self_hosted (qdrant only), or managed
connection: null # managed only: an ApiConnection with the service's URL and API key
isActive: true
index:
dimension: 1536 # must be the embeddings model's
metric: cosine # cosine, euclidean or dotproduct
deletionProtection: false
embeddings:
provider: openai # a Provider with an OpenAI-compatible embeddings API
model: text-embedding-3-small
chunkSize: 1000 # characters per chunk
chunkOverlap: 200
selfHosted: # self_hosted only
storage: 10Gi
cpu: 500m
memory: 1Gi
pinecone: # pinecone only
cloud: aws
region: us-east-1
maintenance:
snapshots: true # Qdrant snapshots, or Pinecone backups, taken by Celery Beat
snapshotIntervalHours: 24
snapshotRetention: 7
- class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreEmbeddings(*, provider: Annotated[str, MinLen(min_length=1)], model: Annotated[str, MinLen(min_length=1)], dimensions: Annotated[int | None, Ge(ge=1), Le(le=20000)] = None, chunkSize: Annotated[int, Ge(ge=100), Le(le=8000)] = 1000, chunkOverlap: Annotated[int, Ge(ge=0), Le(le=2000)] = 200, batchSize: Annotated[int, Ge(ge=1), Le(le=1000)] = 64)[source]
Bases:
SmarterBasePydanticModelVectorstore.spec.embeddings: how documents and queries are turned into vectors.
- model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialize private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
- validate_overlap()[source]
Validate that
chunkOverlapis smaller thanchunkSize.- Return type:
- Returns:
The model.
- Raises:
SAMValidationError – If it is not.
- class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreIndex(*, name: Annotated[str | None, MaxLen(max_length=45)] = None, dimension: Annotated[int, Ge(ge=1), Le(le=20000)], metric: Literal['cosine', 'euclidean', 'dotproduct'] = 'cosine', deletionProtection: bool = False)[source]
Bases:
SmarterBasePydanticModelVectorstore.spec.index: the index (Pinecone), or collection (Qdrant), of the database.
- model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialize private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
- class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreMaintenance(*, snapshots: bool = True, snapshotIntervalHours: Annotated[int, Ge(ge=1), Le(le=720)] = 24, snapshotRetention: Annotated[int, Ge(ge=1), Le(le=100)] = 7)[source]
Bases:
SmarterBasePydanticModelVectorstore.spec.maintenance: what Celery Beat does, besides checking its status.
- model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialize private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
- class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstorePinecone(*, cloud: Literal['aws', 'gcp', 'azure'] = 'aws', region: Annotated[str, MinLen(min_length=1)] = 'us-east-1')[source]
Bases:
SmarterBasePydanticModelVectorstore.spec.pinecone: where Pinecone runs a serverless index.
- model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialize private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
- class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreSelfHosted(*, image: str = 'qdrant/qdrant:v1.19.1', storage: str = '10Gi', storageClass: str | None = None, cpu: str = '500m', memory: str = '1Gi')[source]
Bases:
SmarterBasePydanticModelVectorstore.spec.selfHosted: the Qdrant server that Smarter runs on Kubernetes.
- model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialize private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
- classmethod validate_image(v)[source]
Validate a container image reference.
- Parameters:
v (
str) – The image.- Return type:
- Returns:
The image.
- Raises:
SAMValidationError – If it is not a container image reference.
- classmethod validate_quantity(v)[source]
Validate a Kubernetes quantity, e.g.
500m,1Gior10Gi.- Parameters:
v (
str) – The quantity.- Return type:
- Returns:
The quantity.
- Raises:
SAMValidationError – If it is not a Kubernetes quantity.
- class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreSpec(*, backend: ~typing.Literal['qdrant', 'pinecone'], hosting: ~typing.Literal['self_hosted', 'managed'] = 'managed', connection: str | None = None, isActive: bool = True, index: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreIndex, embeddings: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreEmbeddings, selfHosted: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreSelfHosted | None = None, pinecone: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstorePinecone | None = None, maintenance: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreMaintenance = <factory>)[source]
Bases:
AbstractSAMSpecBaseSmarter API Vectorstore Manifest Vectorstore.spec.
- embeddings: SAMVectorstoreEmbeddings
- index: SAMVectorstoreIndex
- maintenance: SAMVectorstoreMaintenance
- model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialize private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
- pinecone: SAMVectorstorePinecone | None
- selfHosted: SAMVectorstoreSelfHosted | None
Smarter API Manifest - Vectorstore.status.
- class smarter.apps.vectorstore.manifest.models.vectorstore.status.SAMVectorstoreStatus(*, recordLocator: str, created: datetime, modified: datetime, dependencies: List[SAMDependency] | None = None, accountNumber: str, username: str, vectorstoreStatus: str, message: str | None = None, indexName: str | None = None, endpoint: str | None = None, apiKeySecret: str | None = None, vectorCount: int = 0, documentCount: int = 0, snapshotCount: int = 0, deployedAt: datetime | None = None, lastCheckedAt: datetime | None = None, lastSnapshotAt: datetime | None = None, lastMaintenanceAt: datetime | None = None)[source]
Bases:
AbstractSAMStatusBaseSmarter API Vectorstore Manifest - Status class.
Read only.
- model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialize private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
Smarter API Vectorstore Manifest.
- class smarter.apps.vectorstore.manifest.models.vectorstore.model.SAMVectorstore(*, apiVersion: str, kind: str, metadata: SAMVectorstoreMetadata, spec: SAMVectorstoreSpec, status: SAMVectorstoreStatus | None = None)[source]
Bases:
AbstractSAMBaseSmarter API Manifest - Vectorstore.
- metadata: SAMVectorstoreMetadata
- model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialize private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
- spec: SAMVectorstoreSpec
- status: SAMVectorstoreStatus | None
Smarter API Vectorstore Manifest handler.
The broker converts between Vectorstore manifests and the
VectorstoreMeta model, and implements the smarter
CLI commands for Vectorstores:
apply: create or update the Vectorstore. It does not create the database.deploy: create the database: a self-hosted Qdrant server, or a managed index.undeploy: stop serving. A self-hosted server’s data is kept.delete: destroy the database and all of its data, unless deletionProtection is enabled, then delete the Vectorstore.describe,get,logsandexample_manifest.
Account admins, i.e. staff, may apply, deploy, undeploy and delete their own Vectorstores. Anyone may describe and get the Vectorstores that are shared with them.
- class smarter.apps.vectorstore.manifest.brokers.vectorstore.SAMVectorstoreBroker(*args, **kwargs)[source]
Bases:
AbstractBroker[VectorstoreMeta,VectorstoreMeta]Broker for Vectorstore manifests.
See the module’s documentation.
- property ORMMetaModelClass: Type[VectorstoreMeta]
The Django model of a vectorstore.
- property ORMModelClass: Type[VectorstoreMeta]
The Django model of a vectorstore.
- __init__(*args, **kwargs)[source]
Initialize the broker from a manifest, a manifest file, or the Django ORM.
A manifest takes precedence over a file path. With neither, the broker is initialized from the ORM instance named
nameand owned by the user profile, if there is one.- Parameters:
request – The request, or None. It may also be passed positionally in
args.args – Positional arguments, among which the request, user, account, user profile and manifest are found by their types.
name – The resource’s name.
kind – The resource’s SAM kind, e.g.
Guardrail.api_version – The manifest’s api version.
manifest – The manifest, as a Pydantic model or as a dict, e.g. from
parse_manifest().file_path – The path of a manifest file, read when no manifest is passed.
url – Accepted, for compatibility, but not used.
kwargs –
user,accountanduser_profile, and the rest, which are passed on toSmarterRequestMixin.
- Raises:
SmarterValueError – If the kind, name or manifest is invalid.
SAMBrokerError – If the manifest’s kind is not the broker’s kind.
- apply(request, *args, **kwargs)[source]
Create or update the Vectorstore.
It does not create its database: deploy does.
Once it is deployed, its backend, hosting, index, and self-hosted storage cannot change, because the database would no longer match. If its embeddings model changes, its loaded documents are loaded again.
- Return type:
- cache_invalidations()[source]
Invalidate the owner’s cached vectorstores, and the cached vectorstore.
- Return type:
- delete(request, *args, **kwargs)[source]
Destroy the database and its data, unless deletionProtection is enabled, then delete the Vectorstore.
- Return type:
- dependencies()[source]
Return brokers for the Vectorsearches that search this Vectorstore.
- Returns:
A broker for each Vectorsearch whose
spec.vectorstoreis this Vectorstore.- Return type:
- deploy(request, *args, **kwargs)[source]
Create the database.
A self-hosted server takes a minute or two to become ready: follow it with describe.
- Return type:
- deployable: ClassVar[bool] = True
True when the broker implements
deploy().The cli api’s
resourcescommand reports it as each kind’sDeployable, and the smarter cli offersdeployandundeploycommands for those kinds. A broker that sets it must overridedeploy();undeploy()may still raiseSAMBrokerErrorNotImplemented, as CustomDomain’s and LLMHostCompute’s do.
- describe(request, *args, **kwargs)[source]
The Vectorstore as a manifest, with its status.
- Return type:
- example_manifest(request, *args, **kwargs)[source]
An example Vectorstore manifest: a self-hosted Qdrant database for a knowledge base.
- Return type:
- property formatted_class_name: str
Returns a formatted string of the class name for logging purposes.
- get(request, *args, **kwargs)[source]
The Vectorstores that the user may read, optionally filtered by name.
- Return type:
- property manifest: SAMVectorstore | None
The Vectorstore manifest, as a Pydantic model, from the client’s manifest data.
- property orm_instance: VectorstoreMeta | None
The vectorstore, or None if it does not exist.
- property orm_meta_instance: VectorstoreMeta | None
The vectorstore, or None if it does not exist.
- prompt(request, *args, **kwargs)[source]
Not implemented: a Vectorstore cannot be prompted.
- Parameters:
request (
HttpRequest) – The request.- Raises:
SAMBrokerErrorNotImplemented – Always.
- Return type:
- classmethod references(instance)[source]
Return the resources that a Vectorstore uses: its ApiConnection, its embeddings Provider, and the Secret of its api key.
- static reload_documents(vectorstore)[source]
Load the loaded documents again, e.g. with a new embeddings model.
- Return type:
- resolve_connection(name)[source]
Spec.connection: the user’s own ApiConnection, else one shared with them.
- Return type:
ApiConnection|None
- resolve_provider(name)[source]
Spec.embeddings.provider: the user’s own Provider, else the most recently updated one shared with them.
- Return type:
- undeploy(request, *args, **kwargs)[source]
Stop serving.
A self-hosted server’s data is kept, and a managed index is left as it is.
- Return type:
- property vectorstore: VectorstoreMeta | None
The user’s own Vectorstore with the broker’s name, else one shared with them.
It is never created here.