Vectorstore Manifest

Constants for Vectorstore manifest models.

Smarter API Manifest - Vectorstore.metadata

class smarter.apps.vectorstore.manifest.models.vectorstore.metadata.SAMVectorstoreMetadata(*, name: str, description: str | None, version: str | None, tags: List[str] | None = <factory>, annotations: dict[str, str] | None=<factory>)[source]

Bases: AbstractSAMMetadataBase

Smarter API Vectorstore Manifest - Metadata class.

class_identifier: ClassVar[str] = 'Vectorstore.metadata'
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:
  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

Smarter API Manifest - Vectorstore.spec.

A Vectorstore is a vector database for retrieval-augmented generation (RAG): documents are split into chunks, embedded with a Provider’s embeddings model, and loaded into it, and searched by meaning. It is either self-hosted, by Smarter on its Kubernetes cluster, or a managed service reached through an ApiConnection.

spec:
  backend: qdrant            # qdrant or pinecone
  hosting: self_hosted       # self_hosted (qdrant only), or managed
  connection: null           # managed only: an ApiConnection with the service's URL and API key
  isActive: true
  index:
    dimension: 1536          # must be the embeddings model's
    metric: cosine           # cosine, euclidean or dotproduct
    deletionProtection: false
  embeddings:
    provider: openai         # a Provider with an OpenAI-compatible embeddings API
    model: text-embedding-3-small
    chunkSize: 1000          # characters per chunk
    chunkOverlap: 200
  selfHosted:                # self_hosted only
    storage: 10Gi
    cpu: 500m
    memory: 1Gi
  pinecone:                  # pinecone only
    cloud: aws
    region: us-east-1
  maintenance:
    snapshots: true          # Qdrant snapshots, or Pinecone backups, taken by Celery Beat
    snapshotIntervalHours: 24
    snapshotRetention: 7
class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreEmbeddings(*, provider: Annotated[str, MinLen(min_length=1)], model: Annotated[str, MinLen(min_length=1)], dimensions: Annotated[int | None, Ge(ge=1), Le(le=20000)] = None, chunkSize: Annotated[int, Ge(ge=100), Le(le=8000)] = 1000, chunkOverlap: Annotated[int, Ge(ge=0), Le(le=2000)] = 200, batchSize: Annotated[int, Ge(ge=1), Le(le=1000)] = 64)[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.embeddings: how documents and queries are turned into vectors.

batchSize: int
chunkOverlap: int
chunkSize: int
class_identifier: ClassVar[str] = 'Vectorstore.__init__.embeddings'
dimensions: int | None
model: str
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:
  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

provider: str
validate_overlap()[source]

Validate that chunkOverlap is smaller than chunkSize.

Return type:

SAMVectorstoreEmbeddings

Returns:

The model.

Raises:

SAMValidationError – If it is not.

class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreIndex(*, name: Annotated[str | None, MaxLen(max_length=45)] = None, dimension: Annotated[int, Ge(ge=1), Le(le=20000)], metric: Literal['cosine', 'euclidean', 'dotproduct'] = 'cosine', deletionProtection: bool = False)[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.index: the index (Pinecone), or collection (Qdrant), of the database.

class_identifier: ClassVar[str] = 'Vectorstore.__init__.index'
deletionProtection: bool
dimension: int
metric: Literal['cosine', 'euclidean', 'dotproduct']
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:
  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

name: str | None
classmethod validate_name(v)[source]

Validate the index name: lowercase letters, digits and hyphens.

Parameters:

v (str | None) – The name, or None for the default.

Return type:

str | None

Returns:

The name.

Raises:

SAMValidationError – If it has other characters.

class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreMaintenance(*, snapshots: bool = True, snapshotIntervalHours: Annotated[int, Ge(ge=1), Le(le=720)] = 24, snapshotRetention: Annotated[int, Ge(ge=1), Le(le=100)] = 7)[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.maintenance: what Celery Beat does, besides checking its status.

class_identifier: ClassVar[str] = 'Vectorstore.__init__.maintenance'
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:
  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

snapshotIntervalHours: int
snapshotRetention: int
snapshots: bool
class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstorePinecone(*, cloud: Literal['aws', 'gcp', 'azure'] = 'aws', region: Annotated[str, MinLen(min_length=1)] = 'us-east-1')[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.pinecone: where Pinecone runs a serverless index.

class_identifier: ClassVar[str] = 'Vectorstore.__init__.pinecone'
cloud: Literal['aws', 'gcp', 'azure']
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:
  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

region: str
class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreSelfHosted(*, image: str = 'qdrant/qdrant:v1.19.1', storage: str = '10Gi', storageClass: str | None = None, cpu: str = '500m', memory: str = '1Gi')[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.selfHosted: the Qdrant server that Smarter runs on Kubernetes.

class_identifier: ClassVar[str] = 'Vectorstore.__init__.selfHosted'
cpu: str
image: str
memory: str
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:
  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

storage: str
storageClass: str | None
classmethod validate_image(v)[source]

Validate a container image reference.

Parameters:

v (str) – The image.

Return type:

str

Returns:

The image.

Raises:

SAMValidationError – If it is not a container image reference.

classmethod validate_quantity(v)[source]

Validate a Kubernetes quantity, e.g. 500m, 1Gi or 10Gi.

Parameters:

v (str) – The quantity.

Return type:

str

Returns:

The quantity.

Raises:

SAMValidationError – If it is not a Kubernetes quantity.

class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreSpec(*, backend: ~typing.Literal['qdrant', 'pinecone'], hosting: ~typing.Literal['self_hosted', 'managed'] = 'managed', connection: str | None = None, isActive: bool = True, index: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreIndex, embeddings: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreEmbeddings, selfHosted: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreSelfHosted | None = None, pinecone: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstorePinecone | None = None, maintenance: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreMaintenance = <factory>)[source]

Bases: AbstractSAMSpecBase

Smarter API Vectorstore Manifest Vectorstore.spec.

backend: Literal['qdrant', 'pinecone']
class_identifier: ClassVar[str] = 'Vectorstore.__init__'
connection: str | None
embeddings: SAMVectorstoreEmbeddings
hosting: Literal['self_hosted', 'managed']
index: SAMVectorstoreIndex
isActive: bool
maintenance: SAMVectorstoreMaintenance
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:
  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

pinecone: SAMVectorstorePinecone | None
selfHosted: SAMVectorstoreSelfHosted | None
validate_hosting()[source]

Validate the combination of backend, hosting, connection, selfHosted and pinecone.

Return type:

SAMVectorstoreSpec

Smarter API Manifest - Vectorstore.status.

class smarter.apps.vectorstore.manifest.models.vectorstore.status.SAMVectorstoreStatus(*, recordLocator: str, created: datetime, modified: datetime, dependencies: List[SAMDependency] | None = None, accountNumber: str, username: str, vectorstoreStatus: str, message: str | None = None, indexName: str | None = None, endpoint: str | None = None, apiKeySecret: str | None = None, vectorCount: int = 0, documentCount: int = 0, snapshotCount: int = 0, deployedAt: datetime | None = None, lastCheckedAt: datetime | None = None, lastSnapshotAt: datetime | None = None, lastMaintenanceAt: datetime | None = None)[source]

Bases: AbstractSAMStatusBase

Smarter API Vectorstore Manifest - Status class.

Read only.

accountNumber: str
apiKeySecret: str | None
class_identifier: ClassVar[str] = 'Vectorstore.status'
deployedAt: datetime | None
documentCount: int
endpoint: str | None
indexName: str | None
lastCheckedAt: datetime | None
lastMaintenanceAt: datetime | None
lastSnapshotAt: datetime | None
message: str | None
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:
  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

snapshotCount: int
username: str
vectorCount: int
vectorstoreStatus: str

Smarter API Vectorstore Manifest.

class smarter.apps.vectorstore.manifest.models.vectorstore.model.SAMVectorstore(*, apiVersion: str, kind: str, metadata: SAMVectorstoreMetadata, spec: SAMVectorstoreSpec, status: SAMVectorstoreStatus | None = None)[source]

Bases: AbstractSAMBase

Smarter API Manifest - Vectorstore.

class_identifier: ClassVar[str] = 'Vectorstore'
metadata: SAMVectorstoreMetadata
model_config = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:
  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

spec: SAMVectorstoreSpec
status: SAMVectorstoreStatus | None

Smarter API Vectorstore Manifest handler.

The broker converts between Vectorstore manifests and the VectorstoreMeta model, and implements the smarter CLI commands for Vectorstores:

  • apply: create or update the Vectorstore. It does not create the database.

  • deploy: create the database: a self-hosted Qdrant server, or a managed index.

  • undeploy: stop serving. A self-hosted server’s data is kept.

  • delete: destroy the database and all of its data, unless deletionProtection is enabled, then delete the Vectorstore.

  • describe, get, logs and example_manifest.

Account admins, i.e. staff, may apply, deploy, undeploy and delete their own Vectorstores. Anyone may describe and get the Vectorstores that are shared with them.

class smarter.apps.vectorstore.manifest.brokers.vectorstore.SAMVectorstoreBroker(*args, **kwargs)[source]

Bases: AbstractBroker[VectorstoreMeta, VectorstoreMeta]

Broker for Vectorstore manifests.

See the module’s documentation.

property ORMMetaModelClass: Type[VectorstoreMeta]

The Django model of a vectorstore.

property ORMModelClass: Type[VectorstoreMeta]

The Django model of a vectorstore.

property SerializerClass: Type[ModelSerializer]

The serializer of a vectorstore.

__init__(*args, **kwargs)[source]

Initialize the broker from a manifest, a manifest file, or the Django ORM.

A manifest takes precedence over a file path. With neither, the broker is initialized from the ORM instance named name and owned by the user profile, if there is one.

Parameters:
  • request – The request, or None. It may also be passed positionally in args.

  • args – Positional arguments, among which the request, user, account, user profile and manifest are found by their types.

  • name – The resource’s name.

  • kind – The resource’s SAM kind, e.g. Guardrail.

  • api_version – The manifest’s api version.

  • manifest – The manifest, as a Pydantic model or as a dict, e.g. from parse_manifest().

  • file_path – The path of a manifest file, read when no manifest is passed.

  • url – Accepted, for compatibility, but not used.

  • kwargs – user, account and user_profile, and the rest, which are passed on to SmarterRequestMixin.

Raises:
  • SmarterValueError – If the kind, name or manifest is invalid.

  • SAMBrokerError – If the manifest’s kind is not the broker’s kind.

apply(request, *args, **kwargs)[source]

Create or update the Vectorstore.

It does not create its database: deploy does.

Once it is deployed, its backend, hosting, index, and self-hosted storage cannot change, because the database would no longer match. If its embeddings model changes, its loaded documents are loaded again.

Return type:

SmarterJournaledJsonResponse

cache_invalidations()[source]

Invalidate the owner’s cached vectorstores, and the cached vectorstore.

Return type:

None

delete(request, *args, **kwargs)[source]

Destroy the database and its data, unless deletionProtection is enabled, then delete the Vectorstore.

Return type:

SmarterJournaledJsonResponse

dependencies()[source]

Return brokers for the Vectorsearches that search this Vectorstore.

Returns:

A broker for each Vectorsearch whose spec.vectorstore is this Vectorstore.

Return type:

List[AbstractBroker | SAMDependencyReference]

classmethod dependents(instance)[source]

Return the Vectorsearches that search a Vectorstore.

Parameters:

instance (VectorstoreMeta) – The VectorstoreMeta.

Returns:

The Vectorsearches whose spec.vectorstore is the Vectorstore.

Return type:

List[Tuple[str | None, Iterable[Model]]]

deploy(request, *args, **kwargs)[source]

Create the database.

A self-hosted server takes a minute or two to become ready: follow it with describe.

Return type:

SmarterJournaledJsonResponse

deployable: ClassVar[bool] = True

True when the broker implements deploy().

The cli api’s resources command reports it as each kind’s Deployable, and the smarter cli offers deploy and undeploy commands for those kinds. A broker that sets it must override deploy(); undeploy() may still raise SAMBrokerErrorNotImplemented, as CustomDomain’s and LLMHostCompute’s do.

describe(request, *args, **kwargs)[source]

The Vectorstore as a manifest, with its status.

Return type:

SmarterJournaledJsonResponse

django_orm_to_manifest_dict()[source]

The Vectorstore as a manifest, with its status.

Return type:

dict[str, Any] | None

example_manifest(request, *args, **kwargs)[source]

An example Vectorstore manifest: a self-hosted Qdrant database for a knowledge base.

Return type:

SmarterJournaledJsonResponse

property formatted_class_name: str

Returns a formatted string of the class name for logging purposes.

get(request, *args, **kwargs)[source]

The Vectorstores that the user may read, optionally filtered by name.

Return type:

SmarterJournaledJsonResponse

property kind: str

The manifest kind, Vectorstore.

logs(request, *args, **kwargs)[source]

A self-hosted Qdrant server’s recent logs.

Return type:

SmarterJournaledJsonResponse

property manifest: SAMVectorstore | None

The Vectorstore manifest, as a Pydantic model, from the client’s manifest data.

manifest_to_django_orm()[source]

The VectorstoreMeta fields of the manifest.

Return type:

dict[str, Any]

property orm_instance: VectorstoreMeta | None

The vectorstore, or None if it does not exist.

property orm_meta_instance: VectorstoreMeta | None

The vectorstore, or None if it does not exist.

owned_vectorstore(command)[source]

The Vectorstore, if the user is staff and owns it.

Return type:

VectorstoreMeta

prompt(request, *args, **kwargs)[source]

Not implemented: a Vectorstore cannot be prompted.

Parameters:

request (HttpRequest) – The request.

Raises:

SAMBrokerErrorNotImplemented – Always.

Return type:

SmarterJournaledJsonResponse

property ready: bool

A broker is ready if it has a manifest, or an account.

classmethod references(instance)[source]

Return the resources that a Vectorstore uses: its ApiConnection, its embeddings Provider, and the Secret of its api key.

Parameters:

instance (VectorstoreMeta) – The Vectorstore’s VectorstoreMeta.

Returns:

Its ApiConnection, Provider and Secret, of those that it has.

Return type:

List[Tuple[str | None, Iterable[Model]]]

static reload_documents(vectorstore)[source]

Load the loaded documents again, e.g. with a new embeddings model.

Return type:

int

resolve_connection(name)[source]

Spec.connection: the user’s own ApiConnection, else one shared with them.

Return type:

ApiConnection | None

resolve_provider(name)[source]

Spec.embeddings.provider: the user’s own Provider, else the most recently updated one shared with them.

Return type:

Provider

spec_of(vectorstore)[source]

The Vectorstore’s spec, as it was applied.

Return type:

SAMVectorstoreSpec

undeploy(request, *args, **kwargs)[source]

Stop serving.

A self-hosted server’s data is kept, and a managed index is left as it is.

Return type:

SmarterJournaledJsonResponse

property vectorstore: VectorstoreMeta | None

The user’s own Vectorstore with the broker’s name, else one shared with them.

It is never created here.

exception smarter.apps.vectorstore.manifest.brokers.vectorstore.SAMVectorstoreBrokerError(message=None, thing=None, command=None, stack_trace=None)[source]

Bases: SAMBrokerError

Base exception for Smarter API Vectorstore Broker handling.

property get_formatted_err_message

The error’s title.