Vectorstore Manifest

Constants for Vectorstore manifest models.

Smarter API Manifest - Vectorstore.metadata

class smarter.apps.vectorstore.manifest.models.vectorstore.metadata.SAMVectorstoreMetadata(*, name: str, description: str | None, version: str | None, tags: List[str] | None = <factory>, annotations: List[dict[str, str | int | float | bool | ~datetime.date | ~datetime.datetime | ~decimal.Decimal | ~uuid.UUID | bytes | list | dict]] | None=<factory>)[source]

Bases: AbstractSAMMetadataBase

Smarter API Vectorstore Manifest - Metadata class.

class_identifier: ClassVar[str] = 'Vectorstore.metadata'
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Return type:

None

Parameters:

  • self – The BaseModel instance.

  • context – The context.

Smarter API Manifest - Vectorstore.spec.

A Vectorstore is a vector database for retrieval-augmented generation (RAG): documents are split into chunks, embedded with a Provider’s embeddings model, and loaded into it, and searched by meaning. It is either self-hosted, by Smarter on its Kubernetes cluster, or a managed service reached through an ApiConnection.

spec:
  backend: qdrant            # qdrant or pinecone
  hosting: self_hosted       # self_hosted (qdrant only), or managed
  connection: null           # managed only: an ApiConnection with the service's URL and API key
  isActive: true
  index:
    dimension: 1536          # must be the embeddings model's
    metric: cosine           # cosine, euclidean or dotproduct
    deletionProtection: false
  embeddings:
    provider: openai         # a Provider with an OpenAI-compatible embeddings API
    model: text-embedding-3-small
    chunkSize: 1000          # characters per chunk
    chunkOverlap: 200
  selfHosted:                # self_hosted only
    storage: 10Gi
    cpu: 500m
    memory: 1Gi
  pinecone:                  # pinecone only
    cloud: aws
    region: us-east-1
  maintenance:
    snapshots: true          # Qdrant snapshots, or Pinecone backups, taken by Celery Beat
    snapshotIntervalHours: 24
    snapshotRetention: 7
class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreEmbeddings(*, provider: Annotated[str, MinLen(min_length=1)], model: Annotated[str, MinLen(min_length=1)], dimensions: Annotated[int | None, Ge(ge=1), Le(le=20000)] = None, chunkSize: Annotated[int, Ge(ge=100), Le(le=8000)] = 1000, chunkOverlap: Annotated[int, Ge(ge=0), Le(le=2000)] = 200, batchSize: Annotated[int, Ge(ge=1), Le(le=1000)] = 64)[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.embeddings: how documents and queries are turned into vectors.

batchSize: int
chunkOverlap: int
chunkSize: int
class_identifier: ClassVar[str] = 'Vectorstore.__init__.embeddings'
dimensions: int | None
model: str
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Return type:

None

Parameters:

  • self – The BaseModel instance.

  • context – The context.

provider: str
validate_overlap()[source]
Return type:

SAMVectorstoreEmbeddings

class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreIndex(*, name: Annotated[str | None, MaxLen(max_length=45)] = None, dimension: Annotated[int, Ge(ge=1), Le(le=20000)], metric: Literal['cosine', 'euclidean', 'dotproduct'] = 'cosine', deletionProtection: bool = False)[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.index: the index (Pinecone), or collection (Qdrant), of the database.

class_identifier: ClassVar[str] = 'Vectorstore.__init__.index'
deletionProtection: bool
dimension: int
metric: Literal['cosine', 'euclidean', 'dotproduct']
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Return type:

None

Parameters:

  • self – The BaseModel instance.

  • context – The context.

name: str | None
classmethod validate_name(v)[source]
Return type:

Optional[str]

class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreMaintenance(*, snapshots: bool = True, snapshotIntervalHours: Annotated[int, Ge(ge=1), Le(le=720)] = 24, snapshotRetention: Annotated[int, Ge(ge=1), Le(le=100)] = 7)[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.maintenance: what Celery Beat does, besides checking its status.

class_identifier: ClassVar[str] = 'Vectorstore.__init__.maintenance'
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Return type:

None

Parameters:

  • self – The BaseModel instance.

  • context – The context.

snapshotIntervalHours: int
snapshotRetention: int
snapshots: bool
class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstorePinecone(*, cloud: Literal['aws', 'gcp', 'azure'] = 'aws', region: Annotated[str, MinLen(min_length=1)] = 'us-east-1')[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.pinecone: where Pinecone runs a serverless index.

class_identifier: ClassVar[str] = 'Vectorstore.__init__.pinecone'
cloud: Literal['aws', 'gcp', 'azure']
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Return type:

None

Parameters:

  • self – The BaseModel instance.

  • context – The context.

region: str
class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreSelfHosted(*, image: str = 'qdrant/qdrant:v1.19.1', storage: str = '10Gi', storageClass: str | None = None, cpu: str = '500m', memory: str = '1Gi')[source]

Bases: SmarterBasePydanticModel

Vectorstore.spec.selfHosted: the Qdrant server that Smarter runs on Kubernetes.

class_identifier: ClassVar[str] = 'Vectorstore.__init__.selfHosted'
cpu: str
image: str
memory: str
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Return type:

None

Parameters:

  • self – The BaseModel instance.

  • context – The context.

storage: str
storageClass: str | None
classmethod validate_image(v)[source]
Return type:

str

classmethod validate_quantity(v)[source]
Return type:

str

class smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreSpec(*, backend: ~typing.Literal['qdrant', 'pinecone'], hosting: ~typing.Literal['self_hosted', 'managed'] = 'managed', connection: str | None = None, isActive: bool = True, index: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreIndex, embeddings: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreEmbeddings, selfHosted: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreSelfHosted | None = None, pinecone: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstorePinecone | None = None, maintenance: ~smarter.apps.vectorstore.manifest.models.vectorstore.spec.SAMVectorstoreMaintenance = <factory>)[source]

Bases: AbstractSAMSpecBase

Smarter API Vectorstore Manifest Vectorstore.spec.

backend: Literal['qdrant', 'pinecone']
class_identifier: ClassVar[str] = 'Vectorstore.__init__'
connection: str | None
embeddings: SAMVectorstoreEmbeddings
hosting: Literal['self_hosted', 'managed']
index: SAMVectorstoreIndex
isActive: bool
maintenance: SAMVectorstoreMaintenance
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Return type:

None

Parameters:

  • self – The BaseModel instance.

  • context – The context.

pinecone: SAMVectorstorePinecone | None
selfHosted: SAMVectorstoreSelfHosted | None
validate_hosting()[source]

Validate the combination of backend, hosting, connection, selfHosted and pinecone.

Return type:

SAMVectorstoreSpec

Smarter API Manifest - Vectorstore.status.

class smarter.apps.vectorstore.manifest.models.vectorstore.status.SAMVectorstoreStatus(*, recordLocator: str, created: datetime, modified: datetime, accountNumber: str, username: str, vectorstoreStatus: str, message: str | None = None, indexName: str | None = None, endpoint: str | None = None, apiKeySecret: str | None = None, vectorCount: int = 0, documentCount: int = 0, snapshotCount: int = 0, deployedAt: datetime | None = None, lastCheckedAt: datetime | None = None, lastSnapshotAt: datetime | None = None, lastMaintenanceAt: datetime | None = None)[source]

Bases: AbstractSAMStatusBase

Smarter API Vectorstore Manifest - Status class.

Read only.

accountNumber: str
apiKeySecret: str | None
class_identifier: ClassVar[str] = 'Vectorstore.status'
deployedAt: datetime | None
documentCount: int
endpoint: str | None
indexName: str | None
lastCheckedAt: datetime | None
lastMaintenanceAt: datetime | None
lastSnapshotAt: datetime | None
message: str | None
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Return type:

None

Parameters:

  • self – The BaseModel instance.

  • context – The context.

snapshotCount: int
username: str
vectorCount: int
vectorstoreStatus: str

Smarter API Vectorstore Manifest.

class smarter.apps.vectorstore.manifest.models.vectorstore.model.SAMVectorstore(*, apiVersion: str, kind: str, metadata: SAMVectorstoreMetadata, spec: SAMVectorstoreSpec, status: SAMVectorstoreStatus | None = None)[source]

Bases: AbstractSAMBase

Smarter API Manifest - Vectorstore.

class_identifier: ClassVar[str] = 'Vectorstore'
metadata: SAMVectorstoreMetadata
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'from_attributes': True, 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Return type:

None

Parameters:

  • self – The BaseModel instance.

  • context – The context.

spec: SAMVectorstoreSpec
status: SAMVectorstoreStatus | None

Smarter API Vectorstore Manifest handler.

The broker converts between Vectorstore manifests and the VectorstoreMeta model, and implements the smarter CLI commands for Vectorstores:

  • apply: create or update the Vectorstore. It does not create the database.

  • deploy: create the database: a self-hosted Qdrant server, or a managed index.

  • undeploy: stop serving. A self-hosted server’s data is kept.

  • delete: destroy the database and all of its data, unless deletionProtection is enabled, then delete the Vectorstore.

  • describe, get, logs and example_manifest.

Account admins, i.e. staff, may apply, deploy, undeploy and delete their own Vectorstores. Anyone may describe and get the Vectorstores that are shared with them.

class smarter.apps.vectorstore.manifest.brokers.vectorstore.SAMVectorstoreBroker(*args, **kwargs)[source]

Bases: AbstractBroker

Broker for Vectorstore manifests.

See the module’s documentation.

property ORMMetaModelClass: Type[VectorstoreMeta]

Return the Django ORM meta model class for the broker.

Returns:

The Django ORM meta model class definition for the broker.

Return type:

Type[MetaDataWithOwnershipModel]

property ORMModelClass: Type[VectorstoreMeta]

Return the Django ORM model class for the broker.

Returns:

The Django ORM model class definition for the broker.

Return type:

Type[MetaDataWithOwnershipModel]

property SerializerClass: Type[ModelSerializer]

Return the serializer class for the broker.

Returns:

The serializer class definition for the broker.

Return type:

Type[ModelSerializer]

__init__(*args, **kwargs)[source]
apply(request, *args, **kwargs)[source]

Create or update the Vectorstore.

It does not create its database: deploy does.

Once it is deployed, its backend, hosting, index, and self-hosted storage cannot change, because the database would no longer match. If its embeddings model changes, its loaded documents are loaded again.

Return type:

SmarterJournaledJsonResponse

cache_invalidations()[source]

Handle broker specific cache invalidation logic.

Return type:

None

delete(request, *args, **kwargs)[source]

Destroy the database and its data, unless deletionProtection is enabled, then delete the Vectorstore.

Return type:

SmarterJournaledJsonResponse

deploy(request, *args, **kwargs)[source]

Create the database.

A self-hosted server takes a minute or two to become ready: follow it with describe.

Return type:

SmarterJournaledJsonResponse

describe(request, *args, **kwargs)[source]

The Vectorstore as a manifest, with its status.

Return type:

SmarterJournaledJsonResponse

django_orm_to_manifest_dict()[source]

The Vectorstore as a manifest, with its status.

Return type:

Optional[dict]

example_manifest(request, *args, **kwargs)[source]

An example Vectorstore manifest: a self-hosted Qdrant database for a knowledge base.

Return type:

SmarterJournaledJsonResponse

property formatted_class_name: str

Return the logger prefix for the AbstractBroker.

Returns:

The logger prefix for the AbstractBroker.

Return type:

str

get(request, *args, **kwargs)[source]

The Vectorstores that the user may read, optionally filtered by name.

Return type:

SmarterJournaledJsonResponse

property kind: str

The kind of manifest.

Returns:

The kind of manifest.

Return type:

Optional[str]

logs(request, *args, **kwargs)[source]

A self-hosted Qdrant server’s recent logs.

Return type:

SmarterJournaledJsonResponse

property manifest: SAMVectorstore | None

The Vectorstore manifest, as a Pydantic model, from the manifest loader.

manifest_to_django_orm()[source]

The VectorstoreMeta fields of the manifest.

Return type:

dict[str, Any]

property orm_instance: VectorstoreMeta | None

Return the Django ORM model instance for the broker.

There are multiple strategies to retrieve the ORM instance:

  1. If the instance is already cached in self._orm_instance, return it.

  2. If the broker is not ready or the name is not set, log a warning and return None.

  3. Attempt to retrieve the ORM instance using the user_profile and name. If not found, attempt to retrieve using the admin user_profile for the account. If still not found, attempt to retrieve using the Smarter platform admin user_profile.

  4. Cache the retrieved instance for future access.

Returns:

The Django ORM model instance for the broker.

Return type:

Optional[MetaDataWithOwnershipModel]

property orm_meta_instance: VectorstoreMeta | None

Return the Django ORM meta model instance for the broker.

This is a cached property that retrieves the ORM meta instance based on the user_profile and kind. For simple relational models, the ORM meta class is the same as the ORM class, and the meta instance is the same as the ORM instance.

This property is used for resolving more complex ORM relationships where the name and user_profile fields are stored in a parent Django model.

owned_vectorstore(command)[source]

The Vectorstore, if the user is staff and owns it.

Return type:

VectorstoreMeta

prompt(request, *args, **kwargs)[source]

Invoke a prompt operation.

This abstract method should be implemented by subclasses to provide prompt-based interactions with the broker resource.

Parameters:

  • request (HttpRequest) – The HTTP request object.

  • args – Additional positional arguments.

  • kwargs – Additional keyword arguments.

Returns:

A SmarterJournaledJsonResponse containing the prompt response.

Return type:

SmarterJournaledJsonResponse

property ready: bool

A broker is ready if it has a manifest, or an account.

static reload_documents(vectorstore)[source]

Load the loaded documents again, e.g. with a new embeddings model.

Return type:

int

resolve_connection(name)[source]

Spec.connection: the user’s own ApiConnection, else one shared with them.

Return type:

Optional[ApiConnection]

resolve_provider(name)[source]

Spec.embeddings.provider: the user’s own Provider, else the most recently updated one shared with them.

Return type:

Provider

spec_of(vectorstore)[source]

The Vectorstore’s spec, as it was applied.

Return type:

SAMVectorstoreSpec

undeploy(request, *args, **kwargs)[source]

Stop serving.

A self-hosted server’s data is kept, and a managed index is left as it is.

Return type:

SmarterJournaledJsonResponse

property vectorstore: VectorstoreMeta | None

The user’s own Vectorstore with the broker’s name, else one shared with them.

It is never created here.

exception smarter.apps.vectorstore.manifest.brokers.vectorstore.SAMVectorstoreBrokerError(message=None, thing=None, command=None, stack_trace=None)[source]

Bases: SAMBrokerError

Base exception for Smarter API Vectorstore Broker handling.

property get_formatted_err_message