Vectorstore Services

The service layer of vectorstores: their lifecycle, their documents, search, and maintenance.

VectorstoreService binds a VectorstoreMeta to its backend and embeddings model. The manifest broker, the REST API, the Celery tasks and the management commands all go through it.

Lifecycle

deploy(): create the database. A self-hosted Qdrant server takes a minute or two to start, so the collection is created by reconcile() once it is reachable.
reconcile(): Celery Beat runs it every few minutes. It creates a missing index or collection, and records the status and statistics.
undeploy(): stop serving. A self-hosted server’s volume, and so its data, is kept, and a managed index is left as it is.
destroy(): delete the database and its data, unless deletionProtection is enabled.

Documents are added with add_document(), which extracts and stores their text, and loaded by load_document(), normally in a Celery task.

Maintenance: maintain(), run hourly by Celery Beat, takes snapshots on the manifest’s schedule, prunes them to its retention, and retries documents whose task was lost.

class smarter.apps.vectorstore.service.VectorstoreService(vectorstore, backend=None, embeddings=None)[source]

Bases: object

The operations of one vectorstore.

Parameters:

vectorstore (VectorstoreMeta) – The vectorstore.
backend (SmarterVectorstoreBackend | None) – Its backend, e.g. one with an in-memory Qdrant client in tests. By default, get_backend().
embeddings (Embeddings | None) – Its embeddings model, e.g. a deterministic fake in tests. By default, get_embeddings(), created when first needed.

__init__(vectorstore, backend=None, embeddings=None)[source]

add_document(name, data=None, text=None, content_type=None, source=VectorstoreDocumentSource.UPLOAD, metadata=None)[source]

Extract and store a document’s text.

It is loaded by load_document().

Parameters:: data (bytes | None) – The file’s bytes, e.g. a PDF. Or text.
Return type:: tuple[VectorstoreDocument, bool]
Returns:: the document, and whether it is new. A document with the same text is not added again.
Raises:: VectorstoreExtractionError – if its text cannot be extracted.

property backend: SmarterVectorstoreBackend

chunks(document)[source]

A document’s chunks, and their metadata: the document’s, and where the chunk came from.

Return type:: tuple[list[str], list[dict[str, Any]]]

delete_document(document)[source]

Remove a document’s chunks from the database, then the document.

Return type:: None

deploy()[source]

Create the database.

A self-hosted server is created now, and its collection once it is reachable.

Raises:: VectorstoreServiceError – if the vectorstore is inactive.
Return type:: None

destroy()[source]

Delete the database and all of its data.

Raises:: VectorstoreServiceError – if deletionProtection is enabled.
Return type:: None

property embeddings: Embeddings

ensure_api_key()[source]

The Secret with a self-hosted database’s API key, created with a random key the first time.

Return type:: Secret

load_document(document)[source]

Split, embed and load a document’s chunks, replacing any that it had.

Return type:: int
Returns:: the number of chunks.

maintain(now=None)[source]

The scheduled maintenance of a ready database.

Refresh its status and statistics, take a snapshot if one is due, prune old snapshots, and return the documents whose task was lost, so that the caller can queue them again.

Return type:: dict[str, Any]
Returns:: what was done.

prune_snapshots(retention=None)[source]

Delete the oldest snapshots beyond the retention.

Failed snapshots are removed too.

Return type:: int

reconcile()[source]

Bring a deployed database to ready: create its index or collection once it is reachable.

It records the status, and the database’s statistics. It never raises.

Return type:: str
Returns:: the status.

refresh_stats()[source]

Return type:: dict[str, Any]

require_ready()[source]

Return type:: None

restore(snapshot)[source]

Replace the database’s data with a snapshot’s.

Documents are not changed: those loaded after the snapshot are no longer in the database.

Return type:: None

search(query, k=4, search_type='similarity', score_threshold=None, fetch_k=None, lambda_mult=None, metadata_filter=None)[source]

Find the chunks closest in meaning to a query.

See SmarterVectorstoreBackend.search().

Return type:: list[SearchResult]

set_status(status, message='')[source]

Record the vectorstore’s status, and send vectorstore_status_changed if it changed.

Return type:: None

snapshot(scheduled=False)[source]

Take a Qdrant snapshot, or a Pinecone backup.

Return type:: VectorstoreSnapshot

snapshot_due(now=None)[source]

Return type:: bool

stuck_documents(now=None)[source]: Documents whose Celery task was lost: pending, loading or deleting for a while.

text_splitter()[source]

Return type:: RecursiveCharacterTextSplitter

undeploy()[source]

Stop serving.

A self-hosted server is deleted, but its volume and data are kept.

Return type:: None

exception smarter.apps.vectorstore.service.VectorstoreServiceError(message='')[source]

Bases: SmarterException

A vectorstore operation is not possible, e.g. destroying a protected database.