Vectorstore ORM Models

The Vectorstore model: a vector database, its embeddings, and its state.

class smarter.apps.vectorstore.models.vectorstore_meta.VectorstoreBackendKind(*values)

Bases: TextChoices

The vector database products that Smarter supports.

PINECONE = 'pinecone'

QDRANT = 'qdrant'

class smarter.apps.vectorstore.models.vectorstore_meta.VectorstoreHostingKind(*values)

Bases: TextChoices

Who runs the vector database.

MANAGED = 'managed'

SELF_HOSTED = 'self_hosted'

class smarter.apps.vectorstore.models.vectorstore_meta.VectorstoreMeta(*args, **kwargs)

Bases: MetaDataWithOwnershipModel

A vector database, for retrieval-augmented generation (RAG).

It is created by applying a Vectorstore manifest, whose spec is kept in spec. The fields that are queried or displayed are also columns. Its documents are VectorstoreDocument, and its dumps VectorstoreSnapshot.

Parameters:

• id (Unknown) – Primary key: ID

• created_at (Unknown) – Created at

• updated_at (Unknown) – Updated at

• name (Unknown) – Name. Name in camelCase, e.g., ‘apiKey’, no special characters.

• description (Unknown) – Description. A brief description of this resource. Be verbose, but not too verbose.

• version (Unknown) – Version. Semantic version in the format MAJOR.MINOR.PATCH, e.g., ‘1.0.0’.

• annotations (Unknown) – Annotations. Key-value pairs for annotating this resource.

• spec (Unknown) – Spec. The manifest’s spec, as it was applied.

• backend (Unknown) – Backend. The vector database product.

• hosting (Unknown) – Hosting. self_hosted: Smarter runs it on Kubernetes. managed: a service such as Pinecone or Qdrant Cloud.

• is_active (Unknown) – Is active. Whether the vector database may be used.

• dimension (Unknown) – Dimension. The number of dimensions of its vectors.

• metric (Unknown) – Metric

• deletion_protection (Unknown) – Deletion protection. Refuse to destroy the database while it is enabled.

• embeddings_model (Unknown) – Embeddings model. e.g. text-embedding-3-small

• status (Unknown) – Status

• status_message (Unknown) – Status message

• index_name (Unknown) – Index name. The Pinecone index, or Qdrant collection, of the database.

• endpoint_url (Unknown) – Endpoint url. Where the database is reached.

• vector_count (Unknown) – Vector count. The number of vectors, as of last_checked_at.

• stats (Unknown) – Stats. The database’s own statistics.

• deployed_at (Unknown) – Deployed at

• last_checked_at (Unknown) – Last checked at

• last_snapshot_at (Unknown) – Last snapshot at

• last_maintenance_at (Unknown) – Last maintenance at

Relationship fields:

Parameters:

• user_profile (Unknown) – User profile (related name: vectorstoremeta)

• connection (Unknown) – Connection. The ApiConnection of a managed database: its URL, and its API key Secret. (related name: vector_databases)

• embeddings_provider (Unknown) – Embeddings provider. The Provider whose embeddings model turns text into vectors. (related name: vector_databases)

• api_key_secret (Unknown) – Api key secret. The API key of a self-hosted database, generated by Smarter. (related name: vectorstores)

• tags (Unknown) – Tags. Tags for categorizing and organizing this resource. (related name: vectorstoremeta)

• tagged_items (Unknown) – Tagged items (related name: +)

Reverse relationships:

Parameters:

• vectorsearches (Unknown) – All vectorsearches of this Vectorstore (related name of vectorstore)

• documents (Unknown) – All documents of this Vectorstore (related name of vectorstore)

• snapshots (Unknown) – All snapshots of this Vectorstore (related name of vectorstore)

exception DoesNotExist

Bases: ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: MultipleObjectsReturned

exception NotUpdated

Bases: ObjectNotUpdated, DatabaseError

annotations

JSONField

Annotations. Key-value pairs for annotating this resource.

Type:

Type

api_key_secret

ForeignKey to Secret

Api key secret. The API key of a self-hosted database, generated by Smarter. (related name: vectorstores)

Type:

Type

api_key_secret_id

Internal field, use api_key_secret instead.

backend

CharField

Backend. The vector database product.

Choices:

• qdrant

• pinecone

Type:

Type

connection

ForeignKey to ApiConnection

Connection. The ApiConnection of a managed database: its URL, and its API key Secret. (related name: vector_databases)

Type:

Type

connection_id

Internal field, use connection instead.

created_at

DateTimeField

Created at

Timestamp indicating when the model instance was created.

This field is automatically set to the current date and time when the instance is first created. It is indexed in the database for efficient querying.

Type:

Type

default_index_name()

A name for the database’s index or collection that is unique on a shared service.

A managed Pinecone project, or Qdrant Cloud cluster, may be shared by many accounts, so the name includes the account number.

Return type:

str

deletion_protection

BooleanField

Deletion protection. Refuse to destroy the database while it is enabled.

Type:

Type

deployed_at

DateTimeField

Deployed at

Type:

Type

description

TextField

Description. A brief description of this resource. Be verbose, but not too verbose.

Type:

Type

dimension

PositiveIntegerField

Dimension. The number of dimensions of its vectors.

Type:

Type

documents

Reverse ForeignKey from VectorstoreDocument

All documents of this Vectorstore (related name of vectorstore)

Type:

Type

embeddings_model

CharField

Embeddings model. e.g. text-embedding-3-small

Type:

Type

embeddings_provider

ForeignKey to Provider

Embeddings provider. The Provider whose embeddings model turns text into vectors. (related name: vector_databases)

Type:

Type

embeddings_provider_id

Internal field, use embeddings_provider instead.

endpoint_url

CharField

Endpoint url. Where the database is reached.

Type:

Type

get_backend_display(*, field=<django.db.models.CharField: backend>)

Shows the label of the backend. See get_FOO_display() for more information.

classmethod get_cached_object(*args, backend=None, **kwargs)

Retrieve a cached VectorstoreMeta, by name and backend, or by the parent class’s arguments.

Return type:

VectorstoreMeta

classmethod get_cached_vectorstores_for_user(user, invalidate=False)

The vectorstores that the user may read.

Return type:

list[VectorstoreMeta]

get_hosting_display(*, field=<django.db.models.CharField: hosting>)

Shows the label of the hosting. See get_FOO_display() for more information.

get_metric_display(*, field=<django.db.models.CharField: metric>)

Shows the label of the metric. See get_FOO_display() for more information.

get_status_display(*, field=<django.db.models.CharField: status>)

Shows the label of the status. See get_FOO_display() for more information.

hosting

CharField

Hosting. self_hosted: Smarter runs it on Kubernetes. managed: a service such as Pinecone or Qdrant Cloud.

Choices:

• self_hosted

• managed

Type:

Type

id

BigAutoField

Primary key: ID

Type:

Type

index_name

CharField

Index name. The Pinecone index, or Qdrant collection, of the database.

Type:

Type

is_active

BooleanField

Is active. Whether the vector database may be used.

Type:

Type

property is_billable_resource: bool

Indicates whether the model instance is considered a billable resource.

This property can be overridden in subclasses to specify which models are billable. By default, it returns False, indicating that the base TimestampedModel is not billable.

Returns:

True if the instance is billable, False otherwise.

Return type:

bool

property is_deployed: bool

property is_self_hosted: bool

property kubernetes_name: str

The name of a self-hosted database’s Kubernetes resources, e.g. vectorstore-12.

It is derived from the primary key, rather than the name, so that renaming a vectorstore does not lose its server.

last_checked_at

DateTimeField

Last checked at

Type:

Type

last_maintenance_at

DateTimeField

Last maintenance at

Type:

Type

last_snapshot_at

DateTimeField

Last snapshot at

Type:

Type

property maintenance: dict[str, Any]

The manifest’s spec.maintenance.

metric

CharField

Metric

Choices:

• cosine

• euclidean

• dotproduct

Type:

Type

name

CharField

Name. Name in camelCase, e.g., ‘apiKey’, no special characters.

Type:

Type

objects: MetaDataWithOwnershipModelManager = <smarter.apps.account.models.metadata_with_ownership.MetaDataWithOwnershipModelManager object>

snapshots

Reverse ForeignKey from VectorstoreSnapshot

All snapshots of this Vectorstore (related name of vectorstore)

Type:

Type

spec

JSONField

Spec. The manifest’s spec, as it was applied.

Type:

Type

stats

JSONField

Stats. The database’s own statistics.

Type:

Type

status

CharField

Status

Choices:

• pending

• provisioning

• ready

• stopped

• failed

• deleting

Type:

Type

status_message

TextField

Status message

Type:

Type

tagged_items

Reverse GenericRelation from VectorstoreMeta

All + of this tagged item (related name of tagged_items)

Type:

Type

tags = <taggit.managers._TaggableManager object>

updated_at

DateTimeField

Updated at

Timestamp indicating when the model instance was last updated.

This field is automatically updated to the current date and time whenever the instance is saved. It is indexed in the database for efficient querying.

Type:

Type

user_profile

ForeignKey to UserProfile

User profile (related name: vectorstoremeta)

Type:

Type

user_profile_id

Internal field, use user_profile instead.

vector_count

BigIntegerField

Vector count. The number of vectors, as of last_checked_at.

Type:

Type

vectorsearches

Reverse ForeignKey from Vectorsearch

All vectorsearches of this Vectorstore (related name of vectorstore)

Type:

Type

version

CharField

Version. Semantic version in the format MAJOR.MINOR.PATCH, e.g., ‘1.0.0’.

Type:

Type

class smarter.apps.vectorstore.models.vectorstore_meta.VectorstoreMetricKind(*values)

Bases: TextChoices

The distance metric of similarity search.

COSINE = 'cosine'

DOTPRODUCT = 'dotproduct'

EUCLIDEAN = 'euclidean'

class smarter.apps.vectorstore.models.vectorstore_meta.VectorstoreStatus(*values)

Bases: TextChoices

The lifecycle of a vector database.

• pending: applied, but not deployed.

• provisioning: deployed, waiting for the database to be created and to become reachable.

• ready: serving.

• stopped: undeployed. A self-hosted database’s data is kept on its volume.

• failed: see status_message.

• deleting: its database is being destroyed.

DELETING = 'deleting'

FAILED = 'failed'

PENDING = 'pending'

PROVISIONING = 'provisioning'

READY = 'ready'

STOPPED = 'stopped'

The documents of a vectorstore: the text that is split, embedded, and loaded into it.

smarter.apps.vectorstore.models.document.PAGE_BREAK = '\x0c'

Separates the pages of a document’s content, e.g. of a PDF, so that each chunk records its page.

class smarter.apps.vectorstore.models.document.VectorstoreDocument(*args, **kwargs)

Bases: TimestampedModel

A document of a vectorstore, e.g. a PDF.

Its text is extracted when it is added, and kept here, rather than the original file, so that proprietary documents are never written to file storage, and so that a document can be loaded again, e.g. after the embeddings model changes. Its chunks are loaded into the vector database with deterministic ids, chunk_id(), so that they can be replaced and removed.

Parameters:

• id (Unknown) – Primary key: ID

• created_at (Unknown) – Created at

• updated_at (Unknown) – Updated at

• name (Unknown) – Name. e.g. a file name, or a URL.

• source (Unknown) – Source

• content_type (Unknown) – Content type

• content (Unknown) – Content. The extracted text. Pages are separated by form feeds.

• sha256 (Unknown) – Sha256. Of the content, so that a document is not loaded twice.

• size_bytes (Unknown) – Size bytes. Of the original file.

• metadata (Unknown) – Metadata. Stored with each chunk, for filtered search.

• status (Unknown) – Status

• status_message (Unknown) – Status message

• chunk_count (Unknown) – Chunk count. The number of chunks in the vector database.

• loaded_at (Unknown) – Loaded at

Relationship fields:

Parameters:

vectorstore (Unknown) – Vectorstore (related name: documents)

exception DoesNotExist

Bases: ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: MultipleObjectsReturned

exception NotUpdated

Bases: ObjectNotUpdated, DatabaseError

chunk_count

PositiveIntegerField

Chunk count. The number of chunks in the vector database.

Type:

Type

chunk_id(index)

The id of a chunk in the vector database.

Qdrant requires a UUID or an integer, so the id is a UUID derived from the document and the chunk’s index.

Return type:

str

chunk_ids(count=0)

Return type:

list[str]

content

TextField

Content. The extracted text. Pages are separated by form feeds.

Type:

Type

content_type

CharField

Content type

Type:

Type

created_at

DateTimeField

Created at

Timestamp indicating when the model instance was created.

This field is automatically set to the current date and time when the instance is first created. It is indexed in the database for efficient querying.

Type:

Type

get_source_display(*, field=<django.db.models.CharField: source>)

Shows the label of the source. See get_FOO_display() for more information.

get_status_display(*, field=<django.db.models.CharField: status>)

Shows the label of the status. See get_FOO_display() for more information.

id

BigAutoField

Primary key: ID

Type:

Type

loaded_at

DateTimeField

Loaded at

Type:

Type

metadata

JSONField

Metadata. Stored with each chunk, for filtered search.

Type:

Type

name

CharField

Name. e.g. a file name, or a URL.

Type:

Type

objects = <django.db.models.Manager object>

property pages: list[str]

sha256

CharField

Sha256. Of the content, so that a document is not loaded twice.

Type:

Type

size_bytes

PositiveBigIntegerField

Size bytes. Of the original file.

Type:

Type

source

CharField

Source

Choices:

• upload

• url

• text

• file

Type:

Type

status

CharField

Status

Choices:

• pending

• loading

• loaded

• failed

• deleting

Type:

Type

status_message

TextField

Status message

Type:

Type

updated_at

DateTimeField

Updated at

Timestamp indicating when the model instance was last updated.

This field is automatically updated to the current date and time whenever the instance is saved. It is indexed in the database for efficient querying.

Type:

Type

vectorstore

ForeignKey to VectorstoreMeta

Vectorstore (related name: documents)

Type:

Type

vectorstore_id

Internal field, use vectorstore instead.

class smarter.apps.vectorstore.models.document.VectorstoreDocumentSource(*values)

Bases: TextChoices

Where a document came from.

FILE = 'file'

TEXT = 'text'

UPLOAD = 'upload'

URL = 'url'

class smarter.apps.vectorstore.models.document.VectorstoreDocumentStatus(*values)

Bases: TextChoices

The lifecycle of a document.

• pending: its text is stored, and waits to be loaded.

• loading: it is being split, embedded and loaded.

• loaded: its chunks are in the vector database.

• failed: see status_message.

• deleting: its chunks are being removed from the vector database, then it is deleted.

DELETING = 'deleting'

FAILED = 'failed'

LOADED = 'loaded'

LOADING = 'loading'

PENDING = 'pending'

The dumps of a vectorstore: Qdrant snapshots, and Pinecone backups.

class smarter.apps.vectorstore.models.snapshot.VectorstoreSnapshot(*args, **kwargs)

Bases: TimestampedModel

A dump of a vectorstore’s database, kept where the backend keeps it.

A self-hosted Qdrant snapshot is a file on the database’s own volume. A Pinecone backup is kept by Pinecone. Celery Beat takes them on the schedule of the manifest’s spec.maintenance, and prunes the oldest beyond its retention.

Parameters:

• id (Unknown) – Primary key: ID

• created_at (Unknown) – Created at

• updated_at (Unknown) – Updated at

• name (Unknown) – Name. The snapshot’s name, or the backup’s id, in the backend.

• status (Unknown) – Status

• status_message (Unknown) – Status message

• size_bytes (Unknown) – Size bytes

• vector_count (Unknown) – Vector count. At the time of the snapshot.

• scheduled (Unknown) – Scheduled. Taken by Celery Beat, rather than on request.

Relationship fields:

Parameters:

vectorstore (Unknown) – Vectorstore (related name: snapshots)

exception DoesNotExist

Bases: ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: MultipleObjectsReturned

exception NotUpdated

Bases: ObjectNotUpdated, DatabaseError

created_at

DateTimeField

Created at

Timestamp indicating when the model instance was created.

This field is automatically set to the current date and time when the instance is first created. It is indexed in the database for efficient querying.

Type:

Type

get_status_display(*, field=<django.db.models.CharField: status>)

Shows the label of the status. See get_FOO_display() for more information.

id

BigAutoField

Primary key: ID

Type:

Type

name

CharField

Name. The snapshot’s name, or the backup’s id, in the backend.

Type:

Type

objects = <django.db.models.Manager object>

scheduled

BooleanField

Scheduled. Taken by Celery Beat, rather than on request.

Type:

Type

size_bytes

PositiveBigIntegerField

Size bytes

Type:

Type

status

CharField

Status

Choices:

• creating

• ready

• failed

Type:

Type

status_message

TextField

Status message

Type:

Type

updated_at

DateTimeField

Updated at

Timestamp indicating when the model instance was last updated.

This field is automatically updated to the current date and time whenever the instance is saved. It is indexed in the database for efficient querying.

Type:

Type

vector_count

BigIntegerField

Vector count. At the time of the snapshot.

Type:

Type

vectorstore

ForeignKey to VectorstoreMeta

Vectorstore (related name: snapshots)

Type:

Type

vectorstore_id

Internal field, use vectorstore instead.

class smarter.apps.vectorstore.models.snapshot.VectorstoreSnapshotStatus(*values)

Bases: TextChoices

The lifecycle of a snapshot.

CREATING = 'creating'

FAILED = 'failed'

READY = 'ready'