Dashboard Authentication - Download My Data

Download My Data, in the user menu, gives a user a copy of everything that they’ve built on the platform: the manifest of every resource that they own, in every account that they belong to, zipped into one file, and emailed to them as a link that works for 24 hours.

It opens a dialog that first explains what is about to happen: what the zip file holds, that it usually takes a few seconds, where the link is emailed and for how long it works, and that Secrets’ values are left out. Request my data starts an export, and the dialog then follows it, step by step, with an illustration of each one:

  1. Find your resources: every resource that the user owns, from every list of the web console, with each list’s own queryset (smarter.lib.django.listview.listview_registry()), so that the export holds what the user’s lists show them as theirs.

  2. Write their manifests: each resource’s manifest, as smarter describe returns it (smarter.apps.dashboard.resource_detail.manifest_data()), one YAML file per resource, at <account number>/<kind>/<name>.yaml. The dialog counts them as they are written.

  3. Zip them up: the manifests, and a README that says what the zip file holds and how to apply a manifest again, in one zip file. A resource that can’t be described, e.g. because it is broken, is named in the README, with the reason, rather than failing the export.

  4. Store the zip file privately: in the environment’s private storage, infrastructure.storage (see Infrastructure Services), never in its public bucket.

  5. Email you a link: a presigned url to the zip file, which expires after 24 hours. The admin gets no blind copy, because the link is the only key to the file.

A Secret’s value is replaced by a placeholder, because the link travels by email, and an email can be forwarded. Its name, description and other settings are kept, so that its manifest can be applied again once its value is set. Other resources refer to Secrets by name, and their manifests hold no credentials.

The user can close the dialog at any time: the export runs in the Celery task smarter.apps.account.tasks.create_data_export(), and the email comes anyway. Opening the dialog again while an export runs follows it from where it is. A user may start five exports a day. Each one is recorded in a DataExport, which the Django admin lists, read-only, as a record of who exported their data, and when. Private storage deletes each zip file a day after it was saved, and the Celery Beat task smarter.apps.account.tasks.delete_expired_data_exports() deletes any whose link has expired, every six hours.

The dialog calls the console’s account api: GET /api/web/account/data-export/ for what it says before the user starts, POST to start an export, and GET /api/web/account/data-export/<id>/ for an export’s progress. See Web Console Api.

A program does the same with an api key, through the REST api, as the Smarter SDKs do: GET /api/v1/accounts/data-exports/ for the overview, POST to start an export (202 when it starts, 200 with the export that is already in progress, 429 when the user may not start one now), and GET /api/v1/accounts/data-exports/<id>/ for its progress. The link is still emailed to the user, never returned in a response. See The API Contract.

Download My Data: a zip file of the manifest of every resource that a user owns, emailed to them.

The user menu’s Download My Data starts an export with start_data_export(), which queues the Celery task smarter.apps.account.tasks.create_data_export(). The task calls run_data_export(), which takes these steps, and records each one in the export’s DataExport, from which the web console shows its progress:

  1. inventory: the resources that the user owns, in every account that they belong to, from every list of the web console (smarter.lib.django.listview.listview_registry()).

  2. manifests: each resource’s manifest, as smarter describe returns it, one YAML file per resource, at <account number>/<kind>/<name>.yaml. A Secret’s value is redacted, because the zip file is shared by a link in an email.

  3. archive: the manifests, and a README, zipped into one file.

  4. upload: the zip file, saved to the environment’s private storage (infrastructure .storage).

  5. email: a link to the zip file, which expires after DATA_EXPORT_LINK_SECONDS, emailed to the user.

Private storage deletes the zip file a day after it was saved, and delete_expired_data_exports(), which Celery Beat runs, deletes any that are left once their links have expired.

smarter.apps.account.data_export.DATA_EXPORTS_PER_DAY = 5

The most data exports that a user may start in 24 hours.

smarter.apps.account.data_export.DATA_EXPORT_API_MODELS = (<class 'smarter.apps.account.data_export.DataExportOverviewResponse'>, <class 'smarter.apps.account.data_export.DataExportStatusResponse'>)

The models whose JSON Schemas the Smarter api’s OpenAPI document includes.

(
    <class 'smarter.apps.account.data_export.DataExportOverviewResponse'>,
    <class 'smarter.apps.account.data_export.DataExportStatusResponse'>,
)

24 hours.

Type:

How long the emailed link to a data export works

class smarter.apps.account.data_export.DataExportOverviewResponse(**data)[source]

Bases: DataExportModel

What the Download My Data dialog tells the user before they start: where the link goes, and the limits.

classmethod for_user(user)[source]

Return the overview of a user’s data exports.

Parameters:

user (User) – The user.

Return type:

DataExportOverviewResponse

Returns:

The overview.

model_config = {'alias_generator': <function to_camel>, 'extra': 'forbid', 'serialize_by_alias': True, 'validate_by_alias': True, 'validate_by_name': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

exception smarter.apps.account.data_export.DataExportStartRefused[source]

Bases: Exception

A user may not start a data export now, e.g. because they started too many today.

class smarter.apps.account.data_export.DataExportStatusResponse(**data)[source]

Bases: DataExportModel

A data export’s progress, as the web console’s Download My Data dialog reads it.

classmethod from_model(export)[source]

Return the progress of an export.

Parameters:

export (DataExport) – The export.

Return type:

DataExportStatusResponse

Returns:

Its progress.

model_config = {'alias_generator': <function to_camel>, 'extra': 'forbid', 'serialize_by_alias': True, 'validate_by_alias': True, 'validate_by_name': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class smarter.apps.account.data_export.ExportFile(path, content)[source]

Bases: object

A file of the zip file.

__init__(path, content)
content: str

Its text.

path: str

Its path, e.g. 3141-5926-5359/Guardrail/pii_redaction.yaml.

class smarter.apps.account.data_export.ExportResource(kind, instance, account_number)[source]

Bases: object

A resource that a data export includes.

__init__(kind, instance, account_number)
account_number: str

The account number of the resource’s owner, e.g. 3141-5926-5359, its folder in the zip file.

instance: Model

The resource’s Django ORM instance.

kind: str

The resource’s SAM kind, e.g. Guardrail.

property name: str

The resource’s name.

smarter.apps.account.data_export.delete_expired_data_exports()[source]

Delete the zip files of the data exports whose links have expired.

Private storage deletes them a day after they were saved too, so this only catches what that misses. The exports’ rows are kept, without their files, as a record of who exported their data, and when.

Return type:

int

Returns:

How many zip files were deleted.

smarter.apps.account.data_export.latest_data_export(user)[source]

Return the user’s most recent data export.

Parameters:

user (User) – The user.

Return type:

DataExport | None

Returns:

The export, or None if they never started one.

smarter.apps.account.data_export.owned_resources(user)[source]

Return every resource that the user owns, in every account that they belong to.

The resources are found in every list of the web console, with each list’s own queryset, so that the export holds what the user’s lists show them as theirs. A resource that two lists show is included once.

Parameters:

user (User) – The user.

Return type:

list[ExportResource]

Returns:

The resources, by list, the most recently updated first.

smarter.apps.account.data_export.run_data_export(export_id)[source]

Run a data export, step by step, recording each step in its row.

A failure is recorded in the export, with what went wrong, rather than raised, so that the Celery task isn’t retried: the user can start another.

Parameters:

export_id (int) – The export’s id.

Return type:

DataExport

Returns:

The export.

Raises:

DataExport.DoesNotExist – If there is no such export.

smarter.apps.account.data_export.start_data_export(user)[source]

Start a data export for the user, unless one is already in progress.

The Celery task is queued once the export’s row is committed, so that the worker finds it.

Parameters:

user (User) – The user.

Return type:

tuple[DataExport, bool]

Returns:

The export, and True if it was started now, or False if it was already in progress.

Raises:

DataExportStartRefused – If the user has no email address, or started DATA_EXPORTS_PER_DAY exports in the last 24 hours.

A user’s data export: the manifest of every resource that they own, zipped and emailed to them.

The user menu’s Download My Data starts one, and the web console shows its progress, step by step, from this row, which the Celery task smarter.apps.account.tasks.create_data_export() updates as it works. See smarter.apps.account.data_export.

smarter.apps.account.models.data_export.DATA_EXPORT_STALE_AFTER = datetime.timedelta(seconds=900)

An export that has neither finished nor failed after this long is abandoned, e.g. its worker restarted.

A user may then start another.

class smarter.apps.account.models.data_export.DataExport(*args, **kwargs)[source]

Bases: TimestampedModel

A user’s data export.

Parameters:
  • user – The user whose resources are exported, in every account that they belong to.

  • status – One of DataExportStatus.

  • step – The step that it is on, one of DataExportStep.

  • resource_count – How many resources the user owns.

  • manifest_count – How many manifests have been written so far.

  • failed_count – How many resources had no manifest, e.g. because one is broken.

  • size_bytes – The size of the zip file.

  • object_key – The zip file’s key in private storage, until it is deleted.

  • expires_at – When the emailed link stops working.

  • completed_at – When it succeeded or failed.

  • error – Why it failed, or an empty string.

  • id (Unknown) – Primary key: ID

  • created_at (Unknown) – Created at

  • updated_at (Unknown) – Updated at

exception DoesNotExist

Bases: ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: MultipleObjectsReturned

exception NotUpdated

Bases: ObjectNotUpdated, DatabaseError

created_at

DateTimeField

Created at

Timestamp indicating when the model instance was created.

This field is automatically set to the current date and time when the instance is first created. It is indexed in the database for efficient querying.

Type:

Type

id: int

BigAutoField

Primary key: ID

The row’s primary key, which Django adds to every concrete model. Declared, without a value, only so that type checkers know it: Django ignores a bare annotation.

Type:

Type

property is_finished: bool

Whether it succeeded or failed.

property is_in_progress: bool

Whether it is queued or running, and not abandoned.

property is_stale: bool

unfinished for longer than DATA_EXPORT_STALE_AFTER.

Type:

Whether it was abandoned

link_expires_in(now=None)[source]

How long the emailed link still works.

Parameters:

now (datetime | None) – The current time. Defaults to timezone.now().

Return type:

timedelta | None

Returns:

The time left, or None if there is no link, or it has expired.

updated_at

DateTimeField

Updated at

Timestamp indicating when the model instance was last updated.

This field is automatically updated to the current date and time whenever the instance is saved. It is indexed in the database for efficient querying.

Type:

Type

class smarter.apps.account.models.data_export.DataExportStatus(*values)[source]

Bases: TextChoices

Where a data export is in its life.

class smarter.apps.account.models.data_export.DataExportStep(*values)[source]

Bases: TextChoices

The step that a data export is on, in the order that it takes them.

The web console illustrates each one. done follows the last step of an export that succeeded.

The user menu’s Download My Data, under the console’s account api, /api/web/account/data-export/.

  • GET data-export/: what the dialog says before the user starts: the address that the link is emailed to, how long it works, how many exports are left today, and the latest export.

  • POST data-export/: start an export, or return the one in progress. See smarter.apps.account.data_export.start_data_export().

  • GET data-export/<id>/: an export’s progress, which the dialog polls.

Every response is camelCase JSON, from the Pydantic models of smarter.apps.account.data_export.

class smarter.apps.account.api.web.data_export.DataExportDetailView(**kwargs)[source]

Bases: SmarterAuthenticatedNeverCachedWebView

Return the progress of one of the authenticated user’s data exports, which the dialog polls.

get(request, *args, export_id=0, **kwargs)[source]

Return an export’s progress.

Parameters:
  • request – The request.

  • export_id (int) – The export’s id.

Returns:

A json response with its progress, or 404 if the user has no such export.

class smarter.apps.account.api.web.data_export.DataExportView(**kwargs)[source]

Bases: SmarterAuthenticatedNeverCachedWebView

Show the overview of the authenticated user’s data exports (GET), or start one (POST).

POST responds 202 with the new export’s progress, or 200 with the export that is already in progress, or 429 with {"error": ...} when the user may not start one now.

get(request, *args, **kwargs)[source]

Return the overview of the user’s data exports.

Parameters:

request – The request.

Returns:

A json response, see DataExportOverviewResponse.

post(request, *args, **kwargs)[source]

Start a data export, unless one is in progress.

Parameters:

request – The request.

Returns:

A json response with the export’s progress, see DataExportStatusResponse.

The REST api’s Download My Data, under /api/v1/accounts/data-exports/: what the web console’s user menu does.

  • GET data-exports/: the overview: the address that the link is emailed to, how long it works, how many exports are left today, and the latest export.

  • POST data-exports/: start an export, or return the one in progress. The zip file of the user’s manifests is built in the background, and a link to it is emailed to the user. See smarter.apps.account.data_export.start_data_export().

  • GET data-exports/<export_id>/: an export’s progress.

Every user may call them, for their own data. The responses are camelCase JSON, from the Pydantic models of smarter.apps.account.data_export. Errors are Django REST framework’s {"detail": ...}.

class smarter.apps.account.api.v1.views.data_export.DataExportListView(*args, **kwargs)[source]

Bases: SmarterAuthenticatedAPIView

The overview of the user’s data exports (GET), or start one (POST).

get(request, *args, **kwargs)[source]

Return the overview of the user’s data exports.

Parameters:

request (Request) – The request.

Return type:

Response

Returns:

A DataExportOverviewResponse.

post(request, *args, **kwargs)[source]

Start a data export, unless one is in progress.

Parameters:

request (Request) – The request.

Return type:

Response

Returns:

The new export’s progress, 202, or the one in progress, 200, or 429 when the user may not start one now: they have no email address, or started too many today.

class smarter.apps.account.api.v1.views.data_export.DataExportView(*args, **kwargs)[source]

Bases: SmarterAuthenticatedAPIView

The progress of one of the user’s data exports.

get(request, *args, export_id=0, **kwargs)[source]

Return an export’s progress.

Parameters:
  • request (Request) – The request.

  • export_id (int) – The export’s id.

Return type:

Response

Returns:

Its progress, or 404 if the user has no such export.