Dashboard Authentication - Download My Data
Download My Data, in the user menu, gives a user a copy of everything that they’ve built on the platform: the manifest of every resource that they own, in every account that they belong to, zipped into one file, and emailed to them as a link that works for 24 hours.
It opens a dialog that first explains what is about to happen: what the zip file holds, that it usually takes a few seconds, where the link is emailed and for how long it works, and that Secrets’ values are left out. Request my data starts an export, and the dialog then follows it, step by step, with an illustration of each one:
Find your resources: every resource that the user owns, from every list of the web console, with each list’s own queryset (
smarter.lib.django.listview.listview_registry()), so that the export holds what the user’s lists show them as theirs.Write their manifests: each resource’s manifest, as
smarter describereturns it (smarter.apps.dashboard.resource_detail.manifest_data()), one YAML file per resource, at<account number>/<kind>/<name>.yaml. The dialog counts them as they are written.Zip them up: the manifests, and a README that says what the zip file holds and how to apply a manifest again, in one zip file. A resource that can’t be described, e.g. because it is broken, is named in the README, with the reason, rather than failing the export.
Store the zip file privately: in the environment’s private storage,
infrastructure.storage(see Infrastructure Services), never in its public bucket.Email you a link: a presigned url to the zip file, which expires after 24 hours. The admin gets no blind copy, because the link is the only key to the file.
A Secret’s value is replaced by a placeholder, because the link travels by email, and an email can be forwarded. Its name, description and other settings are kept, so that its manifest can be applied again once its value is set. Other resources refer to Secrets by name, and their manifests hold no credentials.
The user can close the dialog at any time: the export runs in the Celery task
smarter.apps.account.tasks.create_data_export(), and the email comes anyway. Opening the dialog
again while an export runs follows it from where it is. A user may start five exports a day. Each one is
recorded in a DataExport, which the Django admin lists, read-only,
as a record of who exported their data, and when. Private storage deletes each zip file a day after it
was saved, and the Celery Beat task smarter.apps.account.tasks.delete_expired_data_exports()
deletes any whose link has expired, every six hours.
The dialog calls the console’s account api: GET /api/web/account/data-export/ for what it says before
the user starts, POST to start an export, and GET /api/web/account/data-export/<id>/ for an
export’s progress. See Web Console Api.
A program does the same with an api key, through the REST api, as the Smarter SDKs do:
GET /api/v1/accounts/data-exports/ for the overview, POST to start an export (202 when it
starts, 200 with the export that is already in progress, 429 when the user may not start one
now), and GET /api/v1/accounts/data-exports/<id>/ for its progress. The link is still emailed to
the user, never returned in a response. See The API Contract.
Download My Data: a zip file of the manifest of every resource that a user owns, emailed to them.
The user menu’s Download My Data starts an export with start_data_export(), which queues the
Celery task smarter.apps.account.tasks.create_data_export(). The task calls
run_data_export(), which takes these steps, and records each one in the export’s
DataExport, from which the web console shows its progress:
inventory: the resources that the user owns, in every account that they belong to, from every list of the web console (
smarter.lib.django.listview.listview_registry()).manifests: each resource’s manifest, as
smarter describereturns it, one YAML file per resource, at<account number>/<kind>/<name>.yaml. A Secret’s value is redacted, because the zip file is shared by a link in an email.archive: the manifests, and a README, zipped into one file.
upload: the zip file, saved to the environment’s private storage (
infrastructure.storage).email: a link to the zip file, which expires after
DATA_EXPORT_LINK_SECONDS, emailed to the user.
Private storage deletes the zip file a day after it was saved, and
delete_expired_data_exports(), which Celery Beat runs, deletes any that are left once their links
have expired.
- smarter.apps.account.data_export.DATA_EXPORTS_PER_DAY = 5
The most data exports that a user may start in 24 hours.
- smarter.apps.account.data_export.DATA_EXPORT_API_MODELS = (<class 'smarter.apps.account.data_export.DataExportOverviewResponse'>, <class 'smarter.apps.account.data_export.DataExportStatusResponse'>)
The models whose JSON Schemas the Smarter api’s OpenAPI document includes.
( <class 'smarter.apps.account.data_export.DataExportOverviewResponse'>, <class 'smarter.apps.account.data_export.DataExportStatusResponse'>, )
- smarter.apps.account.data_export.DATA_EXPORT_LINK_SECONDS = 86400
24 hours.
- Type:
How long the emailed link to a data export works
- class smarter.apps.account.data_export.DataExportOverviewResponse(**data)[source]
Bases:
DataExportModelWhat the Download My Data dialog tells the user before they start: where the link goes, and the limits.
- classmethod for_user(user)[source]
Return the overview of a user’s data exports.
- Parameters:
user (
User) – The user.- Return type:
- Returns:
The overview.
- model_config = {'alias_generator': <function to_camel>, 'extra': 'forbid', 'serialize_by_alias': True, 'validate_by_alias': True, 'validate_by_name': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- exception smarter.apps.account.data_export.DataExportStartRefused[source]
Bases:
ExceptionA user may not start a data export now, e.g. because they started too many today.
- class smarter.apps.account.data_export.DataExportStatusResponse(**data)[source]
Bases:
DataExportModelA data export’s progress, as the web console’s Download My Data dialog reads it.
- classmethod from_model(export)[source]
Return the progress of an export.
- Parameters:
export (
DataExport) – The export.- Return type:
- Returns:
Its progress.
- model_config = {'alias_generator': <function to_camel>, 'extra': 'forbid', 'serialize_by_alias': True, 'validate_by_alias': True, 'validate_by_name': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class smarter.apps.account.data_export.ExportFile(path, content)[source]
Bases:
objectA file of the zip file.
- __init__(path, content)
- class smarter.apps.account.data_export.ExportResource(kind, instance, account_number)[source]
Bases:
objectA resource that a data export includes.
- __init__(kind, instance, account_number)
- smarter.apps.account.data_export.delete_expired_data_exports()[source]
Delete the zip files of the data exports whose links have expired.
Private storage deletes them a day after they were saved too, so this only catches what that misses. The exports’ rows are kept, without their files, as a record of who exported their data, and when.
- Return type:
- Returns:
How many zip files were deleted.
- smarter.apps.account.data_export.latest_data_export(user)[source]
Return the user’s most recent data export.
- smarter.apps.account.data_export.owned_resources(user)[source]
Return every resource that the user owns, in every account that they belong to.
The resources are found in every list of the web console, with each list’s own queryset, so that the export holds what the user’s lists show them as theirs. A resource that two lists show is included once.
- Parameters:
user (
User) – The user.- Return type:
- Returns:
The resources, by list, the most recently updated first.
- smarter.apps.account.data_export.run_data_export(export_id)[source]
Run a data export, step by step, recording each step in its row.
A failure is recorded in the export, with what went wrong, rather than raised, so that the Celery task isn’t retried: the user can start another.
- Parameters:
export_id (
int) – The export’s id.- Return type:
DataExport- Returns:
The export.
- Raises:
DataExport.DoesNotExist – If there is no such export.
- smarter.apps.account.data_export.start_data_export(user)[source]
Start a data export for the user, unless one is already in progress.
The Celery task is queued once the export’s row is committed, so that the worker finds it.
- Parameters:
user (
User) – The user.- Return type:
- Returns:
The export, and True if it was started now, or False if it was already in progress.
- Raises:
DataExportStartRefused – If the user has no email address, or started
DATA_EXPORTS_PER_DAYexports in the last 24 hours.
A user’s data export: the manifest of every resource that they own, zipped and emailed to them.
The user menu’s Download My Data starts one, and the web console shows its progress, step by step, from
this row, which the Celery task smarter.apps.account.tasks.create_data_export() updates as it works.
See smarter.apps.account.data_export.
- smarter.apps.account.models.data_export.DATA_EXPORT_STALE_AFTER = datetime.timedelta(seconds=900)
An export that has neither finished nor failed after this long is abandoned, e.g. its worker restarted.
A user may then start another.
- class smarter.apps.account.models.data_export.DataExport(*args, **kwargs)[source]
Bases:
TimestampedModelA user’s data export.
- Parameters:
user – The user whose resources are exported, in every account that they belong to.
status – One of
DataExportStatus.step – The step that it is on, one of
DataExportStep.resource_count – How many resources the user owns.
manifest_count – How many manifests have been written so far.
failed_count – How many resources had no manifest, e.g. because one is broken.
size_bytes – The size of the zip file.
object_key – The zip file’s key in private storage, until it is deleted.
expires_at – When the emailed link stops working.
completed_at – When it succeeded or failed.
error – Why it failed, or an empty string.
id (Unknown) – Primary key: ID
created_at (Unknown) – Created at
updated_at (Unknown) – Updated at
- exception DoesNotExist
Bases:
ObjectDoesNotExist
- exception MultipleObjectsReturned
Bases:
MultipleObjectsReturned
- exception NotUpdated
Bases:
ObjectNotUpdated,DatabaseError
- created_at
-
Created at
Timestamp indicating when the model instance was created.
This field is automatically set to the current date and time when the instance is first created. It is indexed in the database for efficient querying.
- Type:
Type
- id: int
-
Primary key: ID
The row’s primary key, which Django adds to every concrete model. Declared, without a value, only so that type checkers know it: Django ignores a bare annotation.
- Type:
Type
- property is_finished: bool
Whether it succeeded or failed.
- property is_in_progress: bool
Whether it is queued or running, and not abandoned.
- property is_stale: bool
unfinished for longer than
DATA_EXPORT_STALE_AFTER.- Type:
Whether it was abandoned
- link_expires_in(now=None)[source]
How long the emailed link still works.
- updated_at
-
Updated at
Timestamp indicating when the model instance was last updated.
This field is automatically updated to the current date and time whenever the instance is saved. It is indexed in the database for efficient querying.
- Type:
Type
- class smarter.apps.account.models.data_export.DataExportStatus(*values)[source]
Bases:
TextChoicesWhere a data export is in its life.
- class smarter.apps.account.models.data_export.DataExportStep(*values)[source]
Bases:
TextChoicesThe step that a data export is on, in the order that it takes them.
The web console illustrates each one.
donefollows the last step of an export that succeeded.
The user menu’s Download My Data, under the console’s account api, /api/web/account/data-export/.
GET data-export/: what the dialog says before the user starts: the address that the link is emailed to, how long it works, how many exports are left today, and the latest export.POST data-export/: start an export, or return the one in progress. Seesmarter.apps.account.data_export.start_data_export().GET data-export/<id>/: an export’s progress, which the dialog polls.
Every response is camelCase JSON, from the Pydantic models of smarter.apps.account.data_export.
- class smarter.apps.account.api.web.data_export.DataExportDetailView(**kwargs)[source]
Bases:
SmarterAuthenticatedNeverCachedWebViewReturn the progress of one of the authenticated user’s data exports, which the dialog polls.
- class smarter.apps.account.api.web.data_export.DataExportView(**kwargs)[source]
Bases:
SmarterAuthenticatedNeverCachedWebViewShow the overview of the authenticated user’s data exports (
GET), or start one (POST).POSTresponds 202 with the new export’s progress, or 200 with the export that is already in progress, or 429 with{"error": ...}when the user may not start one now.- get(request, *args, **kwargs)[source]
Return the overview of the user’s data exports.
- Parameters:
request – The request.
- Returns:
A json response, see
DataExportOverviewResponse.
- post(request, *args, **kwargs)[source]
Start a data export, unless one is in progress.
- Parameters:
request – The request.
- Returns:
A json response with the export’s progress, see
DataExportStatusResponse.
The REST api’s Download My Data, under /api/v1/accounts/data-exports/: what the web console’s user menu does.
GET data-exports/: the overview: the address that the link is emailed to, how long it works, how many exports are left today, and the latest export.POST data-exports/: start an export, or return the one in progress. The zip file of the user’s manifests is built in the background, and a link to it is emailed to the user. Seesmarter.apps.account.data_export.start_data_export().GET data-exports/<export_id>/: an export’s progress.
Every user may call them, for their own data. The responses are camelCase JSON, from the Pydantic models of
smarter.apps.account.data_export. Errors are Django REST framework’s {"detail": ...}.
- class smarter.apps.account.api.v1.views.data_export.DataExportListView(*args, **kwargs)[source]
Bases:
SmarterAuthenticatedAPIViewThe overview of the user’s data exports (
GET), or start one (POST).- get(request, *args, **kwargs)[source]
Return the overview of the user’s data exports.
- Parameters:
request (
Request) – The request.- Return type:
Response- Returns:
- post(request, *args, **kwargs)[source]
Start a data export, unless one is in progress.
- Parameters:
request (
Request) – The request.- Return type:
Response- Returns:
The new export’s progress, 202, or the one in progress, 200, or 429 when the user may not start one now: they have no email address, or started too many today.
- class smarter.apps.account.api.v1.views.data_export.DataExportView(*args, **kwargs)[source]
Bases:
SmarterAuthenticatedAPIViewThe progress of one of the user’s data exports.