List Pagination, Search and Sorting

Every resource list in the web console, e.g. the Guardrail, Secret or Prompt list, shows one page of resources at a time, with previous and next page buttons, a page number box, and a search, and its column headers sort it. Its Django list api, at react-integration/api/listview/<owned|shared|all>/, does the paging, the searching and the sorting, so a search covers all of the resources that the user may list, and a sort orders all of them, not only those of the page shown.

The api’s optional query parameters are page, from 1; page_size, up to 100, and 25 by default; search, the text that a resource’s name or description contains, ignoring case; and ordering, the column to sort by, such as name, or -name to sort it in descending order. Without an ordering, the most recently updated resources come first. Its response describes the page that it returns, alongside its objects:

{
    "objects": [{"name": "my_guardrail", "...": "..."}],
    "pagination": {
        "page": 2,
        "pageSize": 25,
        "numPages": 3,
        "count": 61,
        "search": "guard",
        "ordering": "-name",
        "sortFields": ["category", "createdAt", "description", "name", "stage", "updatedAt"]
    }
}

A page past the last is the last, and an invalid page or page size is the default. Every list api implements this with paginate_listview().

Each list api names the columns that it can sort by, in camelCase as its objects name them, and the model field that sorts each one, e.g. "apiKey": "api_key__name" for a Provider’s API key. Every list can sort by name, description, createdAt and updatedAt (DEFAULT_SORT_FIELDS), and most add the columns of their own resource, such as a Guardrail’s stage and category. An unknown ordering is ignored, and sortFields lists the columns that the api can sort by, so that the React list makes only those column headers clickable. Each click on a header cycles its column through ascending, descending and the default order, and returns the list to its first page. Resources whose column has the same value are ordered by their primary key, so that each appears on exactly one page.

Pagination and search for the list apis of the React list views in the Smarter web console.

Every resource list, e.g. the Guardrail or Secret list, is a TabbedListView of @smarter/common, which asks its Django list api for one page of the resources at a time, optionally narrowed by a search, and sorted by one of its columns. paginate_listview() reads those query parameters, filters, sorts and paginates the queryset, and describes the page, so that every list api behaves alike.

The list is sorted by the database, not in the browser, because a page holds only some of the resources: sorting the rows of the page shown would leave the rest of the resources in their default order, on the other pages.

SORT_FIELDS = {**DEFAULT_SORT_FIELDS, "stage": "stage"}

page, pagination = paginate_listview(request, qs.order_by("-updated_at"), sort_fields=SORT_FIELDS)
return JsonResponse(
    {
        "objects": GuardrailSerializer(page, many=True, context={"request": request}).data,
        "pagination": pagination,
    }
)

smarter.lib.django.pagination.DEFAULT_PAGE_SIZE = 25

The resources of a page, unless the list asks for another number.

smarter.lib.django.pagination.DEFAULT_SEARCH_FIELDS = ('name', 'description')

The fields that a search matches, of the MetaDataModel that every Smarter resource inherits.

('name', 'description')

smarter.lib.django.pagination.DEFAULT_SORT_FIELDS: Mapping[str, str] = {'createdAt': 'created_at', 'description': 'description', 'name': 'name', 'updatedAt': 'updated_at'}

The columns that a list may be sorted by, of the MetaDataModel that every Smarter resource inherits.

Each key is a column’s name in the list api’s response, in camelCase, as the ordering query parameter names it. Each value is the field that sorts it, in the queryset’s lookup syntax, such as "api_key__name" for a related resource’s name.

{
    'createdAt': 'created_at',
    'description': 'description',
    'name': 'name',
    'updatedAt': 'updated_at',
}

class smarter.lib.django.pagination.ListViewPagination(**data)

Bases: BaseModel

The description of one page of a list api’s resources, as paginate_listview() returns it.

Its JSON is camelCase, e.g. pageSize.

count: int

model_config = {'alias_generator': <function to_camel>, 'extra': 'forbid', 'serialize_by_alias': True, 'validate_by_alias': True, 'validate_by_name': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

num_pages: int

ordering: str

page: int

page_size: int

search: str

sort_fields: list[str]

smarter.lib.django.pagination.MAX_PAGE_SIZE = 100

The most resources that a page may have.

smarter.lib.django.pagination.MAX_SEARCH_LENGTH = 255

The longest search, in characters.

A longer one is truncated.

smarter.lib.django.pagination.has_field(model, lookup)

Whether model has the field that lookup names, following relations, e.g. "api_key__name".

Parameters:

• model (type[Model]) – The model whose field to look up.

• lookup (str) – A field name, or a path of field names joined by __.

Return type:

bool

Returns:

True if every field of the path exists.

smarter.lib.django.pagination.paginate_listview(request, queryset, search_fields=('name', 'description'), sort_fields={'createdAt': 'created_at', 'description': 'description', 'name': 'name', 'updatedAt': 'updated_at'})

One page of a list api’s resources, which match its search, and a description of the page.

The request’s query parameters are all optional:

• page: from 1. A page past the last is the last, and an invalid one is the first.

• page_size: up to MAX_PAGE_SIZE; DEFAULT_PAGE_SIZE if invalid.

• search: text that one of search_fields contains, ignoring case.

• ordering: the column of sort_fields to sort by, e.g. name, or -name to sort it in descending order. Without one, the resources keep the queryset’s order. See sort_queryset().

Parameters:

• request (HttpRequest) – The list api’s request.

• queryset (QuerySet) – The resources that the user may list, in their default order.

• search_fields (Sequence[str]) – The names of the fields that a search matches.

• sort_fields (Mapping[str, str]) – The columns that the list may be sorted by, and the fields that sort them. See DEFAULT_SORT_FIELDS.

Return type:

tuple[Page, dict[str, Any]]

Returns:

The page, and its description for the list api’s response, a dumped ListViewPagination: {"page": 1, "pageSize": 25, "numPages": 3, "count": 61, "search": "", "ordering": "-name", "sortFields": ["createdAt", "description", "name", "updatedAt"]}. ordering is empty when the default order applies, and sortFields are the columns that the list may be sorted by, so that the list shows only those columns as sortable.

smarter.lib.django.pagination.positive_int(value, default, maximum=None)

An integer query parameter, from 1 up to maximum, if any, or else default.

Parameters:

• value (Any) – The query parameter, a string such as "2", or None when it is missing.

• default (int) – The value for a missing, invalid, or out of range parameter.

• maximum (int | None) – The largest value allowed, if any.

Return type:

int

Returns:

The parameter as an integer, or default.

smarter.lib.django.pagination.search_queryset(queryset, search, search_fields=('name', 'description'))

The resources of queryset that one of search_fields contains search in, ignoring case.

Fields that the queryset’s model does not have are ignored. An empty search, or one with no field to match, returns the queryset unchanged.

Parameters:

• queryset (QuerySet) – The resources to search.

• search (str) – The text to search for.

• search_fields (Sequence[str]) – The names of the fields to search.

Return type:

QuerySet

Returns:

The matching resources.

smarter.lib.django.pagination.sort_queryset(queryset, ordering, sort_fields={'createdAt': 'created_at', 'description': 'description', 'name': 'name', 'updatedAt': 'updated_at'})

The resources of queryset, sorted by the column that ordering names.

ordering is a column of sort_fields, such as "name", or, to sort it in descending order, the column after a minus sign, such as "-name". Empty values sort after the others, in either direction. An empty or unknown ordering leaves the queryset’s own order.

The resources’ primary key breaks ties, in the same direction, so that resources whose column has the same value, e.g. the same stage, are in the same order on every request. Otherwise the database could return them in a different order for each page, and a resource could appear on two pages, or on none.

Parameters:

• queryset (QuerySet) – The resources to sort, in their default order.

• ordering (str) – The column to sort by, optionally after a minus sign.

• sort_fields (Mapping[str, str]) – The columns that the list may be sorted by, and the fields that sort them.

Return type:

tuple[QuerySet, str]

Returns:

The sorted resources, and the ordering that sorted them: ordering, or an empty string if the queryset kept its own order.

smarter.lib.django.pagination.sortable_fields(queryset, sort_fields={'createdAt': 'created_at', 'description': 'description', 'name': 'name', 'updatedAt': 'updated_at'})

The columns of sort_fields that the queryset’s model can be sorted by.

Parameters:

• queryset (QuerySet) – The resources to sort.

• sort_fields (Mapping[str, str]) – The columns that the list may be sorted by, and the fields that sort them.

Return type:

dict[str, str]

Returns:

Those of sort_fields whose field the model has.