Throttling

Request rate limits for every Smarter api, per authenticated user and, for anonymous clients, per IP address. The rates are SMARTER_THROTTLE_RATE_USER and SMARTER_THROTTLE_RATE_ANON.

Request rate limits for Smarter’s apis.

Smarter proxies billable LLM requests, so every api is rate limited, per authenticated user and, for anonymous clients, per IP address. The rates are smarter_settings.throttle_rate_user and smarter_settings.throttle_rate_anon, e.g. 600/minute. Either can be set to none to disable it.

Django REST Framework views are throttled by REST_FRAMEWORK["DEFAULT_THROTTLE_CLASSES"].
The apis that a deployed LLMClient serves are plain Django views, derived from smarter.lib.django.views.SmarterOptionallyAuthenticatedApiView, which calls throttle_wait() in its dispatch().

A client that exceeds a rate receives HTTP 429 Too Many Requests, with a Retry-After header. The counters are kept in the Redis cache, through smarter.lib.cache.lazy_cache, so that every pod shares them.

smarter.lib.drf.throttling.SMARTER_THROTTLE_CLASSES = [<class 'smarter.lib.drf.throttling.SmarterAnonRateThrottle'>, <class 'smarter.lib.drf.throttling.SmarterUserRateThrottle'>]

The throttles of every Smarter api, in the order in which they are checked.

[
    <class 'smarter.lib.drf.throttling.SmarterAnonRateThrottle'>,
    <class 'smarter.lib.drf.throttling.SmarterUserRateThrottle'>,
]

class smarter.lib.drf.throttling.SmarterAnonRateThrottle[source]

Bases: AnonRateThrottle

Limit the request rate of each anonymous client, by IP address.

The rate is smarter_settings.throttle_rate_anon. Authenticated requests are not limited by this throttle, but by SmarterUserRateThrottle.

cache = <smarter.lib.cache.lazy_cache.LazyCache object>

get_rate()[source]

Return the rate, e.g. 120/minute, or None if the throttle is disabled.

Return type:: str | None
Returns:: smarter_settings.throttle_rate_anon.

class smarter.lib.drf.throttling.SmarterUserRateThrottle[source]

Bases: UserRateThrottle

Limit the request rate of each authenticated user.

The rate is smarter_settings.throttle_rate_user. Anonymous requests are limited by IP address, at the same rate, and by SmarterAnonRateThrottle.

cache = <smarter.lib.cache.lazy_cache.LazyCache object>

get_rate()[source]

Return the rate, e.g. 600/minute, or None if the throttle is disabled.

Return type:: str | None
Returns:: smarter_settings.throttle_rate_user.

smarter.lib.drf.throttling.throttle_wait(request, view)[source]

Check a plain Django view’s request against every Smarter throttle.

Django REST Framework views are throttled by their throttle_classes. This applies the same throttles, and counts against the same limits, for views that are not Django REST Framework views. The request’s user decides which throttle applies: an authenticated user is limited by user, everyone else by IP address.

Parameters:

request (HttpRequest) – The request. Its user is the authenticated user, if any.
view – The view that serves the request.

Return type:

int | None

Returns:

None if the request may proceed, or the number of seconds to wait before retrying.