Passthrough Service

The Proxy passthrough: forward a caller’s request to an LLM provider’s API, with the provider’s API key.

resolve_proxy() finds the Proxy that a caller means by its name, and ProxyForwarder forwards the caller’s request:

It checks that the Proxy is active, that the path is one of its allowedPaths, and that no budget forbids the charge.
It removes the caller’s credentials, cookies and hop-by-hop headers, adds the Proxy’s headers, and adds the provider’s API key from the Proxy’s Secret.
It sends the request to the provider, and returns the provider’s response as it is: status, headers and body. Server-sent event streams, e.g. "stream": true, are streamed.
It reads the token usage from the response, and charges it to the Proxy, the caller, and the caller’s account.

The HTTP client is httpx. Tests replace its transport with configure_transport(), and the forwarder refuses to call a real provider from the unit tests.

Security

The provider’s API key is read from its Secret for each request, and is never logged or returned. The caller’s Smarter API key is never forwarded.
A Proxy’s base URL may not be a private, loopback or link-local address, so that a Proxy (or a Provider) cannot be used to reach Smarter’s internal network, e.g. a cloud metadata service. Only Proxies that a superuser owns may.

class smarter.apps.proxy.services.ProxyForwarder(proxy, user_profile)[source]

Bases: object

Forward one caller’s request through one Proxy.

Parameters:

proxy (Proxy) – The Proxy, e.g. from resolve_proxy().
user_profile (UserProfile) – The caller, who is charged for the request.

__init__(proxy, user_profile)[source]

api_key()[source]

The provider’s API key, from the Proxy’s Secret.

Raises:: ProxyConfigurationError – if the Proxy has no Secret, or it is expired or empty.
Return type:: str

check(path)[source]

Check that the request may be forwarded, and return the provider’s URL of the path.

Raises:: ProxyError – if it may not.
Return type:: str

forward(method, path, query_string, headers, body)[source]

Forward a request to the provider, and return its response.

Parameters:

method (str) – e.g. POST.
path (str) – The path, relative to the Proxy’s base URL, e.g. chat/completions.
query_string (str) – The request’s query string, which is forwarded as it is.
headers (Mapping[str, str]) – The request’s headers.
body (bytes) – The request’s body, which is forwarded as it is.

Raises:

ProxyError – if the request is refused, or the provider cannot be reached. A provider’s error response is not an exception: it is returned as it is.

Return type:

HttpResponseBase

class smarter.apps.proxy.services.SSEUsageReader[source]

Bases: object

Reads the token usage of a server-sent event stream, as it passes through, from its data: lines.

__init__()[source]

close()[source]

Read the last line, if the stream did not end with a line break.

Return type:: None

feed(chunk)[source]

Read the complete lines of a chunk, and keep the rest for the next one.

Return type:: None

class smarter.apps.proxy.services.Usage(prompt_tokens=0, completion_tokens=0, total_tokens=0, model=None)[source]

Bases: object

The tokens of one request, as the provider reports them.

__init__(prompt_tokens=0, completion_tokens=0, total_tokens=0, model=None)

completion_tokens: int = 0

merge(other)[source]

Combine the usage of two events of one stream.

Providers report cumulative counts, e.g. Anthropic’s message_start has the input tokens and message_delta the output tokens, and Gemini repeats its usage in every chunk, so the larger of each count is kept.

Return type:: Usage

model: str | None = None

prompt_tokens: int = 0

total_tokens: int = 0

smarter.apps.proxy.services.budget_resource_locators(proxy, user_profile)[source]

The record locators of everything that a request is charged to, and whose budgets it must respect.

Return type:: list[str]

smarter.apps.proxy.services.check_budget(proxy, user_profile)[source]

Refuse the request if a budget’s resource lock forbids charges to the Proxy, its Provider, the caller, or their account.

Raises:: ProxyBudgetExceeded – if one does.
Return type:: None

smarter.apps.proxy.services.check_upstream_host(proxy)[source]

Refuse a base URL whose host is not on the public internet, unless a superuser owns the Proxy.

Raises:: ProxyConfigurationError – if the host cannot be resolved, or is not public.
Return type:: None

smarter.apps.proxy.services.configure_transport(factory)[source]

Replace the HTTP transport to the providers, e.g. with an httpx.MockTransport in tests.

Parameters:: factory (Optional[Callable[[], BaseTransport]]) – Returns the transport for each request, or None to restore the default.
Return type:: None

smarter.apps.proxy.services.extract_usage(data)[source]

The token usage of a response body, or of one event of a stream, as the provider reports it.

OpenAI, and OpenAI-compatible APIs: usage.prompt_tokens and usage.completion_tokens, or, for the Responses API, usage.input_tokens and usage.output_tokens, also in response.usage.
Anthropic: usage.input_tokens and usage.output_tokens, also in message.usage of a stream’s message_start event.
Google Gemini: usageMetadata.promptTokenCount and usageMetadata.candidatesTokenCount.
Cohere: meta.billed_units (v1), or usage.billed_units (v2).

Return type:: Optional[Usage]
Returns:: The usage, or None if the body reports none.

smarter.apps.proxy.services.forwarded_request_headers(proxy, headers, api_key)[source]

The headers to send to the provider.

The caller’s, without those in DROPPED_REQUEST_HEADERS and without the Proxy’s auth header; then the Proxy’s headers; then the provider’s API key.

Return type:: dict[str, str]

smarter.apps.proxy.services.get_transport()[source]

The configured transport, or None for httpx’s default, which calls the provider.

Raises:: ProxyConfigurationError – in the unit tests, if no transport is configured, so that tests never call a real provider with a real API key.
Return type:: Optional[BaseTransport]

smarter.apps.proxy.services.host_addresses(host)[source]

The IP addresses of a host name.

Tests patch it, so that they do not depend on DNS.

Return type:: list[str]

smarter.apps.proxy.services.is_public_address(address)[source]

Whether an IP address is on the public internet: not private, loopback, link-local, or reserved.

Return type:: bool

smarter.apps.proxy.services.proxies_for(user_profile)[source]

The Proxies that a caller may read.

Return type:: QuerySet

smarter.apps.proxy.services.record_charges(proxy, user_profile, usage)[source]

Charge a request’s tokens to the Proxy, its Provider, the caller, and the caller’s account.

The charges are created by a Celery task. A failure is logged, and never fails the request.

Return type:: None

smarter.apps.proxy.services.resolve_proxy(name, user_profile)[source]

The Proxy that a caller means by name: of those that the caller may read, their own, else.

their account’s, else the built-in one, which the Smarter admin owns. Ties go to the most recently updated.

Raises:: ProxyNotFound – if the caller may read no Proxy of that name.
Return type:: Proxy

smarter.apps.proxy.services.returned_response_headers(proxy, headers)[source]

The provider’s response headers to return to the caller, and the name of the Proxy.

Return type:: list[tuple[str, str]]