Passthrough Service
The Proxy passthrough: forward a caller’s request to an LLM provider’s API, with the provider’s API key.
resolve_proxy() finds the Proxy that a caller means by its name, and ProxyForwarder
forwards the caller’s request:
It checks that the Proxy is active, that the path is one of its
allowedPaths, and that no budget forbids the charge.It removes the caller’s credentials, cookies and hop-by-hop headers, adds the Proxy’s
headers, and adds the provider’s API key from the Proxy’s Secret.It sends the request to the provider, and returns the provider’s response as it is: status, headers and body. Server-sent event streams, e.g.
"stream": true, are streamed.It reads the token usage from the response, and charges it to the Proxy, the caller, and the caller’s account.
The HTTP client is httpx. Tests replace its transport with
configure_transport(), and the forwarder refuses to call a real provider from the unit tests.
Security
The provider’s API key is read from its Secret for each request, and is never logged or returned. The caller’s Smarter API key is never forwarded.
A Proxy’s base URL may not be a private, loopback or link-local address, so that a Proxy (or a Provider) cannot be used to reach Smarter’s internal network, e.g. a cloud metadata service. Only Proxies that a superuser owns may.
- class smarter.apps.proxy.services.ProxyForwarder(proxy, user_profile)[source]
Bases:
objectForward one caller’s request through one Proxy.
- Parameters:
proxy (
Proxy) – The Proxy, e.g. fromresolve_proxy().user_profile (
UserProfile) – The caller, who is charged for the request.
- api_key()[source]
The provider’s API key, from the Proxy’s Secret.
- Raises:
ProxyConfigurationError – if the Proxy has no Secret, or it is expired or empty.
- Return type:
- check(path)[source]
Check that the request may be forwarded, and return the provider’s URL of the path.
- Raises:
ProxyError – if it may not.
- Return type:
- forward(method, path, query_string, headers, body)[source]
Forward a request to the provider, and return its response.
- Parameters:
- Raises:
ProxyError – if the request is refused, or the provider cannot be reached. A provider’s error response is not an exception: it is returned as it is.
- Return type:
- class smarter.apps.proxy.services.SSEUsageReader[source]
Bases:
objectReads the token usage of a server-sent event stream, as it passes through, from its
data:lines.
- class smarter.apps.proxy.services.Usage(prompt_tokens=0, completion_tokens=0, total_tokens=0, model=None)[source]
Bases:
objectThe tokens of one request, as the provider reports them.
- __init__(prompt_tokens=0, completion_tokens=0, total_tokens=0, model=None)
- smarter.apps.proxy.services.budget_resource_locators(proxy, user_profile)[source]
The record locators of everything that a request is charged to, and whose budgets it must respect.
- Parameters:
proxy (
Proxy) – The Proxy.user_profile (
UserProfile) – The caller.
- Return type:
- Returns:
The locators of the Proxy, its Provider (if it has one), the caller, and the caller’s account.
- smarter.apps.proxy.services.check_budget(proxy, user_profile)[source]
Refuse the request if a budget’s resource lock forbids charges to the Proxy, its Provider, the caller, or their account.
- Raises:
ProxyBudgetExceeded – if one does.
- Return type:
- smarter.apps.proxy.services.check_upstream_host(proxy)[source]
Refuse a base URL whose host is not on the public internet, unless a superuser owns the Proxy.
- Raises:
ProxyConfigurationError – if the host cannot be resolved, or is not public.
- Return type:
- smarter.apps.proxy.services.configure_transport(factory)[source]
Replace the HTTP transport to the providers, e.g. with an
httpx.MockTransportin tests.
- smarter.apps.proxy.services.extract_usage(data)[source]
The token usage of a response body, or of one event of a stream, as the provider reports it.
OpenAI, and OpenAI-compatible APIs:
usage.prompt_tokensandusage.completion_tokens, or, for the Responses API,usage.input_tokensandusage.output_tokens, also inresponse.usage.Anthropic:
usage.input_tokensandusage.output_tokens, also inmessage.usageof a stream’s message_start event.Google Gemini:
usageMetadata.promptTokenCountandusageMetadata.candidatesTokenCount.Cohere:
meta.billed_units(v1), orusage.billed_units(v2).
- smarter.apps.proxy.services.forwarded_request_headers(proxy, headers, api_key)[source]
The headers to send to the provider.
The caller’s, without those in
DROPPED_REQUEST_HEADERSand without the Proxy’s auth header; then the Proxy’sheaders; then the provider’s API key.
- smarter.apps.proxy.services.get_transport()[source]
The configured transport, or
Nonefor httpx’s default, which calls the provider.- Raises:
ProxyConfigurationError – in the unit tests, if no transport is configured, so that tests never call a real provider with a real API key.
- Return type:
BaseTransport|None
- smarter.apps.proxy.services.host_addresses(host)[source]
The IP addresses of a host name.
Tests patch it, so that they do not depend on DNS.
- smarter.apps.proxy.services.is_public_address(address)[source]
Whether an IP address is on the public internet: not private, loopback, link-local, or reserved.
- Return type:
- smarter.apps.proxy.services.proxies_for(user_profile)[source]
The Proxies that a caller may read.
- Return type:
- smarter.apps.proxy.services.record_charges(proxy, user_profile, usage)[source]
Charge a request’s tokens to the Proxy, its Provider, the caller, and the caller’s account.
The charges are created by a Celery task. A failure is logged, and never fails the request.
- Return type:
- smarter.apps.proxy.services.resolve_proxy(name, user_profile)[source]
The Proxy that a caller means by
name: of those that the caller may read, the nearest.Their own, else their account’s, else that of the nearest account that shares with theirs, ending with the built-in ones, which the Smarter admin owns. Ties go to the most recently updated. See
smarter.apps.account.hierarchy.resolve_by_name().- Raises:
ProxyNotFound – if the caller may read no Proxy of that name.
- Return type:
- smarter.apps.proxy.services.returned_response_headers(proxy, headers)[source]
The provider’s response headers to return to the caller, and the name of the Proxy.
- async smarter.apps.proxy.services.stream_asynchronously(chunks)[source]
Serve a synchronous stream, e.g. a provider’s event stream read with httpx, as the ASGI server reads it.
Django serves a
StreamingHttpResponsewhose content is a synchronous iterator by reading the whole iterator first, which is no stream at all: the client waited for the provider’s whole response, and a thread was held for as long as it lasted. Each chunk is read here in the request’s own thread, the one where its view ran, and passed on as soon as it arrives. When the stream ends, or the client disconnects, the iterator is closed, which runs itsfinallyblock, e.g.ProxyForwarder._stream()’s charges, in the same thread.- Parameters:
- Returns:
The same chunks, as an asynchronous iterator.
- Return type: