OpenAI Compatible Chat Provider

Base class for prompt providers.

smarter.apps.provider.services.text_completion.lib.openai_compatible_chat_provider.BLOCKED_MESSAGE_PLACEHOLDER = '[This message was blocked by a guardrail.]'

What replaces a user message that an input guardrail blocked, in the conversation’s history.

smarter.apps.provider.services.text_completion.lib.openai_compatible_chat_provider.MARKDOWN_SYSTEM_PROMPT = 'Your responses are displayed in a chat window that renders GitHub flavored Markdown: headings, bold, italics, strikethrough, lists, tables, block quotes, inline code, fenced code blocks, links and images. Use Markdown when it makes a response clearer, and name the language of each fenced code block. It also typesets math written in LaTeX: \\( ... \\) for inline math, and \\[ ... \\] or $$ ... $$ for a displayed equation. A single dollar sign is not math, so never use $ ... $ for math. It draws a fenced code block whose language is mermaid as a Mermaid diagram, such as a flowchart, sequence diagram, class diagram, state diagram or entity relationship diagram. Draw one when a process, architecture or set of relationships is easier to understand as a picture, and keep its syntax simple: quote labels that contain punctuation. The chat window does not render HTML, so do not use it. To show an image, use Markdown image syntax, , but only with an image url that the user gave you or that a tool returned. NEVER invent an image url, because a broken image looks worse than no image.'

Added to the system prompt of every request, so that the LLM knows what the chat window renders.

It is the single source of the chat window’s rendering rules, so an LLMClient’s defaultSystemRole need not repeat them, and should only describe the formatting that suits its own persona.

It is added to the request’s messages only (see OpenAISmarterClient.openai_messages), and is never saved in the conversation’s history, nor displayed in the chat.

class smarter.apps.provider.services.text_completion.lib.openai_compatible_chat_provider.OpenAISmarterClient(provider, provider_name, base_url, api_key, default_model, default_system_role='You are a helpful llmclient. When given the opportunity to utilize function calling, you should always do so. This will allow you to provide the best possible responses to the user. If you are unable to provide a response, you should prompt the user for more information. If you are still unable to provide a response, you should inform the user that you are unable to help them at this time.', default_temperature=1.0, default_max_tokens=2048, valid_chat_completion_models=None, add_built_in_tools=False, **kwargs)

Bases: SmarterChatProviderBase

Prompt provider for OpenAI-compatible text completion APIs.

This provider class enables seamless integration with any vendor or service that implements the OpenAI prompt completion API, including both OpenAI and third-party providers that adhere to the same protocol and message formats.

Key Features:

• Supports OpenAI’s prompt completion API and compatible alternatives.

• Handles message formatting, tool calls, plugin integration, and billing.

• Manages multi-step prompt completion workflows, including tool and plugin responses.

• Provides hooks for plugin selection, function registration, and error handling.

Usage:

Inherit from this class to implement a prompt provider that communicates with any OpenAI-compatible API endpoint. This class is suitable for use cases where you want to support multiple LLM vendors with a unified interface.

Example:

class MyProvider(OpenAISmarterClient):
    pass

provider = MyProvider()
response = provider.handler(user_profile, prompt, data)

See also

• https://developers.openai.com/api/reference/overview/prompt

• SmarterChatProviderBase

static add_markdown_system_prompt(messages)

Tell the LLM that its responses may use Markdown.

MARKDOWN_SYSTEM_PROMPT is appended to the first system message, or is the system message of a thread that has none. The messages are copies, so the conversation’s history is unchanged.

Parameters:

messages (list[dict[str, Any]]) – The request’s messages.

Returns:

The request’s messages, with the Markdown instructions.

Return type:

list[dict[str, Any]]

append_guardrail_message(stage, result)

Append a Smarter message that summarizes what the guardrails of a stage did, if anything.

Return type:

None

append_openai_error_response(response, e=None)

Append an error message to the internal message list based on the OpenAI response.

Parameters:

response (ChatCompletion) – The OpenAI prompt completion response containing the error.

Returns:

None

Return type:

None

append_openai_response(response)

Append the OpenAI-compatible response message to the internal message list.

2025-06-20: updated to use model_dump_json() to ensure compatibility with Pydantic v2. 2025-10-02: updated to validate that the response message is indeed a ChatCompletionMessage.

Parameters:

response (ChatCompletion) – The OpenAI-compatible prompt completion response.

Returns:

None

Return type:

None

authorize_budgets()

Check the budgets of everything that a prompt completion is charged to, before calling the LLM.

Raises:

SmarterBudgetExceeded – if a budget’s resource lock forbids charges to the provider, the llmclient, the user, or their account.

Return type:

None

available_functions: dict[str, Any]

completion_tokens: int | None

create_first_completion(completions_kwargs)

Send the first request, which is the only one that offers the LLM tools.

Some reasoning models reject function tools on v1/chat/completions unless reasoning_effort is ‘none’. Requests with tools set reasoning_effort to ‘none’ for the models in TOOLS_REQUIRE_NO_REASONING_MODELS. If another model rejects the request for that reason, it is retried once with reasoning_effort set to ‘none’ and the model is added to the set.

Return type:

ChatCompletion

data: dict[str, Any] | None

first_iteration: dict[str, Any]

first_response: ChatCompletion | None

functions: List[str] | None

guardrail_pipeline: GuardrailPipeline | None = None

The MCP tools of the current prompt.

See handle_mcp_clients().

handle_completion()

Handle prompt completion response.

This method formats the final response to be returned to the client.

Returns:

A dictionary representing the final prompt completion response.

Return type:

dict[str, Any]

handle_function_provided(function)

Handle a function being provided.

Parameters:

function (str) – The name of the function that was provided.

Returns:

None

Return type:

None

handle_input_guardrails()

Run the LLMClient’s input guardrails on the user’s message.

The guardrails are those listed in the LLMClient manifest’s spec.guardrails. If they redact or transform the user’s message, the message is replaced, so that the LLM, the plugins and the persisted history see the changed message.

Raises:

GuardrailBlockedError – If a guardrail blocks the user’s message.

Return type:

None

handle_mcp_clients()

Offer the LLM the tools of the LLMClient’s MCP servers.

The LLMClient’s active MCPClients, listed in its manifest’s spec.mcpClients, are loaded into an MCPToolkit, whose tools are added to the request, and whose servers’ instructions are added to the system prompt. An MCP server that cannot be reached is skipped, and reported in a Smarter message.

Returns:

None

Return type:

None

handle_output_guardrails(response)

Run the LLMClient’s output guardrails on the LLM’s final reply.

Parameters:

response (ChatCompletion) – The LLM’s final response.

Return type:

ChatCompletion

Returns:

The response, with its reply redacted or transformed, or replaced with the guardrail’s message if a guardrail blocks it.

handle_plugin_called(plugin)

Handle a plugin tool call.

example: SqlPlugin, ApiPlugin, StaticPlugin etc.

Parameters:

plugin (PluginBase) – The plugin instance that was called.

Returns:

None

Return type:

None

handle_plugin_selected(plugin)

Handle a plugin being selected.

does the prompt have anything to do with any of the search terms defined in a plugin? The LLMClient’s model, temperature and max_completion_tokens are used whichever plugins are selected: a plugin customizes the prompt’s messages and tools, not the LLM settings. 2025-10-02: updated to validate that messages and tools are lists. 2025-10-02: updated to use plugin.plugin_meta.name for the plugin name.

Parameters:

plugin (PluginBase) – The plugin instance that was selected.

Returns:

None

Return type:

None

handle_response()

Handle internal billing, and append messages to the response for prompt completion and the billing summary.

Returns:

None

Return type:

None

handle_tool_called(function_name, function_args, extra_resource_locators=None)

Handle a built-in tool call.

example: get_current_weather()

Parameters:

• function_name (str) – The name of the tool function called.

• function_args (str) – The arguments passed to the tool function.

• extra_resource_locators (list[str] | None) – The record locators of any other resources that the tool call used, e.g. an MCPClient, to include in the charge.

Returns:

None

Return type:

None

handler(user_profile, prompt, data, plugins=None, functions=None)

Process a prompt prompt request and invoke the appropriate OpenAI-compatible API endpoint.

This method orchestrates the entire prompt completion workflow, including:

• Validating input and internal state.

• Initializing or updating the message thread.

• Selecting and configuring plugins and/or functions (collectively, tool calls) for the LLM.

• Preparing and sending requests to the OpenAI API (or compatible provider).

• Handling tool calls and plugin responses.

• Managing billing, logging, and signal dispatch.

• Returning a formatted HTTP response with the LLM’s output and relevant metadata.

Parameters:

• user_profile (UserProfile) – The user_profile instance making the request.

• prompt (Prompt) – The prompt session instance associated with this request.

• data – The request payload, typically containing a session key and a list of message dictionaries.

• plugins (list[PluginBase] | None) – A list of plugin instances to be considered for selection and presentation to the LLM.

• functions (list[str] | None) – A list of predefined function definitions for tool calls.

Returns:

An HTTP response dictionary (or list) containing the LLM’s output, tool call results, and metadata.

Return type:

dict[str, Any] | SmarterHttpResponseForbidden | SmarterHttpResponseNotFound | SmarterHttpResponseBadRequest | SmarterJournaledJsonErrorResponse | SmarterJournaledJsonResponse

Raises:

• SmarterValueError – If required parameters are missing or invalid.

• SmarterConfigurationError – If there are configuration issues with the provider or plugins.

• SmarterIlligalInvocationError – If the method is invoked in an invalid state.

Note

This method manages both the initial and any required follow-up LLM requests (e.g., for tool calls). It also handles plugin selection logic and ensures that all required signals and billing events are triggered.

See also

_InternalKeys OpenAIMessageKeys PluginBase ChatCompletion ChatCompletionMessageToolCall

Example usage:

response = provider.handler(
    prompt=chat_instance,
    data=request_data,
    plugins=[plugin1, plugin2],
    functions=[function_definition_1, function_definition_2],
    user_profile=current_user
)

input_text: str | None

iteration: int

max_completion_tokens: int | None

mcp_toolkit: MCPToolkit | None = None

The MCP tools of the current prompt.

See handle_mcp_clients().

model: str | None

property new_messages: list[dict[str, Any]]

Return a list of messages that are marked as new.

This property filters the internal message list to return only those messages that have the ‘smarter_is_new’ flag set to True.

Returns:

A list of new messages.

Return type:

list[SmarterDictType]

property openai_messages: list[dict[str, Any]]

Return a sanitized list of messages compatible with OpenAI’s prompt completion API.

This property processes the internal message list, removing Smarter-specific annotations (such as metadata about tool calls and interim completion token charges) to ensure that only valid OpenAI message fields are included. This is essential for avoiding API errors related to unexpected or extraneous fields.

Returns:

A list of dictionaries representing prompt messages, formatted for OpenAI’s API.

Return type:

list[SmarterDictType]

Raises:

SmarterValueError – If the internal message list is not a list.

Example:

[
    {
        "role": "assistant",
        "content": "Welcome to Smarter!",
        "tool_calls": [
            {
                "id": "call_ABC123",
                "type": "function",
                "function": {
                    "name": "smarter_plugin_0000000045",
                    "arguments": "{"description":"AI"}"
                }
            }
        ]
    },
    {
        "role": "tool",
        "name": "smarter_plugin_0000000045",
        "content": "SqlPlugin stackademy_sql response: ...",
        "tool_call_id": "call_ABC123"
    }
]

Important

• OpenAI expects that every assistant message with a tool_calls field is immediately followed by a corresponding tool message for each tool_call_id. Failure to do so will result in an API error.

• If you include Smarter-specific fields (such as smarter_is_new) in the message list, OpenAI’s API may reject the request.

• On the first iteration, tool call responses are excluded from the message list to comply with OpenAI’s requirements.

See also

• https://developers.openai.com/api/reference/overview/prompt/create

• OpenAIMessageKeys

plugins: List[PluginBase] | None

prep_first_request()

Prepare the first request for the prompt completion.

This is called at the beginning of the prompt completion process.

Raises:

SmarterValueError – If the messages are not a list, or if tool definitions are invalid.

Returns:

None

Return type:

None

prep_second_request()

Prepare the second request for the prompt completion.

This is called in response to a tool call that requires a second request to the LLM.

Returns:

None

Return type:

None

process_tool_call(tool_call)

Process a tool call from the LLM.

This method handles both built-in tool calls and plugin tool calls.

Parameters:

tool_call (ChatCompletionMessageFunctionToolCall | ChatCompletionMessageCustomToolCall) – The tool call data from the LLM.

Returns:

None

Return type:

None

prompt_tokens: int | None

reference: str | None

request_meta_data: dict[str, Any]

request_meta_data_factory()

Return a dictionary of request meta data.

This includes the model, temperature, max_completion_tokens, and input_text.

Returns:

A dictionary of request meta data.

Return type:

dict

second_iteration: dict[str, Any] | None

second_response: ChatCompletion | None

serialized_tool_calls: list[dict[str, Any]] | None

temperature: float | None

tool_budget_refusal(function_name)

Check the budget of the plugin or MCP server that a tool call would use.

Return type:

str | None

Returns:

The tool response that tells the LLM why the tool cannot be used, or None if it can.

tool_display_name(function_name)

Return the human-readable name of a tool call’s function, for progress messages.

A plugin’s function name is symbolic, e.g. smarter_plugin_0000000016, so its PluginMeta’s name is returned instead. An MCP function’s name is prefixed with its server’s number, so the MCP tool’s own name is returned instead. Anything else, e.g. a built-in function, or a plugin that can’t be found, is returned as is.

Parameters:

function_name (str) – The function name that the LLM called.

Returns:

The tool’s name, as users know it.

Return type:

str

tools: list[dict[str, Any]] | None

total_tokens: int | None

smarter.apps.provider.services.text_completion.lib.openai_compatible_chat_provider.TOOLS_REQUIRE_NO_REASONING_MODELS: set[str] = {'gpt-6-luna'}

Models whose v1/chat/completions endpoint rejects function tools unless reasoning_effort is ‘none’.

OpenAI returns a 400 for these models when a request has tools and omits reasoning_effort, because their default effort is not ‘none’. The set is not exhaustive: a model that fails this way at runtime is added to it, so each process pays for the failed request at most once per model. reasoning_effort is never sent otherwise, because older models such as gpt-4o-mini, and some OpenAI-compatible providers, reject the parameter.

set(['gpt-6-luna'])

smarter.apps.provider.services.text_completion.lib.openai_compatible_chat_provider.should_log(level)

Check if logging should be done based on the waffle switch.