Smarter LLM Client

Attention

The term ‘LLMClient’ is used interchangeably with ‘Agent’, ‘Workflow Unit’ and ‘Harness’ throughout this documentation.

Overview

The Smarter LLMClient is the platform’s implementation of what is commonly known as an LLM harness. It provides the execution environment surrounding a large language model, managing the prompts, conversation state, tool execution, configuration, security, and runtime behavior required to transform a foundation model into a reliable, production-ready application.

LLMClients support both interactive conversations with human users and fully automated workflows. In addition to orchestrating requests to a language model, they leverage the Smarter Plugin architecture to invoke tools, retrieve data from private systems, access external APIs, execute code, and integrate other runtime capabilities. This allows developers to extend the model far beyond text generation while maintaining consistent interfaces, security boundaries, and execution policies.

Each LLMClient is defined by a YAML-based :doc:Smarter API Manifest (SAM) <../smarter-framework/smarter-api>, which describes its model configuration, prompts, plugins, user interface, behavior, and other runtime characteristics. Because every aspect of an LLMClient is declaratively configured, developers can easily customize and deploy specialized AI applications without modifying framework code.

LLMClients are managed using the Smarter command-line interface (CLI).

See also

Usage

# Smarter Prompt Engineer Workbench
curl -X POST http://localhost:9357/api/v1/llm-clients/9/chat/?session_key=e5c0368d6d7201b60f4f20c470f4b5ba36faf45e80ddbe8b04b6cf20f33167a7

# Deployed Smarter LLMClient - Production
curl -X GET https://stackademy.3141-5926-5359.api.example.com/chat/?session_key=<SESSION-KEY>

Web Pages That Embed It

A deployed LLMClient can be the chat backend of any web page, with Smarter Chat, the @smarter.sh/ui-chat React component. The page calls the LLMClient from the browser, so its origin must be allowed, in spec.config.prompt.allowedOrigins:

spec:
  config:
    prompt:
      allowedOrigins:
        - https://www.example.com
        - http://localhost:5173

Each origin is a scheme, a host and an optional port, without a path or a wildcard. If the LLMClient has an api key, the page must send one; without api keys, the LLMClient is public, and its prompts are charged to its owner. See Embedding Smarter Chat.

Styling the Chat

An LLMClient’s manifest splits spec.config in two. prompt, which is required, is what the LLMClient is: its LLM provider, model and system prompt, and the words of its chat. styles, which is optional, is how its chat looks, in the workbench and on every page that embeds it: its logo, avatars, background image, font and colors.

spec:
  config:
    prompt:
      appName: Stackademy
      appAssistant: Stanley
    styles:
      appLogoUrl: https://cdn.example.com/stackademy/logo.png
      googleFont: Nunito
      conversationHeader:
        backgroundColor: "#14213d"
        color: "#ffffff"
      outgoingMessage:
        backgroundColor: "#fca311"

No CSS or web development is needed. Styling Your LLMClient’s Chat explains every setting, with a crash course in colors, sizes, fonts and images, and a case study.

Example Manifest

Example Smarter Manifest
apiVersion: smarter.sh/v1
kind: LLMClient
metadata:
  name: stackademy_sql
  description: Stackademy University course catalogue inquiries using the Stackademy SQL plugin.
  version: 1.0.0
  tags:
    - stackademy
    - sql
    - llmclient
  annotations:
    - smarter.sh/llm-client/stackademy-sql/creator: QA Team
    - smarter.sh/llm-client/stackademy-sql/purpose: llmclient for Stackademy course catalogue inquiries
    - smarter.sh/llm-client/stackademy-sql/release-date: 2024-06-15
    - smarter.sh/llm-client/stackademy-sql/backward-compatibility: true
    - smarter.sh/llm-client/stackademy-sql/note: |
        Uses SQL plugin to answer course catalogue questions.
        Very fast and accurate when the database is well-structured.
        Leverages cache for improved performance.
spec:
  config:
    prompt:
      deployed: false
      provider: openai
      defaultModel: gpt-6-luna
      defaultSystemRole: >
        You are a helpful assistant. When given the opportunity to utilize
        function calling, you should always do so. This will allow you to
        provide the best possible responses to the user. DO NOT GUESS. IF
          YOU DON'T KNOW THE ANSWER, RESPOND THAT YOU DON'T KNOW.
      defaultTemperature: 1.0
      defaultMaxTokens: 1024
      appName: Stackademy
      appAssistant: Stanley
      appWelcomeMessage: Welcome to Stackademy! How can I help you today?
      appExamplePrompts:
        - "Do you offer any courses on AI?"
        - "My budget is $1,000. What courses can I take?"
        - "I want to study programming. What do you suggest?"
      appPlaceholder: "Ask me anything about Stackademy courses..."
      appInfoUrl: https://stackademy.edu/online-courses
    styles:  # Stackademy's colors
      # appLogoUrl: https://cdn.smarter.sh/images/llmclients/stackademy/logo.png  # coming soon: upload, then uncomment
      # appBackgroundImageUrl: https://cdn.smarter.sh/images/llmclients/stackademy/background.png  # coming soon: upload, then uncomment
      # assistantAvatarUrl: https://cdn.smarter.sh/images/llmclients/stackademy/assistant.png  # coming soon: upload, then uncomment
      googleFont: Nunito
      fontSize: 15px
      borderRadius: 16px
      conversationHeader:
        backgroundColor: '#14213d'
        color: '#ffffff'
        infoColor: '#fca311'
      messageList:
        backgroundColor: rgba(255, 255, 255, 0.85)
        color: '#14213d'
      incomingMessage:
        backgroundColor: '#e5e5e5'
        color: '#14213d'
        linkColor: '#0b57d0'
      outgoingMessage:
        backgroundColor: '#fca311'
        color: '#14213d'
      messageInput:
        backgroundColor: '#ffffff'
        editorBackgroundColor: '#f1f1f1'
        placeholderColor: '#6c757d'
      buttons:
        color: '#fca311'
      typingIndicator:
        color: '#fca311'
  plugins:
    - stackademy_sql
  functions: []

Technical Reference

Sandbox Mode

Smarter LLMClients can be operated in a ‘Sandbox Mode’, which restricts their capabilities to ensure safe experimentation and testing. In this mode, llmclient are only addressable using URL schemes that authenticate with Django sessions. That is, they cannot be accessed via API keys nor will they function using URL schemas such as stackademy.1234-5678-9012.api.example.com.

An example sandbox mode url:

https://platform.smarter.sh/workbench/stackademy-sql/chat/

Deploying

Deploy a Smarter LLMClient using the Smarter CLI. For example:

smarter deploy llmclient stackademy-sql

smarter deploy llmclient -h
Deploys a LLMClient:

smarter deploy llmclient <name> [flags]

The Smarter API will deploy the LLMClient.

Usage:
  smarter deploy llmclient <name> [flags]

Flags:
  -h, --help   help for llmclient

Global Flags:
      --api_key string         Smarter API key to use
      --config string          config file (default is $HOME/.smarter/config.yaml)
      --environment string     environment to use: local, alpha, beta, next, prod. Default is prod
  -o, --output_format string   output format: json, yaml (default "json")
  -v, --verbose                verbose output

Updating

Update a Smarter LLMClient using the Smarter CLI. For example:

smarter apply -f path/to/llmclient-manifest.yaml

Deleting

Delete a Smarter LLMClient using the Smarter CLI. For example:

smarter delete llmclient stackademy-sql

Testing

Test your Smarter LLMClient using the Smarter Workbench while in Sandbox Mode.

Smarter Prompt Engineering Workbench Demo

Monitoring

Use ad hoc Sql queries to monitor your Smarter LLMClient’s production performance and usage.

See:

Scaling

If you use Kubernetes Smarter LLMClients will scale seamlessly with demand. See https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ for more information.

Trouble Shooting

Beyond the Django models listed above, you should also check the smarter pod logs for any errors. Use the following command to view the logs:

kubectl logs -n <namespace> <smarter-pod-name>

for example,

kubectl logs -n smarter-platform-prod smarter-68f445c866-59lmp