Smarter LLM Client

Attention

The term ‘LLMClient’ is used interchangeably with ‘Agent’, ‘Workflow Unit’ and ‘Harness’ throughout this documentation.

Overview

The Smarter LLMClient is the platform’s implementation of what is commonly known as an LLM harness. It provides the execution environment surrounding a large language model, managing the prompts, conversation state, tool execution, configuration, security, and runtime behavior required to transform a foundation model into a reliable, production-ready application.

LLMClients support both interactive conversations with human users and fully automated workflows. In addition to orchestrating requests to a language model, they leverage the Smarter Plugin architecture to invoke tools, retrieve data from private systems, access external APIs, execute code, and integrate other runtime capabilities. This allows developers to extend the model far beyond text generation while maintaining consistent interfaces, security boundaries, and execution policies.

Each LLMClient is defined by a YAML-based :doc:Smarter API Manifest (SAM) <../smarter-framework/smarter-api>, which describes its model configuration, prompts, plugins, user interface, behavior, and other runtime characteristics. Because every aspect of an LLMClient is declaratively configured, developers can easily customize and deploy specialized AI applications without modifying framework code.

LLMClients are managed using the Smarter command-line interface (CLI).

See also

Usage

# Smarter Prompt Engineer Workbench
curl -X POST http://localhost:9357/api/v1/llm-clients/9/chat/?session_key=e5c0368d6d7201b60f4f20c470f4b5ba36faf45e80ddbe8b04b6cf20f33167a7

# Deployed Smarter LLMClient - Production
curl -X GET https://stackademy.3141-5926-5359.api.example.com/chat/?session_key=<SESSION-KEY>

Example Manifest

Example Smarter Manifest
apiVersion: smarter.sh/v1
kind: LLMClient
metadata:
  name: stackademy_sql
  description: Stackademy University course catalogue inquiries using the Stackademy SQL plugin.
  version: 1.0.0
  tags:
    - stackademy
    - sql
    - llmclient
  annotations:
    - smarter.sh/llm-client/stackademy-sql/creator: QA Team
    - smarter.sh/llm-client/stackademy-sql/purpose: llmclient for Stackademy course catalogue inquiries
    - smarter.sh/llm-client/stackademy-sql/release-date: 2024-06-15
    - smarter.sh/llm-client/stackademy-sql/backward-compatibility: true
    - smarter.sh/llm-client/stackademy-sql/note: |
        Uses SQL plugin to answer course catalogue questions.
        Very fast and accurate when the database is well-structured.
        Leverages cache for improved performance.
spec:
  config:
    deployed: false
    provider: openai
    defaultModel: gpt-4o-mini
    defaultSystemRole: >
      You are a helpful assistant. When given the opportunity to utilize
      function calling, you should always do so. This will allow you to
      provide the best possible responses to the user. DO NOT GUESS. IF
        YOU DON'T KNOW THE ANSWER, RESPOND THAT YOU DON'T KNOW.
    defaultTemperature: 0.5
    defaultMaxTokens: 1024
    appName: Stackademy
    appAssistant: Stanley
    appWelcomeMessage: Welcome to Stackademy! How can I help you today?
    appExamplePrompts:
      - "Do you offer any courses on AI?"
      - "My budget is $1,000. What courses can I take?"
      - "I want to study programming. What do you suggest?"
    appPlaceholder: "Ask me anything about Stackademy courses..."
    appInfoUrl: https://stackademy.edu/online-courses
  plugins:
    - stackademy_sql
  functions: []

Technical Reference

Sandbox Mode

Smarter LLMClients can be operated in a ‘Sandbox Mode’, which restricts their capabilities to ensure safe experimentation and testing. In this mode, llmclient are only addressable using URL schemes that authenticate with Django sessions. That is, they cannot be accessed via API keys nor will they function using URL schemas such as stackademy.1234-5678-9012.api.example.com.

An example sandbox mode url:

https://platform.smarter.sh/workbench/stackademy-sql/chat/

Deploying

Deploy a Smarter LLMClient using the Smarter CLI. For example:

smarter deploy llmclient stackademy-sql

smarter deploy llmclient -h
Deploys a LLMClient:

smarter deploy llmclient <name> [flags]

The Smarter API will deploy the LLMClient.

Usage:
  smarter deploy llmclient <name> [flags]

Flags:
  -h, --help   help for llmclient

Global Flags:
      --api_key string         Smarter API key to use
      --config string          config file (default is $HOME/.smarter/config.yaml)
      --environment string     environment to use: local, alpha, beta, next, prod. Default is prod
  -o, --output_format string   output format: json, yaml (default "json")
  -v, --verbose                verbose output

Updating

Update a Smarter LLMClient using the Smarter CLI. For example:

smarter apply -f path/to/llmclient-manifest.yaml

Deleting

Delete a Smarter LLMClient using the Smarter CLI. For example:

smarter delete llmclient stackademy-sql

Testing

Test your Smarter LLMClient using the Smarter Workbench while in Sandbox Mode.

Smarter Prompt Engineering Workbench Demo

Monitoring

Use ad hoc Sql queries to monitor your Smarter LLMClient’s production performance and usage.

See:

Scaling

If you use Kubernetes Smarter LLMClients will scale seamlessly with demand. See https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ for more information.

Trouble Shooting

Beyond the Django models listed above, you should also check the smarter pod logs for any errors. Use the following command to view the logs:

kubectl logs -n <namespace> <smarter-pod-name>

for example,

kubectl logs -n smarter-platform-prod smarter-68f445c866-59lmp