Smarter LLMClient

Attention

The term ‘LLMClient’ is used interchangeably with ‘Agent’ and ‘Workflow Unit’ throughout this documentation.

Overview

Smarter LLMClients are highly advanced conversational agents designed to provide intelligent and context-aware interactions with human users as well as fully automated workflows. They leverage the vanguard of generative AI text completion technology to deliver personalized and efficient responses. Namely, these llm_clients leverage the Smarter Plugin architecture, which provides extensible tool integration capabilities, including secure access to to private, secure data sources and external APIs.

LLMClients are recognized by the yaml-based Smarter API Manifest (SAM) architecture, facilitating both behavioral and visual customization. This allows developers to tailor the llm_client’s functionality and appearance to meet specific use cases and user preferences.

LLMClients are managed with the Smarter command-line interface (CLI).

See also

Usage

# Smarter Prompt Engineer Workbench
curl -X POST http://localhost:9357/api/v1/llm-clients/9/chat/?session_key=e5c0368d6d7201b60f4f20c470f4b5ba36faf45e80ddbe8b04b6cf20f33167a7

# Deployed Smarter LLMClient - Production
curl -X GET https://stackademy.3141-5926-5359.api.example.com/chat/?session_key=<SESSION-KEY>

Example Manifest

Example Smarter Manifest
apiVersion: smarter.sh/v1
kind: LLMClient
metadata:
  name: stackademy_sql
  description: Stackademy University course catalogue inquiries using the Stackademy SQL plugin.
  version: 1.0.0
  tags:
    - stackademy
    - sql
    - llm_client
  annotations:
    - smarter.sh/llm-client/stackademy-sql/creator: QA Team
    - smarter.sh/llm-client/stackademy-sql/purpose: llm_client for Stackademy course catalogue inquiries
    - smarter.sh/llm-client/stackademy-sql/release-date: 2024-06-15
    - smarter.sh/llm-client/stackademy-sql/backward-compatibility: true
    - smarter.sh/llm-client/stackademy-sql/note: |
        Uses SQL plugin to answer course catalogue questions.
        Very fast and accurate when the database is well-structured.
        Leverages cache for improved performance.
spec:
  config:
    deployed: false
    provider: openai
    defaultModel: gpt-4o-mini
    defaultSystemRole: >
      You are a helpful assistant. When given the opportunity to utilize
      function calling, you should always do so. This will allow you to
      provide the best possible responses to the user. DO NOT GUESS. IF
        YOU DON'T KNOW THE ANSWER, RESPOND THAT YOU DON'T KNOW.
    defaultTemperature: 0.5
    defaultMaxTokens: 1024
    appName: Stackademy
    appAssistant: Stanley
    appWelcomeMessage: Welcome to Stackademy! How can I help you today?
    appExamplePrompts:
      - "Do you offer any courses on AI?"
      - "My budget is $1,000. What courses can I take?"
      - "I want to study programming. What do you suggest?"
    appPlaceholder: "Ask me anything about Stackademy courses..."
    appInfoUrl: https://stackademy.edu/online-courses
  plugins:
    - stackademy_sql
  functions: []

Technical Reference

Sandbox Mode

Smarter LLMClients can be operated in a ‘Sandbox Mode’, which restricts their capabilities to ensure safe experimentation and testing. In this mode, llm_clients are only addressable using URL schemes that authenticate with Django sessions. That is, they cannot be accessed via API keys nor will they function using URL schemas such as stackademy.1234-5678-9012.api.example.com.

An example sandbox mode url:

https://platform.smarter.sh/workbench/stackademy-sql/chat/

Deploying

Deploy a Smarter LLMClient using the Smarter CLI. For example:

smarter deploy llm_client stackademy-sql

smarter deploy llm_client -h
Deploys a LLMClient:

smarter deploy llm_client <name> [flags]

The Smarter API will deploy the LLMClient.

Usage:
  smarter deploy llm_client <name> [flags]

Flags:
  -h, --help   help for llm_client

Global Flags:
      --api_key string         Smarter API key to use
      --config string          config file (default is $HOME/.smarter/config.yaml)
      --environment string     environment to use: local, alpha, beta, next, prod. Default is prod
  -o, --output_format string   output format: json, yaml (default "json")
  -v, --verbose                verbose output

Updating

Update a Smarter LLMClient using the Smarter CLI. For example:

smarter apply -f path/to/llm_client-manifest.yaml

Deleting

Delete a Smarter LLMClient using the Smarter CLI. For example:

smarter delete llm_client stackademy-sql

Testing

Test your Smarter LLMClient using the Smarter Workbench while in Sandbox Mode.

Smarter Prompt Engineering Workbench Demo

Monitoring

Use ad hoc Sql queries to monitor your Smarter LLMClient’s production performance and usage.

See:

Scaling

If you use Kubernetes Smarter LLMClients will scale seamlessly with demand. See https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ for more information.

Trouble Shooting

Beyond the Django models listed above, you should also check the smarter pod logs for any errors. Use the following command to view the logs:

kubectl logs -n <namespace> <smarter-pod-name>

for example,

kubectl logs -n smarter-platform-prod smarter-68f445c866-59lmp