manage.py verify_stackademy

Verify that the Stackademy LLMClients work, with a real prompt to each one.

class smarter.apps.llmclient.management.commands.verify_stackademy.Command(*args, **kwargs)[source]

Bases: SmarterCommand

Django manage.py verify_stackademy command.

Verifies that the Stackademy LLMClients of an account work: it sends each one a real, billable prompt, through the chat api, and fails unless the LLM called the LLMClient’s plugin, the plugin returned course catalogue data, and the LLM answered. It creates nothing.

The Stackademy resources are the manifests of the Stackademy project, https://github.com/smarter-sh/stackademy, which manage.py load_from_github applies: manage.py initialize_platform loads it for the Smarter admin, then runs this command. The platform provides what they query: the Stackademy test database, and the course catalogue.

stackademy_sql: its SqlPlugin queries the Stackademy test database, with a SqlConnection.
stackademy_api: its ApiPlugin queries this platform’s Stackademy course catalogue, /api/v1/demos/stackademy/course-catalogue/, with an ApiConnection.

add_arguments(parser)[source]: Add arguments to the command.

handle(*args, **options)[source]: Verify the Stackademy LLMClients.

help = 'Verify that the Stackademy LLMClients work, with a real, billable prompt to each one. Load https://github.com/smarter-sh/stackademy with load_from_github first.'

static is_empty_tool_result(message)[source]

True if a tool message holds no data, e.g. "" or [], which a failed plugin returns.

Parameters:: message (PromptChatApiMessage) – A message whose role is tool.
Returns:: Whether the tool returned nothing.
Return type:: bool

verify_llmclient(name, user_profile, token_key)[source]

Verify that a Stackademy LLMClient answers VERIFICATION_PROMPT with its plugin’s data.

The prompt is sent in-process to the chat api, as a client of version 2 of the chat contract would send it. The LLMClient works if the prompt succeeds, the LLM calls a tool, every tool call returns data, and the LLM answers.

Parameters:

name (str) – The LLMClient’s name.
user_profile (UserProfile) – The owner of the LLMClient.
token_key (str) – An api key of the owner.

Raises:

CommandError – If the LLMClient doesn’t work.

Return type:

None

verify_llmclients(user_profile)[source]

Verify that each of the Stackademy LLMClients works, with a single-use api key.

Parameters:: user_profile (UserProfile) – The owner of the Stackademy LLMClients.
Raises:: CommandError – If an LLMClient doesn’t work.
Return type:: None

smarter.apps.llmclient.management.commands.verify_stackademy.STACKADEMY_LLMCLIENTS = ('stackademy_sql', 'stackademy_api')

The Stackademy LLMClients, in STACKADEMY_REPOSITORY_URL.

('stackademy_sql', 'stackademy_api')

smarter.apps.llmclient.management.commands.verify_stackademy.VERIFICATION_PROMPT = 'Do you offer any courses on AI?': The prompt that each Stackademy LLMClient must answer with its plugin’s course catalogue data.