Image Search LLMTool Plugin

An ImageSearchPlugin gives the LLM an image search. Its tool takes a search query, and returns a JSON list of strings: the https urls of the images that it found, for example ["https://upload.wikimedia.org/.../bald-eagle.jpg"]. It is powered by the Brave Image Search API.

The LLM supplies the search query in each tool call. The manifest’s spec.imageSearchData.queryTerms, if set, are appended to it: with queryTerms: black and white photograph, a query for formula 1 race cars searches for formula 1 race cars black and white photograph. spec.imageSearchData.searchParams map 1:1 to the API’s other query parameters: count, country, searchLang, safesearch (strict, Brave’s default, or off) and spellcheck. Brave cannot filter images by file type, size or license, so spec.imageSearchData.filters are applied by Smarter to Brave’s results:

  • fileType: only these file types, e.g. jpg|png, judged by each url’s file extension, or else by the Content-Type of its response.

  • minWidth and minHeight: only images at least this large. Brave reports the size of most images; those of unknown size are skipped. Use them to skip thumbnails.

  • allowedDomains and blockedDomains: only, or never, images from web pages on these domains. Blocked domains also apply to the image itself, since a website’s localized pages, e.g. on gettyimages.fr, often serve images from its main domain. The allowed domains are also sent to Brave as site: operators, so that Brave searches them. The blocked domains are not, since Brave returns no results for some queries with several -site: operators. Brave’s results are most relevant with one or two allowed domains.

Use them to tailor a plugin to one purpose, such as kid-friendly images from Wikimedia Commons, black and white photographs, technical diagrams or SVG images. The sample manifests in smarter/apps/llmtool/data/sample-plugins/image-search-*.yaml do just that.

Alternatively, let the LLM decide. With llmSearchParams: true, the LLM may set every search parameter in each tool call: count (up to 20), country, search_lang, safesearch and spellcheck, and searchParams are only the defaults. Invalid values are ignored, in favor of the defaults. The sample image_search (smarter/apps/llmtool/data/sample-plugins/image-search.yaml) does this, and its selector is always, so that its tool is offered with every prompt. It is the image search plugin of the example LLMClients.

Offering a tool does not make the LLM call it. Unless told otherwise, most models answer from what they already know and skip an optional image search. So the systemRole and the description (which becomes the tool’s description) of image_search are directive: they tell the LLM to search before answering any question about something that can be seen or drawn, even when the user did not ask for an image, and to skip the search for arithmetic, code and short conversational replies. Also check that the LLMClient’s own defaultSystemRole does not contradict this, for example by allowing only image urls that the user provided. Neither needs to explain Markdown image syntax, or forbid invented image urls: Smarter appends the chat window’s rendering rules, which say both, to the system prompt of every request.

Only https urls of the original images are returned, never Brave’s thumbnails and, unless validateUrls is false, only those that respond to a request with HTTP 200, and not with a web page, such as that of a website that blocks hotlinking.

Brave’s results are cached for cacheTtl seconds, and always come back in the same order. So that a repeated prompt, such as “show me cats”, does not always show the same image, the images that pass the filters are shuffled before each search picks count of them. Each tool call, including one answered from the cache, returns a random selection of the matching images, in random order.

Brave Search API key

The plugin needs a Brave Search API key, in the Smarter Secret that the manifest names in apiKey, which defaults to brave_search_api_key. The Brave samples of the Websearch Plugin use the same Secret.

  1. Sign up at the Brave Search API dashboard, subscribe to a plan that includes image search, and create an api key.

  2. Set SMARTER_BRAVE_SEARCH_API_KEY in .env, and run manage.py initialize_providers, which initialize_platform also runs, to store it as the Secret. Alternatively, apply a Secret manifest named brave_search_api_key.

If the Secret is missing, the plugin returns an empty list, and logs a warning that explains how to create it. If Brave rejects the key, or its plan’s quota is exceeded, the plugin also returns an empty list, and logs why.

Note

Experimental. The ImageSearchPlugin was designed and coded by Claude Code (Anthropic’s Claude Opus 5.5), with Lawrence McDaniel as co-author. It is experimental.

Technical References

Note

This is a complex AI resource that exists within the following class hierarchy

  1. Smarter Secret: a Brave Search API key.

  2. Smarter ImageSearch Plugin: The plugin that configures the image search parameters and filters.

  3. Smarter LLMClient: The prompting resource (LLMClient, Agent, Workflow unit, etcetera) that includes the ImageSearch Plugin.

Note

The ImageSearchPlugin is powered by the Brave Image Search API. Its manifest’s spec.imageSearchData.searchParams map 1:1 to the API’s query parameters, and its spec.imageSearchData.filters are applied by Smarter to the API’s results. The LLM supplies the query, and optionally a smaller number of images. The tool returns a JSON list of strings, e.g. ["https://example.com/cat.png"]:

  • Only the https urls of the original images are returned, never Brave’s thumbnails.

  • Images are filtered by file type, minimum size, and the domain of the web page that they come from.

  • The images that pass the filters are shuffled, so that each search, including one answered from the cache, returns a random selection of them, in random order.

  • Unless validateUrls is false, only urls that respond to a HEAD request with HTTP 200, and not with a web page, are returned. Requests are made with smarter.apps.llmtool.plugin.safe_http.fetch(), so only public hosts are contacted.

  • If the Brave Search API key is unavailable, or the search fails, the tool returns an empty list, and logs a warning. When the api key is missing, the warning explains how to create it.

Example ImageSearch Plugin Manifest
---
# An ImageSearchPlugin for kid-friendly image searches. Brave's safe search is
# strict, and images only come from Wikimedia Commons, whose media are all in the
# public domain or freely licensed. Thumbnails narrower than 400 pixels are
# skipped. The tool returns a JSON list of https image urls, each of which has
# been checked to respond with HTTP 200.
#
# Requires a Brave Search API key, in the Smarter Secret brave_search_api_key.
# Without it the plugin returns an empty list, and logs how to create it.
#
# Experimental: the ImageSearchPlugin was designed and coded by Claude Code (Anthropic's
# Claude Opus 5.5), with Lawrence McDaniel as co-author.
apiVersion: smarter.sh/v1
kind: ImageSearchPlugin
metadata:
  name: safe_image_search
  pluginClass: imagesearch
  description: Finds kid-friendly, freely licensed images from Wikimedia Commons. Use it to illustrate answers for children.
  version: 0.1.0
  tags:
    - imagesearch
    - kid-friendly
  annotations:
    - smarter.sh/safe-image-search/use-case: education
spec:
  selector:
    directive: search_terms
    # Complete search terms that will trigger the llmclient to use your customized system prompt.
    searchTerms:
      - kid-friendly
      - children
      - educational
      - safe
      - family-friendly
      - wholesome
      - child-appropriate
      - age-appropriate
      - child-safe
      - child-friendly
      - family-safe
  prompt:
    provider: openai
    systemRole: >
      You are a friendly assistant for children. When a picture would help to
      explain your answer, search for images, and include one or two of them in
      your answer as Markdown images.
    model: gpt-6-luna
    temperature: 1.0
    maxTokens: 2048
  imageSearchData:
    apiKey: brave_search_api_key
    searchParams:
      count: 5
      safesearch: strict
    filters:
      minWidth: 400
      allowedDomains:
        - wikimedia.org
    validateUrls: true
    timeout: 10

Note

Experimental. The ImageSearchPlugin was designed and coded by Claude Code (Anthropic’s Claude Opus 5.5), with Lawrence McDaniel as co-author. It is experimental.

smarter.apps.llmtool.plugin.image_search.BRAVE_MAX_COUNT = 200

The most images that one Brave Image Search API request returns.

smarter.apps.llmtool.plugin.image_search.BRAVE_MIN_COUNT = 50

The fewest images to request from Brave, since many may be filtered out.

Brave bills per request, not per image.

class smarter.apps.llmtool.plugin.image_search.ImageResult(url, page_url=None, width=None, height=None)[source]

Bases: object

An image found by the Brave Image Search API.

Variables:

  • url – The url of the original image.

  • page_url – The url of the web page on which the image appears, if known.

  • width – The image width in pixels, if known.

  • height – The image height in pixels, if known.

__init__(url, page_url=None, width=None, height=None)
height: int | None = None
page_url: str | None = None
url: str
width: int | None = None
class smarter.apps.llmtool.plugin.image_search.ImageSearchPlugin(*args, manifest=None, **kwargs)[source]

Bases: PluginBase

Implements a plugin that searches for images with the Brave Image Search API, and returns a list of image urls.

Key Features:

  • Image search via the Brave Image Search API, whose query parameters (count, country, language, safe search and spellcheck) are set by the manifest.

  • Filters for file type, minimum size, and the websites that images come from.

  • Returns only https urls, which, by default, are validated to respond with HTTP 200.

  • Returns an empty list, and logs how to create it, when its api key is missing.

  • Caching of search results, from which each search returns a random selection of images.

Example Use Cases:

  • Kid-friendly image search, with strict safe search, from Wikimedia Commons.

  • Technical diagrams, or SVG images, to illustrate documentation.

  • High resolution black and white photographs.

See also

PluginBase PluginDataImageSearch

SAMPluginType

alias of SAMImageSearchPlugin

__init__(*args, manifest=None, **kwargs)[source]

Initialize a PluginBase instance.

Key features:

  • Lazy instantiation of class properties.

  • Supports multiple initialization methods including manifest, database model, and YAML/JSON.

  • Integrates with Django ORM for plugin metadata management.

  • Utilizes Pydantic models for manifest validation and serialization.

  • Provides hooks for plugin selection and prompt customization.

  • Ensures plugin readiness and validation before use.

  • Sends signals upon plugin creation or update.

This constructor supports several ways to create a plugin object:

  • Manifest-based initialization:

    Pass a Pydantic model instance via the manifest argument. This is the preferred method and is typically used when loading plugins from a manifest broker.

  • Database model initialization:

    Provide a Django model plugin ID via plugin_id or a PluginMeta instance via plugin_meta to load an existing plugin from the database.

  • YAML/JSON manifest initialization:

    Supply a YAML or JSON string (or dictionary) via the data argument to create a plugin from a manifest representation.

See ./data/sample-plugins/everlasting-gobstopper.yaml for an example manifest.

The constructor sets up internal state, initializes plugin properties, and triggers plugin creation or update logic as needed. It also sends signals when the plugin is ready.

Parameters:

  • user_profile (Optional[UserProfile]) – The user profile associated with the plugin.

  • selected (bool) – Whether the plugin is initially selected.

  • api_version (Optional[str]) – The API version for the plugin manifest.

  • manifest (SAMImageSearchPlugin | None) – A Pydantic model representing the plugin manifest.

  • plugin_id (Optional[int]) – The Django ORM plugin ID.

  • plugin_meta (Optional[PluginMeta]) – The Django ORM PluginMeta instance.

  • data (Union[SmarterDictType, str, None]) – YAML/JSON manifest as a string or dictionary.

  • name (Optional[str]) – The name of the plugin.

  • kwargs – Additional keyword arguments.

api_key()[source]

Return the Brave Search API key, from a Secret that the plugin’s user may read.

If it is unavailable, a warning explains how to create it.

Return type:

str | None

Returns:

The api key, or None if there is no such Secret that the user may read.

brave_images(search_params)[source]

Search the Brave Image Search API.

More images than wanted are requested, since many may be filtered out. The allowed domains are added to the query as site: operators, so that Brave searches them. The blocked domains are not, since Brave returns no results for some queries with several -site: operators: filtered_images() removes their images instead.

Responses are cached for the plugin’s cacheTtl seconds, keyed on the exact query parameters sent to Brave, so the plugin’s filters, which Smarter applies afterwards, are never part of the key. Failed requests are never cached.

Parameters:

search_params (dict[str, Any]) – The query parameters, from PluginDataImageSearch.search_params().

Return type:

list[ImageResult]

Returns:

The images found, or an empty list if the api key is missing, or the search fails.

property custom_tool: dict[str, Any] | None

Return the plugin tool definition for OpenAI function calling.

Returns:

The tool definition, or None if the plugin is not ready.

Example:

tool = {
    "type": "function",
    "function": {
        "name": "smarter_plugin_0000000042",
        "description": "Kid-friendly image search. Set query to search for images ...",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "maxLength": 400, "description": "..."},
                "count": {"type": "integer", "minimum": 1, "maximum": 10, "description": "..."},
            },
            "required": ["query"],
        },
    },
}
classmethod example_manifest(kwargs=None)[source]

Use Pydantic models to generate an example manifest for an ImageSearchPlugin.

Parameters:

kwargs (dict[str, Any] | None) – Optional keyword arguments to customize the example manifest.

Return type:

dict[str, Any] | None

Returns:

An example manifest as a dictionary.

See Also:

  • SAMImageSearchPlugin

  • smarter/apps/llmtool/data/sample-plugins/image-search-*.yaml for more examples.

filtered_images(images)[source]

Apply the plugin’s filters to the images, except for file types that only a request can determine.

Images are removed if their url is not https or is a duplicate, their web page is not permitted by the domain policy, the image itself is on a blocked domain, they are smaller than the minimum size, or of unknown size when a minimum is set, or their url’s file extension is not a permitted file type. Blocked domains apply to the image too, since a website’s localized pages, e.g. on example.fr, often serve images from its main domain. Images without a file extension are kept for validated_urls(), if url validation is enabled.

Parameters:

images (list[ImageResult]) – The images found.

Return type:

list[ImageResult]

Returns:

The images that pass the filters, in order.

property kind: str

Returns the kind identifier for this plugin.

Returns:

ImageSearchPlugin

Return type:

str

property manifest: SAMImageSearchPlugin | None

Return the Pydantic model representation of the plugin manifest.

If the manifest has not been set but the plugin is in a ready state, it is reconstructed from the Django ORM data, using to_json().

Returns:

The manifest, or None if unavailable.

Return type:

Optional[SAMImageSearchPlugin]

property plugin_data: PluginDataImageSearch | None

Return the plugin data as a Django ORM instance.

Returns:

The image search configuration, or None if unavailable.

Return type:

Optional[PluginDataImageSearch]

property plugin_data_class: Type[PluginDataImageSearch]

Return the Django ORM class used for image search plugin data.

property plugin_data_django_model: dict[str, Any] | None

Transform the Pydantic manifest into a Django ORM-compatible dictionary.

Returns:

A dictionary of PluginDataImageSearch fields, or None if the manifest is not available.

property plugin_data_serializer: PluginImageSearchSerializer | None

Return the serializer instance for the plugin’s image search data.

property plugin_data_serializer_class: Type[PluginImageSearchSerializer]

Return the plugin data serializer class.

search(query, count=None, llm_params=None)[source]

Search for images.

Parameters:

  • query (str) – The search query.

  • count (Any) – The maximum number of images, up to the plugin’s count, or, if the LLM may set every search parameter, up to MAX_COUNT.

  • llm_params (dict[str, Any] | None) – The other search parameters that the LLM set, which are used only if the plugin’s llmSearchParams is true. See PluginDataImageSearch.search_params().

Return type:

list[str]

Returns:

The https urls of a random selection of the images found, in random order. Empty if none were found, the api key is missing, or the search failed.

to_json(version='v1')[source]

Serialize the ImageSearchPlugin to a JSON-compatible dictionary suitable for Pydantic import.

Parameters:

version (str) – The API version to use for serialization. Only “v1” is supported.

Return type:

dict[str, Any] | None

Returns:

A dictionary representing the plugin in JSON format, or None if the plugin is not ready.

Raises:

SmarterPluginError – If an unsupported version is specified.

tool_call_fetch_plugin_response(function_args)[source]

Search for images in response to a tool call.

Example tool call payload:

"arguments": "{\"query\": \"bald eagle\", \"count\": 3}"
"arguments": "{\"query\": \"tour de france\", \"country\": \"fr\", \"search_lang\": \"fr\"}"

Example response:

["https://upload.wikimedia.org/wikipedia/commons/1/1a/bald-eagle.jpg", "https://..."]
Parameters:

function_args (dict[str, Any] | str | None) – The function arguments, as a dict or as the JSON string sent by OpenAI.

Return type:

dict[str, Any] | list | str

Returns:

A list of https image urls. Empty if the arguments are invalid, no images were found, the api key is missing, or the search failed.

Raises:

SmarterImageSearchPluginError – If the plugin is not ready, or has no data.

property tool_description: str

what the plugin does, and how to use it.

Type:

The tool description presented to the LLM

validated_urls(images, count)[source]

Return the urls of up to count images, in order, that respond with HTTP 200, if validation is enabled.

Images whose file type could not be determined from their url are checked against the permitted file types by the Content-Type of their response.

Parameters:

  • images (list[ImageResult]) – The filtered images.

  • count (int) – The maximum number of urls to return.

Return type:

list[str]

smarter.apps.llmtool.plugin.image_search.LLM_SEARCH_PARAMS = ('country', 'search_lang', 'safesearch', 'spellcheck')

The search parameters, besides query and count, that the LLM may set if the plugin’s llmSearchParams is true.

('country', 'search_lang', 'safesearch', 'spellcheck')
smarter.apps.llmtool.plugin.image_search.OVERFETCH_FACTOR = 5

Request at least this many times the number of images wanted from Brave, since many may be filtered out.

smarter.apps.llmtool.plugin.image_search.REJECTED_CONTENT_TYPE_PREFIXES = ('text/', 'application/json', 'application/xml', 'application/xhtml')

Content-Types of responses that are not images, e.g. the html page of a hotlink protected image.

('text/', 'application/json', 'application/xml', 'application/xhtml')
exception smarter.apps.llmtool.plugin.image_search.SmarterImageSearchPluginError(message='')[source]

Bases: SmarterPluginError

Base class for all ImageSearch plugin errors.

exception smarter.apps.llmtool.plugin.image_search.SmarterImageSearchRequestError(message='')[source]

Bases: SmarterImageSearchPluginError

Raised by a cached Brave Image Search API request that failed, so that failures are never cached.

smarter.apps.llmtool.plugin.image_search.as_dimension(value)[source]

Return an image width or height reported by the API as a positive int, or None.

Return type:

int | None

smarter.apps.llmtool.plugin.image_search.describe_brave_error(error)[source]

Describe a failed Brave Image Search API request, without disclosing the api key.

Return type:

str

smarter.apps.llmtool.plugin.image_search.file_type_of_content_type(content_type)[source]

Return the file type of a response, from its Content-Type.

Parameters:

content_type (str | None) – The media type, e.g. image/png.

Return type:

str | None

Returns:

The file type, e.g. png, or None if the Content-Type is not a known image type.

smarter.apps.llmtool.plugin.image_search.file_type_of_url(url)[source]

Return the file type of an image url, from its file extension.

Parameters:

url (str) – The image url.

Return type:

str | None

Returns:

The file type, e.g. jpg, or None if the url has no known image file extension.

smarter.apps.llmtool.plugin.image_search.image_content_type(url, timeout)[source]

Return the Content-Type of an image url, if a HEAD request to it responds with HTTP 200 and not with a web page.

This runs in worker threads, so it neither logs nor touches the database: the waffle switched logger reads its switch from the database, which would open a connection per thread.

Parameters:

  • url (str) – The https url of an image.

  • timeout (float) – Seconds to wait for each request.

Return type:

str | None

Returns:

The Content-Type, which is empty if the server did not send one, or None if the url is not available.

smarter.apps.llmtool.plugin.image_search.is_https_url(url)[source]

Return True if the url is an https url with a host.

Parameters:

url (Any) – The url.

Return type:

bool

smarter.apps.llmtool.plugin.image_search.parse_brave_images(data)[source]

Parse a Brave Image Search API response.

Parameters:

data (Any) – The response, e.g. {"results": [{"url": "https://page", "properties": {"url": "https://image.png"}}]}.

Return type:

list[ImageResult]

Returns:

The images, in Brave’s order. Results without an original image url are skipped.

smarter.apps.llmtool.plugin.image_search.should_log(level)[source]

Check if logging should be done based on the waffle switch.

smarter.apps.llmtool.plugin.image_search.shuffled(images)[source]

Return the images in random order.

Brave’s results for a query, and therefore the cached results, are always in the same order. Shuffling them before ImageSearchPlugin.validated_urls() picks the first count valid images means that repeated searches, e.g. “show me cats”, return a different selection of the cached images each time, rather than always the same ones.

Parameters:

images (list[ImageResult]) – The filtered images.

Return type:

list[ImageResult]

Returns:

A new list of the same images, in random order.