Natural Language Processing (NLP)

A plugin whose selector directive is search_terms is selected for a prompt that refers to one of its search terms. smarter.apps.llmtool.nlp.does_refer_to() decides, trying three kinds of matching in turn, and stopping at the first that succeeds:

  1. Exact: the prompt contains every word of the search term, in any order, ignoring case, punctuation and plurals. Tell me about Gobstoppers refers to Gobstopper. This is always on.

  2. Fuzzy: the prompt contains the search term with a typo or two, depending on its length. Who is Lawrance McDaniel? refers to Lawrence McDaniel. The enable_plugin_fuzzy_matching waffle switch, ENABLE_PLUGIN_FUZZY_MATCHING, controls it, and is on by default.

  3. Thesaurus: the prompt contains a synonym of the search term, or of each of its words, from the WordNet thesaurus. I want to buy an automobile refers to car, and a monochrome photograph refers to black and white. The enable_plugin_thesaurus_matching waffle switch, ENABLE_PLUGIN_THESAURUS_MATCHING, controls it, and is on by default.

Plugin selectors try exact and fuzzy matching on the prompt and on every user message first, and thesaurus matching last, on the prompt only.

Thesaurus matching

Thesaurus matching selects plugins more often than the others, since a word has many synonyms. To turn it off, if it selects plugins for prompts that do not need them, use Django admin, under Waffle > Switches, or:

python manage.py waffle_switch enable_plugin_thesaurus_matching off

To keep plugin selection precise, synonyms come only from the two most common senses of each word, and only from its noun senses if it is a noun, since search terms are almost always nouns. For example, the verb image means see, so Let me see the code does not refer to image. Words of fewer than three letters, mostly pronouns and abbreviations, must match exactly. WordNet does not know brand names, product names or most abbreviations, such as admin, so search terms like these only match exactly, or with a typo.

WordNet is read with NLTK, whose corpus data the Docker image installs into /home/smarter_user/nltk_data. Elsewhere, install it with python -m nltk.downloader wordnet. Without it, thesaurus matching is disabled, with a warning in the log.

Observability

Each successful match sends the search_term_matched signal, with the prompt, the search term, the kind of matching that succeeded (exact, fuzzy or thesaurus), and, for a thesaurus match, the synonym that matched. The plugin app’s receiver logs it, so that you can see why each plugin was selected.

Technical Reference

Natural language processing functions for OpenAI API-compatible providers.

class smarter.apps.llmtool.nlp.MatchMethod[source]

Bases: object

How does_refer_to() matched a search term.

Sent as the method of search_term_matched.

EXACT = 'exact'

Every word of the search term is in the prompt.

See simple_search().

FUZZY = 'fuzzy'

The search term is in the prompt, with a typo or two.

See within_levenshtein_distance().

THESAURUS = 'thesaurus'

The search term, or a synonym of each of its words, is in the prompt.

See thesaurus_match().

smarter.apps.llmtool.nlp.PLURAL_SUFFIXES = ('s', 'es')

Suffixes with which a prompt’s word still matches a search term’s word, e.g. Gobstoppers and Gobstopper.

('s', 'es')
smarter.apps.llmtool.nlp.THESAURUS_MAX_SENSES = 2

The number of a word’s most common senses from which its synonyms come.

A word has many senses in WordNet, ordered from the most to the least common. For example, the fifth noun sense of car is an elevator car. Synonyms of rare senses would select plugins for prompts that have nothing to do with them. For the same reason, a word that is a noun only has the synonyms of its noun senses: the verb image means see, so that let me see the code would otherwise refer to image.

smarter.apps.llmtool.nlp.THESAURUS_MIN_WORD_LENGTH = 3

Search term words shorter than this are matched exactly, never by synonym.

Short words are mostly pronouns and abbreviations, whose WordNet senses are rarely the intended ones. For example, me is the abbreviation of Maine.

smarter.apps.llmtool.nlp.allowed_distance(search_term)[source]

Return the number of typos to tolerate in a search term.

Longer search terms tolerate more typos, and short ones none, because a short word is only a few edits away from many unrelated words. For example, API is within three edits of and, the and any.

  • Search terms of fewer than 5 characters: 0.

  • Search terms of 5 to 8 characters: 1.

  • Longer search terms: 2.

Spaces are not counted.

Parameters:

search_term (str) – The search term.

Returns:

The maximum Levenshtein distance at which the search term still matches.

Return type:

int

smarter.apps.llmtool.nlp.clean_prompt(prompt)[source]

Clean up a prompt by inserting spaces before capital letters.

This function transforms concatenated or camel-cased words into a more readable format by adding spaces before capital letters, except for names starting with “Mc”. Useful for improving prompt clarity in NLP tasks.

Parameters:

prompt (str) – The input string to clean.

Returns:

The cleaned string with spaces before capital letters.

Return type:

str

Note

This is a simple heuristic and may not handle all edge cases perfectly. For example, names starting with “Mc” (e.g., “McDaniel”) are not split.

Tip

Use this function to preprocess user input or model prompts for better readability.

See also

Example usage:

from smarter.apps.llmtool.nlp import clean_prompt

s = "WhoIsLawrenceMcDaniel"
print(clean_prompt(s))
# Output: "Who Is Lawrence McDaniel"
smarter.apps.llmtool.nlp.contains_phrase(prompt_words, phrase)[source]

Check if a list of words contains a phrase, as consecutive words.

Each word matches with word_matches(), so plurals match, e.g. motor vehicles contains the phrase motor vehicle.

Parameters:
Returns:

True if the words contain the phrase, otherwise False.

Return type:

bool

smarter.apps.llmtool.nlp.does_refer_to(prompt, search_term, threshold=None, fuzzy=None, use_thesaurus=None)[source]

Check if the prompt refers to the given string.

This function determines whether a prompt refers to a search term by first cleaning the prompt with clean_prompt(), and then searching it with simple_search(). If typo tolerant matching is enabled, it then also tries within_levenshtein_distance() and, if thesaurus matching is enabled, thesaurus_match(). When the prompt refers to the search term, the search_term_matched signal is sent, with the MatchMethod that matched.

Parameters:

  • prompt (str) – The input string to analyze.

  • search_term (str) – The target string to check for reference.

  • threshold (int | None) – An optional maximum Levenshtein distance for typo tolerant matching. See within_levenshtein_distance().

  • fuzzy (bool | None) – Whether to tolerate typos. Defaults to the enable_plugin_fuzzy_matching waffle switch, which is on by default.

  • use_thesaurus (bool | None) – Whether to match synonyms, from the WordNet thesaurus. Defaults to the enable_plugin_thesaurus_matching waffle switch, which is on by default.

Returns:

True if the prompt refers to the search term, otherwise False.

Return type:

bool

Important

Plugin selectors use this function to decide whether to select a plugin, which adds the plugin’s system prompt and tools to the conversation. Typo tolerant matching selects plugins more often, and it is controlled by SmarterWaffleSwitches.ENABLE_PLUGIN_FUZZY_MATCHING. Thesaurus matching selects plugins more often still, and it is controlled by SmarterWaffleSwitches.ENABLE_PLUGIN_THESAURUS_MATCHING.

See also

Example usage:

from smarter.apps.llmtool.nlp import does_refer_to

prompt = "WhoIsLawranceMcDaniel"
print(does_refer_to(prompt, "Lawrence McDaniel", fuzzy=True))   # True
print(does_refer_to(prompt, "Lawrence McDaniel", fuzzy=False))  # False
print(does_refer_to(prompt, "John Doe", fuzzy=True))            # False
print(does_refer_to("buy an automobile", "car", use_thesaurus=True))  # True
smarter.apps.llmtool.nlp.lower_case_splitter(string_of_words)[source]

Split a string on spaces and return a list of lowercase words.

This function tokenizes a string by spaces and converts each token to lowercase. Useful for case-insensitive text processing, search, and normalization in NLP tasks.

Parameters:

string_of_words (str) – The input string to split and lowercase.

Returns:

List of lowercase words.

Return type:

list

Tip

Use this function to prepare text for matching, searching, or comparison.

See also

Example usage:

from smarter.apps.llmtool.nlp import lower_case_splitter

s = "The Quick Brown Fox"
print(lower_case_splitter(s))
# Output: ['the', 'quick', 'brown', 'fox']
smarter.apps.llmtool.nlp.normalized_words(text)[source]

Split a string into lowercase words, without punctuation.

Parameters:

text (str) – The input string.

Returns:

List of lowercase words, from which punctuation is removed.

Return type:

list[str]

Example usage:

from smarter.apps.llmtool.nlp import normalized_words

print(normalized_words("Who is Lawrence P. McDaniel?"))
# Output: ['who', 'is', 'lawrence', 'p', 'mcdaniel']

Check if the prompt contains every word of the search term.

This function performs a case-insensitive, whole word search. It returns True if each word of the search_term appears in the prompt, in any order, ignoring punctuation. A word also matches its plural, e.g. Gobstoppers matches the search term Gobstopper, but it does not match a longer word that merely contains it, e.g. concatenate does not match the search term cat.

Parameters:

  • prompt (str) – The input string to search within.

  • search_term (str) – The target string or phrase to look for.

Returns:

True if the search term is found in the prompt, otherwise False. An empty search term is never found.

Return type:

bool

Caution

This function does not tolerate typos. For typo tolerant matching, see within_levenshtein_distance().

See also

Example usage:

from smarter.apps.llmtool.nlp import simple_search

prompt = "Find all weather plugins for New York"
print(simple_search(prompt, "weather plugins"))  # True
print(simple_search(prompt, "Weather"))          # True
print(simple_search(prompt, "plugin"))           # True
print(simple_search(prompt, "California"))       # False
print(simple_search(prompt, "the"))              # False
smarter.apps.llmtool.nlp.synonyms(phrase: str, nouns_only: bool = False) → frozenset[str][source]

Return the synonyms of a word or phrase, from the most common senses of WordNet.

Synonyms come from the first THESAURUS_MAX_SENSES noun senses of the phrase, or, if it is not a noun, from the first senses of each of its other parts of speech. They are lower case, and multi-word synonyms have spaces, e.g. motor vehicle. The phrase itself is not included.

Parameters:

  • phrase (str) – A lower case word, e.g. car, or phrase, e.g. black and white.

  • nouns_only (bool) – Whether a phrase that is not a noun has no synonyms, rather than those of its other parts of speech.

Returns:

The synonyms, which are empty if WordNet is not installed, or does not know the phrase.

Return type:

frozenset[str]

Example usage:

from smarter.apps.llmtool.nlp import synonyms

print(sorted(synonyms("car")))
# Output: ['auto', 'automobile', 'machine', 'motorcar', 'railcar', 'railroad car', 'railway car']
print(sorted(synonyms("image")))
# Output: ['mental image', 'persona']
smarter.apps.llmtool.nlp.thesaurus_match(prompt, search_term)[source]

Check if the prompt contains the search term, or synonyms of it, from the WordNet thesaurus.

The prompt matches if it contains a synonym of the whole search term, e.g. monochrome for black and white, or if, for each word of the search term, it contains the word, one of its synonyms(), e.g. automobile for car, or a noun of which it is a synonym. WordNet’s synonyms are not symmetric: image is a synonym of picture, but picture is not a synonym of the noun image. Only nouns count in this direction, since weather is a synonym of the verb endure. Words shorter than THESAURUS_MIN_WORD_LENGTH must match exactly. Synonyms come only from each word’s most common senses, to keep plugin selection precise.

Parameters:

  • prompt (str) – The input string to search within.

  • search_term (str) – The target string or phrase to look for.

Returns:

The words of the prompt that matched, e.g. automobile, or None if the prompt does not match, or WordNet is not installed.

Return type:

str | None

See also

Example usage:

from smarter.apps.llmtool.nlp import thesaurus_match

print(thesaurus_match("I want to buy an automobile", "car"))  # 'automobile'
print(thesaurus_match("show me a picture of a cat", "image"))  # 'picture'
print(thesaurus_match("what is the weather", "car"))           # None
smarter.apps.llmtool.nlp.within_levenshtein_distance(prompt, search_term, threshold=None)[source]

Check if the prompt contains the search term, allowing for typos.

This function compares the search term, case-insensitively and without punctuation, with each run of consecutive words in the prompt that has as many words as the search term. It returns True if any run is within the search term’s allowed_distance() of it, measured as Levenshtein distance. For example, the search term Lawrence McDaniel matches Lawrance McDaniel, and Gobstopper matches Gobbstopper.

Parameters:

  • prompt (str) – The input string to search within.

  • search_term (str) – The target string to compare against.

  • threshold (int | None) – An optional maximum Levenshtein distance, which can only make the match stricter than allowed_distance().

Returns:

True if a run of words in the prompt is within the allowed distance of the search term, otherwise False. Search terms of fewer than 5 characters never match, because they tolerate no typos.

Return type:

bool

See also

Example usage:

from smarter.apps.llmtool.nlp import within_levenshtein_distance

prompt = "Who is Lawrance McDaniel?"
print(within_levenshtein_distance(prompt, "Lawrence McDaniel"))  # True
print(within_levenshtein_distance("any ideas?", "API"))          # False
smarter.apps.llmtool.nlp.word_matches(prompt_word, term_word)[source]

Check if a prompt’s word matches a search term’s word.

The words match if they are equal, or if one is the plural of the other, formed with one of PLURAL_SUFFIXES. Both words are expected to be normalized, as by normalized_words().

Parameters:

  • prompt_word (str) – A word of the prompt.

  • term_word (str) – A word of the search term.

Returns:

True if the words match, otherwise False.

Return type:

bool

Example usage:

from smarter.apps.llmtool.nlp import word_matches

print(word_matches("gobstoppers", "gobstopper"))  # True
print(word_matches("concatenate", "cat"))          # False
smarter.apps.llmtool.nlp.wordnet() → LazyCorpusLoader | None[source]

Return the WordNet corpus reader, loading the corpus on first use.

WordNet is the thesaurus of thesaurus_match(). It is read with NLTK, whose corpus data must be installed separately. If it is not installed, a warning is logged once, and thesaurus matching is disabled.

Return type:

LazyCorpusLoader | None

Returns:

The nltk.corpus.wordnet reader, or None if the corpus is not installed.