Repository Indexer

The indexer of The Smarter Project’s repositories, which copies their text files from GitHub into the database.

The MCPServer’s public endpoint answers questions about the project’s repositories, e.g. “how does custom domain verification work?”, from its own copy of their files, which is searched in the database by smarter.apps.mcpserver.services.search, and read by the tools of smarter.apps.mcpserver.protocol.tools.index. This module makes and refreshes that copy.

index_repository() brings one repository’s RepositoryIndex up to date with its branch:

  1. It asks the GitHub api for the branch’s latest commit. If that commit is already indexed, there is nothing to do, so a repository that has not changed costs one small request.

  2. Otherwise it downloads the repository’s gzipped tar archive at that commit, from GitHub’s codeload service, streaming it to a temporary file whose size is capped.

  3. It reads the archive’s text files, in memory, without extracting anything to disk. It keeps documentation and source code, and skips binaries, generated and vendored files, lock files, and files that are too large to be useful to an LLM. The images of the published Sphinx documentation get their published urls, as Markdown images: see link_documentation_images().

  4. It replaces the repository’s files in one transaction, so that a search never sees a partial index.

The Celery task smarter.apps.mcpserver.tasks.index_repository() calls it, on a schedule and when a tool finds a repository that is not indexed yet. The blog is indexed the same way, from WordPress instead of GitHub, by smarter.apps.mcpserver.services.indexing.blog, which reuses ArchiveFile, markdown_image() and store_files().

The urls are fixed GitHub urls of public repositories, not user-supplied ones, so the requests do not go through smarter.apps.llmtool.plugin.safe_http. GitHub’s api allows 60 unauthenticated requests an hour, and an hourly refresh of every repository needs one request for each.

Note

Experimental. The MCPServer was designed and coded by Claude Code (Anthropic’s Claude Opus 5.5), with Lawrence McDaniel as co-author. It is experimental, and will be documented.

class smarter.apps.mcpserver.services.indexing.github.ArchiveFile(**data)[source]

Bases: BaseModel

A text file read from a repository’s archive, before it is stored as a RepositoryFile.

read_archive() makes one for each file that is indexed, link_documentation_images() may change its content, and store_files() saves it. The blog’s indexer, smarter.apps.mcpserver.services.indexing.blog, makes one for each post.

content: str

The file’s text.

kind: RepositoryFileKind

Whether the file is documentation or code, from file_kind().

line_count: int

The number of lines of the file, from count_lines().

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

path: str

The file’s path in the repository, without the archive’s top-level folder, e.g. docs/source/index.rst.

size: int

The size of the file’s text, in bytes, when encoded as UTF-8.

title: str

A document’s first heading, from document_title(), or an empty string for code.

smarter.apps.mcpserver.services.indexing.github.collect_files(archive)[source]

Return the files of an archive to index, within MAX_FILES and MAX_INDEX_BYTES.

The limits keep one very large repository from filling the database. The files are read in the archive’s order, and the first file that would exceed a limit ends the list.

Parameters:

archive (IO[bytes]) – The archive.

Return type:

list[ArchiveFile]

Returns:

The files. Files beyond the limits are left out, and a warning is logged.

Raises:

SmarterMCPServerIndexError – If the archive cannot be read.

smarter.apps.mcpserver.services.indexing.github.count_lines(content)[source]

Count the lines of a file’s text, as an editor numbers them.

A last line without a newline counts as a line, so "a\nb" and "a\nb\n" both have two.

Parameters:

content (str) – The text.

Return type:

int

Returns:

The number of lines.

smarter.apps.mcpserver.services.indexing.github.document_title(content)[source]

Return a document’s first heading, in reStructuredText or Markdown.

Only the first 60 lines are read: a heading further down is a section’s, not the document’s. The title is stored with the file, and a search ranks a file whose title matches a term higher.

Parameters:

content (str) – The document.

Return type:

str

Returns:

The heading, cut at MAX_TITLE_LENGTH characters, or an empty string if it has none.

smarter.apps.mcpserver.services.indexing.github.documentation_image_url(repository, target)[source]

Return the published url of an image of a repository’s Sphinx documentation.

Sphinx copies a local image to the _images/ folder of the built site, by its file name. A target that is already a url is kept, unless it is a status badge.

Parameters:

  • repository (ProjectRepository) – The repository.

  • target (str) – The image directive’s argument, e.g. /_static/api-key-claude.png, or a url.

Return type:

str | None

Returns:

The url, or None for a status badge, an empty target, a target with characters that would break a Markdown image, ()<>, or a local image of a repository without published documentation.

smarter.apps.mcpserver.services.indexing.github.download_archive(repository, commit_sha, destination)[source]

Download a repository’s gzipped tar archive at a commit.

The archive is streamed, a chunk at a time, so that it is never wholly in memory, and the download stops as soon as it is larger than MAX_ARCHIVE_BYTES.

Side effects: one HTTP request to GitHub’s codeload service, and writes to destination.

Parameters:

  • repository (ProjectRepository) – The repository.

  • commit_sha (str) – The commit.

  • destination (IO[bytes]) – A binary file to write the archive to. It is left positioned at its start, ready for read_archive().

Return type:

int

Returns:

The archive’s size, in bytes.

Raises:

SmarterMCPServerIndexError – If the download fails, or the archive is larger than MAX_ARCHIVE_BYTES.

smarter.apps.mcpserver.services.indexing.github.file_kind(path)[source]

Return what a file of a repository is, or None if it is not indexed.

A file is documentation or code by its extension or its name (see DOCUMENTATION_EXTENSIONS and CODE_EXTENSIONS). A file in an excluded folder, or that matches an excluded pattern, is not indexed, whatever its extension.

Parameters:

path (str) – The file’s path in the repository.

Return type:

RepositoryFileKind | None

Returns:

Documentation, code, or None.

smarter.apps.mcpserver.services.indexing.github.index_repository(name, force=False)[source]

Bring a repository’s index up to date with the latest commit of its branch.

If the latest commit is already indexed, only a previous error is cleared. Otherwise the archive is downloaded to a temporary file, which is deleted afterwards, and its files replace the index’s (see the module’s description for the steps).

A failure never raises: the index keeps its files, and stays ready if it was, so that the tools can go on answering from the previous commit, and the failure is recorded in its error.

Side effects: HTTP requests to GitHub (see latest_commit_sha() and download_archive()), a temporary file, database writes (the index’s row is created if it does not exist, and see store_files()), and a signal: mcpserver_repository_indexed when the files are replaced, whose receiver makes the cached tool results stale, or mcpserver_repository_index_failed.

Parameters:

  • name (str) – The repository’s name, e.g. smarter-cli.

  • force (bool) – Index the repository even if its latest commit is already indexed.

Return type:

RepositoryIndex

Returns:

The repository’s index. If indexing fails, its error says why, and an existing index is kept.

Raises:

SmarterMCPServerIndexError – If there is no such repository.

Example usage:

index = index_repository("smarter-cli")
index.status  # 'ready'
smarter.apps.mcpserver.services.indexing.github.latest_commit_sha(repository)[source]

Return the latest commit of a repository’s branch, from the GitHub api.

The request asks for the application/vnd.github.sha media type, so that GitHub answers with the 40 characters of the sha alone, not the commit’s JSON.

Side effects: one unauthenticated HTTP request to the GitHub api.

Parameters:

repository (ProjectRepository) – The repository.

Return type:

str

Returns:

The commit’s sha.

Raises:

SmarterMCPServerIndexError – If GitHub cannot be reached, or does not answer with a commit.

Append a Markdown image to each image and figure directive of a Sphinx source file.

An LLM cannot turn .. image:: /_static/x.png into a link, so the line that names the image gets its published url, as a Markdown image whose description is the directive’s :alt:, or else a figure’s caption. The image is appended to the line, rather than added on a line of its own, so that line numbers still match the file on GitHub.

Parameters:
Return type:

ArchiveFile

Returns:

A copy of the file, with its images linked, or the file unchanged if it is not a reStructuredText file of the repository’s published documentation, or has no image to link.

smarter.apps.mcpserver.services.indexing.github.markdown_image(description, url, prefix_length=0)[source]

Return a Markdown image, ![description](url), that fits on one line of a search’s passage.

The LLM cannot see an image, so its description, e.g. its alt text, is what tells the LLM what it shows, and Smarter Chat displays the image when the LLM includes it in an answer. A search clips a line longer than MAX_IMAGE_LINE_LENGTH, which would break the image’s Markdown, so a long description is shortened at a word, rather than the line clipped.

Parameters:

  • description (str) – What the image shows. Brackets become parentheses, which Markdown allows. An empty description becomes Illustration.

  • url (str) – The image’s url.

  • prefix_length (int) – The length of the text before the image on its line.

Return type:

tuple[str, bool]

Returns:

The image, and whether its description is complete. A description that does not fit within MAX_IMAGE_LINE_LENGTH is shortened, and ends with an ellipsis. If fewer than 20 characters are left for it, it is replaced by Illustration.

smarter.apps.mcpserver.services.indexing.github.read_archive(archive)[source]

Read the indexable text files of a repository’s gzipped tar archive.

GitHub’s archives contain one top-level folder, <repository>-<commit>/, which is removed from the paths. Only regular files are read: links, which could point outside the repository, are skipped. Nothing is extracted to disk. A file is skipped if file_kind() does not index it, it is larger than MAX_FILE_BYTES, it is binary, or it is blank.

Parameters:

archive (IO[bytes]) – The archive, positioned at its start.

Return type:

Iterator[ArchiveFile]

Returns:

The files, in the archive’s order.

Raises:

SmarterMCPServerIndexError – If the archive cannot be read.

smarter.apps.mcpserver.services.indexing.github.store_files(index, commit_sha, files)[source]

Replace a repository’s indexed files, in one transaction.

The old files are deleted and the new ones inserted, in batches of BATCH_SIZE, and the index is marked ready, at the commit, with its counts, and with no error. Since it is one transaction, a search sees either the old files or the new ones, never a mix.

Side effects: database writes: deletes and inserts of RepositoryFile rows, and an update of the index.

Parameters:

  • index (RepositoryIndex) – The repository’s index.

  • commit_sha (str) – The commit that the files are of. The blog’s indexer passes a fingerprint of its posts instead.

  • files (list[ArchiveFile]) – The files.

Return type:

RepositoryIndex

Returns:

The index, ready to search.