Repository Indexer

The indexer of The Smarter Project’s repositories.

index_repository() brings one repository’s RepositoryIndex up to date with its branch:

It asks the GitHub api for the branch’s latest commit. If that commit is already indexed, there is nothing to do, so a repository that has not changed costs one small request.
Otherwise it downloads the repository’s gzipped tar archive at that commit, from GitHub’s codeload service, streaming it to a temporary file whose size is capped.
It reads the archive’s text files, in memory, without extracting anything to disk. It keeps documentation and source code, and skips binaries, generated and vendored files, lock files, and files that are too large to be useful to an LLM.
It replaces the repository’s files in one transaction, so that a search never sees a partial index.

The urls are fixed GitHub urls of public repositories, not user-supplied ones, so the requests do not go through smarter.apps.plugin.plugin.safe_http. GitHub’s api allows 60 unauthenticated requests an hour, and an hourly refresh of every repository needs one request for each.

Note

Experimental. The MCPServer was designed and coded by Claude Code (Anthropic’s Claude Opus 5.5), with Lawrence McDaniel as co-author. It is experimental, and will be documented.

class smarter.apps.mcpserver.indexer.ArchiveFile(**data)[source]

Bases: BaseModel

A text file read from a repository’s archive, before it is stored as a RepositoryFile.

content: str

kind: RepositoryFileKind

line_count: int

model_config = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

path: str

size: int

title: str

smarter.apps.mcpserver.indexer.collect_files(archive)[source]

Return the files of an archive to index, within MAX_FILES and MAX_INDEX_BYTES.

Parameters:: archive (IO[bytes]) – The archive.
Return type:: list[ArchiveFile]
Returns:: The files. Files beyond the limits are left out, and a warning is logged.

smarter.apps.mcpserver.indexer.document_title(content)[source]

Return a document’s first heading, in reStructuredText or Markdown.

Parameters:: content (str) – The document.
Return type:: str
Returns:: The heading, or an empty string if it has none.

smarter.apps.mcpserver.indexer.download_archive(repository, commit_sha, destination)[source]

Download a repository’s gzipped tar archive at a commit.

Parameters:

repository (ProjectRepository) – The repository.
commit_sha (str) – The commit.
destination (IO[bytes]) – A binary file to write the archive to.

Return type:

int

Returns:

The archive’s size, in bytes.

Raises:

SmarterMCPServerIndexError – If the download fails, or the archive is larger than MAX_ARCHIVE_BYTES.

smarter.apps.mcpserver.indexer.file_kind(path)[source]

Return what a file of a repository is, or None if it is not indexed.

Parameters:: path (str) – The file’s path in the repository.
Return type:: RepositoryFileKind | None
Returns:: Documentation, code, or None.

smarter.apps.mcpserver.indexer.index_repository(name, force=False)[source]

Bring a repository’s index up to date with the latest commit of its branch.

Parameters:

name (str) – The repository’s name, e.g. smarter-cli.
force (bool) – Index the repository even if its latest commit is already indexed.

Return type:

RepositoryIndex

Returns:

The repository’s index. If indexing fails, its error says why, and an existing index is kept.

Raises:

SmarterMCPServerIndexError – If there is no such repository.

Example usage:

index = index_repository("smarter-cli")
index.status  # 'ready'

smarter.apps.mcpserver.indexer.latest_commit_sha(repository)[source]

Return the latest commit of a repository’s branch, from the GitHub api.

Parameters:: repository (ProjectRepository) – The repository.
Return type:: str
Returns:: The commit’s sha.
Raises:: SmarterMCPServerIndexError – If GitHub does not answer with a commit.

smarter.apps.mcpserver.indexer.read_archive(archive)[source]

Read the indexable text files of a repository’s gzipped tar archive.

GitHub’s archives contain one top-level folder, <repository>-<commit>/, which is removed from the paths. Only regular files are read: links, which could point outside the repository, are skipped. Nothing is extracted to disk.

Parameters:: archive (IO[bytes]) – The archive, positioned at its start.
Return type:: Iterator[ArchiveFile]
Returns:: The files, in the archive’s order.
Raises:: SmarterMCPServerIndexError – If the archive cannot be read.

smarter.apps.mcpserver.indexer.store_files(index, commit_sha, files)[source]

Replace a repository’s indexed files, in one transaction.

Parameters:

index (RepositoryIndex) – The repository’s index.
commit_sha (str) – The commit that the files are of.
files (list[ArchiveFile]) – The files.

Return type:

RepositoryIndex

Returns:

The index, ready to search.