Blog Indexer

The indexer of The Smarter Project’s blog, which copies its posts from WordPress into the database.

index_blog() brings the blog’s RepositoryIndex up to date with its published posts, so that the MCP server’s search_blog and read_blog_post tools can read them as they read a repository’s documentation:

It reads every published post from the blog’s public WordPress REST api, /wp-json/wp/v2/posts, 100 posts a request.
It fingerprints the posts by their ids and modification times. If the fingerprint is already indexed, there is nothing to do.
Otherwise it converts each post’s html to Markdown-like text, with its illustrations as Markdown images, ![alt](url), which Smarter Chat displays, and replaces the blog’s files in one transaction, with store_files(). A post’s path is its permalink’s path, e.g. 2026/10/26/release-smarter-api-v0-6, and the fingerprint is stored as the index’s commit_sha.

The blog is stored like a repository, under the blog’s name, so that search_files() searches it like any other. The Celery task smarter.apps.mcpserver.tasks.index_blog() calls index_blog().

The url is the fixed url of the project’s public blog, not a user-supplied one, so the requests do not go through smarter.apps.llmtool.plugin.safe_http.

Note

Experimental. The MCPServer was designed and coded by Claude Code (Anthropic’s Claude Opus 5.5), with Lawrence McDaniel as co-author. It is experimental, and will be documented.

smarter.apps.mcpserver.services.indexing.blog.WORDPRESS_POSTS = TypeAdapter(list[WordPressPost]): Validates a page of the WordPress REST api’s posts, a JSON list, into WordPressPost objects.

class smarter.apps.mcpserver.services.indexing.blog.WordPressPost(**data)[source]

Bases: BaseModel

A post, as the WordPress REST api returns it.

Only the fields that the indexer reads are modeled. WordPress and its plugins add others, e.g. protected and raw, which are ignored.

content: WordPressRendered: The post’s body, as html.

date: datetime: When the post was published.

id: int: The post’s WordPress id, which the fingerprint uses.

link: str: The post’s permalink, e.g. https://blog.smarter.sh/2026/10/26/release-smarter-api-v0-6/.

model_config = {'extra': 'ignore'}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

modified: datetime

When the post was last changed.

A change makes the fingerprint change, and the blog is indexed again.

slug: str: The post’s slug, e.g. release-smarter-api-v0-6, its path if its permalink has none.

title: WordPressRendered: The post’s title, as html.

smarter.apps.mcpserver.services.indexing.blog.fetch_posts(blog)[source]

Return the blog’s published posts, newest first, from its WordPress REST api.

The posts are read a page of POSTS_PER_PAGE at a time, until the last page, which WordPress names in its X-WP-TotalPages header, an empty page, or MAX_PAGES pages.

Side effects: one unauthenticated HTTP request to the blog for each page.

Parameters:: blog (ProjectBlog) – The blog.
Return type:: list[WordPressPost]
Returns:: The posts.
Raises:: SmarterMCPServerIndexError – If the blog does not answer with posts.

smarter.apps.mcpserver.services.indexing.blog.html_to_text(content, base_url='')[source]

Return a post’s html as Markdown-like text, for search and for an LLM to read.

Parameters:

content (str) – The post’s rendered html.
base_url (str) – The url that relative image urls are relative to, e.g. the blog’s.

Return type:

str

Returns:

The text, one block per paragraph, heading, list, table, code block or image, separated by blank lines.

smarter.apps.mcpserver.services.indexing.blog.index_blog(force=False, blog=None)[source]

Bring the blog’s index up to date with its published posts.

If the posts’ fingerprint is already indexed, only a previous error is cleared. Two posts with the same path are indexed once, the newer one.

A failure never raises: the index keeps its posts, and stays ready if it was, so that the tools can go on answering from them, and the failure is recorded in its error.

Side effects: HTTP requests to the blog (see fetch_posts()), database writes (the index’s row is created if it does not exist, and see store_files()), and a signal: mcpserver_repository_indexed when the posts are replaced, whose receiver makes the cached tool results stale, or mcpserver_repository_index_failed.

Parameters:

force (bool) – Index the posts even if they are already indexed.
blog (ProjectBlog | None) – The blog. Defaults to project_blog().

Return type:

RepositoryIndex

Returns:

The blog’s index. If indexing fails, its error says why, and an existing index is kept.

Example usage:

index = index_blog()
index.file_count  # the number of posts

smarter.apps.mcpserver.services.indexing.blog.post_file(blog, post)[source]

Return a post as a file of the blog’s index.

The text starts with the post’s title, dates and url, so that a reader of any part of it can cite it. A post longer than MAX_FILE_BYTES is truncated.

Parameters:

blog (ProjectBlog) – The blog.
post (WordPressPost) – The post.

Return type:

ArchiveFile

Returns:

The file.

smarter.apps.mcpserver.services.indexing.blog.posts_fingerprint(posts)[source]

Return a fingerprint of the posts, which changes when a post is published, changed or removed.

It is a SHA-1 of each post’s id and modification time, in the order of their ids, and is not used for security. It is stored as the index’s commit_sha, which has room for a sha.

Parameters:: posts (list[WordPressPost]) – The posts.
Return type:: str
Returns:: 40 hexadecimal characters, the size of a commit’s sha.