AI discovery: llms.txt, robots.txt and WebMCP

Besides the MCP server for agents with credentials, Basehim makes every site easy for AI crawlers and in-browser agents to find, read and use. Site owners control all of it under Settings → AI Agents. Themes and apps can extend it with a few helpers and filters.

What core publishes

bh_head() adds links to the catalog and to llms.txt, plus schema.org WebSite data with a search action on the home page. bh_footer() registers the WebMCP tools. A theme that calls both functions needs nothing else.

Crawler classes

Settings → AI Agents lets a site owner allow or block three classes of crawler separately:

For example, a site can stay out of training data while still appearing in AI search answers.

Extending llms.txt, the catalog and robots.txt

// In an app's boot(): add a section to /llms.txt.
$this->addFilter('ai.llms_sections', function (array $sections): array {
    $sections[] = [
        'title' => 'Developer docs',
        'text'  => 'Guides for building themes and apps.',
        'links' => [
            ['App development', 'https://example.com/docs/apps', 'Start here'],   // title, URL, note
        ],
    ];
    return $sections;
});

// Add a line to robots.txt.
$this->addFilter('ai.robots', fn(string $txt) => $txt . "Disallow: /private-reports/\n");

ai.catalog_entries works the same way for the ARD catalog. Entries that aren't valid are dropped.

WebMCP: tools in the visitor's browser

WebMCP lets an AI agent running in the visitor's browser use a page through named tools. Basehim supports both kinds. While WebMCP is a Chrome origin trial, it runs only on registered sites: add your token under Settings → AI Agents.

Declarative: forms

Core's comment form is already described for agents. With Describe every form for agents turned on, a small script describes every other form as well: it names the form and each field from its label. Forms with a password field are skipped.

For better descriptions, write the attributes yourself in a theme:

<form action="<?= $base ?>/newsletter" method="post"<?= bh_webmcp_form('subscribe_newsletter', 'Subscribe an email address to the weekly newsletter.') ?>>
    <input type="email" name="email" required<?= bh_webmcp_param('Email address to subscribe') ?>>
    <button>Subscribe</button>
</form>

Pass ['autosubmit' => true] as the third argument of bh_webmcp_form() only for harmless, read-only forms such as search. Both helpers return '' while WebMCP is off, so they're safe to leave in.

Imperative: tools

Core registers list_recent_articles and list_topics. An app adds its own with the ai.tools filter:

$this->addFilter('ai.tools', function (array $tools): array {
    $tools[] = [
        'name'        => 'check_stock',
        'description' => 'Check whether a product is in stock.',
        'inputSchema' => [
            'type'       => 'object',
            'properties' => ['sku' => ['type' => 'string', 'description' => 'Product code']],
            'required'   => ['sku'],
        ],
        'annotations' => ['readOnlyHint' => true],
        'endpoint'    => '/shop/stock.json',   // same-origin GET; input becomes the query string; JSON comes back
    ];
    return $tools;
});

A template can do the same with bh_webmcp_tool([...]) while it renders.

Each tool needs:

• a name and a description;
• an object inputSchema whose required fields exist in its properties;
• exactly one of endpoint (a same-origin URL) or result (a fixed answer).

A tool that doesn't validate is left out, without breaking the others. Keep tools few and read-only where possible. Mark anything with real-world effects with 'annotations' => ['consequentialHint' => true].

Note: In Basehim 1.2.53 and earlier, bh_webmcp_tool() doesn't exist yet while apps boot. Calling it from an app's boot() stops the rest of that boot(). Use the ai.tools filter in apps.

Checking your site

PageSpeed Insights and Chrome DevTools' Lighthouse have an Agentic Browsing category. It checks llms.txt, the catalog, WebMCP form coverage, registered tools and schema validity. Settings → AI Agents links to it.