lans.cloud Blog
guide6 min read

AI Search Optimization, Part 3: llms.txt and the Agentic Web (An Honest Take)

#llms-txt#ai-crawlers#mcp#agentic-web#indexnow#bing

If you've been anywhere near an SEO feed in the last year, you've seen the advice: "Ship an llms.txt and you'll show up in ChatGPT." I shipped one for lans.cloud. It was the right call. But almost none of the reasons people give for it are true, and I'd rather tell you the honest version than sell you the hype one.

This post is the boring, load-bearing middle of the series: the file that gets all the attention (llms.txt) does very little today, and the plumbing nobody blogs about (Bing indexing, IndexNow, open crawlers) does almost all the real work. Let's separate them.

What llms.txt actually is

llms.txt is a proposed convention from llmstxt.org. The idea is clean: put a markdown file at /llms.txt that gives an LLM a curated, machine-readable index of your site. The format is deliberately simple — an H1 title, a blockquote summary, then sections of links:

# lans.cloud

> Fast, privacy-first, single-purpose web tools. No sign-up, no tracking, runs in your browser.

## Classroom
- [Classroom Timer](https://lans.cloud/classroom-timer): Fullscreen countdown timer for lessons, no ads in the way.
- [Random Name Picker](https://lans.cloud/random-name-picker): Pick a student fairly from a class list.

## Everyday
- [Coin Flip](https://lans.cloud/coin-flip): Flip a virtual coin, heads or tails, with history.

The pitch is that instead of an LLM scraping your rendered HTML — nav, footer, cookie banner, ad slots and all — it reads a hand-curated table of contents in the format models parse best. Reasonable idea. The problem is adoption on both sides.

The honest adoption reality

Let's do the numbers, because they're the whole story.

Publisher side, it's catching on. One study of ~300k domains put adoption around 10%; a separate look at the top 10k domains put it lower, around 5.6%. So plenty of sites are shipping the file. Fine.

Consumer side, it's a ghost town. In a study of 500M+ AI-bot requests over 90 days, only about 408 requests targeted /llms.txt. Four hundred and eight. Out of half a billion. That's not "early" — that's effectively nobody is fetching it.

And the big engines have been explicit. Google's Gary Illyes said publicly in July 2025 that Google doesn't support llms.txt and has no plans to. No major LLM provider has committed to it as a citation signal. So when someone tells you an llms.txt will get you cited in ChatGPT, Perplexity, or AI Overviews — today, in 2026 — the evidence says it does close to nothing for that.

I want to be precise here: "does nothing for citations" is not "does nothing." It's a category error to judge llms.txt as a search-ranking file. It isn't one.

Who does use it

llms.txt is plumbing for the agentic web, not the answer-engine web. The things that actually reach for it:

  • IDE and coding agents — Cursor, Claude Code, GitHub Copilot, Windsurf. When an agent is working against your docs or your API, a clean llms.txt (or the fuller llms-full.txt) is a genuinely useful map.
  • MCP servers and in-product assistants that need a structured entry point into a site rather than a crawl.

That's a real audience, just not the one the hype implies. It's developer tooling, not search traffic. Which is exactly why I still shipped one: the cost is near zero, the scope is correct, and if the agentic tier grows, we're already legible to it.

Here's the actual Next.js route that serves ours. The key design choice: it's generated from the same tool registry that builds our sitemap, so it can never drift out of sync with what's live (that drift-proofing instinct is the whole theme of Part 2):

export function GET() {
  return new Response(buildLlmsTxt(), {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=0, must-revalidate, s-maxage=3600',
    },
  });
}

buildLlmsTxt() walks the registry, groups tools by cluster (classroom, gaming, dev utilities, and so on), emits a hub link per cluster and one [name](url): description line per tool. Because the registry is the single source of truth, adding a tool updates the sitemap, the internal-link graph, and llms.txt in one commit. You can read the live output at https://lans.cloud/llms.txt.

That's the whole story on the file: ship it, keep it generated, and move on. Don't spend a second more optimizing it, because the return isn't there yet.

What actually moves AI visibility: the plumbing

Here's where the real near-term wins live. None of it is glamorous.

Bing is ChatGPT's index

This is the single most underrated fact in AI SEO. ChatGPT's web search runs on Bing's index. If Bing hasn't indexed a page, ChatGPT can't cite it — full stop. So Bing indexing is a hard prerequisite for showing up in ChatGPT's sources, in a way that no llms.txt will ever substitute for.

The fast path: set up Bing Webmaster Tools and use the import from Google Search Console option. It pulls your verified properties and sitemaps over in a couple of clicks — no re-verification dance. If you've only ever thought about Google, this is probably the highest-leverage hour you can spend on AI visibility.

IndexNow: one ping, several engines

IndexNow is a shared "please recrawl these URLs" protocol. You POST a list of changed URLs to one endpoint, and Bing, Yandex, and Seznam all consume it. One request, multiple indexes nudged.

We wired it into the deploy flow so a shipped or changed page announces itself:

# after a deploy that ships or changes pages:
npm run indexnow   # submits the live sitemap URLs to IndexNow

The caveat, and it's a big one: Google does not support IndexNow. For Google you still lean on a fresh sitemap plus the occasional manual "Request indexing" in Search Console. IndexNow is a Bing-family tool — which, given the point above, is exactly the family that feeds ChatGPT. That's the connection people miss: IndexNow → Bing → ChatGPT's sources.

Don't block the AI crawlers

The most expensive mistake in this whole space is invisible: a robots.txt (or a WAF rule, or a Cloudflare bot-fight toggle) that blocks the crawlers the answer layer depends on. Keep these user-agents allowed:

  • GPTBot and OAI-SearchBot (OpenAI — training and search, respectively)
  • PerplexityBot
  • ClaudeBot
  • Google-Extended

Blocking them isn't privacy; it's opting out of the answer layer entirely. If your instinct is to wall off the AI bots, at least do it knowing the trade is "these engines will never quote me." For a tools site whose growth strategy is being cited, that trade makes no sense — so our robots.txt stays open to them. (This is the AI-crawler flavor of the crawl/index discipline I covered in the discovery layer.)

Takeaway

Ship the llms.txt. It's cheap, it's correct for the agentic tier, and generating it from your registry costs you nothing to maintain. But be honest about what it is: a map for coding agents and MCP clients, not a traffic lever, and not a citation signal that any major engine reads today.

The near-term wins are the unglamorous plumbing:

  1. Bing Webmaster (import from GSC) — because Bing is ChatGPT's index.
  2. IndexNow on deploy — one ping recrawls the Bing family.
  3. Open crawlers — never block GPTBot, PerplexityBot, ClaudeBot, and friends.

Get that layer right and you've earned the eligibility to be cited. In Part 4 we'll get into how to actually measure whether any of it is working — because "did ChatGPT cite us?" is a genuinely hard question to answer, and most of the dashboards selling you an answer are guessing.