The question was innocent enough: "Is our webshop's SEO complete? Can we list it on Google Search Console?"
The shop in question is a Next.js storefront — digital downloads, licenses, subscriptions. It had all the SEO furniture you'd expect from a project that took itself seriously: a robots.ts, a sitemap that paged through the whole catalogue, canonical URLs on product pages, Product JSON-LD with the price in the right currency. Someone had clearly done this properly.
So the audit should have been a formality. It mostly was — right up until it checked the footer.
The footer that ate Googlebot
Every page of the shop has a footer with an "About" and a "Contact" link. Standard stuff. Both pages existed, both had proper metadata, canonicals, the lot.
Both also redirected every logged-out visitor to /login.
The shop's edge middleware is deny-by-default: any route not on a public allowlist requires an auth cookie. That's the right default for an app with an admin area — but the About and Contact pages had been added after the allowlist was last touched, and nobody noticed, because everyone who works on the shop is always logged in. Googlebot is never logged in. It would have followed those sitewide footer links straight into a 307 to a login form, on every single page of the crawl.
Fail-closed auth is a feature. Fail-closed auth plus "we test while logged in" is how you ship a login wall to a search engine. The git history even shows the mechanism: the pages landed in a "data-driven footer" commit that never touched the middleware.
The rest of the gap list was more ordinary — the four static pages missing from the sitemap, no canonicals on the two highest-value pages (home and the catalogue), no OG images anywhere, no Search Console verification hook. All fixable in an afternoon. Which left the more interesting half of the brief.
What does "SEO for AI agents" actually mean?
The owner's follow-up: the shop should also be optimized for AI agents. Not as a buzzword — as a concrete question about who reads your site now. Coding agents, in-product assistants, and MCP-connected tools genuinely do fetch pages and act on them; a storefront that's legible to them is a storefront they can recommend, price-check, and link correctly.
Three things made the cut, in increasing order of effort:
1. An /llms.txt manifest, generated from the database. The llms.txt convention is markdown: a title, a summary, sections of links. Ours is a route handler, not a static file — it reads the live product table on every request, so a new product or a price change shows up with zero manual steps:
# Lans Shop
> Digital products, SaaS subscriptions, services, and software licenses...
## Products
- [Forge CLI — Indie License](https://shop.lans.cloud/products/indie-tool-license): software license, €49.00. A perpetual single-seat license...
- [Pulse Analytics — Pro Plan](https://shop.lans.cloud/products/premium-saas-subscription): subscription, €29.99/month. Privacy-first product analytics...
Honest scope, because it matters: the big answer engines don't read llms.txt for citations today — they crawl your HTML like everyone else. The file serves the agentic tier that does fetch it: IDE agents, assistants, scripts. Cheap to serve, clearly useful to exactly that audience, and no downside.
2. JSON-LD that agents (and Google) can trust — emitted through one escaping function. Structured data was already there, but every emission now goes through a single serializer that escapes <, >, and & to \uXXXX sequences. JSON.stringify does not escape <, so a product description containing </script> would otherwise break out of the tag and execute. "The content is admin-authored and trusted" is true right up until the day it isn't; the escape costs nothing and the output is still valid JSON. If you want to sanity-check what your own pages emit, paste the script block into a JSON formatter — it should parse cleanly, escapes and all.
3. Open Graph cards rendered from live data. This is where the "automate it" requirement got interesting.
Why we didn't copy our own pipeline
We already had prior art: my tools site pre-renders its OG cards with a script — SVG templates rasterized to PNGs, committed to git, 112 of them. I went in assuming we'd extract that "pipeline" and reuse it.
The survey said no. That pipeline is manual — an npm script you remember to run, plus a test that fails if a card is missing. That's the right shape for content that lives in git: the card changes when a file changes, and a human is already in the loop. A shop's products live in a database and change through an admin UI. No human runs scripts there. Committed PNGs would drift out of date the first time someone renamed a product.
So the shop got the opposite design: a dynamic opengraph-image.tsx per product, rendered by satori from the live row at request time. New product → correct card. Edited price → correct card. Nobody runs anything.
Three constraints cost real debugging time before they became API documentation:
- Satori cannot parse woff2. The font files your framework serves for CSS are useless to it — vendor actual TTFs.
- Satori cannot parse
oklch()or CSS custom properties. Our theme tokens are oklch; the card template carries them pre-converted to hex, with a comment pointing back at the source of truth. - Next's standalone output tracing cannot see through
readFile. The fonts load at runtime vianode:fs, which means the production build silently omits them unless you add the directory tooutputFileTracingIncludes. Locally everything works; only the deployed container 500s. We caught it by checking the standalone output before shipping, not after.
The generation code that wasn't shop-specific — the JSON-LD escaper, the llms.txt builder, the font loading with those three constraints written into its docstrings — moved into a small private package, so the tools site can adopt it next. (The survey also found that site's five JSON-LD emitters inject unescaped JSON.stringify, which is now the first item on its backlog. The best argument for extracting a shared function is discovering the copies disagree about safety.)
What the adversarial review caught
House rule: substantive work doesn't ship until a fresh agent — primed to refute, not to review — has attacked it. This one earned its keep four times over.
The product pages were silently losing og:site_name. Next.js merges metadata shallowly: a child page's openGraph object replaces the layout's, no deep merge. Our product pages declared their own openGraph (title, description, URL) and thereby wiped the siteName set at the root. One restated field fixes it — but nothing warns you, and the pages most likely to be shared were the ones affected.
The OG route was a CPU amplifier. First version: force-dynamic, and unknown slugs got a friendly fallback card. Reviewer's reading: an unauthenticated endpoint that will do a database lookup plus a 1200×630 PNG rasterization for any string a stranger puts in the URL, uncached. Now it's cached for an hour and junk slugs get a plain 404 — no render at all.
robots.txt banned what llms.txt advertised. The manifest's "machine-readable" section pointed agents at two JSON endpoints; robots.txt disallowed /api wholesale. Telling agents to fetch URLs you tell crawlers not to touch is at best incoherent. Specific Allow entries for exactly those two paths resolve it — most-specific match wins.
And the kicker: CI had been red for days, and our local gate couldn't see it. The CI quality job runs prettier --check; our local pre-flight gate didn't. When we added it, prettier failed — on the backlog file, which no recent commit had touched. Better still, prettier turned out to be non-convergent on that file: run --write, and --check still fails, forever. Two deeply-indented markdown blocks oscillate between formattings. The fix wasn't to fight it; a hand-authored narrative doc gains nothing from mechanical reflow. It's in .prettierignore now, with a comment explaining why, and the local gate matches CI so this class of drift can't happen silently again.
Was the SEO "complete"?
After the batch: the crawl surface verified live — footer pages return 200 logged-out, the sitemap covers everything, every card renders, junk URLs 404, and an agent fetching /llms.txt gets the real catalogue. Listing on Search Console is now a ten-minute human task (verify the domain, submit the sitemap) plus one genuinely human decision about when — the catalogue is still demo data, and whatever Google indexes first, it remembers.
Three things I'd tell past-me:
- Crawl your own site logged out. Every "works for me" was a session cookie talking. The bot has no cookie, and neither does a first-time visitor from a search result.
- Match the automation to where the content lives. Git-authored content → build-time artifacts and a human in the loop. Database content → request-time generation, or it will drift.
- Your local gate and your CI must run the same checks. Any gap between them is a place where red builds hide — ours hid a formatter that literally cannot say yes.