lans.cloud Blog
guide3 min read

A Practical SEO Guide, Part 2: The Discovery Layer

#seo#sitemap#robots-txt#canonical-urls#nextjs

In Part 1 we made sure a crawler can read the content. Now we make sure it can find all of it — and doesn't waste its budget on pages you don't want indexed. Three small files do this: robots.txt, sitemap.xml, and canonical URLs.

robots.txt: the front door

robots.txt is the first thing most crawlers request. Missing it isn't fatal, but it means every crawler gets a 404 on the way in and has no pointer to your sitemap. A minimal, correct one:

User-Agent: *
Allow: /
Disallow: /admin

Host: https://cv.lans.cloud
Sitemap: https://cv.lans.cloud/sitemap.xml

Two things worth being deliberate about:

  • Disallow the routes that shouldn't be indexed — admin panels, login pages, transactional endpoints. (Note: Disallow is a crawl hint, not a security control. Anything truly private needs auth, not a robots rule. It also isn't a reliable noindex — for that, use the meta tag, covered in Part 5.)
  • Point at your sitemap so a crawler that arrives cold knows where the inventory lives.

Frameworks increasingly generate this for you. In Next.js App Router it's a file that returns an object:

// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/', disallow: '/admin' },
    sitemap: 'https://cv.lans.cloud/sitemap.xml',
    host: 'https://cv.lans.cloud',
  };
}

sitemap.xml: the inventory

The sitemap is the canonical list of URLs you want indexed, each with a lastmod date so crawlers know what changed. For a small site it can be a single entry; for a blog it should enumerate every post.

The important design decision is build-time vs. request-time. If your content changes without a rebuild — say, posts on a mounted volume, or a CMS — a sitemap generated at build time starts lying the moment you publish. In that case, generate it per request by walking your live content:

// A request-time sitemap that can't go stale (Astro endpoint shown; the
// same idea works as a Next route handler).
export const GET = async ({ site }) => {
  const posts = await getAllPosts();
  const urls = [
    { loc: site.href, lastmod: posts[0]?.date, priority: '1.0' },
    ...posts.map((p) => ({ loc: new URL(p.slug, site).href, lastmod: p.date })),
  ];
  return new Response(renderSitemapXml(urls), {
    headers: { 'Content-Type': 'application/xml' },
  });
};

For a truly static site, a build-time sitemap is perfectly fine — just make sure a deploy actually regenerates it. Either way, verify the output is real XML:

curl -s https://cv.lans.cloud/sitemap.xml | head -5
# <?xml version="1.0" encoding="UTF-8"?>
# <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
# <url><loc>https://cv.lans.cloud</loc>...

Canonical URLs: stop splitting your own ranking

This is the one people skip, and it quietly costs them. The same content is often reachable at multiple URLs:

  • https://site.com and https://www.site.com
  • https://site.com/page and https://site.com/page?utm_source=twitter
  • http:// and https://

To a search engine those can look like different pages with duplicate content, splitting your ranking signals across all of them. A canonical tag says "no matter which URL you arrived by, this is the real one":

<link rel="canonical" href="https://cv.lans.cloud/" />

In Next.js you set it declaratively and it also needs a metadataBase so relative URLs resolve to absolute ones:

export const metadata = {
  metadataBase: new URL('https://cv.lans.cloud'),
  alternates: { canonical: 'https://cv.lans.cloud/' },
};

One subtlety: strip volatile query strings when you compute the canonical, so /?ref=hn and /?utm=x all canonicalize to the same clean URL. Otherwise you've reintroduced the exact duplication you were trying to kill.

The payoff

These three files together mean a crawler doesn't have to discover your site by luck. It arrives, reads robots.txt, pulls the sitemap, indexes exactly the URLs you nominated, and consolidates all ranking signal onto the canonical version of each. Cheap to add, and they compound with everything else.

Next, the part that changes how your links look in search results and social feeds: structured data and social cards.

Next: Part 3: Structured Data & Social Cards →