Sitemaps and robots.txt

A sitemap is a machine-readable list of the URLs you want indexed, with hints about each one's freshness. Most sites hand-maintain sitemap.xml until the day a CMS publish silently isn't in it. The App Router inverts the workflow: export an array from app/sitemap.ts and Next renders it to /sitemap.xml, built from the same route data your pages already query.

// app/sitemap.ts
import type { MetadataRoute } from "next";
import { getAllPosts } from "@/lib/posts";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getAllPosts();
  const base = "https://acme.com";

  return [
    {
      url: base,
      lastModified: new Date(),
      changeFrequency: "daily",
      priority: 1,
    },
    ...posts.map((post) => ({
      url: `${base}/blog/${post.slug}`,
      lastModified: post.updatedAt,
      changeFrequency: "weekly" as const,
      priority: 0.7,
    })),
  ];
}

The return type is the contract, and each field's honesty matters more than its value:

Because the function runs on the server with your data layer, the sitemap cannot drift from the content: unpublish a post and the next generation has no entry, with no file to forget. On a prerendered deployment the XML is produced at build — every deploy refreshes it — while a route with dynamic data can serve a freshly computed map on each request. For the common CMS-backed site, the build-time generation from a query you already run is the whole feature: both sitemap and robots.txt are generated at build time from route data you already have — no new infrastructure, no sync job.

Two habits keep the generator honest as it grows. First, centralize the origin — read a SITE_URL env var at generation time instead of scattering hard-coded https://acme.com literals, so a staging build cannot leak staging URLs into the crawl map and a domain migration stays a one-line change. Second, keep the entry set to canonical, indexable pages only: exclude noindexed routes, redirect sources, and filtered search permutations (?sort=price), each of which wastes crawl requests on pages you never wanted listed. Both habits are data hygiene rather than framework features — the generator simply makes whatever query you write the site's official statement of what deserves crawling.

The static export (output: "export") has no request-time generation, so its sitemap is frozen at build like everything else — fine for a docs site, wrong for a catalog that publishes hourly without deploys. In that case the same array logic can back a route handler (app/sitemap.xml/route.ts) or a regeneration triggered by the CMS webhook of Incremental Static Regeneration fame.

Two API details before moving on: entries are typed by MetadataRoute.Sitemap, so your editor catches typos in field names — use it rather than memorizing the XML. And multiple locales map to the alternates field, which emits the hreflang links search engines use to serve the right language variant — the same alternates object shape you would set in page metadata, kept consistent because both come from the record's language list.

robots.txt is the first file most crawlers fetch from a site: a short set of rules saying which paths a well-behaved crawler may request. The App Router generates it from app/robots.ts the same way it generates the sitemap:

// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: "*",
        allow: "/",
        disallow: ["/admin", "/api", "/checkout", "/drafts/"],
      },
      {
        userAgent: "GPTBot",
        disallow: "/",
      },
    ],
    sitemap: "https://acme.com/sitemap.xml",
  };
}

Read the emitted semantics precisely, because the rules have an order and a grammar. allow and disallow are path prefixes; the most specific matching rule wins, so allow: "/" plus disallow: "/admin" lets everything except the admin tree through, and a bare disallow: "/" for a named userAgent blocks that crawler entirely — the pattern used to opt specific AI training crawlers out of a site. * in userAgent means every crawler; the field is a request for voluntary compliance, which is the pivot of this chapter.

Disallowing a path is not protecting it. Any agent can ignore robots.txt — the file is read before requesting, by convention and by good faith. It is invisible-door policy, not a lock: disallowing /admin keeps the admin login page out of search results and out of well-behaved crawlers' request queues, but anyone with the URL still loads it. Anything genuinely private needs real authentication, and anything genuinely unindexable additionally carries robots: { index: false } at the page level from The Metadata API — the two mechanisms are complementary (crawl guidance in one file, per-page index directives in the head) and neither replaces server-side access control.

There is also a trap in what you disallow: a path blocked from crawling can still be indexed if other sites link to it, because the crawler cannot read your noindex directive for a page it may not fetch. Disallowing a page you want de-listed is therefore counterproductive; the correct recipe for "remove this page from search" is the reverse — allow crawling, send index: false, and canonicalize or delete the URL. Teams learn this rule the expensive way during a staging-URL cleanup.

The sitemap field in the same file is the handshake: it tells crawlers where to find your crawl map even when they never visited the root. Keep both files in agreement — a sitemap listing paths the robots file disallows wastes crawl budget on requests that return nothing useful, and the audit that catches it is one curl of each file side by side:

curl -s https://acme.com/robots.txt
curl -s https://acme.com/sitemap.xml | grep -c '<loc>'   # how many URLs are advertised

For i18n sites, remember that robots rules apply per path prefix; a language-prefixed staging tree (/fr-staging/) is exactly the kind of thing a single disallow line cleanly covers. And since both files are plain generated text, they belong in the same deploy-verification checklist as the build output: if robots.txt 404s, every crawler defaults to permissive crawling of your whole origin — often how private-ish routes show up in search in the first place.

Sitemaps have hard physical limits — 50,000 URLs or 50 MB per file, whichever comes first — and soft limits that matter more: crawlers fetch sitemaps rarely (Googlebot might read yours daily or weekly), so a URL's freshness in the index lags by however often you regenerate. Both facts push large sites toward sharding, which Next.js exposes through generateSitemaps:

// app/sitemap/[id]/sitemap.ts — sharded generation
import type { MetadataRoute } from "next";

export async function generateSitemaps() {
  const shards = Math.ceil((await countProducts()) / 10_000);
  return Array.from({ length: shards }, (_, i) => ({ id: i }));
}

export default async function sitemap({
  params,
}: {
  params: Promise<{ id: string }>;
}): Promise<MetadataRoute.Sitemap> {
  const { id } = await params;
  const products = await getProductsPage(Number(id), 10_000);
  return products.map((p) => ({
    url: `https://acme.com/product/${p.slug}`,
    lastModified: p.updatedAt,
  }));
}

Each returned { id } becomes its own shard (/sitemap/0, /sitemap/1, …); a large site's root sitemap lists the shards, which is the sitemap-index pattern search engines expect. Shard boundaries should follow your data partitions (category ranges, ID windows) so one publish updates one shard, not all of them.

lastModified deserves the harshest scrutiny in this topic, because it drives crawl allocation: engines prioritize re-crawling URLs whose lastmod moved. A generator that stamps new Date() on every entry at each build claims "everything changed" on every deploy and forfeits the signal entirely — engines learn the noise and back off. Emit the record's actual updatedAt; if your CMS cannot provide one, omit the field rather than fake it.

Freshness of the files themselves follows their generation mode:

Treat crawl reports as operational telemetry once the files are live. Search Console's crawl stats show how often Googlebot fetches, which statuses it hits, and how much of the budget goes to non-indexable or redirected URLs — a rising share there means the sitemap is advertising paths that 404 or redirect, or robots rules are blocking something the map promises. The corrective loop is the same every time: fix the data (redirect or remove the entry), regenerate, and let the next crawl window show the correction. Teams that read those reports monthly catch drift the week it happens; teams that check yearly rediscover it during a traffic dip.

The operational checklist, in the order it pays for itself: verify both files respond 200 with the correct content types (text/xml-ish for the sitemap, text/plain for robots); grep that every disallowed prefix has no sitemap entries advertising it; confirm lastmod values track real updatedAts; and register the sitemap in Google Search Console / Bing Webmaster Tools so you can watch crawl errors instead of inferring them. Registration is the feedback loop the whole phase has been building toward: metadata drives how the link renders, cards drive how it previews, and these two files drive whether it gets crawled at all — the last gate before search engines can act on any of it. Above those files, sitting in every page's head, is the structured markup that decides how a crawled page renders in results: JSON-LD Structured Data.