# Metadata, sitemaps and robots with the SDK

> Give every page its title, description, canonical link, share image, hreflang and structured data from the CMS, and serve sitemaps, a sitemap index and robots.txt for one region or many, including on staging.

Source: https://docs.bifrostorbit.com/cms/developers/seo-with-the-sdk/

Editors manage SEO in the CMS: titles, descriptions, share images, no-index switches, redirects and the sitemap. Your website's job is to output it. The SDK does almost all of it. This page shows each part, building on [Connect a website](https://docs.bifrostorbit.com/cms/developers/connect-a-website/).

## Page metadata

```tsx
// app/[[...slug]]/page.tsx
import type { Metadata } from 'next'
import { toMetadata } from '@bifrost-orbit-cms/sdk/seo'

export async function generateMetadata(props: Props): Promise<Metadata> {
  const result = await resolveFor(props)
  return result.kind === 'page' ? toMetadata(result.page) : {}
}
```

`toMetadata(page)` reads `page.seo`, which the CMS has already completed with the space's defaults, and returns:

| Metadata | From | Written as |
| --- | --- | --- |
| `title` | The page's SEO title, or its name, with the title template from **SEO › Settings** applied | `<title>` |
| `description` | The page's description, or the space's default. Left out when empty. | `<meta name="description">` |
| `alternates.canonical` | The page's canonical address, from **Configuration › Domains** and the region prefix, or the one editors set | `<link rel="canonical">` |
| `alternates.languages` | The same page in other languages and regions. Left out when there are none. | `<link rel="alternate" hreflang>` |
| `robots` | `index` and `follow`, from the page's no-index and no-follow switches | `<meta name="robots">` |
| `openGraph` | Type, title, description, address, site name, locale and share image | `og:` tags |
| `twitter` | Card, the site's account, title, description and image | `twitter:` tags |
| `verification` | Google and Bing verification codes from **SEO › Settings**. Bing's is written as `msvalidate.01`. | `<meta name="google-site-verification">` and `<meta name="msvalidate.01">` |

CMS Orbit checks the result on the live website about once a day: it loads each published page in Production on a verified domain and compares its `<link rel="alternate" hreflang>` and `<link rel="canonical">` tags with what it sent. Pages that leave them out, or change them, show up under **Languages** in **SEO › Health**. The check identifies itself as `CMS Orbit SEO check` in its user agent, and reads only the page's `<head>`.

### Combine it with your own metadata

Next.js merges a page's metadata into the layout's, one top-level key at a time. A few things to know:

- **Title templates.** The CMS title already has the template from **SEO › Settings** applied, such as "Hiking boots | Acme Outdoors". Do not also set `title.template` in your layout, or the site name appears twice. If you must, return `title: { absolute: page.seo.title }`.
- **`metadataBase`.** Set it in the root layout to your website's address, so relative image addresses in your own metadata become full ones. The CMS's addresses are already full.
- **Open Graph is replaced, not merged.** If the layout sets `openGraph` and the page returns one too, the page's wins entirely. Add your own fields to the page's: `{ ...meta, openGraph: { ...meta.openGraph, siteName: 'Acme Outdoors' } }`.
- **Not found and redirects.** Return `{}` or a short title for anything that is not a page, as above.

```ts
const meta = toMetadata(result.page)
return {
  ...meta,
  icons: { icon: '/icon.svg' },
  other: { 'theme-color': '#ffffff' },
}
```

### Staging and preview websites

robots.txt on a non-indexable environment, such as Staging, refuses everything (see below). To be sure nothing on a staging website is indexed even when a page is reached directly, override `robots` there:

```ts
const indexable = process.env.SITE_INDEXABLE === 'true'
return { ...meta, robots: indexable ? meta.robots : { index: false, follow: false } }
```

Set `SITE_INDEXABLE=true` only on the live website. Canonical links on staging still point at the live addresses, which is what you want: search engines that do find a staging page are told the live one is the real one.

Without the SDK, Pages Router websites and other frameworks can write the same tags from `page.seo`; [Delivery API](https://docs.bifrostorbit.com/cms/developers/delivery-api/) lists every field.

## Structured data

```tsx
import { JsonLd } from '@bifrost-orbit-cms/sdk/seo'

<JsonLd page={result.page} />
```

`JsonLd` writes `page.seo.jsonLd` as one `<script type="application/ld+json">`: a single object when there is one, a list when there are several. It writes nothing when there are none. Characters that could end the script early are escaped, so editors' text cannot break out of it.

The CMS builds the structured data from the page and the space's settings, such as breadcrumbs, the organisation and the page type. Place `JsonLd` anywhere in the page; search engines read it wherever it is.

To add your own, write another script next to it:

```tsx
<script type="application/ld+json" dangerouslySetInnerHTML={{ __html: JSON.stringify(mine).replace(/</g, '\\u003c') }} />
```

`page.seo.breadcrumbs` has the same trail as a list of `{ name, url }`, for drawing visible breadcrumbs.

## robots.txt

```ts
// app/robots.txt/route.ts
import { createRobotsRoute } from '@bifrost-orbit-cms/sdk/next'
import { cms } from '@/lib/cms'

export const GET = createRobotsRoute(cms)
```

It serves the environment's robots.txt as plain text, cached for five minutes:

- **Production** gets the robots.txt editors set in **SEO › Settings**, with `Sitemap:` lines for each sitemap.
- **Any environment that is not indexable**, such as Staging, always gets `User-agent: *` and `Disallow: /`, whatever the settings say.

So the same code is safe on every environment, as long as each website uses a key from its own environment. `cms.getRobots()` answers `{ indexable, content }` if you would rather build it yourself.

Remove `app/robots.ts` if you have one; Next.js does not allow both.

## Sitemaps

The CMS lists every published page with an address, skipping pages that are no-index or left out of the sitemap by editors. Each entry has the page's full address, its last change, change frequency and priority if set, its images and its other languages. Addresses come from **Configuration › Domains**, so set those up first.

A sitemap file holds up to 50,000 addresses. Bigger regions are split into several files.

### One region

```ts
// app/sitemap.xml/route.ts
import { createSitemapRoute } from '@bifrost-orbit-cms/sdk/next'
import { cms } from '@/lib/cms'

export const GET = createSitemapRoute(cms)
```

It reads the client's `defaultRegion` in its default language and serves the XML with image and hreflang entries, cached for five minutes. `createSitemapRoute(cms, { locale: 'en-DE' })` serves another language.

For a region with more than 50,000 pages, serve the extra files too. `createSitemapRoute` reads the file number from a name like `sitemap-2.xml`, in the route's `file` parameter or the address. Send those names to a route with a rewrite, because a dynamic segment at the top level would clash with the page route:

```ts
// next.config.ts
const nextConfig = {
  async rewrites() {
    return [{ source: '/sitemap-:n(\\d+).xml', destination: '/api/sitemap/sitemap-:n.xml' }]
  },
}
export default nextConfig
```

```ts
// app/api/sitemap/[file]/route.ts
import { createSitemapRoute } from '@bifrost-orbit-cms/sdk/next'
import { cms } from '@/lib/cms'

export const GET = createSitemapRoute(cms)
```

`cms.getSitemap({ page: 1 })` answers `totalPages`, so you know how many files there are.

### Several regions or languages

Each region and language has its own sitemap at its own address, such as `/de/sitemap.xml`, and one sitemap index lists them all.

```ts
// app/sitemap.xml/route.ts
import { createSitemapIndexRoute } from '@bifrost-orbit-cms/sdk/next'
import { cms } from '@/lib/cms'

export const GET = createSitemapIndexRoute(cms)
```

The index lists `sitemap.xml`, `sitemap-2.xml` and so on under each region's address, such as `https://www.example.com/de/sitemap.xml`. A region and language with no domain is left out.

Then serve each region's sitemap. With regions mapped from the address prefix as in [Regions and languages](https://docs.bifrostorbit.com/cms/developers/regions-and-languages/):

```ts
// app/[region]/sitemap.xml/route.ts
import { sitemapXml } from '@bifrost-orbit-cms/sdk/seo'
import { cms } from '@/lib/cms'
import { regionFor } from '@/lib/regions'

export async function GET(_request: Request, { params }: { params: Promise<{ region: string }> }) {
  const region = regionFor((await params).region)
  if (!region) return new Response('Not found', { status: 404 })
  const sitemap = await cms.getSitemap({ region: region.key })
  return new Response(sitemapXml(sitemap), {
    headers: { 'Content-Type': 'application/xml; charset=utf-8', 'Cache-Control': 'public, max-age=300' },
  })
}
```

`sitemap.xml` is a fixed folder name, so it takes priority over `[[...slug]]` under `[region]`. If your address prefixes are the regions' keys or short codes, `export const GET = createSitemapRoute(cms)` does the same in one line: it passes the `region` parameter straight to the API.

For each language of a region, pass `locale` to `getSitemap`, at an address of your choice, and make sure **Configuration › Domains** builds the same address for that language.

### With Next.js's own sitemap file

`app/sitemap.ts` is the other way, and lets you change addresses, for example to your staging host:

```ts
// app/sitemap.ts
import type { MetadataRoute } from 'next'
import { cms } from '@/lib/cms'

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const { entries } = await cms.getSitemap()
  return entries.map((entry) => ({
    url: entry.loc,
    ...(entry.lastmod ? { lastModified: entry.lastmod } : {}),
    ...(entry.priority !== null ? { priority: entry.priority } : {}),
    alternates: { languages: Object.fromEntries(entry.alternates.map((alt) => [alt.hreflang, alt.href])) },
  }))
}
```

Use either this or `app/sitemap.xml/route.ts`, not both.

### Keeping sitemaps fresh

Sitemap answers are tagged `cms:sitemap`, which the webhook refreshes whenever a page is published, unpublished or deleted and when a release runs. The five-minute `Cache-Control` lets search engines and CDNs keep a copy briefly.

Tell search engines where the sitemap is once, in Google Search Console and Bing Webmaster Tools. The `Sitemap:` lines in robots.txt tell everyone else.

## What editors control

Most SEO needs no code. Editors set, in the CMS:

- each page's SEO title, description, share image, canonical address and no-index and no-follow switches;
- whether a page is left out of the sitemap;
- redirects, which `resolve` follows for you;
- the title template, default description and share image, social profiles, robots.txt and verification codes in **SEO › Settings**.

**SEO › Health** shows pages with problems, such as missing descriptions. See [SEO health](https://docs.bifrostorbit.com/cms/editors/seo-health/) and [SEO metadata](https://docs.bifrostorbit.com/cms/editors/seo-metadata/).