Rule Details

For a page to appear in Google Search, it must be crawlable and indexable. Google's indexing controls guide (opens in new tab) and your broader robots meta strategy both matter here, because three separate mechanisms can block indexing: noindex meta tags, X-Robots-Tag HTTP headers, and robots.txt rules.

Code Examples

❌ Avoid — important page accidentally noindexed

<!-- Product page that should rank in search -->
<meta name="robots" content="noindex, nofollow">
<!-- This page will never appear in Google Search -->

❌ Avoid — Next.js metadata with indexing disabled

// app/products/[slug]/page.tsx — accidental noindex
export const metadata = {
  robots: {
    index: false,   // This blocks indexing for ALL product pages!
    follow: false,
  },
}

✅ Correct — indexable page (no noindex)

<!-- Default: no meta robots tag = index, follow -->
<!-- Or be explicit: -->
<meta name="robots" content="index, follow">

✅ Correct — Next.js metadata enabling indexing

export const metadata = {
  // For pages you want indexed, either omit 'robots' or set explicitly:
  robots: {
    index: true,
    follow: true,
  },
}

✅ Limiting noindex to genuinely private content

// pages/api/preview-post.ts — only noindex preview/draft pages
export async function generateMetadata({ params }) {
  const post = await getPost(params.slug)
 
  return {
    robots: post.published
      ? { index: true, follow: true }
      : { index: false, follow: false },  // Draft only
  }
}

✅ Testing indexability

# Check X-Robots-Tag header
curl -I https://yoursite.com/important-page | grep -i 'x-robots'
 
# Check meta robots in HTML
curl -s https://yoursite.com/important-page | grep -i 'robots'

Why It Matters

• No index = no ranking: A page with noindex cannot appear in search results under any circumstances.
• Common accidents: CMS platforms often default new pages to noindex (e.g., draft mode, staging environments accidentally promoted, category pages in WordPress).
• Silent failure: Indexing blocks cause no visible errors on the frontend. Only Google Search Console (opens in new tab) or log analysis reliably surfaces the problem, and those same checks help uncover indexability conflicts.

The Three Indexing Mechanisms

Exceptions

• Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
• Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
• When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.

Standards

• Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
• Check the implementation against Google Search Central: Prevent Google from indexing pages before treating the rule as satisfied.
• Check the implementation against Google Search Central: Robots meta tag before treating the rule as satisfied.

Verification

Automated Checks

• Use Google Search Console Coverage report → "Excluded" tab to find pages marked "Indexed, though blocked by robots.txt" or "Noindex page".
• Use URL Inspection in Search Console for specific URLs.

Manual Checks

• Export your top 50–100 pages by importance (traffic, revenue, links).
• For each, check: meta robots, X-Robots-Tag header, robots.txt.
• Fix blocks and request re-indexing.