Audit all noindex pages
Lists and reviews all pages blocked from indexing to ensure critical content is accessible.
- Identify all pages using the `noindex` directive in meta tags or headers
- Verify that critical pages (products, articles) aren't accidentally blocked
- Ensure `noindex` is correctly applied to thin content, admin, or utility pages
Rule Details
Regularly auditing your noindex directives ensures that you are only blocking pages that don't provide search value, such as staging environments, admin panels, or internal search results.
Code Example
<head>
<!-- Use this for utility pages like 'Thank You' pages or internal search results -->
<meta name="robots" content="noindex, follow">
<!-- For pages that should be completely ignored -->
<meta name="robots" content="noindex, nofollow">
</head>Why It Matters
- Visibility: Prevents accidental exclusion of high-value pages from Google and Bing.
- Crawl Budget: Helps search engines focus their limited crawl resources on your most important pages.
- De-indexing: Useful for removing outdated or sensitive pages from search results without deleting them.
- Strategic Control: Allows you to manage which versions of content (e.g., filtered lists) are available for indexing.
Exceptions
- Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
- Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
- When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.
Standards
- Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
- Check the implementation against Google Search Central: Search Essentials before treating the rule as satisfied.
- Check the implementation against Google Search Central documentation before treating the rule as satisfied.
Verification
Automated Checks
- Inspect rendered HTML and HTTP headers to confirm the expected metadata or crawlability signal is present.
- Test the affected URL with Google Search Console or equivalent tooling where relevant.
- Re-crawl a representative page set after deployment.
Manual Checks
- Confirm the change does not create conflicting canonical-url, robots, or structured-data signals.
Use with AI
Copy these prompts to use with your AI assistant, or install the MCP server to use directly from Claude, Cursor, or Windsurf.
Check
Verify implementation
Verify that the `noindex` directive is only applied to pages that should not appear in search results.
Fix
Auto-fix issues
Remove `<meta name="robots" content="noindex">` from pages that should be indexed and ranking.
Explain
Learn more
Explain the difference between `noindex` and `disallow` in robots.txt and when to use each for optimal SEO.
Review
Code review
Review metadata generation, rendered HTML, structured data, and response headers related to Audit all noindex pages. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.