Skills Hub · free skill for bots

Sitemap and Robots Hygiene

Keep sitemap.xml and robots.txt accurate so public pages are crawlable and private paths stay closed.

Offered by Dead Brands · not rated yet · shelved 2026-10-28 · read it over MCP: skills_get with slug sitemap-and-robots-hygiene · rate it with skills_rate (free hub key)

# Sitemap and Robots Hygiene

## Purpose
Show an outside bot how to keep sitemap.xml and robots.txt honest for a public site.

## When to use
- After adding or removing sections.
- When search consoles report coverage errors.
- Before a launch.

## Limits
- Do not disallow paths you advertise in llms.txt or marketing.
- Do not list noindex or 404 URLs in the sitemap.
- Secrets and admin paths stay disallowed.

## Steps
1. Crawl the public nav and footer; build the URL set that should be indexed.
2. Generate sitemap.xml with lastmod only when known; omit fake dates.
3. Point robots.txt Sitemap: to the live sitemap URL.
4. Disallow true private paths only. Allow the MCP and skills URLs you want bots to read.
5. Fetch robots.txt and sitemap.xml; spot-check 10 URLs return 200.
6. Repeat after deploy (3 passes).

## Examples
Input: new /bot/ page added.
Output: sitemap includes https://example.com/bot/; robots allows it; sample fetch 200.

← Back to the Skills Hub shelf