Skip to content
Free Tool by Peplio

Robots.txt & Sitemap.xml Generator

Enter your domain and I'll crawl your site, generate a proper robots.txt (with AI bot blocking) and a full sitemap.xml — ready to download. No signup. Works up to 150 pages.

⚙️ Advanced options ▼
Prevents ChatGPT/Claude/Perplexity from training on your content
Real site crawl Up to 150 URLs AI bot blocking Instant download
Generator output

Enter your domain to start

Free

Fetching your homepage…

URLs Found
—
Pages Crawled
—
Load Time
—
Existing robots.txt —
Existing sitemap.xml —
HTTPS enabled —
➕ Add more URLs manually Have URLs the crawler didn't find? Paste them here (one per line) and click Regenerate.
  • Enter your website URL above and I'll crawl up to 150 pages to build both files.
  • The robots.txt includes optional AI bot blocking — a growing SEO best practice for 2026.
  • The sitemap lists discovered URLs. Review them before publishing. Modification dates are omitted because this tool cannot verify when pages changed.

Draft files, then review

This free robots.txt and sitemap.xml generator discovers a limited set of public URLs and creates draft files. Review your existing rules and the URL list before publishing. A sitemap helps discovery but does not guarantee indexing or rankings.

Our test on 16 November 2026

For https://peplio.com/, the crawl returned 150 discovered URLs and reported one page crawled. This is a capped discovery result, not a complete website inventory. Existing sitemaps or homepage links can contribute URLs without every page being crawled.

We verified that the revised generator leaves AI training opt-out unchecked by default, does not include OAI-SearchBot in that opt-out, and omits unverified modification dates from XML.

How to generate a draft safely

  1. Enter your public website address.
  2. Open Advanced options and select rules that fit your site. AI training opt-out is optional.
  3. Generate the files and inspect each URL. Keep only canonical, indexable pages that return successfully.
  4. Add missing URLs manually, then regenerate and review again.
  5. Back up any existing files before replacing them. If WordPress or your SEO plugin already maintains a working sitemap, keep that automatic sitemap rather than replacing it with a static draft.

What the robots options mean

  • WordPress rules: Optional rules restrict administrative and utility paths. Plugin and wp-includes resources are not blocked by this preset because search engines may need them to render a page.
  • Cart and filter rules: Check the generated patterns against your actual shop URLs before use.
  • AI training opt-out: Adds rules for selected training crawlers. It does not add blocks for OAI-SearchBot, ChatGPT-User or PerplexityBot. Different crawler controls have different purposes; robots rules depend on voluntary compliance.
  • Image crawling: Image crawlers follow the standard rules. The generator does not add a separate allow-all group that could override your path restrictions.
  • Crawl delay: Support varies. Do not rely on this option to control Googlebot or protect private data.

Why this sitemap omits lastmod and priority

The tool does not know when each page last changed, so it does not label every URL as modified today. It also omits priority and changefreq. Google says it ignores those two values and uses lastmod only when it is consistently accurate. See Google’s sitemap guidance.

For a WordPress site, a sitemap generated by the site’s SEO plugin is usually easier to keep current as pages change. Avoid maintaining competing static and dynamic URL lists unless you have a clear reason.

Limits and troubleshooting

  • The crawler is capped at 150 discovered URLs. It may miss JavaScript links, unlinked pages, alternate languages or blocked resources.
  • A discovery result is not a full status-code, canonical or noindex audit. Validate URLs before submission.
  • If crawling fails, the draft contains only the homepage and the tool now says so. Add verified URLs manually.
  • Robots.txt is publicly readable and cannot secure private information.
  • Blocking crawling is not a reliable method for removing an already indexed URL. Review Google’s robots.txt documentation before changing live rules.

Keep your checks focused

Use the AI crawler access checker for crawler rules, the AEO checker for pasted content, or browse all Peplio tools.

Frequently asked questions

S

Peplio tool notes

Published by Sougan Kumar Mandi, founder of Peplio. Updated after the tests described above on 16 November 2026.