Create a clean robots.txt file for your website, control crawler access, add your sitemap, validate common rules and download the final file instantly.
Choose your crawler rules, enter your sitemap URL and generate a ready-to-save robots.txt file. You can also edit the generated file manually before downloading.
Browser-based generation and validation — no data is uploaded.
Choose which crawler the rules apply to.
Select common rules to add to your robots.txt.
One path per line. Example: /private/
Optional paths to explicitly allow.
Use the full canonical sitemap URL.
Optional — used for validation and preview.
Optional. Add extra valid robots.txt directives manually.
User-agent: * Disallow: /wp-admin/ Disallow: /?s= Sitemap: https://example.com/sitemap_index.xml
A robots.txt file can communicate crawler access preferences at the root of a website. Use it carefully and keep important, indexable content accessible.
Define User-agent rules and specify paths that crawlers should or should not request.
Keep private, temporary or low-value crawl areas out of crawler request paths where appropriate.
Reference your XML sitemap so supporting crawlers can discover the sitemap location.
Check common formatting problems, duplicate rules, invalid sitemap URLs and risky directives.
Use the generated output as a starting point and add advanced directives when required.
Copy the output or download a plain-text robots.txt file ready for your website root.
These are the core building blocks used by the generator. Always review rules before publishing them on a production website.
Identifies the crawler to which a group of rules applies. The wildcard * represents all crawlers.
Specifies a URL path that the matching crawler is instructed not to request.
Can be used to explicitly permit a path when a crawler's rules otherwise restrict a broader path.
Provides the location of an XML sitemap. Use a full absolute URL and keep the sitemap itself crawlable.
Use these checks to reduce the risk of accidentally restricting useful website content.
Blocking crawling is different from controlling indexation. Indexed URLs can still appear without crawl access in some circumstances.
Search engines may need accessible resources to understand and render pages properly.
If you reference a sitemap, make sure the URL is correct, returns successfully and is not blocked by robots.txt.
A path such as / can prevent crawling of an entire site for matching crawlers. Avoid broad rules unless intentional.
Do not use robots.txt to protect confidential information. Restricted URLs may still be discovered or accessed directly.
After uploading the file to your site's root, verify the live file and test important URLs with appropriate crawler tools.
Common questions about creating and using a robots.txt file.
A robots.txt file is a plain-text file placed at the root of a website that communicates crawler access rules for matching user-agents.
For a standard HTTPS website, robots.txt is normally placed at the root, such as https://example.com/robots.txt.
Not reliably. Robots.txt primarily controls crawling. If the goal is to prevent a page from being indexed, use an appropriate indexation control such as a noindex directive where applicable.
You can reference an XML sitemap using the Sitemap directive. Use the full sitemap URL and make sure the sitemap is accessible to crawlers.
Yes. The generator includes common WordPress paths as optional quick rules, but you should review them against your actual site's setup before publishing.
No. Robots.txt provides crawler instructions; it should never be used to protect confidential files, passwords or sensitive information.
Yes. The generated output is editable through the advanced rules field before you generate the final file. You can also review the output and copy or download it as plain text.
I help businesses improve crawlability, indexation, site architecture, technical SEO and overall search visibility.
LET'S TALK ABOUT SEO →