Quick Presets
User-Agent Rules
Define crawling rules for different bots
Disallow Paths
Allow Paths
Copy this content to your robots.txt file
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Allow: / Allow: /products/ Allow: /categories/ Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/products-sitemap.xml
Save the generated content as "robots.txt" and upload it to your website's root directory (e.g. https://yoursite.com/robots.txt).
Use Google Search Console's robots.txt Tester to verify your file works correctly.
Use Disallow: / to block all crawlers, or list specific paths. Allow: / can override a broader Disallow. Always add your Sitemap URL(s).
