Sintassi ed esempi robots.txt

The robots.txt file is the first thing search engine crawlers check when visiting your website. A properly configured robots.txt can protect private content, save crawl budget, and improve SEO. This guide covers every directive, wildcard pattern, and real-world example you need.

What is robots.txt?

robots.txt is a plain text file placed at the root of your website (e.g., https://example.com/robots.txt) that tells web crawlers which URLs they are allowed or forbidden to access. It follows the Robots Exclusion Protocol (REP), first proposed in 1994 and now an internet standard (RFC 9309).

How crawlers use robots.txt:

1. A crawler arrives at your domain and requests /robots.txt before crawling any other page.
2. If the file exists, the crawler parses rules for its specific User-agent.
3. The crawler follows matching Disallow and Allow directives when deciding which URLs to fetch.
4. If no robots.txt is found (404), the crawler assumes everything is allowed.

Syntax Rules & Directives

A robots.txt file is composed of one or more rule groups. Each group starts with a User-agent line followed by directives:

• User-agent — User-agent — Specifies which crawler the rules apply to. Use * for all crawlers.
• Disallow — Disallow — Blocks a URL path. An empty value (Disallow:) means nothing is blocked.
• Allow — Allow — Explicitly permits a URL path, overriding a broader Disallow. Supported by Googlebot and most modern crawlers.
• Sitemap — Sitemap — Points to your XML sitemap. Can appear anywhere in the file, outside rule groups.
• Crawl-delay — Crawl-delay — Requests a delay (in seconds) between successive requests. Supported by Bing and Yandex but NOT by Google.

Formatting rules:

• One directive per line.
• Lines starting with # are comments.
• Blank lines separate rule groups.
• Paths are case-sensitive (/Admin is different from /admin).
• The file must be named exactly robots.txt and placed at the domain root.

Basic syntax example:

# This is a comment
User-agent: *
Disallow: /private/
Disallow: /tmp/
Allow: /private/public-page.html

# Slow down Bingbot
User-agent: Bingbot
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

Wildcard Patterns

Google, Bing, and most major crawlers support two wildcard characters in robots.txt paths:

• * — * (asterisk) — Matches any sequence of characters (including empty).
• $ — $ (dollar sign) — Anchors the match at the end of the URL.

Wildcard pattern examples:

User-agent: *

# Block all URLs containing "?sort="
Disallow: /*?sort=

# Block all .pdf files
Disallow: /*.pdf$

# Block all URLs with query parameters
Disallow: /*?*

# Block all .json API responses
Disallow: /*.json$

# Block paths containing /temp/ anywhere
Disallow: /*/temp/

# Allow specific .xml files (sitemaps)
Allow: /*.xml$

Common robots.txt Examples

Block all crawlers from the entire site:

User-agent: *
Disallow: /

Allow all crawlers (explicit):

User-agent: *
Disallow:

Block a specific crawler:

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

Block a specific directory:

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /staging/

Block all but allow a specific path:

User-agent: *
Disallow: /api/
Allow: /api/public/

Multiple rule groups:

# Default rules for all crawlers
User-agent: *
Disallow: /admin/
Disallow: /private/

# Googlebot gets special access
User-agent: Googlebot
Disallow: /admin/
Allow: /private/google-partner/

# Block aggressive SEO bots entirely
User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

WordPress robots.txt

WordPress sites have specific directories and files that should typically be blocked from crawlers to save crawl budget and prevent indexing of admin/utility pages:

User-agent: *
# Block WordPress admin and login
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Block WordPress includes
Disallow: /wp-includes/

# Block XML-RPC (security best practice)
Disallow: /xmlrpc.php

# Block trackbacks and pingbacks
Disallow: /trackback/
Disallow: /*/trackback/

# Block comment feeds
Disallow: /*/feed/
Disallow: /*/comments/

# Block search results pages
Disallow: /?s=
Disallow: /search/

# Block author archives (optional)
Disallow: /author/

# Block tag pages with thin content (optional)
Disallow: /tag/

# Allow all media uploads
Allow: /wp-content/uploads/

Sitemap: https://example.com/sitemap_index.xml

Next.js / React SPA robots.txt

Modern JavaScript frameworks serve static assets and API routes that should not be indexed. Here is a recommended robots.txt for Next.js applications:

User-agent: *
# Block Next.js internal routes
Disallow: /_next/
Disallow: /api/

# Block internal utility pages
Disallow: /404
Disallow: /500

# Block query parameter variations
Disallow: /*?*

# Allow static assets that search engines need
Allow: /_next/static/
Allow: /_next/image/

# Allow public API endpoints (if any)
Allow: /api/public/

Sitemap: https://example.com/sitemap.xml

E-commerce robots.txt

E-commerce sites need careful robots.txt configuration to prevent crawling of user accounts, checkout flows, and faceted navigation that creates duplicate content:

User-agent: *
# Block user account pages
Disallow: /account/
Disallow: /my-account/
Disallow: /login/
Disallow: /register/
Disallow: /password-reset/

# Block checkout and cart
Disallow: /cart/
Disallow: /checkout/
Disallow: /order-confirmation/

# Block wishlist and compare
Disallow: /wishlist/
Disallow: /compare/

# Block internal search results
Disallow: /search/
Disallow: /*?q=
Disallow: /*?search=

# Block faceted navigation (duplicate content)
Disallow: /*?sort=
Disallow: /*?order=
Disallow: /*?filter=
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?price=
Disallow: /*?page=

# Block review/rating sort variations
Disallow: /*?rating=

# Allow product and category pages
Allow: /products/
Allow: /category/
Allow: /collections/

Sitemap: https://example.com/sitemap.xml

Block AI Crawlers

With the rise of large language models, many site owners want to prevent AI training crawlers from scraping their content. Here are the known AI crawler user agents and how to block them:

Known AI crawler user agents:

• GPTBot — GPTBot — OpenAI's crawler for training data
• ChatGPT-User — ChatGPT-User — OpenAI's crawler for ChatGPT browsing feature
• Google-Extended — Google-Extended — Google's AI training crawler (Gemini/Bard)
• anthropic-ai — anthropic-ai — Anthropic's crawler for Claude training data
• ClaudeBot — ClaudeBot — Anthropic's web crawler
• CCBot — CCBot — Common Crawl bot (used by many AI companies)
• Bytespider — Bytespider — ByteDance's crawler
• Amazonbot — Amazonbot — Amazon's crawler

# Block all known AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

# Still allow regular search engine crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Sitemap Directive

The Sitemap directive tells crawlers where to find your XML sitemap. This is especially useful because it does not require a specific User-agent block — it applies globally:

# Single sitemap
Sitemap: https://example.com/sitemap.xml

# Multiple sitemaps
Sitemap: https://example.com/sitemap-pages.xml
Sitemap: https://example.com/sitemap-posts.xml
Sitemap: https://example.com/sitemap-products.xml

# Sitemap index file
Sitemap: https://example.com/sitemap_index.xml

Sitemap directive rules:

• Must use a full absolute URL (including https://).
• Can list multiple Sitemap directives for multiple sitemaps.
• Can be placed anywhere in the file (not tied to a User-agent group).
• Supports sitemap index files that reference other sitemaps.

Testing & Validation

Always test your robots.txt before deploying to production. A single typo can accidentally block your entire site from search engines.

• Google Search Console — Google Search Console — The "robots.txt Tester" tool lets you test specific URLs against your rules.
• curl — curl command — Quickly check if your robots.txt is accessible:
• Online — Online validators — Tools like Google's robots.txt tester or Merkle's robots.txt analyzer.

# Check if robots.txt is accessible
curl -I https://example.com/robots.txt

# View the full content
curl https://example.com/robots.txt

# Check what Googlebot sees (simulate Googlebot)
curl -A "Googlebot" https://example.com/robots.txt

# Test a specific URL against robots.txt (Python)
pip install robotexclusionrulesparser
python -c "
import robotexclusionrulesparser as rerp
rp = rerp.RobotExclusionRulesParser()
rp.fetch('https://example.com/robots.txt')
print(rp.is_allowed('Googlebot', '/private/page'))
"

Common mistakes to avoid:

• Placing robots.txt in a subdirectory instead of the domain root.
• Using relative URLs in the Sitemap directive (must be absolute).
• Blocking CSS/JS files that search engines need for rendering.
• Forgetting that Disallow: / blocks EVERYTHING including your homepage.
• Not testing after changes — always verify with Google Search Console.
• Using incorrect line endings or BOM characters (use UTF-8 without BOM).

robots.txt vs meta robots vs X-Robots-Tag

There are three main ways to control crawler behavior. Each serves a different purpose:

FAQ