Does Disallow remove a page from Google?

No, and this catches people out. Disallow stops the page being fetched, not listed — a blocked URL that other sites link to can still appear in results, showing only the URL because Google was never allowed to read a title. To remove a page, let it be crawled and put a noindex tag on it.

Is robots.txt a security measure?

The opposite. It is a public file that well-behaved crawlers choose to obey, and listing /admin/ in it advertises exactly where your admin panel lives to anyone who reads it — which is anyone. Protect private areas with authentication; use robots.txt only for crawl efficiency.

What does an empty Disallow line mean?

Nothing is blocked — it is the explicit way of saying "allow everything". "Disallow: /" with the slash is the opposite and blocks the whole site, which is why the difference between those two characters is worth double-checking before you deploy.

Does Google respect crawl-delay?

No, Google ignores it entirely; crawl rate is adjusted in Search Console instead. Bing and Yandex do honour it. Leave it empty unless crawler traffic is genuinely straining your server.

Robots.txt Generator

Robots.txt is the first file a crawler asks for, and one careless line in it can take a whole site out of search. This builds the file from presets — allow everything, block the usual admin and search-result paths, or block a staging site entirely — and adds your own rules and sitemap reference on top. Every line it writes is standard syntax, so there are no vendor-specific directives to trip over.

Starting point

User-agent

* means every crawler. Name one, like Googlebot, to write a rule just for it.

Extra paths to block

One path per line, each starting with /.

Exceptions to allow

Sitemap URL

Crawl delay (seconds)

Leave empty unless crawlers are overloading your server. Google ignores this.

Result

User-agent: *
Disallow:

Save this as robots.txt at the root of your domain — it only works at /robots.txt, not in a subfolder. An empty Disallow line means “nothing is blocked”.

Nothing is uploaded — this tool runs inside your browser, so your files never leave your device.

Help us improve

Why use this tool

Presets for the three cases that cover almost every site
Custom disallow and allow paths, one per line
Sitemap reference and optional crawl-delay
Runs in your browser — nothing uploaded

How it works

1Choose a starting point, then add any paths of your own.
2Add your sitemap URL so crawlers can find your page list.
3Save the result as robots.txt in the root of your domain.

Robots.txt Generator — frequently asked questions

At the very root of the domain — https://example.com/robots.txt and nowhere else. Crawlers only look there, so a copy in a subfolder does nothing. Each subdomain needs its own: blog.example.com is not covered by the file on example.com.

More seo tools

Meta Tags Open Graph Schema Keyword Density Sitemap Canonical Tag

Looking for something else? Browse all seo tools

Explore the other tools

Same deal everywhere on the site — free, no sign-up, and no watermark on anything you download.

QR Code Tools Barcode Tools PDF Tools Image Tools Developer Tools Text Tools Calculators Unit Converters Color Tools File Tools Business Tools AI Tools Utilities

Or see every tool on one page.