What Is a Robots.txt File? How It Works and How to Write One
A robots.txt file is a small plain-text file at the root of your website that tells search engine crawlers which URLs they may and may not fetch. Its main job is to manage crawler traffic and keep bots away from unimportant pages. It is not a way to hide a page from Google, and that single misunderstanding causes most robots.txt mistakes.
If you only remember two things: the file must live at yoursite.com/robots.txt, and blocking a URL from crawling is not the same as keeping it out of search results. The rest of this guide shows you how to read one, write one, test it, and avoid the errors that quietly hurt rankings.
What Is a Robots.txt File?
Think of robots.txt as a set of house rules posted at the front door of your site. Well-behaved crawlers, such as Googlebot, read the rules before they fetch anything else. Each rule says which crawler it applies to and which paths that crawler should stay away from.
According to Google's documentation, the file is mainly for managing crawler traffic so your server is not overloaded, and for steering crawlers away from duplicate or low-value URLs. It is part of the Robots Exclusion Protocol, a long-standing convention that reputable crawlers follow voluntarily.
What Robots.txt Can and Cannot Do
Because it is a polite request and not a lock, robots.txt has real limits. This table shows the difference.
| You want to... | Use robots.txt? | Better option |
|---|---|---|
| Reduce crawling of filter or search URLs | Yes | A Disallow rule |
| Keep a page out of Google results | No | A noindex tag, or password protection |
| Protect private or sensitive files | No | Password protection or server permissions |
| Point crawlers to your sitemap | Yes | A Sitemap line |
Google also notes that a blocked URL can still appear in search results, without a description, if other sites link to it. And since the file is publicly readable by anyone, never list secret folder names in it.
Where Does Robots.txt Go?
The file must be named exactly robots.txt and sit at the root of the host it governs. A copy inside a subfolder is ignored. A few more rules from Google's guidance:
- A site can have only one robots.txt file.
- Each subdomain can have its own file, and a file applies only to the protocol, host and port where it lives.
- Save it as plain text in UTF-8. Word processors can add hidden formatting that confuses crawlers.
If you use a hosted builder such as Wix or Blogger, you may not be able to edit the file directly. In that case, use the platform's own search visibility settings.
Robots.txt Syntax: The Four Lines You Need
Almost every robots.txt file is built from four directives.
- User-agent: names the crawler a group of rules applies to. An asterisk (
*) means all crawlers, except Google's AdsBot crawlers, which must be named explicitly. - Disallow: a path the crawler should not fetch. Paths start with a slash.
- Allow: an exception inside a blocked area, such as one page in a blocked folder.
- Sitemap: the full URL of your XML sitemap. You can list more than one.
Lines starting with # are comments. Anything that is not blocked is allowed by default, so you do not need an Allow line for your whole site.
A Simple Robots.txt Example
Here is a safe starting point for a typical small business or blog site:
User-agent: *
Disallow: /cart/
Disallow: /search
Sitemap: https://www.example.com/sitemap.xmlThis tells every crawler to skip the cart and internal search result pages, and shows where the sitemap is. Everything else stays open. If you have not built a sitemap yet, our guide to the XML sitemap walks through it.
How Crawlers Read the Rules
Understanding a few processing rules prevents most confusion:
- A crawler follows only the single most specific group that matches its name. Rules written for other crawlers do not stack on top.
- If the same crawler name appears in two groups, the groups are merged.
- Paths are case-sensitive, so
/Photos/and/photos/are different. - Most directives accept the
*wildcard to match a pattern, such as every URL that contains a parameter. - Lines that are not a supported directive are ignored.
Common Robots.txt Mistakes to Avoid
These are the errors that most often cause trouble for beginners:
- Blocking the whole site. A line reading
Disallow: /underUser-agent: *tells every crawler to stay out of everything. It is sometimes left over from a development site and forgotten at launch. - Using it to hide pages. As covered above, use noindex for that. A page blocked in robots.txt cannot even be crawled, which means Google cannot see a noindex tag on it.
- Blocking CSS and JavaScript. Search engines need to load these files to render your pages properly. Blocking them can make pages look broken to the crawler.
- Typos and wrong location. A file named
Robots.txtin a subfolder will not work. - Forgetting the file after a redesign or domain move. Always re-check it, and make sure the Sitemap line points to the current domain.
If pages are missing from search, a robots.txt block is one of several possible causes. Our checklist on why a website is not showing on Google covers the others.
Robots.txt vs Noindex vs Sitemap
These three get mixed up constantly, yet each does a different job.
| Tool | What it controls | Where it lives |
|---|---|---|
| Robots.txt | Which URLs crawlers may fetch | Root of the site |
| Noindex | Whether a page may appear in results | Inside the page itself |
| XML sitemap | Which URLs you want discovered | A separate file you reference |
Combining crawling and indexing rules can create conflicts, so decide what you actually want for each page first. Crawl it and index it, crawl it but do not index it, or keep crawlers away entirely.
How to Test Your Robots.txt File
After you edit the file, check it before you walk away:
- Open
yoursite.com/robots.txtin a browser and confirm it loads as plain text. - Read it line by line and confirm no important section is blocked.
- Check Google Search Console for crawl or indexing warnings over the following days. If you are new to it, start with our Search Console beginner's guide.
Google finds and uses a changed file automatically, though it may take a little time for its cached copy to refresh.
Frequently Asked Questions
Do I need a robots.txt file?
No. Without one, crawlers assume everything is allowed. A file becomes useful when you want to block certain sections or point to your sitemap.
Can robots.txt remove a page from Google?
No. It only controls crawling. To keep a page out of results, use a noindex directive or password protection.
Is robots.txt a security tool?
No. Reputable crawlers obey it, but others may not, and anyone can read the file. Never rely on it to protect private content.
Where do I find my robots.txt file?
Add /robots.txt to the end of your domain. If you see a 404 page, you do not have one.
Does robots.txt affect rankings?
Indirectly. It does not boost rankings itself, but a mistaken block can stop important pages from being crawled and found.
The Bottom Line
Keep your robots.txt short, test it after every change, and use it for crawl management only. For anything that must stay out of search results, use noindex or a password instead. If you would like a second pair of eyes on your site's technical setup, you are welcome to get in touch.