A robots.txt file is a plain text file sitting at the root of your website, at yourdomain.com/robots.txt, that tells search engine crawlers which parts of your site they’re welcome to access. Its main job is signalling crawl preferences to well behaved bots and cutting down unnecessary server load, not locking anything away from public view.
Want to check yours right now? Open a browser tab and add /robots.txt to the end of your domain. If nothing loads, you either don’t have one or it’s misplaced.
- A robots.txt file lives at your site’s root, never in a subfolder.
- It guides crawler behaviour; it doesn’t enforce security.
- Checking it takes ten seconds: type yourdomain.com/robots.txt into your browser.
Key Takeaways
A robots.txt file works by giving well behaved crawlers advisory instructions on which paths to skip, and it only delivers value when the syntax is precise and the file sits at the true site root.
| Point | Details |
|---|---|
| Definition and location | A plain text file at yourdomain.com/robots.txt that guides crawler behaviour. |
| It’s advisory, not enforceable | Reputable bots comply; malicious scrapers can ignore the file entirely. |
| Never list sensitive paths | Public visibility means listing /admin paths creates a roadmap for attackers. |
| Test before and after edits | Use Search Console, Bing Webmaster Tools, or curl to confirm real crawler behaviour. |
| Get professional support | CantyDigital audits robots.txt, sitemaps, and crawl budget together as part of ongoing SEO plans. |
Table of Contents
- How does robots.txt work?
- What are the common robots.txt directives and syntax?
- How do you create and upload a robots.txt file?
- Is robots.txt a security risk if used incorrectly?
- Does robots.txt affect crawl budget and indexing?
- How do you fix a “blocked by robots.txt” error?
- What do practical robots.txt examples look like?
- Is a robots.txt file necessary, and is it legal?
- Robots.txt best practices checklist
- Sources
How does robots.txt work?
Robots.txt runs on an honour system. The Robots Exclusion Protocol, formalised in 2022 as RFC 9309, is an advisory standard. Reputable crawlers like Googlebot and Bingbot check it before crawling, but nothing forces compliance. Malicious scrapers and some aggressive bots ignore it outright, which is a fact worth sitting with before you treat this file as a wall.
When a well behaved crawler visits your site, it fetches /robots.txt first, before touching any other page. It then parses the file into groups, each starting with a User-agent line followed by rules that apply to that specific bot. If your file targets Googlebot separately from all other crawlers, Google’s bot follows its own group and ignores the generic one.
Parsing has quirks that trip people up constantly:
- Rules are case-sensitive, so /Private/ and /private/ are treated as different paths.
- When multiple rules could apply to the same URL, the most specific match wins, not the first one listed in the file.
- If robots.txt is missing entirely, crawlers assume everything is open for business.
A robots.txt file with no rules under its last group effectively means “everything permitted.” Absence of restriction is the default, not an edge case.
Picture the process as four steps: fetch the file, parse it into groups, match the requested URL against the most specific applicable rule, then crawl whatever survives that filter. Sitemaps declared inside robots.txt don’t gate crawling; they just point bots toward the pages you actually want indexed, which is a separate concern from what you’re blocking.
What are the common robots.txt directives and syntax?
The file itself has strict requirements. It must be named robots.txt in lowercase, saved as plain text, served with a text/plain content type, and ideally encoded in UTF-8. Get any of that wrong and crawlers may misread or ignore the whole thing.
Five directives cover almost everything you’ll need:
- User-agent identifies which crawler the following rules apply to (use * for all bots).
- Disallow blocks a path from being crawled.
- Allow carves out an exception within a blocked path.
- Sitemap points crawlers to your XML sitemap location.
- Crawl-delay asks bots to space out requests, though Google ignores this directive entirely.
Wildcards make rules flexible. An asterisk (*) matches any sequence of characters, and a dollar sign ($) anchors a rule to the end of a URL.
User-agent: *
Disallow: /checkout*
Disallow: /search$
That second line blocks a page ending exactly in /search, but leaves /search-results untouched, because the $ anchor is specific.
Here’s a common pattern for blocking a folder while allowing one file inside it:
User-agent: *
Disallow: /private/
Allow: /private/press-kit.pdf
Sitemap: https://example.com/sitemap.xml
Pro Tip: When a Disallow and an Allow rule both match the same URL, the crawler applies whichever pattern is longer and more specific, not whichever line comes first in the file.
| Directive | What it does |
|---|---|
| User-agent | Targets a specific crawler or all crawlers |
| Disallow | Blocks a path from crawling |
| Allow | Creates an exception inside a blocked path |
| Sitemap | Points to your XML sitemap |
| Crawl-delay | Requests slower crawling (unsupported by Google) |
How do you create and upload a robots.txt file?
Building one takes minutes once you know the checklist.
- Open any plain text editor (Notepad, TextEdit in plain text mode, or a code editor).
- Write your directives using the syntax above.
- Save the file as robots.txt in lowercase, with UTF-8 encoding.
- Upload it to your site’s root directory, not a subfolder, so it’s reachable at yourdomain.com/robots.txt.
- Verify it loads by visiting that URL directly in a browser.
Platform quirks matter here. Static HTML sites and self-hosted WordPress installs give you direct root access via FTP or a hosting file manager. WordPress with certain SEO plugins generates a virtual robots.txt you edit through the plugin dashboard instead. Shopify and Wix historically restricted direct root file access, though Shopify now allows edits through a robots.txt.liquid template, worth checking against current platform documentation before assuming you can’t touch it, particularly if you’re running Shopify SEO alongside other technical fixes.
- Only one robots.txt file is recognised per host, protocol, and port.
- Subdomains each need their own file; blog.example.com and www.example.com are treated as separate hosts.
Pro Tip: If your CMS doesn’t give you root access, search for a robots.txt override setting before assuming you’re stuck with the default.
Is robots.txt a security risk if used incorrectly?
Robots.txt is a public file. Anyone, including attackers, can read it by visiting the URL directly, which means it was never designed as an access control mechanism.
Blocking a page in robots.txt doesn’t guarantee it disappears from search results either. If other sites link to that blocked URL, Google can still index the URL itself, just without a title or description snippet, because the crawler was never permitted to read the page content to generate one.
The real danger shows up when site owners list sensitive paths hoping to hide them. Writing Disallow: /admin-login or Disallow: /backup-files in a publicly readable file is effectively publishing a map of your weak points.
Listing a private directory in robots.txt to keep it “hidden” achieves the opposite. You’ve handed anyone who checks the file a shortlist of what you’re protecting.
Pro Tip: Use server-side authentication or a noindex meta tag for anything genuinely private. Keep robots.txt limited to crawl management, and never treat it as a lock.
Does robots.txt affect crawl budget and indexing?
Crawl budget is the number of pages a search engine is willing to crawl on your site within a given timeframe. On large sites, this budget is finite, and disallowing low value paths helps redirect that attention toward pages that actually matter for rankings.
Block the wrong thing, though, and you create a different problem. Disallowing folders containing CSS or JavaScript files can stop crawlers from rendering your pages properly, which affects how search engines understand layout and content, sometimes hurting the exact pages you wanted indexed.
Sensible candidates for blocking include staging environments accidentally left public, internal search result pages that generate infinite URL variations, and faceted navigation on ecommerce sites where filter combinations can produce thousands of near-duplicate URLs. Large archive-heavy sites often disallow predictable low-value patterns like calendar pages while still exposing their canonical content through sitemaps.
- Block: staging subdomains, internal search results, duplicate filter combinations.
- Don’t block: CSS, JavaScript, images needed to render your key pages.
- Always pair disallow rules with a sitemap directive so crawlers know where the good content lives.
Getting this balance right is one of the more technical parts of ongoing SEO work, and it’s easy to get backwards without realising until rankings slip.
How do you fix a “blocked by robots.txt” error?
Start with the basics before assuming something complex is wrong.
- Visit yourdomain.com/robots.txt directly and confirm the file loads and displays correctly.
- Check the file is served as plain text, not wrapped in HTML by your CMS.
- Test the specific blocked URL using Google Search Console’s URL Inspection tool or the robots.txt tester.
- Cross-check with Bing Webmaster Tools if Bing crawling is also affected.
- Fetch the raw file with curl or wget from the command line to rule out caching issues showing you an outdated version.
Most “blocked by robots.txt” errors trace back to one of a handful of causes: an overly broad Disallow rule written to block one page but catching a whole directory, a wildcard pattern matching more than intended, or a rule left over from a staging environment that never got removed before launch.
- Double check path syntax for typos or missing slashes.
- Confirm you’re testing the correct protocol and host (http versus https, www versus non-www).
- Remove accidental blanket Disallow rules.
- Resubmit your sitemap in Search Console once the fix is live.
What do practical robots.txt examples look like?
Four scenarios cover most situations you’ll face.
An empty file, or one with no Disallow rules, permits full crawling. This suits small brochure sites with nothing to hide and no crawl budget concerns:
User-agent: *
Allow: /
Blocking everything is riskier and should only apply to sites not yet ready for public discovery, like a pre-launch staging environment:
User-agent: *
Disallow: /
- Use this only on non-production environments, never your live domain.
- Remove it immediately before launch, because forgetting to delete this exact block is one of the most common self-inflicted SEO disasters agencies see.
A more typical setup blocks one folder while carving out an exception, and declares a sitemap:
User-agent: *
Disallow: /private/
Allow: /private/media-kit.html
Sitemap: https://example.com/sitemap.xml
Pro Tip: Keep separate User-agent groups only when you genuinely need different rules per bot. Duplicating identical rules across multiple groups just adds room for error.
Is a robots.txt file necessary, and is it legal?
Small brochure sites with few pages can often skip robots.txt entirely; crawlers will index everything by default anyway. Larger sites, ecommerce platforms, and anything with staging environments or faceted navigation genuinely need one to manage crawl behaviour.
On legality, robots.txt is a publicly readable file and a voluntary protocol, not an access control law. It carries no legal weight forcing bots to comply.
- Don’t rely on robots.txt to hide personal data or regulated information; that requires proper authentication.
- A missing robots.txt isn’t an error; it simply means “crawl everything” by default.
Robots.txt best practices checklist
Run through this before publishing any changes:
- File is named robots.txt, lowercase, saved in UTF-8.
- It sits at the true root of your domain, not a subfolder.
- Every blocked path has been tested individually, not assumed.
- No admin, login, or backup paths appear anywhere in the file.
- A Sitemap directive points to your current XML sitemap.
- You’ve checked Search Console for crawl errors after publishing.
Pro Tip: Test changes with a quick curl yourdomain.com/robots.txt from the command line before and after editing. It shows you exactly what a crawler sees, with no caching or rendering quirks in the way.
Practical notes from Canty Digital’s own SEO work
CantyDigital has spent years untangling robots.txt problems that clients didn’t know existed until rankings dropped. One recurring pattern: a staging site’s blanket Disallow rule migrating to production during a site relaunch, quietly blocking pages that had ranked for years. Another: large content archives crawling inefficiently until predictable low-value URL patterns, like tag pages generating duplicate content, got disallowed and sitemap coverage improved.
- Always check Search Console within days of any robots.txt edit, not weeks later.
- Favour a sitemap-first approach over broad Disallow rules wherever possible.
- Audit robots.txt every time you migrate platforms or relaunch a site.
Pro Tip: A quick spreadsheet-based crawl audit before and after any robots.txt change makes it easy to spot pages that dropped out of the index unexpectedly.
When does a robots.txt issue need professional eyes?
In my experience reviewing sites, most robots.txt problems are simple to fix once spotted, a stray rule, a wrong path, a leftover staging block. Where it gets genuinely tricky is large sites with faceted navigation, multiple subdomains, or a history of platform migrations, where one misplaced rule can quietly suppress thousands of pages for months. If that sounds like your situation, it’s worth having someone audit the whole crawl setup rather than guessing at individual fixes.
How CantyDigital handles technical SEO like this
Robots.txt errors rarely show up alone. They tend to travel with sitemap gaps, crawl budget waste, and indexing issues that a one-off fix won’t catch. CantyDigital runs full technical audits as part of ongoing SEO growth plans, covering robots.txt, Search Console configuration, and crawl health together rather than patching one file in isolation.
That matters because a robots.txt fix without ongoing monitoring just resets the clock until the next platform update or migration breaks something again. CantyDigital’s approach includes Search Console setup, crawl budget reviews, and root file support as standard parts of a monthly plan rather than a one-off consulting fee. If you’ve found an issue while reading this and want it looked at properly, check CantyDigital’s SEO FAQs for a straightforward next step, or reach out for a technical audit.
Sources
- robots.txt specification and guidance | Google
- RFC 9309 — The Robots Exclusion Protocol
- robots.txt configuration – Security – MDN Web Docs
- What Is Robots.txt? | How a Robots.txt File Works — Cloudflare Learning
