Robots.txt is a plain text file placed at the root of a website (e.g. yoursite.com/robots.txt) that instructs search engine crawlers which pages or sections they may or may not crawl, per the specification documented on Google Search Central (opens in new tab). It does not remove pages from search results by itself — it only controls crawling access.
A Common Confusion Worth Clearing Up
Blocking a page in robots.txt stops a crawler from visiting it, but that’s not the same as removing it from search results — a blocked page that’s still linked to from elsewhere can technically still show up, just without a proper description. For genuinely hiding a page from search, a noindex tag on the page itself is the more reliable tool, with robots.txt reserved for controlling crawl access to things like admin areas.
A misconfigured robots.txt file can also accidentally block an entire website from being crawled, which is why it’s worth checking after any site migration or redesign.
People also ask
Does robots.txt hide pages from Google?
Not reliably — blocking a page in robots.txt only stops crawling, but the URL can still appear in search results without a description if it’s linked elsewhere. Use a noindex tag to actually prevent indexing.
See also
How TAP-ONE helps
A fast, secure site that brings in leads while you focus on running the business.
Explore Get a WebsiteMore in Website & Technical
301 Redirect
A permanent redirect that sends visitors and search engines from an old URL to a new one, passing along ranking value.
404 Error
The "page not found" error shown when a requested URL doesn’t exist on a website.
Alt Text
A short written description of an image, used by screen readers and search engines when the image can’t be seen.
Backlink
A link from another website pointing to your website, treated as a vote of trust by search engines.