DigiKitHow to Create a robots.txt File
robots.txt is a small text file that tells search engine bots which parts of a site not to crawl. Blocking a page there doesn't hide it from search entirely though, it just stops the crawl.
Summary
robots.txt is a small text file that tells search engine bots which parts of a site not to crawl. Blocking a page there doesn't hide it from search entirely though, it just stops the crawl.
Step by step
- Place the file at the site's root. Anywhere else, it won't work.
- Decide which folders shouldn't be crawled and add them with Disallow lines.
- Add the sitemap's location at the end of the file so bots can find it easily.
- If you actually want a page removed from search results entirely, robots.txt isn't enough. You need a noindex tag on the page itself.
Why it matters
Search engine crawlers don't spend unlimited time on your site. If they keep fetching thousands of low-value URLs, like filter combinations, internal search results, cart pages or session parameters, your real content can end up waiting in line. robots.txt is the simplest way to cut that noise. The flip side is that one wrong line can make the whole site uncrawlable, so even a tiny file deserves care every time you touch it.
What a basic file looks like
Each group of rules starts with a User-agent line. An asterisk means all bots. Under it you add Disallow lines for paths you don't want crawled, and Allow lines for exceptions if you need them. The last line is usually Sitemap, followed by the full URL of your sitemap. Paths are relative to the site root and case-sensitive, so /Admin/ and /admin/ are treated as different folders.
Common mistakes
- Shipping a leftover Disallow: / from a staging site to production. That single line blocks everything, and it can take days before anyone notices.
- Blocking CSS and JavaScript folders. If Google can't render the page the way a visitor sees it, it may judge the content as incomplete.
- Blocking a page in robots.txt and adding noindex to it at the same time. The crawler can't fetch the page, so it never sees the noindex tag.
- Treating robots.txt as security. The file is public. Listing the path to a hidden admin panel there is basically advertising it.
- Forgetting subdomains. blog.example.com needs its own robots.txt. The one on the main domain doesn't cover it.
A realistic scenario
An online store's product listings generate endless URLs through color, size and price filters. In Search Console, the number of pages crawled but not indexed keeps climbing. The fix: block the filter parameters and the /cart/ folder with Disallow, and leave category and product pages open. Before going live, the team checks the robots.txt report in Search Console to make sure the file is read correctly, then tests a handful of important product URLs one by one to confirm they are still crawlable.
After you make a change
Once the new file is live, open yourdomain.com/robots.txt in a browser and confirm you're seeing the updated version. A cache or CDN can keep serving the old one. Watch the crawl stats in Search Console for a few days. If important pages suddenly drop, this file is the first place to look.