Why Every Website Needs a robots.txt File

Your Website's Traffic Cop

Every time you search for something on Google, Bing, or any search engine, a small program called a web crawler (also called a bot or spider) visits websites to read their pages and add them to the search engine's index. Think of it like a librarian walking through a library, reading every book cover to know what's inside.

A robots.txt file is a tiny text file that sits in your website's root folder (the main folder, like public_html or www). Its job is to give instructions to those crawlers: "You can look at these pages, but stay out of these ones."

It's not a security tool — a robots.txt file doesn't actually block hackers or prevent people from visiting pages. It's more like a polite sign on a door that says "Staff Only." Most well-behaved crawlers will follow it. Bad actors won't.

Without It, You're Leaving Things to Chance

If you don't have a robots.txt file, every crawler that visits your site gets to read everything — including pages you probably don't want showing up in search results. This can waste your "crawl budget" (the limited number of pages a search engine will index from your site each visit), and it can accidentally expose private or unfinished pages.

For example, you might have:

• Admin pages meant only for you to log in and manage the site
• Staging or dev versions of your site that aren't ready for the public
• Internal search results pages that look messy in search results
• Duplicate pages that dilute your SEO by splitting traffic across multiple versions of the same content

The Two Commands That Do Most of the Work

A robots.txt file is just plain text. You don't need any special software — just a text editor. Here are the two most important commands:

• User-agent: — Who the rule applies to. Use * for "all crawlers."
• Disallow: — Which pages or folders the crawler should not visit.

You can also use Allow: to explicitly permit access to something inside a blocked folder.

# Block all crawlers from these folders
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /tmp/

# Allow crawlers to see the rest of the site
Allow: /

The # symbol creates a comment — notes for humans that crawlers ignore. Key rules to remember:

• File location must be: yoursite.com/robots.txt — root of the domain, not a subfolder
• Disallow: with just / blocks everything — use with caution!
• You can target specific crawlers: User-agent: Googlebot targets only Google's crawler

A Real robots.txt for a Small Business Site

Imagine you run a small business website with a public blog and a hidden admin area. Here's a robots.txt that protects the admin folder while letting search engines index the rest:

# robots.txt for YourBusiness.com
# Last updated: July 2026

# Block all crawlers from admin and private folders
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /cgi-bin/
Disallow: /private/
Disallow: /checkout/

# Let Google crawl the shop even though others are blocked
User-agent: Googlebot
Allow: /shop/

# Block everyone else from the shop too
User-agent: *
Disallow: /shop/

Notice User-agent blocks can appear more than once. The more specific one (targeting Googlebot) overrides the general one for that crawler. You can set different rules for different search engines this way.

Knowledge Check

Test what you learned with this quick quiz.