Why It Matters
Without a robots.txt file, search engine bots will attempt to crawl everything on your site, which may include pages you do not want indexed. This can waste crawl budget on unimportant pages, expose duplicate content that confuses search engines, or leak internal tools into search results. A well-configured robots.txt file keeps bots focused on the pages that matter for your visibility. It is also a core part of technical SEO, and getting it wrong has real consequences: a single incorrect line can accidentally block your entire site from appearing in Google. It is a small file with outsized consequences, so it needs to be set up correctly and reviewed periodically.
Example
A business discovers that Google has indexed their WordPress login page, their internal site search results, and several draft landing pages. None of these should be visible in search results. They update their robots.txt file to disallow bots from crawling the login directory, the internal search path, and the draft folder. Within a few weeks, those pages disappear from search results, and Google’s crawlers focus their time on the real content pages instead.
For more technical terms explained in plain English, see the glossary.
