What it is
Robots.txt is a small text file at the root of your domain that tells search engine crawlers where they may and may not go. It is the first file Google reads on a visit. Used well, it keeps crawlers out of cart, checkout, and account pages and points them to your sitemap. One stray line, a leftover Disallow from a staging site, can also tell Google to skip pages you very much want indexed.
How common it is
Just over half of audited stores (57%) have a robots.txt that helps rather than harms. The audit checks that the file exists, that it lets the Google crawlers through, that it names your sitemap, and that it blocks private paths like cart and login while leaving products and categories open. A file that exists but blocks the wrong thing fails here, and that is the dangerous case.
Why it costs you
This is the rare SEO mistake that can switch off a whole section of traffic at once. A single “Disallow: /” left over from development tells every crawler to ignore the entire site, and search traffic falls over the following weeks while everything on the store looks completely normal. Because the file is plain text and rarely reviewed, the error can sit there for months before anyone connects the lost traffic back to it.
Check it in 30 seconds
Open yourstore.com/robots.txt and read it. A “Disallow: /” with nothing after the slash is blocking your whole site, and any Disallow line covering your product or category URLs may be hiding those pages. For the full picture, paste the file into the robots.txt report in Google Search Console, which shows what it would block.
Read next: Ecommerce SEO: Turn Search Visibility Into Revenue
Run the free audit to confirm this one file is not keeping pages out of search.