What a robots.txt line means

A robots.txt file is a short set of instructions at the root of a host. Each line is either a comment, a field, or empty. The robots.txt checker reads the text you paste. It does not fetch a live site.

User-agent names the crawler the following rules apply to. An asterisk means “any crawler that honors the file.” A later User-agent block starts a new group.

Disallow names a path prefix that the named crawler is asked not to fetch. An empty Disallow means “no path is disallowed in this group.” Allow names a prefix that may be fetched even when a broader Disallow exists. Crawlers differ in how they resolve conflicts; this page does not simulate every bot.

Sitemap points to a sitemap URL. It is a hint, not a command to index every URL in that file.

Misspellings matter because a crawler that does not recognize the field will ignore the line. Dissallow is a common example. The checker underlines that kind of typo. A file with no typos can still be the wrong policy for the site; syntax is not an audit of the server.