robots.txt is a text file websites publish at the top level of each host (e.g., example.com/robots.txt) that specifies which pages or sections search engine crawlers and other bots are allowed or disallowed from accessing. It's a voluntary standard; well-behaved crawlers respect it, but it has no technical enforcement mechanism of its own.

robots.txt is commonly checked by ethical scraping and crawling operations as a first step, as a courtesy to the site owner. It only controls crawler access to pages, though, it says nothing about the legality of scraping data that's already publicly accessible, which is a separate and more complex question.

A robots.txt file can set different rules for different crawlers by name, so a site might allow one search engine's crawler broad access while blocking others, or block AI crawlers specifically, entirely.

Respecting robots.txt is a matter of convention rather than enforcement, and it's one of the baseline checks that separates ethical web scraping from scraping that ignores a site's stated access rules.