The file names a user agent and lists the paths it is allowed or disallowed to fetch. Since 2023 it is also where a site decides which AI crawlers it accepts: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended each read their own section. A site that wants to be cited by assistants allows their search crawlers here.
robots.txt blocks fetching, not indexing: a blocked page can still appear in results if other sites link to it, shown without a description. To keep a page out of the index, let it be crawled and add a noindex rule. Mistakes in this file are the fastest way to disappear from search, so every change deserves a test.