Skip to content
Gridlode
Glossary

What is robots.txt?

A file at the root of a site telling crawlers which paths they may fetch.

robots.txt (RFC 9309) lists rules per user agent, such as GPTBot or Googlebot. It is a request, not access control, but reputable crawlers and data vendors honor it. Gridlode's URL engines respect it by default.

Related terms

  • Certificate Transparency: Public, append-only logs of every certificate issued by public authorities.
  • llms.txt: A Markdown file at /llms.txt that gives AI assistants a curated map of a site.
  • XML sitemap: A file listing a site's URLs so search engines can discover them.