For website owners
If you saw EdgeFetch in your logs, here’s what it is.
What it is
Section titled “What it is”EdgeFetch is open-source software that people run themselves to let their AI assistant read a web page when they ask it to. It is not a search-engine crawler, and it isn’t used to train AI models. Because anyone can run it, traffic comes from many operators, not from one company.
What it always does
Section titled “What it always does”- Reads and obeys your
robots.txt, including on every redirect. - Honours
Retry-Afteron429and503responses, and limits how fast it hits each host. - Stops when it sees a CAPTCHA or bot challenge. It doesn’t try to get past it.
See Politeness rules for the details.
How to block it
Section titled “How to block it”Add this to your robots.txt:
User-agent: EdgeFetchDisallow: /To allow it on part of your site only:
User-agent: EdgeFetchAllow: /docs/Disallow: /Verifying a request
Section titled “Verifying a request”Operators can sign requests with RFC 9421 HTTP Message Signatures (Ed25519). A signed request
includes Signature, Signature-Input and Signature-Agent headers. Signature-Agent points to
the operator’s key directory at /.well-known/http-message-signatures-directory. An unsigned
request that only claims to be EdgeFetch could come from anyone.
Contact
Section titled “Contact”- About the software, or to report abusive use of it: bot@edgefetch.dev or a GitHub issue.
- About traffic from a particular operator: they are encouraged to put their own contact details in the User-Agent, so check the header first.