The Retry-After header is an HTTP response header that tells a client how long to wait before making a follow-up request. A server can include it in a 429 Too Many Requests response, the status code for rate limiting. For a scraper, honoring this value means pausing for as long as the server asked, instead of guessing and risking more blocked requests.

The header is optional, so a 429 response may arrive without it. RFC 6585 also does not define how a server identifies the user or counts requests. A limit may be tied to login credentials or a cookie, not only to an IP address.

Retry-After Value Formats

  • Delay in seconds: A non-negative whole number, such as Retry-After: 120.
  • HTTP date: A date after which to retry, such as Retry-After: Wed, 21 Oct 2015 07:28:00 GMT.
  • Where it appears: With 429 responses, with 503 Service Unavailable to show how long the outage should last, and with redirects such as 301 to set a minimum wait.

Other Rate Limit Headers

  • Vendor headers: Some APIs add their own fields. GitHub's REST API, for example, sends x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-used and x-ratelimit-reset.
  • Proposed standard: An IETF Internet-Draft defines RateLimit and RateLimit-Policy fields so servers can advertise quota policies and remaining quota. It is still a draft, not a published RFC.
  • Retry logic: GitHub advises waiting the number of seconds in retry-after when present, and increasing the wait exponentially if requests keep failing.

Who Honors Retry-After

  • Support varies: MDN notes that client and server support for Retry-After is still inconsistent.
  • Search crawlers: Some crawlers, such as Googlebot, honor it. Google says a site can send Retry-After with a 503 or 429 response to tell its crawler when to retry.
  • Keep it short: Google advises against serving 500, 503 or 429 to slow crawling for more than 1-2 days. If Googlebot sees these codes on the same URL for several days, it may drop that URL from its index.