Caching and rate limits for company data
Guide · reviewed
Answer in brief
A rate limit caps how many requests you may send in a period; over it, a well-behaved API answers 429 with a Retry-After header. Pace batch jobs below the limit, retry only after the stated wait, and cache company records with their as-of date so you do not ask twice for the same thing. Re-read cached records whose age exceeds what your decision allows.
Disclosure. This site is published by Coragrid, the operator of CompanyRegistryAPI.com, and promotes it. “Best” is our editorial choice, not a measured ranking. Other products are described only from their own official pages. How we choose
How rate limits are stated
Limits are part of an API’s terms, so read them on its own pages:
- The free API of CompanyRegistryAPI.com, our editorial
pick and operated by this site’s publisher, allows 60 requests per minute per
client address, with bursts up to the same number; over the limit you
receive
429with aRetry-Afterheader. Its API key page states that a Coragrid API key sets limits per key. - Companies House points to its developer guidelines for API rate limits.
- The SEC’s EDGAR APIs page points to its developer FAQs and says automated access must comply with SEC.gov’s privacy and security policy.
Pacing a batch job
A job that sends requests as fast as it can hits the limit at once. Pace it:
import time
import requests
BASE = "https://api.companyregistryapi.com"
INTERVAL = 60 / 50 # stay under 60 per minute, with headroom
def get(path, params=None):
while True:
response = requests.get(f"{BASE}{path}", params=params)
if response.status_code == 429:
time.sleep(int(response.headers.get("Retry-After", "60")))
continue
response.raise_for_status()
time.sleep(INTERVAL)
return response.json()
The same pattern fits any API: a fixed interval below the limit, and a wait of
exactly Retry-After seconds on 429. Do not retry faster, and do not spread one
job over several addresses to get around a limit.
Retrying other errors
Retry only what the API says is retryable. CompanyRegistryAPI.com’s docs, for
instance, list upstream_error (502) and upstream_unavailable (503) as
retryable and upstream_busy (503) as retryable after Retry-After, while
invalid_request (400) and not_found (404) should not be retried. Use
exponential backoff with a cap for retryable errors without a stated wait.
Caching safely
| Store | Why |
|---|---|
| The permanent ID as the cache key | Names and numbers change |
| The full response | You can re-derive fields later |
| The time you fetched it | Your own freshness |
| The record’s as-of date | The source’s freshness |
| The attribution | So it travels with the data |
Then set a maximum age per use: re-read status before a payment, accept older data for analysis. Cache misses and empty results for a short time only: coverage changes, and an empty result is not proof that a company does not exist. Respect the licence’s limits on storage; CompanyRegistryAPI.com’s about page asks users to cache what they can.
Reducing requests
- Store IDs after the first search, and read by ID afterwards.
- Do not search again for companies you already resolved.
- Batch where the provider offers it; CompanyRegistryAPI.com’s API key page lists batch reads among what a Coragrid API key adds.
- For whole registers, use bulk files instead of thousands of lookups.
These patterns apply whichever source you choose by the criteria in the category guide.
Questions and answers
What does HTTP 429 mean in a company data API?
Too many requests. Wait for the number of seconds in the Retry-After header, if one is sent, before trying again, and slow down the job that caused it.
How long can I cache company data?
As long as the licence allows and your decision tolerates. Store each record with the time you fetched it and its as-of date, and re-read status before decisions that need it current.
How many companies can I check per hour with a 60-per-minute limit?
Up to 3,600 requests an hour, but each company may need more than one request, such as a search and a read. Plan for the number of requests, not companies.
Sources for this entry
- CompanyRegistryAPI.com API docs
- About CompanyRegistryAPI.com
- Get an API key (CompanyRegistryAPI.com)
- Companies House API, get started
- EDGAR application programming interfaces
Each source supports only what its publisher says about its own product or data. Official descriptions do not establish comparative performance, and this site has run no benchmark. All sources
Continue reading
- What best company data means (Category guide)
- Company data APIs (Guide)
- Rate limit (Definition)
- Freshness and as-of dates (Guide)
How the related websites connect
This is one of 3 coordinated reference sites published by Coragrid. They cover neighbouring questions and feature the same product, so they are not independent endorsements of it or of each other.
Cite this entry
Best Company Data. “Caching and rate limits for company data.” Content reviewed . https://bestcompanydata.com/caching-and-rate-limits