Rate Limiting and Throttling for APIs

Langdock currently imposes limits like 60,000 tokens per second, among others. On the surface, that sounds generous, but when you’re working against it, it quickly becomes restrictive. Processing files, handling multiple requests, rapid queries, lack of conversation history support from the API provider—there are plenty of reasons you can suddenly find yourself bumping into the ceiling. If you’re running an API service yourself, you need to keep it stable and can’t let an aggressive client starve everyone else out. What can you do about it?

API Rate Limiting and Throttling

Rate limiting and throttling protect APIs from overload, abuse, and unfair usage patterns by controlling the number of requests allowed within a given period.

Quick Overview

Rate limiting and throttling are mechanisms that control how many requests hit an API within a time window. Rate limiting sets a hard ceiling—once exceeded, requests are rejected, typically with a 429 Too Many Requests status code. Throttling, by contrast, slows down request processing instead of outright rejecting them. This shields backend systems during traffic spikes and ensures consistent response times. Both defend against brute-force attacks, scraping, unintended traffic bursts, and denial-of-service attempts. Implementations rely on algorithms like Fixed Window, Sliding Window, Token Bucket, or Leaky Bucket. Supporting measures include informative HTTP headers, Retry-After guidance, per-endpoint or per-user differentiation, and active monitoring.

Key Components

Rate Limiting

Rate limiting caps the number of requests a client can send to an API within a defined time window. Requests beyond the quota are rejected. Limits can apply globally, per API, per endpoint, per client, or per user account.

Throttling

Throttling reduces the rate at which requests are processed without immediately rejecting them. Use it when backend systems face temporary overload or you want to enforce steady-state processing. Throttling works through queuing, backoff strategies, or slowed handling.

Fixed Window

The Fixed Window algorithm counts requests across fixed time intervals—say, per minute or per hour. It’s straightforward to implement but can create traffic bursts at window boundaries.

Sliding Window

The Sliding Window algorithm uses a moving time window, avoiding bursts at boundaries. It’s fairer than Fixed Window but slightly more complex to code.

Token Bucket

The Token Bucket algorithm fills a bucket with tokens at a constant rate. Each request consumes one token. As long as tokens remain, requests go through. When empty, requests are rejected or delayed. This allows short bursts while keeping long-term throughput stable.

Leaky Bucket

The Leaky Bucket algorithm queues incoming requests and lets them out at a constant rate. It smooths traffic spikes effectively and prevents bursts, making it ideal for steady processing. The trade-off is potential wait times.

HTTP Status Code 429

When a client exceeds its rate limit, the server responds with 429 Too Many Requests. The Retry-After header tells the client when to retry. This enables fair backoff and reduces unnecessary load.

Limit Headers

Informative headers like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset help clients understand their current quota and plan requests accordingly. They improve the developer experience and reduce accidental overages.

Per-User and Per-Endpoint Limits

Global limits are simple but often too coarse. Finer-grained limits per authenticated user, per client, or per endpoint work better. Free tiers can have stricter limits while paying customers get higher quotas.

Load Shedding

Load shedding is a throttling tactic that actively drops requests during extreme load to keep the system alive. Prioritizing important operations and discarding less critical ones help maintain stability.

Distributed Rate Limiting

In distributed systems, rate limits must stay consistent across multiple instances. Achieve this with centralized stores like Redis or databases. Alternatively, use approximation techniques such as Sliding Window Log in Redis.

Monitoring and Alerting

Track how often limits trigger, which clients hit them, and whether spikes indicate attacks or failures. Alerts on unusual patterns enable quick response to abuse or capacity issues.

Practical Example

An API provider enforces rate limits using the Token Bucket algorithm. Each authenticated user gets 100 tokens per minute with a maximum burst of 20 tokens.

Request:

GET /api/v1/products
Authorization: Bearer USER_TOKEN

Successful response:

HTTP/1.1 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 87
X-RateLimit-Reset: 1751300000

Response when limit is exceeded:

HTTP/1.1 429 Too Many Requests
Retry-After: 45
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1751300000

{
  "type": "https://api.example.com/problems/rate-limit-exceeded",
  "title": "Rate Limit Exceeded",
  "status": 429,
  "detail": "You have exceeded the limit of 100 requests per minute."
}

The client uses the headers and Retry-After value to determine when the next request can be sent.

FAQ: Rate Limiting and Throttling

1. What is rate limiting?

Rate limiting restricts the number of requests a client can send to an API within a time period. Excess requests are typically rejected with a 429 Too Many Requests response.

2. What is throttling?

Throttling slows the rate of request processing to protect backend systems and maintain consistent response times. Requests are delayed or queued rather than immediately rejected.

3. What is the difference between rate limiting and throttling?

Rate limiting rejects requests when the quota is exceeded. Throttling slows processing without direct rejection. Both defend against overload.

4. What is the Token Bucket algorithm?

Token Bucket fills a bucket with tokens at a constant rate. Each request consumes one token. When the bucket empties, requests are rejected. The algorithm permits short bursts while maintaining stable long-term throughput.

5. What is the Leaky Bucket algorithm?

Leaky Bucket queues incoming requests and releases them at a constant rate. It smooths traffic spikes effectively but may introduce wait times.

6. What is Fixed Window?

Fixed Window counts requests within fixed time intervals. It is simple to implement but can allow traffic bursts at window boundaries.

7. What is Sliding Window?

Sliding Window uses a moving time window and is fairer than Fixed Window. It prevents boundary bursts but requires more complex logic.

8. What does HTTP 429 mean?

HTTP 429 Too Many Requests signals that the client has exceeded its rate limit. The server can use Retry-After to indicate when retry is allowed.

9. What is Retry-After?

Retry-After is an HTTP header that tells a client how long to wait before retrying a request. It is used with 429 and 503 responses.

10. What is load shedding?

Load shedding drops requests during extreme load to preserve system stability. Critical operations are prioritized while non-essential ones are rejected.

11. How do you implement rate limiting in distributed systems?

Use a centralized store like Redis to keep limits consistent across all instances. Algorithms such as Sliding Window Log or Token Bucket can be implemented in Redis.

12. What are per-user limits?

Per-user limits apply to individual authenticated users. They are fairer than global limits because they account for individual behavior and isolate abuse.

13. What is a burst?

A burst is a sudden spike of requests in a short window. Algorithms like Token Bucket intentionally allow bursts as long as the sustained rate stays bounded.

14. Should rate limits differ by endpoint?

Yes. Different endpoints have different costs. Write operations and data-heavy queries typically warrant stricter limits than simple read operations.

15. Why is monitoring important for rate limiting?

Monitoring reveals how often limits are triggered, which clients are affected, and whether spikes indicate attacks or failures. This enables optimization of limits and capacity.

Next in the API Learning Path

The next article in the API Learning Path covers API Monitoring and Observability 2026 — how to monitor your APIs using metrics, logs, and Distributed Tracing.

References

Recommended Books on API Development

If you’d like to explore Rate Limiting, API design, and security further, we recommend the following books:

API Development

Books about API design, REST, GraphQL, OpenAPI and API architecture

Designing Data-Intensive Applications von Martin Kleppmann

Bei Amazon ansehen

Affiliate-Link: Bei einem Kauf erhalten wir möglicherweise eine Provision.

API Design Patterns von JJ Geewax