Design a Rate Limiter: System Design Interview Walkthrough

The rate limiter is one of the most asked system design questions because it's small enough to finish in 45 minutes yet touches algorithms, distributed state and failure handling. For a broader framework, see the system design cheat sheet.

1. Clarify requirements

• Limit by user, IP, API key or endpoint?
• Hard (reject) or soft (throttle/queue)?
• Single server or distributed?
• Scale: requests per second, number of distinct clients?
• Should rules be configurable (e.g., 100/min for free, 1000/min for paid)?

2. The algorithms

Algorithm	How it works	Pros	Cons
Fixed window	Counter per window, reset each period	Simple, cheap	Up to 2x burst at window boundaries
Sliding window log	Store timestamps, count those in the last window	Accurate	Memory heavy
Sliding window counter	Weighted blend of current and previous window counts	Good accuracy, low memory	Approximate
Token bucket	Bucket refills at a fixed rate; each request takes a token	Allows bursts, simple	Needs tuning of size and rate
Leaky bucket	Queue drained at a constant rate	Smooth output	Bursts are delayed or dropped

Pick token bucket for most APIs and justify it.

3. High-level design

Client -> API Gateway / Middleware -> Rate Limiter -> Service
                                          |
                                      Redis (counters)
                                          |
                                     Rules config

• Key: ratelimit:{client_id}:{endpoint}
• On each request, load rule, update the counter, allow or reject.
• Rejected requests return 429 with Retry-After and optionally X-RateLimit-Remaining.

4. Making it correct when distributed

The classic bug is a race: two servers read the counter, both see room, both increment. Fixes:

• Atomic Redis operations: INCR with EXPIRE, or a Lua script that reads and updates in one step.
• Sharding by key so a client's counter lives on one Redis node.
• Local + global: servers keep local counters and sync periodically, trading accuracy for fewer calls.

5. Failure and edge cases

• Redis down: fail open (allow traffic, risk overload) or fail closed (block, risk outage). Most public APIs fail open with alerts.
• Clock skew: use Redis time, not server time.
• Hot keys: a single abusive client concentrating load on one shard.
• Race on rule changes, IPv6 and NAT (many users behind one IP).

6. Follow-ups interviewers ask

• How would you support different limits per plan?
• How do you rate limit by both user and IP?
• How would you avoid the limiter becoming a bottleneck?
• How do you test it under load?

Practise with the other questions in 12 common system design interview questions.

Worked example: token bucket with Redis

A token bucket stores two values per key: the current token count and the last refill time. On each request:

1. Read both values.
2. Compute tokens to add: elapsed time multiplied by the refill rate, capped at the bucket size.
3. If at least one token is available, subtract one and allow the request. Otherwise reject.
4. Write the new count and timestamp.

Steps 1 to 4 must be one atomic operation, so implement them as a Lua script executed by Redis. Set an expiry on the key slightly longer than the time to refill a full bucket, so idle clients do not accumulate keys.

Sizing the numbers

Say you serve 10 million daily active users, each with a bucket entry of about 100 bytes: that is roughly 1 GB of state, easily held by one Redis node, and sharded across a few for throughput and failover. The load is the request rate, not the storage. At 50,000 requests per second, one Lua call per request is within what a small Redis cluster handles, but add a local in-memory pre-check if you want to cut calls.

Choosing limits

Dimension	Example	Notes
Per user	100 requests per minute	Fair use
Per IP	1,000 per minute	Catches unauthenticated abuse; beware shared NAT
Per endpoint	5 per minute for login	Protects sensitive routes
Per plan	Free 60, Paid 600	Config-driven from a rules store
Global	Total backend capacity	Protects against surges

What to return to the client

• 429 Too Many Requests
• Retry-After with seconds until the next token
• Optional headers: remaining tokens and reset time, so well-behaved clients can back off

Where the limiter sits

• API gateway: one place for all services; the common answer
• Service middleware: finer control, duplicated logic
• Client SDK hints: cooperative only, never trust alone

Failure modes to discuss

Failure	Choice
Redis unavailable	Fail open for public APIs (with alerts), fail closed for sensitive ones like login
Network partition	Local fallback counters with generous limits
Clock drift	Use the store's clock
Sudden hot key	Replicate reads or shard a user's counters by sub-key

Interview tips

Lead with clarifying questions, state the algorithm and why, then draw the architecture. Mention that you would test with a load generator that simulates bursts at window boundaries, which exposes fixed-window flaws.

Related reading

• Design a URL shortener
• Design a notification system
• Low-level design and machine coding guide