Understanding How Rate Limiters Work Internally

The Basic Problem

Rate limiting is one of those things every backend engineer eventually has to build or configure. At its core, you are just deciding if a specific user (or IP address) is allowed to make another request right now. If they are, you let it pass. If not, you return a 429 Too Many Requests response.

The Token Bucket Algorithm

The most common way to do this is the Token Bucket algorithm. Imagine a bucket that holds a maximum number of tokens. Every time a request comes in, you check if the bucket has at least one token. If it does, you remove a token and let the request proceed. If the bucket is empty, you reject the request.

Tokens are added to the bucket at a fixed rate over time. This allows for bursts of traffic while enforcing a long-term average rate.

Implementation Example

In a real system, you usually store this state in Redis because it is fast and supports atomic operations. Here is a simplified version of what that logic looks like in Node.js:

1
async function isAllowed(userId) {
2
  const now = Date.now();
3
  const bucketKey = `rate_limit:${userId}`;
4

5
  const [tokens, lastRefill] = await redis.hmget(bucketKey, 'tokens', 'lastRefill');
6

7
  const refillRate = 10; // tokens per second
8
  const maxTokens = 50;
9
  const timePassed = (now - lastRefill) / 1000;
10
  const newTokens = Math.min(maxTokens, tokens + (timePassed * refillRate));
11

12
  if (newTokens >= 1) {
13
    await redis.hmset(bucketKey, {
14
      tokens: newTokens - 1,
15
      lastRefill: now
16
    });
17
    return true;
18
  }
19

20
  return false;
21
}

Sliding Window Logs

While Token Bucket is great for general traffic, sometimes you need more precision. A sliding window log keeps track of the actual timestamps of every request a user makes. If the number of requests in the last 60 seconds exceeds your limit, you block the user.

This is more accurate than the token bucket, but it uses more memory because you have to store a list of timestamps for every user. For high-scale systems, this often becomes a performance bottleneck.

Fixed Window Counters

The simplest approach is just counting requests in fixed blocks of time, like per minute. You increment a counter for each request and reset it at the start of the next minute.

The downside here is the edge case at the boundary of the window. A user could technically send their full quota at the very end of one minute and the very beginning of the next, effectively doubling their allowed rate in a short window. Most systems settle for Token Bucket or Sliding Window to avoid this.

Why it Matters

Rate limiting is not just about stopping bad actors. It is about protecting your database from being hammered by a buggy frontend loop or a scraping bot. It provides predictability to your system design. When you have a hard limit on requests, you can calculate the maximum load your services might face, which helps with resource allocation and prevents cascading failures.

Start simple. If you are just getting started, a basic counter in your local memory is often enough. As you scale, move that state to Redis or a dedicated sidecar proxy like Envoy to keep your application code clean.