Specialty · System designLesson 5 of 5Free to read

Queues, retries and idempotency · System Design

What happens when requests arrive faster than you can handle them?

Why you need it

Work rarely arrives evenly. A queue lets a service absorb a burst and catch up later, and it lets slow jobs (emails, video processing) run in the background. But a queue that grows without limit just turns overload into long waits, and clients that retry carelessly make an outage worse. These patterns keep a busy system standing.

The intuition

Queues. Producers add work, workers take it off. Each second, the backlog changes by arrivals − served:

a short burst above capacity is absorbed and then drained
if arrivals stay above capacity, the backlog grows forever, and so does the wait (roughly backlog ÷ service rate)

So bound the queue: when it's full, reject new work quickly. That signal to slow down is called back-pressure.

Retries. Many failures are temporary, so retrying helps, but not all at once:

exponential backoff: wait base × 2^attempt, up to a cap (0.5 s, 1 s, 2 s, 4 s…)
jitter: wait a random part of that delay, so a thousand clients don't retry at the same instant

Idempotency. A retried "charge €30" must not charge twice. The client sends a unique idempotency key with the request; the server remembers keys it has processed and doesn't repeat the work.

Definition

Message queue

A buffer where producers put work and workers take it, so the two can run at different speeds.

Back-pressure

Pushing back on producers (for example by rejecting work) when consumers can't keep up.

Exponential backoff

Waiting twice as long after each failed attempt, up to a cap.

Idempotency key

A unique id sent with a request so the server can recognise a retry and not do the work twice.

Examples

A backlog builds and drains

Try it yourself

Workers handle 8 per second; a burst of 12 per second arrives.

arrivals = [5, 12, 12, 12, 3, 0, 0]
per_second = 8
backlog = 0
for second, n in enumerate(arrivals):
    backlog += n
    backlog -= min(backlog, per_second)
    print(second, backlog)

Output

Step by step

L5New work joins the queue.
L6Workers take up to 8, but not more than there is.
L74 extra each busy second piles up to 12, then drains once the burst ends.

Backoff with a cap

Try it yourself

The delay doubles each attempt until it reaches 8 seconds.

base, cap = 0.5, 8.0
for attempt in range(7):
    delay = min(cap, base * 2 ** attempt)
    print(attempt + 1, delay)

Output

Step by step

L30.5 × 2^attempt, but never more than the cap.
L4Real clients then sleep for random.uniform(0, delay): that's the jitter.

Idempotent payments

Try it yourself

The client retried after a timeout; the customer is charged once.

charged = {}

def pay(key, amount):
    if key in charged:
        return "already done"
    charged[key] = amount
    return "charged"

print(pay("order-17", 30))
print(pay("order-17", 30))
print(sum(charged.values()))

Output

charged
already done
30

Step by step

L4Seen this key before? It's a retry; don't charge again.
L10The retry gets an answer, but no second charge.already done
L11Only 30 was charged in total.30

Common mistakes

Unbounded queues

The queue hides the overload until memory runs out or every request waits minutes. Bound it and reject early.

Retrying immediately, in a loop

Every client hammering a struggling service keeps it down. Back off exponentially, add jitter and give up after a few attempts.

Retrying work that isn't idempotent

A timeout doesn't mean the first attempt failed; it may have succeeded. Without an idempotency key, a retry can charge or send twice.

Quick quiz

1. What does this print?

backlog = 0
for n in [10, 10, 10]:
    backlog += n
    backlog -= min(backlog, 6)
print(backlog)

A
```
12
```
B
```
0
```
C
```
18
```

Show the answer

Answer: A

4 left over each second: 4, 8, 12.

2. Why add random jitter to retry delays?

ASo many clients don't retry at exactly the same moment
BTo make each retry faster
CBecause servers reject evenly spaced requests

Show the answer

Answer: A

Without jitter, clients that failed together retry together and overload the service again.

3. A payment request timed out. What makes retrying it safe?

ASending the same idempotency key, so the server can spot a repeat
BWaiting exactly one second
CSending it to a different server

Show the answer

Answer: A

The first attempt may have gone through; the key lets the server answer without charging again.

Practice

3 exercises in three steps: predict what a program prints, change a program to meet a new requirement, then write one from scratch. Your code runs in the browser and is checked the moment you submit.

PredictA backoff schedule
Type exactly what this prints.
ModifyBound the queue
simulate ignores limit. After adding the arrivals each second, cap the backlog at limit (the extra requests are rejected), then let the workers serve.
WriteCharge once
Write process(requests). Each request is (idempotency_key, amount). Return a list with one entry per request: "charged" the first time a key is seen, "duplicate" for any later request with the same key.
```
process([("a", 10), ("b", 5), ("a", 10)])   # ['charged', 'charged', 'duplicate']
```

Practise this lesson

Open the interactive lesson to edit and run these examples and solve the exercises with instant feedback. Free to start, no sign-up.

Open the interactive lesson Course map

Interactive lessons open in order. If you have not done the earlier ones yet, you start at the course map.

Terms in this lesson

Term	Chinese	Definition
Message queue	消息队列	A buffer between producers and workers.
Back-pressure	背压	Slowing producers down when consumers can't keep up.
Exponential backoff	指数退避	Doubling the wait after each failure, up to a cap.
Idempotency	幂等性	Doing the same request twice has the same effect as once.