Specialty · Cloud and SRELesson 5 of 5Free to read

SLOs and error budgets · Cloud and Reliability

How reliable should a service be, and how do you know when it's not?

Why you need it

"Never down" is impossible and chasing it freezes a team. Site reliability engineering (SRE) replaces it with a number everyone agrees on, the SLO, and turns the difference from 100% into an error budget: room for releases and experiments. When the budget burns too fast, you get paged; when it's healthy, you ship.

The intuition

SLI (indicator): what you measure, as good events ÷ all events. Example: requests answered successfully in under 300 ms ÷ all requests.
SLO (objective): the target for the SLI over a window. Example: 99.9% over 30 days.
Error budget: 1 − SLO. 99.9% over 30 days allows 0.1% bad: 30 × 24 × 60 × 0.001 = 43.2 minutes of full downtime, or the same share of failed requests.

Burn rate says how fast you're spending it: error rate ÷ (1 − SLO). A burn rate of 1 spends exactly the budget over the whole window; 10 spends it in a tenth of the time.

The Google SRE Workbook suggests paging someone when 2% of a 30-day budget burns in one hour, which is a burn rate of 0.02 × 720 h ÷ 1 h = 14.4. Slower burns become tickets for working hours instead.

Each extra 9 makes the budget ten times smaller: 99.99% leaves about 4.3 minutes a month.

Definition

SLI

Service level indicator: the share of good events, such as successful fast requests ÷ all requests.

SLO

Service level objective: the target for an SLI over a time window, like 99.9% over 30 days.

Error budget

1 − SLO: how much failure is acceptable in the window.

Burn rate

Error rate ÷ (1 − SLO). Above 1, the budget runs out before the window ends.

Examples

Budgets in minutes

Try it yourself

How much downtime each SLO allows in 30 days.

minutes = 30 * 24 * 60
for slo in [0.99, 0.999, 0.9999]:
    budget = (1 - slo) * minutes
    print(f"{slo:.2%}: {budget:.1f} minutes")

Output

99.00%: 432.0 minutes
99.90%: 43.2 minutes
99.99%: 4.3 minutes

Step by step

L143,200 minutes in 30 days.
L3The budget is the share you're allowed to miss.
L4Each extra 9 cuts the budget by ten.

Burn rate

Try it yourself

Compare the current error rate with what the SLO allows.

def burn_rate(errors, requests, slo):
    return (errors / requests) / (1 - slo)

print(round(burn_rate(50, 100_000, 0.999), 2))
print(round(burn_rate(1_500, 100_000, 0.999), 2))

Output

0.5
15.0

Step by step

L2Error rate divided by the allowed error rate.
L40.05% errors against 0.1% allowed: half speed, fine.0.5
L51.5% errors: 15 times too fast. Above 14.4: page someone.15.0

Common mistakes

Aiming for 100%

Users' own networks fail more often than that, and every change risks it. An SLO a little above what users need leaves room to improve the product.

Measuring the wrong thing

CPU and memory aren't what users feel. Measure from the user's side: successful, fast responses.

Alerting on every error

Pages for things that don't threaten the SLO teach people to ignore pages. Alert on fast burn rates.

Quick quiz

1. An SLO is 99.5% over 30 days. How much full downtime does the error budget allow?

A216 minutes
B43.2 minutes
C5 minutes

Show the answer

Answer: A

0.005 × 43,200 minutes = 216 minutes, a bit over 3.5 hours.

2. What does this print?

1
2
3

slo = 0.999
errors, requests = 300, 100_000
print(round((errors / requests) / (1 - slo), 1))

A
```
3.0
```
B
```
0.3
```
C
```
30.0
```

Show the answer

Answer: A

0.3% errors ÷ 0.1% allowed = 3: the budget will be gone in a third of the window.

Practice

3 exercises in three steps: predict what a program prints, change a program to meet a new requirement, then write one from scratch. Your code runs in the browser and is checked the moment you submit.

PredictA month of requests
Type exactly what this prints.
ModifyFix the budget
budget_minutes(slo, days) should return how many minutes of full downtime the SLO allows over days days, rounded to 1 decimal. It returns huge numbers instead. Fix it.
WriteShould we page?
Write alert(errors, requests, slo) that returns:
- "page" if the burn rate is 14.4 or more
- "ticket" if it's 1 or more (but under 14.4)
- "ok" otherwise, including when there were no requests
Burn rate = (errors / requests) / (1 - slo). Round the burn rate to 6 decimals before comparing, to avoid tiny float errors.

Practise this lesson

Open the interactive lesson to edit and run these examples and solve the exercises with instant feedback. Free to start, no sign-up.

Open the interactive lesson Course map

Interactive lessons open in order. If you have not done the earlier ones yet, you start at the course map.

Terms in this lesson

Term	Chinese	Definition
SRE	站点可靠性工程	Site reliability engineering: running services with software and agreed reliability targets.
SLO	服务水平目标	The reliability target, such as 99.9% over 30 days.
Error budget	错误预算	1 − SLO: the failure you can afford.
Burn rate	消耗速率	How fast the error budget is being spent.