Specialty · SecurityLesson 8 of 9Free to read

Finding attacks in logs · Cyber Security Basics

Someone is guessing passwords on our server. How do we find them in the logs?

Why you need it

Prevention fails sometimes, so defenders also need detection. Servers write a line for every login attempt, and a few lines of Python can turn thousands of log lines into a short list of suspicious addresses. This is the everyday work of a security analyst, and the same skill (parse, count, threshold) works for any log.

The intuition

A log line is a sentence with a fixed shape: time, result, user, address. A regular expression describes that shape once and pulls out the parts:

\S+ means "one or more characters that aren't spaces"
(...) captures a part so you can read it with m.group(1), m.group(2) and so on
(FAIL|OK) means "FAIL or OK"

Then count failures per IP address in a dictionary, and flag every address over a threshold. One more pattern matters a lot: many failures followed by a success from the same address may mean the attacker got in.

(The addresses here come from ranges reserved for documentation, such as 203.0.113.0/24, so they belong to nobody.)

Definition

Regular expression

A pattern that describes the shape of text. `re.match(pattern, line)` returns a match whose groups hold the captured parts, or None if the line doesn't fit.

Brute-force detection

Counting failed logins per source over a period and alerting when the count passes a threshold.

Examples

Pull fields out of a line

Try it yourself

One pattern describes every line. Groups 2 and 4 are the result and the IP.

import re
lines = [
    "2024-05-01 10:00:01 FAIL user=root ip=203.0.113.5",
    "2024-05-01 10:00:02 FAIL user=admin ip=203.0.113.5",
    "2024-05-01 10:00:04 FAIL user=amy ip=198.51.100.7",
    "2024-05-01 10:00:05 FAIL user=test ip=203.0.113.5",
    "2024-05-01 10:00:09 OK user=amy ip=198.51.100.7",
]
pattern = r"(\S+ \S+) (FAIL|OK) user=(\S+) ip=(\S+)"

for line in lines:
    m = re.match(pattern, line)
    print(m.group(2), m.group(3), m.group(4))

Step by step

L9Four groups: time, result, user, ip.
L12re.match checks the line against the pattern from the start.
L13group(0) would be the whole line; groups count from 1.FAIL root 203.0.113.5

Count failures per address

Try it yourself

counts.get(ip, 0) gives 0 the first time an address appears.

import re
lines = [
    "2024-05-01 10:00:01 FAIL user=root ip=203.0.113.5",
    "2024-05-01 10:00:02 FAIL user=admin ip=203.0.113.5",
    "2024-05-01 10:00:04 FAIL user=amy ip=198.51.100.7",
    "2024-05-01 10:00:05 FAIL user=test ip=203.0.113.5",
    "2024-05-01 10:00:09 OK user=amy ip=198.51.100.7",
]
pattern = r"(\S+ \S+) (FAIL|OK) user=(\S+) ip=(\S+)"

fails = {}
for line in lines:
    m = re.match(pattern, line)
    if m.group(2) == "FAIL":
        ip = m.group(4)
        fails[ip] = fails.get(ip, 0) + 1
for ip, n in fails.items():
    print(ip, n, "ALERT" if n >= 3 else "")

Step by step

L14Only failed attempts are counted.
L16Add one to this address's count.
L18Three failures reach the threshold of 3.203.0.113.5 3 ALERT
L18One failure, then a success: probably amy mistyping.

Common mistakes

Only looking at failures

Failures tell you someone is trying; a success right after many failures tells you they may have got in. Always check what each noisy address did next.

No time window

Ten failures in two seconds is an attack; ten failures over a month is a forgetful user. Real detection counts within a window, such as 5 minutes.

Quick quiz

1. What does this print?

import re
m = re.match(r"(\S+) user=(\S+)", "FAIL user=root extra")
print(m.group(1), m.group(2))
print(re.match(r"OK", "FAIL user=x"))

A
```
FAIL root
None
```
B
```
FAIL root extra
None
```
C
```
FAIL user=root
False
```

Show the answer

Answer: A

\S+ stops at the first space, so group 2 is just root. A line that doesn't fit gives None.

2. Which log pattern is the most worrying?

AOne address fails once a day
BOne address fails 40 times in a minute, then succeeds
CForty different users each log in successfully

Show the answer

Answer: B

Many fast failures are guessing; the success afterwards suggests a guess was right. Investigate that account first.

Practice

3 exercises in three steps: predict what a program prints, change a program to meet a new requirement, then write one from scratch. Your code runs in the browser and is checked the moment you submit.

PredictCount the users
Type exactly what this program prints.
ModifyAlert list
The program prints the failure count of every address. Change it so it prints only addresses with 3 or more failures, as ALERT <ip> <count>, with the biggest count first.
WriteDid they get in?
Write compromised(lines, limit). Each line looks like "<time> FAIL|OK user=<name> ip=<ip>", in time order. Return a sorted list of the IP addresses that had at least limit failures and after that a successful login (OK).

Practise this lesson

Open the interactive lesson to edit and run these examples and solve the exercises with instant feedback. Free to start, no sign-up.

Open the interactive lesson Course map

Interactive lessons open in order. If you have not done the earlier ones yet, you start at the course map.

Terms in this lesson

Term	Chinese	Definition
Log	日志	A record a program writes for each event, one line at a time.
Regular expression	正则表达式	A pattern describing the shape of text, used to match and extract parts.
Threshold	阈值	The value above which an alert is raised.
Indicator of compromise	入侵迹象	Evidence that an attacker may have got in, such as a success after many failures.