Python: Regular Expressions

What Are Regular Expressions?

A regular expression (regex) is a sequence of characters that defines a search pattern. You specify what text you are looking for in abstract terms, and the regex engine finds all text that fits that description. Instead of writing code that checks whether a string starts with a digit, contains exactly ten characters, and ends with a letter, you write a single pattern that expresses all of that at once.

Regex shows up in almost every domain of software development: validating user input, extracting structured data from logs, scraping text from web pages, replacing substrings while respecting their structure, and tokenising source code. The syntax is almost the same across programming languages, so learning it in Python transfers directly to JavaScript, Java, Rust, or any shell tool.

The re Module

Python's standard library includes the re module, which provides all regex functionality. You import it once and then call its functions directly or compile patterns into reusable objects. The full reference lives atdocs.python.org/3/library/re.html, but this tutorial covers every function and feature you will encounter in practice.

Raw Strings in Regex

Regex patterns make heavy use of backslashes: \d means "a digit",\s means "whitespace", \b means "a word boundary". The problem is that Python's string parser also treats backslashes as escape sequences. Writing"\d" gives you just d, because \d is not a recognised Python escape and gets silently reduced. To pass a literal backslash to the regex engine, you would need "\\d".

Raw strings solve this cleanly. Prefix a string with r and Python passes every character literally, backslashes included. r"\d" is exactly two characters: a backslash and a d, which is what the regex engine expects. The convention in Python is to always write regex patterns as raw strings.

Basic Patterns and Special Characters

Regex patterns are built from two kinds of characters: literals (which match themselves) and metacharacters (which have special meaning). Learning the metacharacters is the bulk of learning regex.

re.match(), re.search(), re.findall(), re.finditer()

The four most commonly used functions each have a distinct purpose. Choosing the right one avoids a category of subtle bugs.

re.match() checks for a match at the very start of the string. If the pattern does not match at position 0, it returns None, regardless of whether it would match somewhere else. re.search() scans the entire string and returns the first match it finds anywhere. re.findall() returns all non-overlapping matches as a flat list of strings. re.finditer() does the same but returns an iterator of match objects, which lets you access position information for each match rather than just the matched text.

re.sub() and re.subn()

re.sub(pattern, replacement, string, count=0) replaces every match (or up tocount matches) with the replacement. The replacement can be a plain string or a callable. When it is a callable, Python calls it with each match object and uses the return value as the replacement text, which gives you the full power of Python code inside a substitution.

re.subn() works identically but returns a tuple of the new string and the number of substitutions made, which is useful when you want to know whether anything was replaced.

re.split()

re.split(pattern, string) splits a string wherever the pattern matches, which is far more flexible than str.split(). If the pattern contains a capturing group, the text matched by that group is included in the resulting list. This is useful when you want to split on a separator but also keep track of what the separator was.

Compiling Patterns with re.compile()

Every time you call re.search(pattern, text), Python compiles the pattern string into an internal finite automaton before running the search. If you use the same pattern thousands of times, that compilation cost adds up. re.compile()compiles the pattern once and returns a compiled pattern object. You then call the same methods (match, search, findall, etc.) directly on that object, paying the compilation cost only once.

Python does cache recently compiled patterns internally, so the practical difference is small for simple scripts. In tight loops or long-running applications, compiling explicitly is both more efficient and more readable.

Groups and Capturing Groups

Parentheses in a regex do two things: they group sub-patterns so quantifiers can apply to the whole group, and they capture the matched text so you can retrieve it later. Groups are numbered left to right by their opening parenthesis, starting at 1.match.group(0) or match.group() returns the entire match.match.group(1) returns the text captured by the first group, and so on.match.groups() returns a tuple of all captured strings.

Named Groups

Numbered groups work fine for simple patterns, but when a pattern has many groups, keeping track of which number refers to which field is error-prone. Named groups solve this: the syntax (?P<name>pattern) assigns a name to the group, and you retrieve it with match.group("name") or via the match.groupdict()dictionary. Named groups still have numbers as well, so both access methods work simultaneously.

Non-capturing Groups

Sometimes you need parentheses for grouping (to apply a quantifier to a sub-pattern, for example) but you do not want that group to be captured or to shift the numbering of other groups. The syntax (?:pattern) creates a non-capturing group. It behaves exactly like a regular group for matching purposes but does not add to the group count and is not accessible via group(n).

Lookahead and Lookbehind

Lookaheads and lookbehinds are zero-width assertions: they check for a condition at the current position without consuming any characters. The matched text they assert is not included in the overall match. This lets you write patterns that say "this word, only when it is followed by X" or "this number, only when it is preceded by a dollar sign" without including X or the dollar sign in the match.

Flags: Modifying Match Behaviour

Flags change how the pattern engine interprets the string being matched. You pass them as the flags argument to any re function, or tore.compile(). You can combine multiple flags with the bitwise OR operator. Alternatively, you can embed flags inside the pattern itself using the inline syntax(?i), (?m), etc., which is useful when working with compiled patterns stored as strings.

Greedy vs Lazy Matching

By default, quantifiers like *, +, ?, and {m,n} are greedy: they match as many characters as possible while still allowing the overall pattern to succeed. This can produce surprising results when your pattern sits inside repeating content such as HTML tags.

Adding a ? after any quantifier makes it lazy: it matches as few characters as possible and expands only if the rest of the pattern requires it. So *?, +?, and ?? are the lazy equivalents of *,+, and ?.

Common Regex Patterns

Rather than deriving these from scratch every time, the patterns below are reliable starting points for the most frequent validation tasks. Each one comes with a brief explanation of the choices made. In production code, always test edge cases specific to your requirements: email validation in particular has many valid formats that simple patterns miss.

Quiz - Test Your Knowledge

Ten questions on raw strings, function differences, quantifiers, groups, lookarounds, flags, and greedy versus lazy matching. Some questions hinge on details that are easy to overlook, so read carefully.