Python Reg Expressions

Python Regular Expressions (re module)

Regular expressions (regex or RE) are powerful sequences of characters that define a search pattern. They are used for pattern matching within strings, enabling tasks like finding specific text formats (emails, URLs), validation, searching, and replacing substrings.

Python’s built-in re module provides comprehensive support for regular expressions, largely compatible with Perl’s syntax. If an error occurs during regex compilation or usage, the re module raises the re.error exception.

It’s highly recommended to use Python’s raw string notation (r"...") for regular expression patterns. This prevents backslashes (\) within the regex pattern from being interpreted as Python string escapes.

Core Functions: match() vs search()

The re module offers several functions for pattern matching. Two fundamental ones are match() and search().

1. re.match(pattern, string, flags=0)

Attempts to match the pattern only at the beginning of the string. If the pattern matches the start of the string, it returns a match object; otherwise, it returns None.

Parameters:

Parameter	Description
pattern	The regular expression string (use raw strings r'').
string	The string to search within.
flags	Optional modifiers (e.g., re.IGNORECASE, re.MULTILINE) combined using bitwise OR (|). Default is 0.

2. re.search(pattern, string, flags=0)

Scans through the entire string looking for the first location where the pattern produces a match. If a match is found anywhere in the string, it returns a match object; otherwise, it returns None.

Parameters are the same as re.match().

Match Objects

Both match() and search() return a Match object upon success. Match objects have useful methods:

Method	Description
group(num=0)	Returns the entire matched string, or a specific capturing group num (1-based index). group(0) or group() returns the whole match.
groups()	Returns a tuple containing all captured subgroups (empty tuple if none).
start(num=0)	Returns the starting index of the match (or subgroup num).
end(num=0)	Returns the ending index (exclusive) of the match (or subgroup num).
span(num=0)	Returns a tuple (start, end) for the match (or subgroup num).

Example: match() vs search()

import re

text = "The quick brown fox jumps over the lazy dog."
pattern = r"fox"

# match() - Fails because 'fox' is not at the beginning
match_obj = re.match(pattern, text)
if match_obj:
    print(f"match() found: '{match_obj.group()}' at index {match_obj.start()}")
else:
    print("match() found nothing.")

# search() - Succeeds because 'fox' is found within the string
search_obj = re.search(pattern, text)
if search_obj:
    print(f"search() found: '{search_obj.group()}' at index {search_obj.start()}")
else:
    print("search() found nothing.")

# Example with capturing groups
line = "Cats are smarter than dogs"
pattern_groups = r'(.*) are (.*?) .*' # Capture groups with ()
search_groups = re.search(pattern_groups, line, re.IGNORECASE)

if search_groups:
    print(f"\nFull match: {search_groups.group(0)}")
    print(f"Group 1: {search_groups.group(1)}")
    print(f"Group 2: {search_groups.group(2)}")
    print(f"All groups: {search_groups.groups()}")
else:
    print("Pattern with groups not found!")

Output:

match() found nothing.
search() found: 'fox' at index 16

Full match: Cats are smarter than dogs
Group 1: Cats
Group 2: smarter
All groups: ('Cats', 'smarter')

Other Useful re Functions

re.findall(pattern, string, flags=0)

Finds all non-overlapping matches of pattern in string and returns them as a list of strings. If the pattern includes capturing groups, it returns a list of tuples, where each tuple contains the captured group(s) for a match.

re.finditer(pattern, string, flags=0)

Similar to findall(), but returns an iterator yielding Match objects for each match. This is more memory-efficient for a large number of matches.

re.sub(pattern, repl, string, count=0, flags=0)

Replaces occurrences of pattern in string with repl. repl can be a string (allowing backreferences like \1, \g<name>) or a function that takes a match object and returns the replacement string. If count is specified and non-zero, only the first count occurrences are replaced. Returns the modified string.

Example: findall() and sub()

import re

text = "Contact us at info@example.com or support@example.org for help."

# Find all email addresses
email_pattern = r'[\w.-]+@[\w.-]+\.\w+'
emails = re.findall(email_pattern, text)
print(f"Emails found: {emails}")

# Replace domain names
phone = "Call 2004-959-559 for support # This is a Phone Number"

# Delete comments
num_no_comment = re.sub(r'#.*$', '', phone).strip()
print(f"Phone without comment: '{num_no_comment}'")

# Remove non-digits
num_digits_only = re.sub(r'\D', '', phone)
print(f"Phone digits only: '{num_digits_only}'")

Output:

Emails found: ['info@example.com', 'support@example.org']
Phone without comment: 'Call 2004-959-559 for support'
Phone digits only: '2004959559'

Regular Expression Flags

Flags modify how patterns are interpreted:

Flag	Alias	Description
re.IGNORECASE	re.I	Performs case-insensitive matching.
re.MULTILINE	re.M	Makes ^ match the start of each line (after a newline) and $ match the end of each line (before a newline), in addition to the start/end of the entire string.
re.DOTALL	re.S	Makes the dot (.) special character match any character at all, including a newline; without this flag, . matches anything except a newline.
re.VERBOSE	re.X	Allows you to write more readable regex patterns by ignoring whitespace (except when escaped or within character sets) and treating # as a comment marker.
re.ASCII	re.A	Makes \w, \W, \b, \B, \s, \S, \d, \D perform ASCII-only matching instead of full Unicode matching (default in Python 3).
re.UNICODE	re.U	Deprecated in Python 3, as Unicode matching is the default. Included for backward compatibility context.

Common Regex Metacharacters and Syntax

Regex patterns are built using literal characters and special metacharacters:

Pattern	Description
.	Matches any character except newline (unless re.DOTALL is used).
^	Matches the beginning of the string (or line if re.MULTILINE).
$	Matches the end of the string (or line if re.MULTILINE).
*	Matches 0 or more repetitions of the preceding expression (greedy).
+	Matches 1 or more repetitions of the preceding expression (greedy).
?	Matches 0 or 1 repetition of the preceding expression (greedy).
*?, +?, ??	Non-greedy (lazy) versions of *, +, ?. Match as few characters as possible.
{m}	Matches exactly m repetitions of the preceding expression.
{m,n}	Matches between m and n repetitions (inclusive, greedy).
{m,n}?	Non-greedy version of {m,n}.
[...]	Character Set: Matches any single character within the brackets. [aeiou] matches any vowel. Ranges like [a-z] or [0-9] can be used.
[^...]	Negated Character Set: Matches any single character NOT within the brackets.
\	Escape Character: Escapes special characters (e.g., \., \*, \\) or signals special sequences.
\d	Matches any Unicode decimal digit (equivalent to [0-9] if re.ASCII).
\D	Matches any character that is not a decimal digit.
\s	Matches any Unicode whitespace character (space, tab, newline, etc.).
\S	Matches any character that is not whitespace.
\w	Matches Unicode word characters (alphanumeric plus underscore).
\W	Matches any character that is not a word character.
\b	Matches an empty string at the beginning or end of a word (word boundary).
\B	Matches an empty string not at a word boundary.
(...)	Capturing Group: Matches the expression inside and captures the matched text. Can be retrieved with group(n).
(?:...)	Non-capturing Group: Groups expressions but doesn’t capture the match.
|	Alternation (OR): Matches either the expression before or the expression after the |. cat|dog matches ‘cat’ or ‘dog’.
(?P<name>...)	Named Capturing Group: Captures the match and assigns it a name. Access via group('name').
(?=...)	Positive Lookahead Assertion: Matches if the subpattern matches next, but doesn’t consume any characters.
(?!...)	Negative Lookahead Assertion: Matches if the subpattern does not match next.
(?<=...)	Positive Lookbehind Assertion: Matches if the subpattern matches immediately before the current position.
(?<!...)	Negative Lookbehind Assertion: Matches if the subpattern does not match immediately before.

For more details and advanced patterns, consult the official Python re module documentation.