Sed Regular Expressions

Sed - Mastering Regular Expressions

Regular expressions are the cornerstone of Sed’s power and efficiency, enabling it to perform complex text manipulations. A solid understanding of regular expressions (often abbreviated as regex) is invaluable for any command-line user.

Like many GNU/Linux utilities, Sed supports regular expressions. This chapter delves into the intricacies of regex syntax used in Sed. We’ll cover Basic Regular Expressions (BREs), which Sed uses by default, POSIX character classes, and special GNU Sed metacharacters. We will also touch upon Extended Regular Expressions (EREs) which can simplify many patterns.

Understanding Basic vs. Extended Regular Expressions (BRE vs. ERE)

Sed, by default, uses Basic Regular Expressions (BRE). In BRE, certain characters like ?, +, {}, |, (, and ) lose their special meaning and must be escaped with a backslash (e.g., \?, \+, \{\}, \|, \(\)).

Most modern Sed versions (especially GNU Sed) support Extended Regular Expressions (ERE) via the -E or -r (for GNU Sed) command-line option. In ERE, these characters regain their special meaning without needing a backslash, leading to cleaner and more readable patterns. This tutorial will primarily use BRE syntax as it’s Sed’s default, but we’ll often point out how ERE can simplify the expressions.

Core Regular Expression Metacharacters (BRE)

Start of Line (^)

The caret (^) symbol matches the beginning of the pattern space (typically, a line of text). The following example prints all lines from books.txt that start with the word “The”.

[user]$ sed -n '/^The/p' books.txt

Assuming books.txt contains:

1. A Storm of Swords, J. R. R. Tolkien
2. The Two Towers, J. R. R. Tolkien
3. The Alchemist, Paulo Coelho
4. The Fellowship of the Ring, J. R. R. Tolkien
5. The Pilgrimage, Paulo Coelho

Executing the command yields:

2) The Two Towers, J. R. R. Tolkien
3) The Alchemist, Paulo Coelho
4) The Fellowship of the Ring, J. R. R. Tolkien
5) The Pilgrimage, Paulo Coelho

End of Line ($)

The dollar ($) symbol matches the end of the pattern space. The following example prints lines ending with “Coelho”.

[user]$ sed -n '/Coelho$/p' books.txt

Executing the command yields:

3) The Alchemist, Paulo Coelho
5) The Pilgrimage, Paulo Coelho

Single Character (.)

The dot (.) matches any single character except a newline. The following example prints three-letter words ending with ‘t’ from the input stream.

[user]$ echo -e "cat\nbat\nrat\nmat\nbatting\nrats\nmats" | sed -n '/^..t$/p'

Executing the command yields:

cat
bat
rat
mat

Character Set ([…])

Square brackets [] define a character set, matching any single character within the set. For example, /[CT]all/ matches “Call” or “Tall”.

[user]$ echo -e "Call\nTall\nBall" | sed -n '/[CT]all/p'

Executing the command yields:

Call
Tall

Exclusive Character Set ([^…])

A caret ^ as the first character inside square brackets negates the set, matching any single character not in the set. /[^CT]all/ matches “Ball” but not “Call” or “Tall”.

[user]$ echo -e "Call\nTall\nBall" | sed -n '/[^CT]all/p'

Executing the command yields:

Ball

Character Range ([-])

A hyphen - within a character set specifies a range. /[C-Z]all/ matches any uppercase letter from C to Z followed by “all”.

[user]$ echo -e "Call\nTall\nBall" | sed -n '/[C-Z]all/p'

Executing the command yields:

Call
Tall

Modifying the range to /[A-P]all/ changes the match:

[user]$ echo -e "Call\nTall\nBall" | sed -n '/[A-P]all/p'

Executing the command yields:

Call
Ball

Quantifiers (Repetition)

Quantifiers specify how many times the preceding character or group can occur. Remember to escape these in BRE.

Zero or One Occurrence (?)

The escaped question mark \? matches zero or one occurrence of the preceding character. (In ERE, use ?). This makes the character optional. The example matches “Behaviour” and “Behavior”.

[user]$ echo -e "Behaviour\nBehavior" | sed -n '/Behaviou\?r/p'

Executing the command yields:

Behaviour
Behavior

One or More Occurrences (+)

The escaped plus \+ matches one or more occurrences of the preceding character. (In ERE, use +). The example matches lines containing one or more ‘2’s.

[user]$ echo -e "111\n22\n123\n234\n456\n222" | sed -n '/2\+/p'

Executing the command yields:

22
123
234
222

Zero or More Occurrences (*)

The asterisk * matches zero or more occurrences of the preceding character. (It does not need escaping in BRE for this basic usage). The example matches “ca”, “cat”, “catt”, etc.

[user]$ echo -e "ca\ncat\ncaz" | sed -n '/cat*/p'

Executing the command yields:

ca
cat

Exactly N Occurrences ({N})

The escaped curly braces \{N\} match exactly N occurrences of the preceding character. (In ERE, use {N}). The example prints only three-digit numbers from numbers.txt.

[user]$ cat numbers.txt
1
10
100
1000
10000

[user]$ sed -n '/^[0-9]\{3\}$/p' numbers.txt

Executing the command yields:

100

At Least N Occurrences ({N,})

\{N,\} matches at least N occurrences. (In ERE, use {N,}). The example prints numbers with five or more digits.

[user]$ sed -n '/^[0-9]\{5,\}$/p' numbers.txt

Executing the command yields:

10000

N to M Occurrences ({N,M})

\{N,M\} matches at least N and at most M occurrences. (In ERE, use {N,M}). The example prints numbers with at least three but not more than four digits.

[user]$ sed -n '/^[0-9]\{3,4\}$/p' numbers.txt

Executing the command yields:

100
1000

Alternation (|)

The escaped pipe \| (within escaped parentheses \(...\)) acts as a logical OR, matching either the expression on its left or its right. (In ERE, use | and (...)). The example matches “str1” or “str3”.

[user]$ echo -e "str1\nstr2\nstr3\nstr4" | sed -n '/str\(1\|3\)/p'

Executing the command yields:

str1
str3

Note: For complex alternations or grouping, using -E (or -r) for Extended Regular Expressions is highly recommended for readability: sed -E -n '/str(1|3)/p'.

Escaping Special Characters

To match characters that have special meaning in regex (like . * [ $ ^), you must escape them with a backslash \. For example, to match a literal dot, use \.. Some sequences like \n (newline) or \t (tab) represent non-printing characters, though their interpretation in Sed patterns can be tricky and often it’s better to use them in the replacement part of an s command or rely on POSIX classes.

Escaping Backslash (\)

To match a literal backslash, you need to escape it: \\.

[user]$ echo 'path\to\file' | sed -n '/to\\file/p'

Executing the command yields:

path\to\file

Escaping Other Special Characters (e.g., \. for literal dot)

To match a literal dot (.), you must escape it as \.. Otherwise, . matches any character.

[user]$ echo -e "file.txt\nfile_txt" | sed -n '/file\.txt/p'

Executing the command yields:

file.txt

Representing Non-Printable Characters (ASCII values)

While Sed primarily works with text lines, some versions (especially GNU Sed) allow matching characters by their ASCII values. This is less common and often platform-dependent.

Decimal ASCII (\dNNN - GNU Sed)

\dNNN matches a character by its decimal ASCII value NNN. Example: \d97 for ‘a’.

[user]$ echo -e "a\nb\nc" | sed -n '/\d97/p' # GNU Sed specific

Executing the command (with GNU Sed) yields:

a

Octal ASCII (\oNNN - GNU Sed)

\oNNN matches a character by its octal ASCII value NNN. Example: \o142 for ‘b’.

[user]$ echo -e "a\nb\nc" | sed -n '/\o142/p' # GNU Sed specific

Executing the command (with GNU Sed) yields:

b

Hexadecimal ASCII (\xHH - GNU Sed)

\xHH matches a character by its hexadecimal ASCII value HH. Example: \x63 for ‘c’.

[user]$ echo -e "a\nb\nc" | sed -n '/\x63/p' # GNU Sed specific

Executing the command (with GNU Sed) yields:

c

POSIX Character Classes

POSIX character classes are special keywords enclosed in [: and :] within a character set []. They provide a portable way to match common character types, respecting locale settings.

[:alnum:] - Alphanumeric Characters

Matches letters and digits. Example: /[[:alnum:]]/ matches lines containing at least one letter or digit.

[user]$ echo -e "One\n123\n@#\n\t" | sed -n '/[[:alnum:]]/p'

Executing the command yields:

One
123

[:alpha:] - Alphabetic Characters

Matches letters only. Example: /[[:alpha:]]/ matches lines containing at least one letter.

[user]$ echo -e "One\n123\n@#\n\t" | sed -n '/[[:alpha:]]/p'

Executing the command yields:

One

[:blank:] - Blank Characters

Matches space and tab characters. The example matches the line containing only a tab. cat -vte is used to visualize non-printable characters (^I for tab, $ for end of line).

[user]$ echo -e "Word\n \n\t" | sed -n '/^[[:blank:]]\+$/p' | cat -vte

Executing the command yields (lines with only spaces or tabs):

 ^I$

[:digit:] - Digits

Matches decimal digits (0-9). Equivalent to [0-9] in many locales. Example: /[[:digit:]]/ matches lines containing at least one digit.

[user]$ echo -e "abc\n123\n@#\n\t" | sed -n '/[[:digit:]]/p'

Executing the command yields:

123

[:lower:] - Lowercase Letters

Matches lowercase letters. Example: /[[:lower:]]/ matches lines containing at least one lowercase letter.

[user]$ echo -e "one\nTWO\n123\n\t" | sed -n '/[[:lower:]]/p'

Executing the command yields:

one

[:upper:] - Uppercase Letters

Matches uppercase letters. Example: /[[:upper:]]/ matches lines containing at least one uppercase letter.

[user]$ echo -e "one\nTWO\n123\n\t" | sed -n '/[[:upper:]]/p'

Executing the command yields:

TWO

[:punct:] - Punctuation Characters

Matches punctuation characters (e.g., . , ; ! ?). Example: /[[:punct:]]/ matches lines containing at least one punctuation mark.

[user]$ echo -e "One,Two\nThree Four\nEnd!" | sed -n '/[[:punct:]]/p'

Executing the command yields:

One,Two
End!

[:space:] - Whitespace Characters

Matches whitespace characters, including space, tab, newline, carriage return, form feed, and vertical tab. The example below uses cat -vte to visualize a line containing space, form feed (^L), and tab (^I).

[user]$ echo -e "Word\nLine with space \f\t" | sed -n '/[[:space:]]/p' | cat -vte

Executing the command yields (all input lines match as they contain spaces or other whitespace):

Word$
Line with space ^L^I$

GNU Sed Metacharacters (Perl-like)

GNU Sed offers additional metacharacters, often inspired by Perl, which can simplify common matching tasks. These are GNU extensions and may not be available in other Sed versions. When using these, it’s good practice to be aware of potential portability issues or use the --posix option if strict POSIX compliance is needed.

Word Boundary (\b)

\b matches a zero-width boundary between a word character ([:alnum:] plus _) and a non-word character, or at the start/end of a line if adjacent to a word character. For example, \bthe\b matches “the” as a whole word, but not in “these” or “other”.

[user]$ echo -e "these\nthe theme\nthey\nthen" | sed -n '/\bthe\b/p'

Executing the command yields:

the theme

Non-Word Boundary (\B)

\B matches a zero-width position that is not a word boundary. For example, the\B matches “the” when it’s part of a larger word like “these” or “they”, but not “the” as a whole word.

[user]$ echo -e "these\nthe\nthey" | sed -n '/the\B/p'

Executing the command yields:

these
they

Whitespace Character (\s)

\s matches any whitespace character (equivalent to [[:space:]]). The example matches “Line” followed by any whitespace.

[user]$ echo -e "Line\t1\nLine 2\nLine3" | sed -n '/Line\s/p'

Executing the command yields:

Line\t1
Line 2

Non-Whitespace Character (\S)

\S matches any non-whitespace character (equivalent to [^[:space:]]). The example matches “Line” followed by a non-whitespace character.

[user]$ echo -e "Line\t1\nLine 2\nLine3" | sed -n '/Line\S/p'

Executing the command yields:

Line3

Word Character (\w)

\w matches any word character (alphanumeric characters plus underscore: [[:alnum:]_]).

[user]$ echo -e "One\n123\n1_2\n&;#" | sed -n '/^\w\+$/p'

Executing the command yields lines consisting only of word characters:

One
123
1_2

Non-Word Character (\W)

\W matches any non-word character (opposite of \w; [^[:alnum:]_]).

[user]$ echo -e "One\n123\n1_2\n&;#" | sed -n '/\W/p'

Executing the command yields lines containing non-word characters:

1_2
&;#

Beginning of Pattern Space (`)

\`` (backslash grave accent) matches the beginning of the pattern space. This is similar to ^but has different behavior in some advanced multi-line scenarios with commands likeD. For most single-line processing, ^` is preferred and more portable.

[user]$ echo -e "One\nTwo One" | sed -n '/\`One/p'

Executing the command yields:

One

End of Pattern Space (’)

\' (backslash apostrophe) matches the end of the pattern space. Similar to $ but with nuances in multi-line processing. For single-line processing, $ is standard.

[user]$ echo -e "End Note\nNote" | sed -n '/Note\'/p'

Executing the command yields:

End Note
Note

Further Learning: For more in-depth information on regular expressions, consult the regex(7) man page on your system or online resources like regex101.com (select PCRE/Perl flavor for patterns similar to GNU extensions, or POSIX ERE/BRE for standard Sed).