Notes on sed

Notes on sed

ed

When you open a file using ed, it displays the number of characters in the file and positions you at the last line.

By default, a command affects only the current line.

Commands:

p: print, d: delete

n: move to nth line

Move to line containing word: /word/

/regular/d, 1d

g/regular/d: delete all the lines that contain the regular expression

Substituting text:

[address]s /pattern/replacement/flag

s/regular/complex/g

This command changes all occurrences on the current line.

To make it apply to all lines, use the global command, putting g before the address.

g/regular/s/regular/complex/g

The "g" at the beginning is the global command that means make the changes on all lines matched by the address. The "g" at the end is a flag that means change each occurrence on a line, not just the first.

If the address and the pattern are the same, it can be written like:

g/regular/s//complex/g

ed test < ed-script

g/re/p stands for "global regular expression print."


Sed

SYNOPSIS

-n, --quiet, --silent suppress automatic printing of pattern space

-e script, --expression=script add the script to the commands to be executed

-f script-file, --file=script-file

add the contents of script-file to the commands to be executed

-i[suffix], --in-place[=suffix]

edit files in place (makes backup if extension supplied)

-l N, --line-length=N

specify the desired line-wrap length for the `l' command

-r, --regexp-extended

use extended regular expressions in the script.

-s, --separate

consider files as separate rather than as a single continuous long stream.

-u, --unbuffered

load minimal amounts of data from the input files and flush the output buffers more often

COMMAND SYNOPSIS

Zero-address “commands”

: label Label for b and t commands.

#comment The comment extends until the next newline (or the end of a -e script fragment).

} The closing bracket of a { } block.

Zero- or One- address commands

= Print the current line number.

a \

text

Append text, which has each embedded newline preceded by a backslash.

i \

text

Insert text, which has each embedded newline preceded by a backslash.

q

Immediately quit the sed script without processing any more input, except that if auto-print is not disabled the current pattern space will be printed.

Q

Immediately quit the sed script without processing any more input.

r filename Append text read from filename.

R filename Append a line read from filename.

Commands which accept address ranges

{ Begin a block of commands (end with a }).

b label Branch to label; if label is omitted, branch to end of script.

t label

If a s/// has done a successful substitution since the last input line was read and since the last t or T command, then branch to label; if label is omitted, branch to end of script.

T label

If no s/// has done a successful substitution since the last input line was read and since the last t or T command, then branch to label; if label is omitted, branch to end of script.

c \

text

Replace the selected lines with text, which has each embedded newline preceded by a backslash.

d

Delete pattern space. Start next cycle.

D

Delete up to the first embedded newline in the pattern space. Start next cycle, but skip reading from the input if there is still data in the pattern space.

h H

Copy/append pattern space to hold space.

g G

Copy/append hold space to pattern space.

x

Exchange the contents of the hold and pattern spaces.

l

List out the current line in a “visually unambiguous” form.

n N

Read/append the next line of input into the pattern space.

p

Print the current pattern space.

P(capital)

Print up to the first embedded newline of the current pattern space.

s/regexp/replacement/

Attempt to match regexp against the pattern space. If successful, replace that portion matched with replacement. The replacement may contain the special character & to refer to that portion of the pattern space which matched, and the special escapes \1 through \9 to refer to the corresponding matching sub-expressions in the regexp.

w filename

Write the current pattern space to filename.

W filename

Write the first line of the current pattern space to filename.

y/source/dest/

Transliterate the characters in the pattern space which appear in source to the corresponding character in dest.

Sed was meant to execute scripts exclusively and cannot be used interactively. Sed differs from ed primarily in that it is stream-oriented. By default, all of the input to sed passes through and goes to standard output. The input file itself is not changed.

Sed is stream-oriented, and ed is not. In ed, a command without an address affects only the current line. But Sed goes through the file, a line at a time, such that each line becomes the current line, and the commands are applied to it. The result is that sed applies a command without an address to every line in the file.

In ed you use addressing to expand the number of lines that are the object of a command; in sed, you use addressing to restrict the number of lines affected by a command.

Command-Line Syntax

sed [-e] ’instruction’ file

The -e option is necessary only when you supply more than one instruction on the command line. It tells sed to interpret the next argument as an instruction. When there is a single instruction, sed is able to make that determination on its own.

sed ’s/MA/Massachusetts/’ list

To specify multiple instructions on the command line:

sed ’s/ MA/, Massachusetts/; s/ PA/, Pennsylvania/’ list

sed -e ’s/ MA/, Massachusetts/’ -e ’s/ PA/, Pennsylvania/’ list

sed -f scriptfile inputfile

Scripting

In sed and awk, each instruction has two parts: a pattern and a procedure. The patter n is a regular expression delimited with slashes (/). A procedure specifies one or more actions to be performed.

Saving output

sed -f sedscr list > newlist

Do not redir ect the output to the file you are editing or you will clobber it. (The “>” redir ection operator truncates the file before the shell does anything else.) If you want the output file to replace the input file, you can do that as a separate step, using the mv command.

Suppressing automatic display of input lines

The default operation of sed is to output every input line. The -n option suppresses the automatic output. When specifying this option, each instruction intended to produce output must contain a print command, p.

sed -n -e ’s/MA/Massachusetts/p’ list

Mixing options (POSIX)

You can build up a script by combining both the -e and -f options on the command

line. The script is the combination of all the commands in the order given.

Using sed and awk Together

awk -F, ’{

print $4 ", " $0

}’ $* |

sort |

awk -F, ’

$1 == LastState {

print "\t" $2

}

$1 != LastState {

LastState = $1

print $1

print "\t" $2

}’


Basic principles how sed works:

All editing commands in a script are applied in order to each line of input.

Commands are applied to all lines (globally) unless line addressing restricts the lines affected by editing commands.

The original input file is unchanged; the editing commands modify a copy of original input line and the copy is sent to standard output.


Sed is implicitly global, will apply commands to every input line.

Line addresses are used to supply context for, or restrict, an operation.

/Sebastopol/s/CA/California/g


A sed command can specify zero, one, or two addresses. An address can be a regular expression describing a pattern, a line number, or a line addressing symbol.

If no addr ess is specified, then the command is applied to each line.

If ther e is only one address, the command is applied to any line matching the address.

If two comma-separated addresses are specified, the command is performed on the first line matching the first address and all succeeding lines up to and including a line matching the second address.

If an addr ess is followed by an exclamation mark (!), the command is applied to all lines that do not match the address.

1d,$d

The line number refers to an internal line count maintained by sed. This counter is not reset for multiple input files. Thus, no matter how many files were specified as input, there is only one line 1 and only one last line in the input stream. Last line can be specified using the addressing symbol $.

/ˆ$/d deletes only blank lines

50,$d, 1,/ˆ$/d


An exclamation mark (!) following an address reverses the sense of the match. The following script deletes all lines except those inside tbl input:

/ˆ\.TS/,/ˆ\.TE/!d


Grouping Commands

Braces ({}) are used in sed to nest one address inside another or to apply multiple commands at the same address. You can nest addresses if you want to specify a range of lines and then, within that range, specify another address.

To delete blank lines only inside blocks of tbl input, use the following command:

/ˆ\.TS/,/ˆ\.TE/{

/ˆ$/d

s/ˆ\.ps 10/.ps 8/

s/ˆ\.vs 12/.vs 10/

}

The opening curly brace must end a line and the closing curly brace must be on a line by itself. Be sure there are no spaces after the braces.

Basic sed Commands: d (delete), a (append), i (insert), and c (change)

[address]command

[line-address]command

Placing multiple commands on the same line is highly discouraged because sed scripts are difficult enough to read even when each command is written on its own line.

Comment #

Substitution

[address]s/pattern/replacement/flags

where the flags that modify the substitution are:

n A number (1 to 512) indicating that a replacement should be made for only the n th occurrence of the patter n.

g Make changes globally on all occurrences in the pattern space. Normally only the first occurrence is replaced.

p Print the contents of the pattern space.

w file Write the contents of the pattern space to file.

In the replacement section, only the following characters have special meaning:

& Replaced by the string matched by the regular expression.

\n Matches the n th substring (n is a single digit) previously specified in the pattern using “\(” and “\)”.

\ Used to escape the ampersand (&), the backslash (\), and the substitution command’s delimiter when they are used literally in the replacement section. In addition, it can be used to escape the newline and create a multiline replacement string.


The print and write flags are typically used when the default output is suppressed (the -n option). In addition, if a script contains multiple substitute commands that match the same line, multiple copies of that line will be printed or written to file.

Append, Insert, and Change

The syntax of these commands is unusual for sed because they must be specified over multiple lines.

append [line-address]a\

text

insert [line-address]i\

text

change [address]c\

text


Each of these commands requires a backslash following it to escape the first end of line. The text must begin on the next line. To input multiple lines of text, each successive line must end with a backslash, with the exception of the very last line.

/<Larry’s Address>/i\

4700 Cross Court\

French Lick, IN

The append and insert commands can be applied only to a single line address, not a range of lines. The change command, however, can address a range of lines. In this case, it replaces all addressed lines with a single copy of the text.

/ˆFrom /,/ˆ$/c\

<Mail Header Removed>

removes the entire mail-message header and replaces it with the line “<Mail Header Removed>.” Note that you will see the opposite behavior when the change command is one of a group of commands, enclosed in braces, that act on a range of lines.

/ˆFrom /,/ˆ$/{

s/ˆFrom //p

c\

<Mail Header Removed>

}

will output “<Mail Header Removed>” for each line in the range.


The change command clears the pattern space, having the same effect on the pattern space as the delete command. No command following the change command in the script is applied.

The insert and append commands do not affect the contents of the pattern space. The supplied text will not match any address in subsequent commands in the script, nor can those commands affect the text. No matter what changes occur to alter the pattern space, the supplied text will still be output appropriately. This is also true when the default output is suppressed — the supplied text will be output even if the pattern space is not. Also, the supplied text does not affect sed’s internal line counter.


List

The list command (l) displays the contents of the pattern space, showing nonprinting characters as two-digit ASCII codes.

Transform

[address]y/abc/xyz/

The replacement is made by character position, “a” is replaced by “x” anywhere on the line, regardless of whether or not it is followed by a “b”.

Print command (p)

/ˆ\.Ah/{

p

s/ "//g

s/ˆ\.Ah //p

}

The substitute command’s print flag differs from the print command in that it is conditional upon a successful substitution.


Print Line Number

An equal sign (=) following an address prints the line number of the matched line.

[line-address]=

This command cannot operate on a range of lines.

#n print line number and line with if statement

/ if/{

=

p

}

#n suppr esses the default output of lines.

Next

The next command (n) outputs the contents of the pattern space and then reads the next line of input without retur ning to the top of the script. Its syntax is:

[address]n

The next command changes the normal flow control, the next command causes the next line of input to replace the current line in the pattern space. Subsequent commands in the script are applied to the replacement line, not the current line. If the default output has not been suppressed, the current line is printed before the replacement takes place.

/ˆ\.H1/{

n

/ˆ$/d

}

Match any line beginning with the string ‘.H1’, then print that line and read in the next line. If that line is blank, delete it.

Reading and Writing Files

[line-address]r file

[address]w file

The read command reads the contents of file into the pattern space after the addressed line. It cannot operate on a range of lines. The write command writes the contents of the pattern space to the file.

The read command will not complain if the file does not exist. The write command will create a file if it does not exist; if the file already exists, the write command will overwrite it each time the script is invoked. If there are multiple instructions writing to the same file in one script, then each write command appends to the file.

sed ’$r closing’ $* | pr | lp

/ˆ<Company-list>/r company.list

/ˆ<Company-list>/d

2.

/Northeast$/{

s///

w region.northeast

}

The substitute command matches the same pattern as the address and removes it.

Quit(q)

The quit command (q) causes sed to stop reading new input lines (and stop sending them to the output). Its syntax is:

[line-address]q

sed ’100q’ test

sed -n "

/ˆ\.de *$mac/,/ˆ\.\.$/{

p

/ˆ\.\.$/q

}" $file


Advanced sed Commands

Multiline Pattern Space

Sed has the ability to look at more than one line in the pattern space. This allows you to match patterns that extend over multiple lines. The three multiline commands (N,D,P) all correspond to lowercase basic commands (n,d,p).


Append Next Line:N

The multiline Next (N) command creates a multiline pattern space by reading a new line of input and appending it to the contents of the pattern space. The original contents of pattern space and the new input line are separated by a newline. The embedded newline character can be matched in patterns by the escape sequence “\n”. In a multiline pattern space, the metacharacter “ˆ” matches the very first character of the pattern space, and not the character(s) following any embedded newline(s). Similarly, “$” matches only the final newline in the pattern space, and not any embedded newline(s). After the Next command is executed, control is then passed to subsequent commands in the script.

The Next command differs from the next command, which outputs the contents of the pattern space and then reads a new line of input. The next command does not create a multiline pattern space.

s/Owner and Operator Guide/Installation Guide/

/Owner/{

N

s/ *\n/ /

s/Owner and Operator Guide */Installation Guide\

/

}


$!N excludes the last line ($) from the Next command.


Multiline Delete

The delete command (d) deletes the contents of the pattern space and causes a new line of input to be read with editing resuming at the top of the script. The Delete command (D) works slightly differently: it deletes a portion of the pattern space, up to the first embedded newline. It does not cause a new line of input to be read; instead, it returns to the top of the script, applying these instructions to what remains in the pattern space.


# reduce multiple blank lines to one; version using d command

/^$/{

N

/^\n$/d

}

When a blank line is encountered, the next line is appended to the pattern space. Then we try to match the embedded newline. Note that the positional metacharacters, ˆ and $, match the beginning and the end of the pattern space, respectively.


Where ther e was an even number of blank lines, all the blank lines were removed. Only when there was an odd number was a single blank line preserved. That is because the delete command clears the entire pattern space. Once the first blank line is encountered, the next line is read in, and both are deleted. If a third blank line is encountered, and the next line is not blank, the delete command is not applied, and thus a blank line is output. If we use the multiline Delete command (D rather than d), we get the result we want:

/ˆ$/{

N

/ˆ\n$/D

}


Multiline Print:(P)

The multiline Print command outputs the first portion of a multiline pattern space, up to the first embedded newline.

The Print command(P) frequently appears after the Next command and before the Delete command. These three commands can set up an input/output loop that maintains a two-line pattern space yet outputs only one line at a time. The purpose of this loop is to output only the first line in the pattern space, then return to the top of the script to apply all commands to what had been the second line in the pattern space.

/UNIX$/{

N

/\nSystem/{

s// Operating &/

P

D

}

}


Examples

# Scribe font change script.

s/@f1(\([^)]*\))/\\fB\1\\fR/g

/@f1(.*/{

N

s/@f1(\(.*\n[^)]*\))/\\fB\1\\fR/g

P

D

}

This can be translated as: Once making a substitution across two lines, print the first line and then delete it from the pattern space. With the second portion remaining in the pattern space, control passes to the top of the script where we see if there is an “@f1(” remaining on the line.


Hold That Line(hold space)

The pattern space is a buffer that contains the current input line. There is also a set-aside buffer called the hold space. The contents of the pattern space can be copied to the hold space and the contents of the hold space can be copied to the patter n space. A group of commands allows you to move data between the hold space and the pattern space. The hold space is used for temporary storage.

The most frequent use of the hold space is to have it retain a duplicate of the current input line while you change the original in the pattern space.

Command Abbreviation Function

Hold h or H Copy or append contents of pattern space to hold space.

Get g or G Copy or append contents of hold space to pattern space.

Exchange x Swap contents of hold space and pattern space.


Each of these commands can take an address that specifies a single line or a range of lines.

# Reverse flip

/1/{

h

d

}

/2/{

G

}

The hold command followed by the delete command is a fairly common pairing. Without the delete command, control would reach the bottom of the script and the contents of the pattern space would be output.


A Capital Transformation

# capitalize statement names

/the .* statement/{

h

s/.*the \(.*\) statement.*/\1/

y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/

G

s/\(.*\)\n\(.*the \).*\( statement.*\)/\2\1\3/

}


Building Blocks of Text

The hold space can be used to collect a block of lines before outputting them.

/ˆ$/!{

H

d

}

/ˆ$/{

x

s/ˆ\n/<p>/

s/$/<\/p>/

G

}


Examples:

sed -n '10,20p' access.log


Resources

sed & awk, 2nd Edition

http://linux.about.com/od/commands/l/blcmdl1_sed.htm


Post a Comment

Labels

Java (159) Lucene-Solr (112) Interview (61) All (58) J2SE (53) Algorithm (45) Soft Skills (39) Eclipse (33) Code Example (31) Linux (24) JavaScript (23) Spring (22) Windows (22) Web Development (20) Tools (19) Nutch2 (18) Bugs (17) Debug (16) Defects (14) Text Mining (14) J2EE (13) Network (13) Troubleshooting (13) PowerShell (11) Problem Solving (10) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) UIMA (9) html (9) Http Client (8) Maven (8) Security (8) bat (8) blogger (8) Big Data (7) Continuous Integration (7) Google (7) Guava (7) JSON (7) Shell (7) ANT (6) Coding Skills (6) Database (6) Lesson Learned (6) Programmer Skills (6) Scala (6) Tips (6) css (6) Algorithm Series (5) Cache (5) Dynamic Languages (5) IDE (5) System Design (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Life (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) How to Interview (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Logging (2) Python (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) Invest (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts