Download Awk is a powerful command language that allows the user to

Awk is a powerful command language that allows the user to manipulate files containing
columns of data. Many applications of awk resemble those done on PC spreadsheets.
There are two ways to run awk. A simple awk command can be run from a single command line.
More complex awk scripts should be written to a command file.
Awk takes each line of input and tries to match the 'pattern' (see below), and if it succeeds it will
do whatever you tell it to do within the {} (called the action). Awk works best on files that have
columns of numbers or strings that are separated by white space (tabs or spaces), though on most
machines you can use the -F option if your columns are set apart by another character. Awk
refers to the first column as $1, the second column as $2, etc., and the whole line as $0.
Example Problem 1: Suppose a file called 'file1' that has 2 columns of numbers, and you want
to make a new file called 'file2' that has columns 1 and 2 as before, but also adds a third column
which is the ratio of the numbers in columns 1 and 2. Further, suppose you want the new 3column file (file2) to contain only those lines with column 1 smaller than column 2.
Solution:
awk '$1 < $2 {print $0, $1/$2}' file1 > file2
Or equivalently,
cat file1 | awk '$1 < $2 {print $0, $1/$2}' > file2
Let's look at the second one. You all know that 'cat file1' prints the contents of file1 to your
screen. The | (called a pipe) directs the output of 'cat file1', which normally goes to your screen,
to the command awk. Awk considers the input from 'cat file1' one line at a time, and tries to
match the 'pattern'. The pattern is whatever is between the first ' and the {, in this case the pattern
is $1 < $2. If the pattern is false, awk goes on to the next line. If the pattern is true, awk does
whatever is in the {}. In this case we have asked awk to check if the first column is less than the
second. If there is no pattern, awk assumes the pattern is true, and goes onto the action contained
in the {}.
What is the action? Almost always it is a print statement of some sort. In this case we want awk
to print the entire line, i.e. $0, and then print the ratio of columns 1 and 2, i.e. $1/$2. We close
the action with a }, and close the awk command with a '. Finally, to store the final 3-column
output into file2 (otherwise it prints to the screen), we add a '> file2'.
Example Problem 2: Suppose you have multiple files (e.g., hundreds) you want to move into a
new directory and rename by appending an extension .old to the filenames. You could do this
one by one (several hours), or use awk (several seconds). Suppose the files are named hw* (* is
wildcard for any sequence of characters), and need to be moved to subdirectory called 620 of the
current directory and have the extension '.old' appended to the name.
Solution:
ls hw* | awk '{print "mv "$0" ./620/"$0".old"}' | csh
ls hw* lists the filenames, and this output is piped into awk instead of going to your screen.
There is no pattern (nothing between the ' and the {), so awk proceeds to print something for
each line. For example, if the first three lines from 'ls hw*' produced hw1, hw2 and hw3,
respectively, then awk would print:
mv hw1 ./620/hw1.old
mv hw2 ./620/hw2.old
mv hw3 ./620/hw3.old
At this point the mv commands are simply printed to the screen. To execute the command we
take the output of awk and pipe it back into the operating system (the C-shell). Hence, to finish
the statement we add a ' | csh'.
More complex awk scripts need to be run from a file. The syntax for such cases is:
cat file1 | awk -f commands.awk > file2
where file1 is the input file, file2 is the output file, and commands.awk is a file containing awk
commands. Examples below that contain more than one line of awk need to be run from files.
Some useful awk variables defined for you are NF (number of columns), NR (the current line
that awk is working on), END (true if awk reaches the EOF), BEGIN (true before awk reads
anything), and length (number of characters in a line or a string). There is also looping
capability, a search (/) command, a substring command (extremely useful), and formatted
printing available. There are logical variables || (or) and && (and) that can be used in 'pattern'.
You can define and manipulate your own user-defined variables. More examples are given
below.
EXAMPLES
'column'
# is the comment character for awk.
# Print first two fields in opposite order:
awk '{ print $2, $1 }' file
# Print lines longer than 72 characters:
awk 'length > 72' file
# Print length of string in 2nd column
awk '{print length($2)}' file
'field' means
# Add up first column, print sum and average:
{ s += $1 }
END { print "sum is", s, " average is", s/NR }
# Print fields in reverse order:
awk '{ for (i = NF; i > 0; --i) print $i }' file
# Print the last line
{line = $0}
END {print line}
# Print the total number of lines that contain the word Pat
/Pat/ {nlines = nlines + 1}
END {print nlines}
# Print all lines between start/stop pairs:
awk '/start/, /stop/' file
# Print all lines whose first field is different from previous one:
awk '$1 != prev { print; prev = $1 }' file
# Print column 3 if column 1 > column 2:
awk '$1 > $2 {print $3}' file
# Print line if column 3 > column 2:
awk '$3 > $2' file
# Count number of lines where col 3 > col 1
awk '$3 > $1 {print i + "1"; i++}' file
# Print sequence number and then column 1 of file:
awk '{print NR, $1}' file
# Print every line after erasing the 2nd field
awk '{$2 = ""; print}' file
# Replace every field by its absolute value
{ for (i = 1; i <= NF; i=i+1) if ($i < 0) $i = -$i print}
References: These notes on awk are a modification of a discussion given in the website
sparky.rice.edu/~hartigan/awk.html