30+ awk examples for beginners / awk command tutorial in Linux/Unix

Table of Contents

In this article we will learn about AWK command used in Linux and Unix. I will share some basic awk examples in one liner commands, awk script examples and other scenarios in Linux or Unix environment.

 

What is AWK in Linux or Unix?

AWK is an interpreted programming language designed for text processing and report generation. It is typically used for data manipulation, such as searching for items within data, performing arithmetic operations, and restructuring raw data for generating reports in most Unix-like operating systems.

Programs in awk are different from programs in most other languages, because awk programs are data driven (i.e., you describe the data you want to work with and then what to do when you find it).

When you run awk, you specify an awk program that tells awk what to do. The program consists of a series of rules. Syntactically, a rule consists of a pattern followed by an action. The action is enclosed in braces to separate it from the pattern.

 

Types of AWK

The AWK language was originally implemented as an AWK utility on Unix. Today, most Linux distributions provide GNU implementation of AWK (GAWK), and a symlink for AWK is created from the original GAWK binary.

  • AWK
  • NAWK
  • GAWK

 

AWK command syntax for action and pattern structure

AWK programs are sequences of one or more patterns followed by action statements. These action-pattern statements are separated by newlines. Both actions (AWK commands) and patterns (search patterns) are optional, hence we use { } to distinguish them: Below is the awk command syntax for action and pattern structure:

/ search pattern / { action / awk-commands }/ search pattern / { action / awk-commands }

 

AWK command syntax for pattern-only statements

The awk command syntax with a pattern only is as follows:

awk   '/ pattern /'    inputfilename

In the given awk command examples, all lines of the passwd file are processed, and those that contain the mail pattern are printed:

# awk '/mail/' /tmp/passwd
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin

 

AWK command syntax for action-only statements

The awk command syntax with an action only is as follows:

awk '{ action statements / awk-commands }' inputfilenames

In the given awk command examples, all employee names are printed with awk print column on the screen as $2, representing the second column of each input line.

# awk '{print $2}' /tmp/userdata.txt
Name
Deepak
Rahul
Amit
Sumit

 

1. AWK examples to print each input line or record

We can print each input record of the input file in multiple ways. Lets see some more awk examples.

# awk '//' /tmp/userdata.txt
# awk '{print}' /tmp/userdata.txt
# awk '{print $0}' /tmp/userdata.txt
# awk '//{print}' /tmp/userdata.txt

All the given awk command examples will produce the same result by printing all the input records of the input file /tmp/userdata.txt.

id   Name    Age  username
1    Deepak  31   deepak
2    Rahul   32   rahul
3    Amit    33   amit
4    Sumit   33   sumit

 

2. Using the BEGIN and END blocks construct

AWK contains two special keywords, BEGIN and END, for patterns where an action is required. Both of them are optional and are used without slashes. They don't match any input lines.

Below awk examples shows usage of the BEGIN and END blocks:

# awk 'BEGIN { print "==Employee Info==" }  	          <-- begin block 
> { print }						  <-- body block 
> END { print "==ends here==" }' /tmp/userdata.txt	  <-- end block
==Employee Info==
id   Name    Age  username
1    Deepak  31   deepak
2    Rahul   32   rahul
3    Amit    33   amit
4    Sumit   33   sumit
==ends here==

 

3. Running awk from source file

When AWK programs are long, it is more convenient to put them in a separate file. Putting AWK programs in a file reduces errors and retyping.

The -f option tells the AWK utility to read the AWK commands from source_file. Any filename can be used in place of source_file. Let's take some awk examples, we can create an awk script example cmd.awk text file containing the AWK commands, as follows:

# cat cmd.awk
BEGIN { print "=== Emp Info ===" }
{ print }

Now, we instruct AWK to read the commands from the cmd.awk file and perform the given actions:

# awk -f cmd.awk userdata.txt
=== Emp Info ===
id   Name    Age  username
1    Deepak  31   deepak
2    Rahul   32   rahul
3    Amit    33   amit
4    Sumit   33   sumit

 

4. AWK script examples for programming

We can write self-contained AWK scripts to execute AWK commands, like we have with shell scripts to execute shell commands. We create the AWK scripts by using #!, followed by the absolute path of the AWK interpreter and the -f optional argument.

Below is one small awk script example:

# cat cmd.awk
# awk script Examples for AWK Tutorial
#!/bin/awk -f

BEGIN { print "=== Emp Info ===" }
{ print }

Give this awk script example file executable permissions

# chmod u+x /tmp/cmd.awk

Next simply run ./cmd.awk userdata.txt at the shell and the system will run AWK as if you had typed awk -f cmd.awk userdata.txt:

# ./cmd.awk userdata.txt
=== Emp Info ===
id   Name    Age  username
1    Deepak  31   deepak
2    Rahul   32   rahul
3    Amit    33   amit
4    Sumit   33   sumit

 

5. Comments in AWK

A comment is some text that is included in a program for documentation or human information. It is not an executable part of the program. It explains what a program does and how it does it. We will create an awk script examples with some comments.

Lets see some awk script examples, we can put the following in cmd.awk:

# cat cmd.awk
# awk script Examples for AWK Tutorial
#!/bin/awk -f

# Info        : This program displays the employees information
# Date        : 22 Jan 2020
# Version     : 1.0

# This is BEGIN block comment
BEGIN { print "=== Emp Info ===" }
# Body block comment
{ print }
# End block comment
END { print "=== Information Database ends here ===" }

Next execute the awk script example scrip and here is the output:

# ./cmd.awk userdata.txt
=== Emp Info ===
id   Name    Age  username
1    Deepak  31   deepak
2    Rahul   32   rahul
3    Amit    33   amit
4    Sumit   33   sumit
=== Information Database ends here ===

 

6. AWK match pattern and print without action statement

In this awk examples, the program has a pattern, but we don't specify any action statements.

# awk '/rahul/' /tmp/userdata.txt
2    Rahul   32   rahul

In this case, AWK selects only those input lines that contain the rahul pattern/string in them. When we don't specify any action, AWK assumes the action is to print the whole line.

 

7. AWK print column without specifying any pattern

In this awk examples, we will not include any pattern but instead using awk print column. The given AWK command prints the first column ($1) and second column ($2) of each input line that is separated by a space (the output field separator, indicated by a comma):

# awk '{ print $1, $2 }' /tmp/userdata.txt
id Name
1 Deepak
2 Rahul
3 Amit
4 Sumit

 

8. AWK print column with matching patterns

In this awk examples, we will include both actions and patterns and with awk print column. The given AWK command prints the first column ($1) separated by tab (specifying \t as the output separator) with the second ($2) and third column ($3) of input lines, which contain the "deepak" string in them:

# awk '/deepak/{ print $1 "\t" $2 "\t" $3}' /tmp/userdata.txt
1       Deepak  31

 

9. AWK print column in any order with custom text

In this awk examples, with awk print column in different orders. Here, in the action statement, we put some text in between the row content

# awk '{ print $1 " is mapped to " $2 ", with age " $3 ", and username " $4 }' /tmp/userdata.txt
id is mapped to Name, with age Age, and username username
1 is mapped to Deepak, with age 31, and username deepak
2 is mapped to Rahul, with age 32, and username rahul
3 is mapped to Amit, with age 33, and username amit
4 is mapped to Sumit, with age 33, and username sumit

We can also add a pattern match for the action and then with awk print column. Here we search for deepak string and then print the text with values from different columns.

# awk '/deepak/{ print $1 " is mapped to " $2 ", with age " $3 ", and username " $4 }' /tmp/userdata.txt
1 is mapped to Deepak, with age 31, and username deepak

 

10. Printing the number of columns in a line

AWK has built-in variables to count and store the number of fields in the current input line, for example, NF. So, in the given awk examples, we will print the number of the columns for each input line, followed by the first column and the last column (accessed using NF):

# awk '{ print NF, $1, $NF }' /tmp/userdata.txt
4 id username
4 1 deepak
4 2 rahul
4 3 amit
4 4 sumit

Since we have 4 columns so NF prints 4 and since we are printing variable value of $NF which is 4, so awk prints fourth column content

We can put text in between the fields and then awk print column

# awk '{ print " No of fields: " NF ", and the last field contains "$NF }' /tmp/userdata.txt
 No of fields: 4, and the last field contains username
 No of fields: 4, and the last field contains deepak
 No of fields: 4, and the last field contains rahul
 No of fields: 4, and the last field contains amit
 No of fields: 4, and the last field contains sumit

 

11. Deleting empty lines using NF

We can print all the lines with at least 1 field using NF > 0. This is the easiest method to remove empty lines from the file using AWK:
Here I have added an empty line to my input file after 3rd row

# awk '//' /tmp/userdata.txt
id   Name    Age  username
1    Deepak  31   deepak
2    Rahul   32   rahul
3    Amit    33   amit

4    Sumit   33   sumit

To remove empty line using awk with NF

# awk 'NF > 0 { print }' /tmp/userdata.txt
id   Name    Age  username
1    Deepak  31   deepak
2    Rahul   32   rahul
3    Amit    33   amit
4    Sumit   33   sumit

 

12. Printing line numbers in the output

AWK has a built-in variable known as NR. It counts the number of input lines read so far. We can use NR to prefix $0 in the print statement to display the line numbers of each line that has been processed:

# awk '{ print NR, $0 }' /tmp/userdata.txt
1 id   Name    Age  username
2 1    Deepak  31   deepak
3 2    Rahul   32   rahul
4 3    Amit    33   amit
5
6 4    Sumit   33   sumit

 

13. AWK sum column by counting the numbers of lines in a file using NR

In our next awk examples, we will count the number of lines in a file using NR and use awk sum column. As NR stores the current input line number, we need to process all the lines in a file for awk sum column. We also don't want to print the line numbers for each line, as our requirement is to just awk sum column. Since the END block is executed after processing the input line is done, we will print NR in the END block to print the total number of lines in the file using awk sum column:

# awk ' END { print "The total number of lines in file are : " NR } ' /tmp/userdata.txt
The total number of lines in file are : 6

 

14. Printing numbered lines exclusively from the file

We know NR contains the line number of the current input line. You can easily print any line selectively, by matching the line number with the current input line number stored in NR, as follows in below awk examples:

# awk  'NR==2 { print NR, $0 }' /tmp/userdata.txt
2 1    Deepak  31   deepak

 

15. AWK print row with even-numbered lines in a file

Using NR, we can easily print even-numbered files by specifying expressions (divide each line number by 2 and find the remainder) in pattern space, as shown in the following awk examples:

# awk  'NR % 2 == 0{ print NR, $0 }' /tmp/userdata.txt
2 1    Deepak  31   deepak
4 3    Amit    33   amit
6 4    Sumit   33   sumit

 

16. AWK print row with odd-numbered lines in a file

Similarly, we can use NR with awk print column to list odd-numbered lines in a file using NR, by performing basic arithmetic operations in the pattern space:

# awk  'NR % 2 == 1{ print NR, $0 }' /tmp/userdata.txt
1 id   Name    Age  username
3 2    Rahul   32   rahul
5

 

17. AWK print row using the range operator (,) and NR

We can combine the range operator (,) and NR to print a group of lines from a file based on their line numbers. The next awk examples displays the lines 2 to 4 from the userdata.txt file:

# awk  'NR==2, NR==4 { print NR, $0 }' /tmp/userdata.txt
2 1    Deepak  31   deepak
3 2    Rahul   32   rahul
4 3    Amit    33   amit

 

18. AWK print row using the range operator and patterns

We can also combine the range operator (,) and string in pattern space to print a group of lines in a file starting from the first pattern, up to the second pattern.

# awk '/deepak/,/amit/ { print NR, $0 }' /tmp/userdata.txt
2 1    Deepak  31   deepak
3 2    Rahul   32   rahul
4 3    Amit    33   amit

 

19. Print selection using the match operator (~)

The match operator (~) is used for matching a pattern in a specified field in the input line of a file. In the next awk examples, we will select and print all lines containing 'a' in the second column of the input line, as follows:

# awk ' $2 ~ /a/ {print NR, $0 } ' /tmp/userdata.txt
1 id   Name    Age  username
2 1    Deepak  31   deepak
3 2    Rahul   32   rahul

 

20. Print selection using the match operator (~) and anchor (^)

The caret (^) in regular expressions (also known as anchor) is used to match at the beginning of a line. In the next awk examples, we combine it with the match operator (~) to print all the lines in which the second field begins with the 'A' character, as follows

# awk ' $2 ~ /^A/ {print NR, $0 } ' /tmp/userdata.txt
4 3    Amit    33   amit

 

21. Print selection using the match operator (~) and character classes ([ ])

The character classes, [ ], in regular expressions are used to match a single character out of those specified within square brackets. Here, we combine the match operator (~) with character classes (/^[AD]/) to print all the lines in which the second field begins with the 'A' or 'D' character, as follows:

# awk ' $2 ~ /^[AD]/ {print NR, $0 } ' /tmp/userdata.txt
2 1    Deepak  31   deepak
4 3    Amit    33   amit

 

22. Print selection using the match operator (~) and anchor ($):

The dollar sign ($) in regular expression (also known as anchor) is used to match at the end of a line. In the next awk examples, we combine it with the match operator (~) to print all the lines in the second field end with the 'k' character, as follows:

# awk ' $2 ~ /k$/ {print NR, $0 } ' /tmp/userdata.txt
2 1    Deepak  31   deepak

 

23. Print selection by numeric comparison

You can use relation operators (==, >=, <=, >, <, !=) for performing numeric comparison. Here, we perform a numeric greater than or equal (>=) to print the lines that have the 32 or higher value in the third field, as follows:

# awk ' $3 >= 32 {print NR, $0 } ' /tmp/userdata.txt
1 id   Name    Age  username
3 2    Rahul   32   rahul
4 3    Amit    33   amit
6 4    Sumit   33   sumit

 

24. Print selection by text content/string matching in a field

Besides numeric matches, we can use string matches to find the lines containing a particular string in a field. String content for matches should be given in double quotes as a string

In our next awk examples, we print all the lines that contain "deepak" in the fourth field ($4), as follows:

# awk ' $4 == "deepak" {print NR, $0 } ' /tmp/userdata.txt
2 1    Deepak  31   deepak

 

25. Print selection by combining patterns

You can combine patterns with parentheses and logical operators, &&, ||, and !, which stand for AND, OR, and NOT. Here, we print the lines containing a value greater than or equal to 30 in the third field and a value less than or equal to 32 in the third field.

# awk ' $3 >= 30 && $3 <= 32  {print NR, $0 } ' /tmp/userdata.txt
2 1    Deepak  31   deepak
3 2    Rahul   32   rahul

 

26. Using AWK BEGIN to print headings

The AWK BEGIN block can be used for printing headings, initialising variables, performing calculations, or any other task that you want to be executed before AWK starts processing the lines in the input file.

# /tmp/header.awk /tmp/userdata.txt
Field1 Field2 Field3 Field4

id     Name    Age   username
1      Deepak  31    deepak
2      Rahul   32    rahul
3      Amit    33    amit
4      Sumit   33    sumit

 

27. Using AWK END to print the last input line

The AWK END block is executed after the processing of the last line of the last file is completed, and $0 stores the value of each input line processed, but its value is not retained in the END block. The following is one way to print the last input line:

# awk '{ last = $0 } END { print last }' /tmp/userdata.txt
4    Sumit   33   sumit

And to print the total number of lines in a file we use NR, because it retains its value in the END block, as follows:

# awk 'END { print "Total no of lines in file: ", NR}' /tmp/userdata.txt
Total no of lines in file:  6

 

28. Count the number of characters using length function

By default, the length function stores the count of the number of characters in the input line. In the next example, we will prefix each line with the number of characters in it using the length function, as follows:

# awk '{print length, $0}' /tmp/passwd
31 root:x:0:0:root:/root:/bin/bash
32 bin:x:1:1:bin:/bin:/sbin/nologin
39 daemon:x:2:2:daemon:/sbin:/sbin/nologin
36 adm:x:3:4:adm:/var/adm:/sbin/nologin
40 lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
31 sync:x:5:0:sync:/sbin:/bin/sync
44 shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
32 halt:x:7:0:halt:/sbin:/sbin/halt
46 mail:x:8:12:mail:/var/spool/mail:/sbin/nologin

 

29. Changing the field separator using FS

The fields in the awk examples we have discussed so far have been separated by space characters. The default behaviour of FS is any number of space or tab characters; we can change it to regular expressions or any single or multiple characters using the FS variable or the -F option.

In this, we use the /etc/passwd file of Linux, which delimits fields with colons (:). So, we change the input of FS to a colon before reading any data from the file, and print the list of usernames, which is stored in the first field of the file, as follows:

# awk 'BEGIN { FS = ":"} {print $1}' /etc/passwd
root
bin
daemon
adm
lp
sync
shutdown

 

30. Control structures (Example: AWK while loop)

AWK supports control (flow) statements, which can be used to change the order of the execution of commands within an AWK program. Different constructs, such as the if...else, while, and for control structures are supported by AWK. In addition, the break and continue statements work in combination with the control structures to modify the order of execution of commands.

Let's try a basic example of a while loop to print a list of numbers under 5:

# awk 'BEGIN{ n=1; while (n < 5 ) { print n; n++; } }'
1
2
3
4

Alternatively you can also create an awk script examples

# cat /tmp/while1.awk
# awk script Examples for AWK Tutorial
#!/bin/awk -f

BEGIN { n=1
while ( n < 5 )
 {
  print n;
  n++;
  }
}

provide executable permission to the awk script example file

# chmod u+x /tmp/while1.awk

Execute the script

# /tmp/while1.awk
1
2
3
4

 

References:
Learn AWK programming
man page of awk

 

Related Searches: Linux AWK tutorial. awk command examples. awk print column. awk script examples in Linux and Unix. what is awk in Unix. awk command syntax. awk examples in Linux or Unix.

Leave a Comment

Please use shortcodes <pre class=comments>your code</pre> for syntax highlighting when adding code.