In this article we will learn about AWK command used in Linux and Unix. I will share some basic awk examples in one liner commands, awk script examples and other scenarios in Linux or Unix environment.
What is AWK in Linux or Unix?
AWK is an interpreted programming language designed for text processing and report generation. It is typically used for data manipulation, such as searching for items within data, performing arithmetic operations, and restructuring raw data for generating reports in most Unix-like operating systems.
Programs in awk are different from programs in most other languages, because awk programs are data driven (i.e., you describe the data you want to work with and then what to do when you find it).
When you run awk, you specify an awk program that tells awk what to do. The program consists of a series of rules. Syntactically, a rule consists of a pattern followed by an action. The action is enclosed in braces to separate it from the pattern.
Types of AWK
The AWK language was originally implemented as an AWK utility on Unix. Today, most Linux distributions provide GNU implementation of AWK (GAWK), and a symlink for AWK is created from the original GAWK binary.
- AWK
- NAWK
- GAWK
AWK command syntax for action and pattern structure
AWK programs are sequences of one or more patterns followed by action statements. These action-pattern statements are separated by newlines. Both actions (AWK commands) and patterns (search patterns) are optional, hence we use { } to distinguish them: Below is the awk command syntax for action and pattern structure:
/ search pattern / { action / awk-commands }/ search pattern / { action / awk-commands }
AWK command syntax for pattern-only statements
The awk command syntax with a pattern only is as follows:
awk '/ pattern /' inputfilename
In the given awk command examples, all lines of the passwd file are processed, and those that contain the mail pattern are printed:
# awk '/mail/' /tmp/passwd mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
AWK command syntax for action-only statements
The awk command syntax with an action only is as follows:
awk '{ action statements / awk-commands }' inputfilenames
In the given awk command examples, all employee names are printed with awk print column on the screen as $2, representing the second column of each input line.
# awk '{print $2}' /tmp/userdata.txt
Name
Deepak
Rahul
Amit
Sumit
1. AWK examples to print each input line or record
We can print each input record of the input file in multiple ways. Lets see some more awk examples.
# awk '//' /tmp/userdata.txt
# awk '{print}' /tmp/userdata.txt
# awk '{print $0}' /tmp/userdata.txt
# awk '//{print}' /tmp/userdata.txt
All the given awk command examples will produce the same result by printing all the input records of the input file /tmp/userdata.txt.
id Name Age username 1 Deepak 31 deepak 2 Rahul 32 rahul 3 Amit 33 amit 4 Sumit 33 sumit
2. Using the BEGIN and END blocks construct
AWK contains two special keywords, BEGIN and END, for patterns where an action is required. Both of them are optional and are used without slashes. They don't match any input lines.
Below awk examples shows usage of the BEGIN and END blocks:
# awk 'BEGIN { print "==Employee Info==" } <-- begin block > { print } <-- body block > END { print "==ends here==" }' /tmp/userdata.txt <-- end block ==Employee Info== id Name Age username 1 Deepak 31 deepak 2 Rahul 32 rahul 3 Amit 33 amit 4 Sumit 33 sumit ==ends here==
3. Running awk from source file
When AWK programs are long, it is more convenient to put them in a separate file. Putting AWK programs in a file reduces errors and retyping.
The -f option tells the AWK utility to read the AWK commands from source_file. Any filename can be used in place of source_file. Let's take some awk examples, we can create an awk script example cmd.awk text file containing the AWK commands, as follows:
# cat cmd.awk
BEGIN { print "=== Emp Info ===" }
{ print }
Now, we instruct AWK to read the commands from the cmd.awk file and perform the given actions:
# awk -f cmd.awk userdata.txt === Emp Info === id Name Age username 1 Deepak 31 deepak 2 Rahul 32 rahul 3 Amit 33 amit 4 Sumit 33 sumit
4. AWK script examples for programming
We can write self-contained AWK scripts to execute AWK commands, like we have with shell scripts to execute shell commands. We create the AWK scripts by using #!, followed by the absolute path of the AWK interpreter and the -f optional argument.
Below is one small awk script example:
# cat cmd.awk
# awk script Examples for AWK Tutorial
#!/bin/awk -f
BEGIN { print "=== Emp Info ===" }
{ print }
Give this awk script example file executable permissions
# chmod u+x /tmp/cmd.awk
Next simply run ./cmd.awk userdata.txt at the shell and the system will run AWK as if you had typed awk -f cmd.awk userdata.txt:
# ./cmd.awk userdata.txt
=== Emp Info ===
id Name Age username
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
4 Sumit 33 sumit
5. Comments in AWK
A comment is some text that is included in a program for documentation or human information. It is not an executable part of the program. It explains what a program does and how it does it. We will create an awk script examples with some comments.
Lets see some awk script examples, we can put the following in cmd.awk:
# cat cmd.awk
# awk script Examples for AWK Tutorial
#!/bin/awk -f
# Info : This program displays the employees information
# Date : 22 Jan 2020
# Version : 1.0
# This is BEGIN block comment
BEGIN { print "=== Emp Info ===" }
# Body block comment
{ print }
# End block comment
END { print "=== Information Database ends here ===" }
Next execute the awk script example scrip and here is the output:
# ./cmd.awk userdata.txt === Emp Info === id Name Age username 1 Deepak 31 deepak 2 Rahul 32 rahul 3 Amit 33 amit 4 Sumit 33 sumit === Information Database ends here ===
6. AWK match pattern and print without action statement
In this awk examples, the program has a pattern, but we don't specify any action statements.
# awk '/rahul/' /tmp/userdata.txt
2 Rahul 32 rahul
In this case, AWK selects only those input lines that contain the rahul pattern/string in them. When we don't specify any action, AWK assumes the action is to print the whole line.
7. AWK print column without specifying any pattern
In this awk examples, we will not include any pattern but instead using awk print column. The given AWK command prints the first column ($1) and second column ($2) of each input line that is separated by a space (the output field separator, indicated by a comma):
# awk '{ print $1, $2 }' /tmp/userdata.txt
id Name
1 Deepak
2 Rahul
3 Amit
4 Sumit
8. AWK print column with matching patterns
In this awk examples, we will include both actions and patterns and with awk print column. The given AWK command prints the first column ($1) separated by tab (specifying \t as the output separator) with the second ($2) and third column ($3) of input lines, which contain the "deepak" string in them:
# awk '/deepak/{ print $1 "\t" $2 "\t" $3}' /tmp/userdata.txt
1 Deepak 31
9. AWK print column in any order with custom text
In this awk examples, with awk print column in different orders. Here, in the action statement, we put some text in between the row content
# awk '{ print $1 " is mapped to " $2 ", with age " $3 ", and username " $4 }' /tmp/userdata.txt
id is mapped to Name, with age Age, and username username
1 is mapped to Deepak, with age 31, and username deepak
2 is mapped to Rahul, with age 32, and username rahul
3 is mapped to Amit, with age 33, and username amit
4 is mapped to Sumit, with age 33, and username sumit
We can also add a pattern match for the action and then with awk print column. Here we search for deepak string and then print the text with values from different columns.
# awk '/deepak/{ print $1 " is mapped to " $2 ", with age " $3 ", and username " $4 }' /tmp/userdata.txt 1 is mapped to Deepak, with age 31, and username deepak
10. Printing the number of columns in a line
AWK has built-in variables to count and store the number of fields in the current input line, for example, NF. So, in the given awk examples, we will print the number of the columns for each input line, followed by the first column and the last column (accessed using NF):
# awk '{ print NF, $1, $NF }' /tmp/userdata.txt 4 id username 4 1 deepak 4 2 rahul 4 3 amit 4 4 sumit
Since we have 4 columns so NF prints 4 and since we are printing variable value of $NF which is 4, so awk prints fourth column content
We can put text in between the fields and then awk print column
# awk '{ print " No of fields: " NF ", and the last field contains "$NF }' /tmp/userdata.txt No of fields: 4, and the last field contains username No of fields: 4, and the last field contains deepak No of fields: 4, and the last field contains rahul No of fields: 4, and the last field contains amit No of fields: 4, and the last field contains sumit
11. Deleting empty lines using NF
We can print all the lines with at least 1 field using NF > 0. This is the easiest method to remove empty lines from the file using AWK:
Here I have added an empty line to my input file after 3rd row
# awk '//' /tmp/userdata.txt
id Name Age username
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
4 Sumit 33 sumit
To remove empty line using awk with NF
# awk 'NF > 0 { print }' /tmp/userdata.txt
id Name Age username
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
4 Sumit 33 sumit
12. Printing line numbers in the output
AWK has a built-in variable known as NR. It counts the number of input lines read so far. We can use NR to prefix $0 in the print statement to display the line numbers of each line that has been processed:
# awk '{ print NR, $0 }' /tmp/userdata.txt 1 id Name Age username 2 1 Deepak 31 deepak 3 2 Rahul 32 rahul 4 3 Amit 33 amit 5 6 4 Sumit 33 sumit
13. AWK sum column by counting the numbers of lines in a file using NR
In our next awk examples, we will count the number of lines in a file using NR and use awk sum column. As NR stores the current input line number, we need to process all the lines in a file for awk sum column. We also don't want to print the line numbers for each line, as our requirement is to just awk sum column. Since the END block is executed after processing the input line is done, we will print NR in the END block to print the total number of lines in the file using awk sum column:
# awk ' END { print "The total number of lines in file are : " NR } ' /tmp/userdata.txt The total number of lines in file are : 6
14. Printing numbered lines exclusively from the file
We know NR contains the line number of the current input line. You can easily print any line selectively, by matching the line number with the current input line number stored in NR, as follows in below awk examples:
# awk 'NR==2 { print NR, $0 }' /tmp/userdata.txt 2 1 Deepak 31 deepak
15. AWK print row with even-numbered lines in a file
Using NR, we can easily print even-numbered files by specifying expressions (divide each line number by 2 and find the remainder) in pattern space, as shown in the following awk examples:
# awk 'NR % 2 == 0{ print NR, $0 }' /tmp/userdata.txt 2 1 Deepak 31 deepak 4 3 Amit 33 amit 6 4 Sumit 33 sumit
16. AWK print row with odd-numbered lines in a file
Similarly, we can use NR with awk print column to list odd-numbered lines in a file using NR, by performing basic arithmetic operations in the pattern space:
# awk 'NR % 2 == 1{ print NR, $0 }' /tmp/userdata.txt 1 id Name Age username 3 2 Rahul 32 rahul 5
17. AWK print row using the range operator (,) and NR
We can combine the range operator (,) and NR to print a group of lines from a file based on their line numbers. The next awk examples displays the lines 2 to 4 from the userdata.txt file:
# awk 'NR==2, NR==4 { print NR, $0 }' /tmp/userdata.txt 2 1 Deepak 31 deepak 3 2 Rahul 32 rahul 4 3 Amit 33 amit
18. AWK print row using the range operator and patterns
We can also combine the range operator (,) and string in pattern space to print a group of lines in a file starting from the first pattern, up to the second pattern.
# awk '/deepak/,/amit/ { print NR, $0 }' /tmp/userdata.txt 2 1 Deepak 31 deepak 3 2 Rahul 32 rahul 4 3 Amit 33 amit
19. Print selection using the match operator (~)
The match operator (~) is used for matching a pattern in a specified field in the input line of a file. In the next awk examples, we will select and print all lines containing 'a' in the second column of the input line, as follows:
# awk ' $2 ~ /a/ {print NR, $0 } ' /tmp/userdata.txt 1 id Name Age username 2 1 Deepak 31 deepak 3 2 Rahul 32 rahul
20. Print selection using the match operator (~) and anchor (^)
The caret (^) in regular expressions (also known as anchor) is used to match at the beginning of a line. In the next awk examples, we combine it with the match operator (~) to print all the lines in which the second field begins with the 'A' character, as follows
# awk ' $2 ~ /^A/ {print NR, $0 } ' /tmp/userdata.txt 4 3 Amit 33 amit
21. Print selection using the match operator (~) and character classes ([ ])
The character classes, [ ], in regular expressions are used to match a single character out of those specified within square brackets. Here, we combine the match operator (~) with character classes (/^[AD]/) to print all the lines in which the second field begins with the 'A' or 'D' character, as follows:
# awk ' $2 ~ /^[AD]/ {print NR, $0 } ' /tmp/userdata.txt 2 1 Deepak 31 deepak 4 3 Amit 33 amit
22. Print selection using the match operator (~) and anchor ($):
The dollar sign ($) in regular expression (also known as anchor) is used to match at the end of a line. In the next awk examples, we combine it with the match operator (~) to print all the lines in the second field end with the 'k' character, as follows:
# awk ' $2 ~ /k$/ {print NR, $0 } ' /tmp/userdata.txt 2 1 Deepak 31 deepak
23. Print selection by numeric comparison
You can use relation operators (==, >=, <=, >, <, !=) for performing numeric comparison. Here, we perform a numeric greater than or equal (>=) to print the lines that have the 32 or higher value in the third field, as follows:
# awk ' $3 >= 32 {print NR, $0 } ' /tmp/userdata.txt 1 id Name Age username 3 2 Rahul 32 rahul 4 3 Amit 33 amit 6 4 Sumit 33 sumit
24. Print selection by text content/string matching in a field
Besides numeric matches, we can use string matches to find the lines containing a particular string in a field. String content for matches should be given in double quotes as a string
In our next awk examples, we print all the lines that contain "deepak" in the fourth field ($4), as follows:
# awk ' $4 == "deepak" {print NR, $0 } ' /tmp/userdata.txt 2 1 Deepak 31 deepak
25. Print selection by combining patterns
You can combine patterns with parentheses and logical operators, &&, ||, and!, which stand for AND, OR, and NOT. Here, we print the lines containing a value greater than or equal to 30 in the third field and a value less than or equal to 32 in the third field.
# awk ' $3 >= 30 && $3 <= 32 {print NR, $0 } ' /tmp/userdata.txt 2 1 Deepak 31 deepak 3 2 Rahul 32 rahul
26. Using AWK BEGIN to print headings
The AWK BEGIN block can be used for printing headings, initialising variables, performing calculations, or any other task that you want to be executed before AWK starts processing the lines in the input file.
# /tmp/header.awk /tmp/userdata.txt Field1 Field2 Field3 Field4 id Name Age username 1 Deepak 31 deepak 2 Rahul 32 rahul 3 Amit 33 amit 4 Sumit 33 sumit
27. Using AWK END to print the last input line
The AWK END block is executed after the processing of the last line of the last file is completed, and $0 stores the value of each input line processed, but its value is not retained in the END block. The following is one way to print the last input line:
# awk '{ last = $0 } END { print last }' /tmp/userdata.txt 4 Sumit 33 sumit
And to print the total number of lines in a file we use NR, because it retains its value in the END block, as follows:
# awk 'END { print "Total no of lines in file: ", NR}' /tmp/userdata.txt Total no of lines in file: 6
28. Count the number of characters using length function
By default, the length function stores the count of the number of characters in the input line. In the next example, we will prefix each line with the number of characters in it using the length function, as follows:
# awk '{print length, $0}' /tmp/passwd 31 root:x:0:0:root:/root:/bin/bash 32 bin:x:1:1:bin:/bin:/sbin/nologin 39 daemon:x:2:2:daemon:/sbin:/sbin/nologin 36 adm:x:3:4:adm:/var/adm:/sbin/nologin 40 lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin 31 sync:x:5:0:sync:/sbin:/bin/sync 44 shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown 32 halt:x:7:0:halt:/sbin:/sbin/halt 46 mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
29. Changing the field separator using FS
The fields in the awk examples we have discussed so far have been separated by space characters. The default behaviour of FS is any number of space or tab characters; we can change it to regular expressions or any single or multiple characters using the FS variable or the -F option.
In this, we use the /etc/passwd file of Linux, which delimits fields with colons (:). So, we change the input of FS to a colon before reading any data from the file, and print the list of usernames, which is stored in the first field of the file, as follows:
# awk 'BEGIN { FS = ":"} {print $1}' /etc/passwd
root
bin
daemon
adm
lp
sync
shutdown
30. Control structures (Example: AWK while loop)
AWK supports control (flow) statements, which can be used to change the order of the execution of commands within an AWK program. Different constructs, such as the if...else, while, and for control structures are supported by AWK. In addition, the break and continue statements work in combination with the control structures to modify the order of execution of commands.
Let's try a basic example of a while loop to print a list of numbers under 5:
# awk 'BEGIN{ n=1; while (n < 5 ) { print n; n++; } }'
1
2
3
4
Alternatively you can also create an awk script examples
# cat /tmp/while1.awk
# awk script Examples for AWK Tutorial
#!/bin/awk -f
BEGIN { n=1
while ( n < 5 )
{
print n;
n++;
}
}
provide executable permission to the awk script example file
# chmod u+x /tmp/while1.awk
Execute the script
# /tmp/while1.awk 1 2 3 4
References:
Learn AWK programming
man page of awk
Related Searches: Linux AWK tutorial. awk command examples. awk print column. awk script examples in Linux and Unix. what is awk in Unix. awk command syntax. awk examples in Linux or Unix.
please provide sed command
Thanks for your feedback, we will try to create one tutorial on sed also.
And try to do on facl also, thanks for this content it seems hard work.