10+ practical examples to learn python subprocess module

Running and spawning a new system process can be useful to system administrators who want to automate specific operating system tasks or execute a few commands within their scripts. Python provides many libraries to call external system utilities, and it interacts with the data produced. In this tutorial we will learn about one such python subprocess() module

The first library that was created is the OS module, which provides some useful tools to invoke external processes, such as os.system, os.spwan, and os.popen*. It lacks some essential functions, however, so Python developers have introduced the subprocess module which is intended to replace functions such as os.system(), os.spawnv(), the variations of popen() in the os, popen2 modules, and the commands module.

 

Content of python subprocess module

Access contents of python subprocess() module

#!/usr/bin/env python3

import subprocess

# Access contents of subprocess module
print(dir(subprocess))

Output from this script:

# python3 exec_system_commands.py
['CalledProcessError', 'CompletedProcess', 'DEVNULL', 'PIPE', 'Popen', 'STDOUT', 'SubprocessError', 'TimeoutExpired', '_PIPE_BUF', '_PLATFORM_DEFAULT_CLOSE_FDS', '_PopenSelector', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_active', '_args_from_interpreter_flags', '_cleanup', '_mswindows', '_optim_args_from_interpreter_flags', '_posixsubprocess', '_time', 'builtins', 'call', 'check_call', 'check_output', 'errno', 'getoutput', 'getstatusoutput', 'io', 'list2cmdline', 'os', 'run', 'select', 'selectors', 'signal', 'sys', 'threading', 'time', 'warnings']

 

Using python subprocess.Popen() function

• Your Python program can start other programs on your computer with the Popen() function in the built-in subprocess module.
• The P in the name of the Popen() function stands for process.
• If you have multiple instances of an application open, each of those instances is a separate process of the same program.
• For example, if you open multiple windows of your web browser at the same time, each of those windows is a different process of the web browser program

 

The general syntax to use subprocess.Popen

subprocess.Popen(cmd,shell=True/False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

In this syntax we are storing the command output (stdout) and command error (stderr) in the same variable i.e. sp

This is a very basic example where we execute "ls -ltr" using python subprocess, similar to the way one would execute it on a shell terminal

#!/usr/bin/env python3

import subprocess

# Define command as string
cmd = 'ls -ltr'

# Use shell to execute the command and store it in sp variable
sp = subprocess.Popen(cmd,shell=True)

# Store the return code in rc variable
rc=sp.wait()

# Print the content of sp variable
print(sp)

The output from this script:

# python3 exec_system_commands.py
total 308256
-rw-r--r--. 1 root root 315632268 Jan  1  2020 large_file
-rw-r--r--. 1 root root       577 Apr  1 00:00 my-own-rsa-key.pub
-rw-r--r--. 1 root root      2610 Apr  1 00:00 my-own-rsa-key
-rwxr--r--  1 root root       428 Jun  8 22:04 create_enum.py
-rwxr--r--  1 root root       176 Jun 11 06:33 check_string.py
-rw-r--r--  1 root root       525 Jul 11 19:29 exec_system_commands.py
<subprocess.Popen object at 0x7ff99a7842e8>

Did you observe the last line "<subprocess.Popen object at 0x7ff99a7842e8>", this is because we are not storing the output from the system command and instead just printing it on the console.

 

Reading stdin, stdout, and stderr with python subprocess.communicate()

In subprocess, Popen() can interact with the three channels and redirect each stream to an external file, or to a special value called PIPE. An additional method, called communicate(), is used to read from the stdout and write on the stdin.

The spawned processes can communicate with the operating system in three channels:

• Standard input (stdin)
• Standard output (stdout)
• Standard error (stderr)

The communicate() method can take input from the user and return both the standard output and the standard error, as shown in the following code snippet:

#!/usr/bin/env python3

import subprocess

# Define command as string and then split() into list format
cmd = 'ping -c 5 google.com'

# Use shell to execute the command, store the stdout and stderr in sp variable
sp = subprocess.Popen(cmd,
        shell=True,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE)

# Store the return code in rc variable
rc=sp.wait()

# Separate the output and error by communicating with sp variable.
# This is similar to Tuple where we store two values to two different variables
out,err=sp.communicate()

print('Return Code:',rc,'\n')
print('output is: \n', out)
print('error is: \n', err)

In this code if you observe we are storing the STDOUT and STDERR into the sp variable and later using communicate() method, we separate the output and error individually into two different variables

The output from this script:

# python3 exec_system_commands.py
Return Code: 0

output is:
 b b'PING google.com (142.250.67.142) 56(84) bytes of data.\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=1 ttl=115 time=70.8 ms\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=2 ttl=115 time=108 ms\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=3 ttl=115 time=103 ms\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=4 ttl=115 time=106 ms\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=5 ttl=115 time=71.4 ms\n\n--- google.com ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 24ms\nrtt min/avg/max/mdev = 70.791/91.788/107.585/16.948 ms\n'
error is:
 b''

• As you observe, the return code is 0 which means the command was executed successfully
• But the output is not clear, because by default file objects are opened in binary mode. Observe the 'b' in the starting of output which means the output is in byte code, we will get to this later.
• The error code is also empty, this is again because our command was successful.

 

Convert bytes to string

There are a couple of methods you can use to convert the byte into string format for subprocess.Popen output:

• decode("utf-8")
• universal_newlines
• text (This is supported with Python 3.7+, text was added as a more readable alias for universal_newlines)

We will use universal_newlines=True in our script

#!/usr/bin/env python3

import subprocess

# Define command as string and then split() into list format
cmd = 'ping -c 5 google.com'

# Use shell to execute the command
sp = subprocess.Popen(cmd,
        shell=True,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        universal_newlines=True)

# Separate the output and error.
# This is similar to Tuple where we store two values to two different variables
out,err=sp.communicate()

# Store the return code in rc variable
rc=sp.wait()

print('Return Code:',rc,'\n')
print('output is: \n', out)
print('error is: \n', err)

and now the script output is more readable:

# python3 exec_system_commands.py
Return Code: 0

output is:
 PING google.com (172.217.26.238) 56(84) bytes of data.
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=1 ttl=115 time=90.8 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=2 ttl=115 time=89.9 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=3 ttl=115 time=79.10 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=4 ttl=115 time=127 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=5 ttl=115 time=127 ms

--- google.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 170ms
rtt min/avg/max/mdev = 79.954/103.012/127.346/20.123 ms

error is:

 

Difference between shell=True or shell=False, which one to use?

Using subprocess.Popen with shell=True

• Whenever we use shell=True, Python is going to open a new shell to execute the provided command.
• The benefit of using this is that you can give the command in plain text format and Python will execute the same in the provided format.
• This can be useful for Linux system commands

In this python code, I am just trying to list the content of current directory using "ls -lrt" with shell=True

#!/usr/bin/env python3

import subprocess
cmd = "ls -lrt"

# Use shell to execute the command
sp = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True)

# Store the return code in rc variable
rc=sp.wait()

# Separate the output and error.
# This is similar to Tuple where we store two values to two different variables
out,err=sp.communicate()

print('Return Code: ',rc)
print('output is: \n', out)
print('error is: \n', err)

Now if we execute our script:

# python3 exec_system_commands.py
Return Code:  0
output is:
 total 308256
-rw-r--r--. 1 root root 315632268 Jan  1  2020 large_file
-rw-r--r--. 1 root root       577 Apr  1 00:00 my-own-rsa-key.pub
-rw-r--r--. 1 root root      2610 Apr  1 00:00 my-own-rsa-key
-rwxr--r--  1 root root       428 Jun  8 22:04 create_enum.py
-rwxr--r--  1 root root       176 Jun 11 06:33 check_string.py
-rw-r--r--  1 root root       475 Jul 11 16:52 exec_system_commands.py

error is:

The output is similar to what we get when we manually execute "ls -lrt" from the shell terminal. Now let me intentionally fail this script by giving some wrong command, and then the output from our script:

# python3 exec_system_commands.py
Return Code:  127
output is:
 b''
error is:
 b'/bin/sh: lsss: command not found\n'

• In our script I changed 'ls' to 'lsss' to intentionally fail the script.
• Now the script has an empty output under "output is:"
• The return code is non zero which means error
• While the error contains the error output from the provided command

 

Using subprocess.Popen with shell=False

In this section we will use shell=False with python subprocess.Popen to understand the difference with shell=True
So if you define shell=True, you are asking Python to execute your command under a new shell but with shell=False you must provide the command in List format instead of string format as we did earlier.

So for example we used below string for shell=True

cmd = 'ls -lrt'

Now to be able to use this command with shell=False, we must convert into List format, this can be done manually:

cmd = ['ls', '-lrt']

Or if it is too complex for you, use split() method (I am little lazy) which should convert the string into list

cmd = 'ls -lrt'.split()

and this should convert your command into string which can be further used with shell=False

So let us take the same example, and convert the command into list format to be able to use with Python subprocess and shell=False

#!/usr/bin/env python3

import subprocess
# Define command as string and then split() into list format
cmd = 'ls -ltr'.split()

# Check the list value of cmd
print('command in list format:',cmd,'\n')

# Use shell=False to execute the command
sp = subprocess.Popen(cmd,shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True)

# Store the return code in rc variable
rc=sp.wait()

# Separate the output and error.
# This is similar to Tuple where we store two values to two different variables
out,err=sp.communicate()

print('Return Code:',rc,'\n')
print('output is: \n', out)
print('error is: \n', err)

The output from this script:

# python3 exec_system_commands.py
command in list format: ['ls', '-ltr']

Return Code: 0

output is:
 total 308256
-rw-r--r--. 1 root root 315632268 Jan  1  2020 large_file
-rw-r--r--. 1 root root       577 Apr  1 00:00 my-own-rsa-key.pub
-rw-r--r--. 1 root root      2610 Apr  1 00:00 my-own-rsa-key
-rwxr--r--  1 root root       428 Jun  8 22:04 create_enum.py
-rwxr--r--  1 root root       176 Jun 11 06:33 check_string.py
-rw-r--r--  1 root root       623 Jul 11 17:10 exec_system_commands.py

error is:

 

When should I use shell=True or shell=False?

Now you must be wondering, when should I use which method? As ultimately both seem to be doing the same thing, one way or the other. We will understand this in our next example.

In this example script, we will try to use our system environment variable with shell=True

#!/usr/bin/env python3

import subprocess
# Define command as string and then split() into list format
cmd = 'echo $PATH'

# Check the list value of cmd
print('command in list format:',cmd,'\n')

# Use shell to execute the command
sp = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True)

# Store the return code in rc variable
rc=sp.wait()

# Separate the output and error.
# This is similar to Tuple where we store two values to two different variables
out,err=sp.communicate()

print('Return Code:',rc,'\n')
print('output is: \n', out)
print('error is: \n', err)

Output from this script is as expected, it is printing our PATH variable content

# python3 exec_system_commands.py
command in list format: echo $PATH

Return Code: 0

output is:
 /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

error is:

Now let us try to get the same using shell=False

#!/usr/bin/env python3

import subprocess
# Define command as string and then split() into list format
cmd = 'echo $PATH'.split()

# Check the list value of cmd
print('command in list format:',cmd,'\n')

# Use shell=False to execute the command
sp = subprocess.Popen(cmd,shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True)

# Store the return code in rc variable
rc=sp.wait()

# Separate the output and error.
# This is similar to Tuple where we store two values to two different variables
out,err=sp.communicate()

print('Return Code:',rc,'\n')
print('output is: \n', out)
print('error is: \n', err)

Here the script output will just print $PATH variable as a string instead of the content of $PATH variable.

# python3 exec_system_commands.py
command in list format: ['echo', '$PATH']

Return Code: 0

output is:
 $PATH

error is:

• So whenever you are planning to use any system variable, you must use shell=True as shell=False will not be able to interpret the system variable and would consider it as just another string.
• Also one of the demerit of shell=True method is that, every time you use this method, Python needs to open a new shell terminal to execute the command, get the status and then close the terminal which takes up some additional time while with shell=False, you can get comparatively faster results.

So what did we learned? To summarise:

• If shell=True then your command is considered as a string (similar to your os command)
• If shell=False then your command is considered as a List
• To convert a string with shell=False, you can use split() method for easier operation
• shell=False doesn't recognise OS system environment variables

 

Practical Example

Let us take a practical example from real time scenario. In this python script we aim to get the list of failed services. In RHEL 7/8 we use "systemctl --failed" to get the list of failed services

# systemctl --failed
  UNIT               LOAD   ACTIVE SUB    DESCRIPTION
● kdump.service      loaded failed failed Crash recovery kernel arming
● nfs-server.service loaded failed failed NFS server and services

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Now here I only wish to get the service name, instead of complete output.

Below is our sample script:

  1 #!/usr/bin/env python3
  2
  3 import subprocess
  4
  5 # Define command as string and then split() into list format
  6 cmd = 'systemctl --failed'.split()
  7
  8 # Check the list value of cmd
  9 print('command in list format:',cmd)
 10
 11 # Use shell=False to execute the command
 12 sp = subprocess.Popen(cmd,shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True)
 13
 14 # Store the return code in rc variable
 15 rc=sp.wait()
 16
 17 # Separate the output and error by communicating with sp variable.
 18 # This is similar to Tuple where we store two values to two different variables
 19 out,err=sp.communicate()
 20
 21 if rc == 0:
 22     for line in out.splitlines():
 23         if "failed" in line:
 24             print(line.split()[1])
 25 else:
 26     print('The command returned an error: ',err)

Here,
Line 3: We import subprocess module
Line 6: We define the command variable and use split() to use it as a List
Line 9: Print the command in list format, just to be sure that split() worked as expected
Line 12: The subprocess.Popen command to execute the command with shell=False. Store the output and error, both into the same variable. Return the output with newline char instead of byte code
Line 15: This may not be required here but it is a good practice to use wait() as sometimes the subprocess may take some time to execute a command for example some SSH process, in such case wait() will make sure the subprocess command is executed successfully and the return code is stored in wait()
Line 19: We need communicate() to get the output and error value from subprocess.Popen and store it in out and err variable respectively
Line 21: If return code is 0, i.e. if the command execution was success
Line 22: Use for loop and run on each line. Here we will use python splitlines() method which splits the string based on the lines. It breaks the string at line boundaries and returns a list of splitted strings.
Line 23: Use "in" operator to search for "failed" string in "line"
Line 24: If "failed" is found in the "line"
Line 25: The split the found line into list and then we print the content of string with "1" index number
Line 25: In case the command fails to execute
Line 26: Print the provided error message from err variable which we stored using communicate()

Output from this script:

# python3 exec_system_commands.py
command in list format: ['systemctl', '--failed']
kdump.service
nfs-server.service

So we were able to print only the failed service using python subprocess module

 

Using python subprocess.call() function

• The subprocess module provides another function that makes process spawning a safer operation than using Popen().
• The subprocess.call() function waits for the called command/program to finish reading the output.
• It supports the same arguments as the Popen() constructor, such as shell, executable, and cwd, but this time, your script will wait for the program to complete and populate the return code without the need to communicate().
• You can fetch the exit status using Popen.returncode
• To suppress stdout or stderr, supply a value of subprocess.DEVNULL which indicates that the special file os.devnull will be used.

• In this sample python code, we will check the availability of eth0 interface using "ip link show eth0"
• The output of the command will be stored in /tmp/dataFile and if eth0 is available we print "Yes, eth0 is available on this server"
• For error condition also, the output of "ip link show eth0" will be written in /tmp/dataFile, you can choose to use another file for error output.

#!/usr/bin/env python3

import subprocess

# Define command as string and then split() into list format
cmd = 'ip link show eth0'.split()

# Check the list value of cmd
print('command in list format:',cmd)

# Open the /tmp/dataFile and use "w" to write into the file
myfile = open("/tmp/dataFile", "w")

# Use shell=False to execute the command
is_eth0_present = subprocess.call(cmd, shell=False, stdout=myfile, stderr=subprocess.STDOUT)

if is_eth0_present == 0:
    print('Yes, eth0 is available on this server')
else:
    print('No, eth0 is not available on this server')

myfile.close()

The output from this script (when eth0 is available)

# python3 exec_system_commands.py
command in list format: ['ip', 'link', 'show', 'eth0']
Yes, eth0 is available on this server

Content of /tmp/dataFile

# cat /tmp/dataFile
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:5a:d3:83 brd ff:ff:ff:ff:ff:ff

The output from this script (when eth0 is NOT available)

# python3 exec_system_commands.py
command in list format: ['ip', 'link', 'show', 'eth0']
No, eth0 is not available on this server

Content of /tmp/dataFile

# cat /tmp/dataFile
Device "eth0" does not exist.

 

Using python subprocess.check_call() function

• This is another function which is part of subprocess module which can run command with arguments.
• check_call will wait for command execution to complete.
• If the execution is successful then the function will return zero then return, otherwise raise CalledProcessError.
• The CalledProcessError object will have the return code in the returncode attribute.

I will try to use subprocess.check_now just to print the command execution output:

#!/usr/bin/env python3

import subprocess

# Define command as string and then split() into list format
cmd = 'ping -c2 google.com'.split()

# Check the list value of cmd
print('command in list format:',cmd)

sp = subprocess.check_call(cmd, shell=False)
print(sp)

The output from this script (when returncode is zero):

# python3 exec_system_commands.py
command in list format: ['ping', '-c2', 'google.com']
PING google.com (172.217.26.238) 56(84) bytes of data.
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=1 ttl=115 time=579 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=2 ttl=115 time=90.1 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 68ms
rtt min/avg/max/mdev = 90.125/334.576/579.028/244.452 ms
0

The output from this script (when returncode is non-zero):

# python3 exec_system_commands.py
command in list format: ['ping', '-c2', 'google.co12m']
ping: google.co12m: Name or service not known
Traceback (most recent call last):
  File "exec_system_commands.py", line 11, in <module>
    sp = subprocess.check_call(cmd, shell=False)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ping', '-c2', 'google.co12m']' returned non-zero exit status 2.

As you see we get subprocess.CalledProcessError for non-zero return code.

So we should use try and except for subprocess.check_now as used in the below code:

#!/usr/bin/env python3

import subprocess

# Define command as string and then split() into list format
cmd = 'ping -c2 google.c12om'.split()

# Check the list value of cmd
print('command in list format:',cmd)

try:
    sp = subprocess.check_call(cmd, shell=False)
except subprocess.CalledProcessError:
    print('Host is not reachable')

The output from this script (when returncode is non-zero):

# python3 exec_system_commands.py
command in list format: ['ping', '-c2', 'google.c12om']
ping: google.c12om: Name or service not known
Host is not reachable

So now this time we don't get CalledProcessError, instead the proper stderr output along with out print statement is printed on the console

 

Using subprocess.run() function

• The subprocess.run() function was added in Python 3.5
• Wait for command to complete, then return a subprocess.CompletedProcess instance.
• The full function signature is largely the same as that of the Popen constructor - most of the arguments to this function are passed through to that interface. (timeout, input, check, and capture_output are not.)
• If capture_output is true, stdout and stderr will be captured.
• When used, the internal Popen object is automatically created with stdout=PIPE and stderr=PIPE.
• The stdout and stderr arguments may not be supplied at the same time as capture_output.
• If you wish to capture and combine both streams into one, use stdout=PIPE and stderr=STDOUT instead of capture_output.

In this sample python code we will try to check internet connectivity using subprocess.run()

#!/usr/bin/env python3

import subprocess

# Define command as string and then split() into list format
cmd = 'ping -c2 google.c12om'.split()

# Check the list value of cmd
print('command in list format:',cmd)

sp = subprocess.run(cmd, shell=False, check=True, capture_output=True, text=True)
print("stdout: ", sp.stdout)
print("stderr: ", sp.stderr)

For successful output:

# python3 /tmp/script.py
command in list format: ['ping', '-c2', 'google.com']
stdout:  PING google.com (172.217.160.142) 56(84) bytes of data.
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=1 ttl=115 time=199 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=2 ttl=115 time=80.8 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 80.756/139.980/199.204/59.224 ms

stderr:

For failed output i.e. when the command returns non-zero exit code:

# python3 /tmp/script.py
command in list format: ['ping', '-c2', 'google.c12om']
Traceback (most recent call last):
  File "/tmp/script.py", line 11, in <module>
    sp = subprocess.run(cmd, shell=False, check=True, capture_output=True, text=True)
  File "/usr/local/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ping', '-c2', 'google.c12om']' returned non-zero exit status 2.

You can use check=false if you don't want to print any ERROR on the console, in such case the output will be:

# python3 exec_system_commands.py
command in list format: ['ping', '-c2', 'google.c12om']
ping: google.c12om: Name or service not known
stdout:  b''
stderr:  None

 

Using python subprocess.check_output() function

• The subprocess.check_output() is similar to subprocess.run(check=True)
• By default, this function will return the data as encoded bytes so you can use text=True or universal_newlines=True to get string value as output
• This function will run command with arguments and return its output.
• If the return code was non-zero it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute and any output in the output attribute.

#!/usr/bin/env python3

import subprocess

# Define command as string and then split() into list format
cmd = 'ping -c2 google.c2om'.split()

# Check the list value of cmd
print('command in list format:',cmd)

try:
    output = subprocess.check_output(cmd, shell=False, stderr=subprocess.STDOUT, universal_newlines=True)
except subprocess.CalledProcessError as e:
    print('output before error: ',e.output)
    print('Return Code: ',e.returncode)

Output from the script for non-zero exit code:

# python3 exec_system_commands.py
command in list format: ['ping', '-c2', 'google.c2om']
output before error:  ping: google.c2om: Name or service not known

Return Code:  2

 

Which subprocess module function should I use?

• The functions run(), call(), check_call(), and check_output() are wrappers around the Popen class.
• Using Popen directly gives more control over how the command is run, and how its input and output streams are processed.

 

Wait for the command to complete

• Use subprocess.call or subprocess.run to run the command described by args. Wait for command to complete, then return the returncode attribute.
• Use subprocess.Popen with wait() to wait for the command to complete

Here we use subprocess.call to check internet connectivity and then print "Something"

#!/usr/bin/env python3

import subprocess
cmd='ping -c5 google.com'.split()
sp = subprocess.call(cmd, shell=False)
print('Something')

Output from this script:

# python3 exec_system_commands.py
PING google.com (172.217.160.142) 56(84) bytes of data.
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=1 ttl=115 time=102 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=2 ttl=115 time=325 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=3 ttl=115 time=85.4 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=4 ttl=115 time=249 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=5 ttl=115 time=81.0 ms

--- google.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 94ms
rtt min/avg/max/mdev = 81.022/168.509/324.751/99.872 ms
Something

So we know subprocess.call is blocking the execution of the code until cmd is executed. Now we do the same execution using Popen (without wait()).

#!/usr/bin/env python3

import subprocess
cmd='ping -c5 google.com'.split()
sp = subprocess.Popen(cmd, shell=False)
print('Something')

If you observe, "Something" was printed immediately while ping was still in process, so call() and run() are non-blocking function.

# python3 exec_system_commands.py
Something
[root@client scripts]# PING google.com (172.217.160.142) 56(84) bytes of data.
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=1 ttl=115 time=107 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=2 ttl=115 time=136 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=4 ttl=115 time=122 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=5 ttl=115 time=121 ms

--- google.com ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 396ms
rtt min/avg/max/mdev = 106.956/121.745/136.476/10.442 ms

 

Capture output from command

• The standard input and output channels for the process started by call() are bound to the parent’s input and output.
• That means the calling program cannot capture the output of the command. Use check_output() to capture the output for later processing.
• To prevent error messages from commands run through check_output() from being written to the console, set the stderr parameter to the constant STDOUT.

 

Error Handling

• The call() function returns the program's exit code, which is an integer that has a program-defined meaning, which is usually used to know whether the program has succeeded or failed.
• The check_call() function works like call(), except that the exit code is checked, and if it indicates an error happened, then a CalledProcessError exception is raised.
• The check_output() function returns whatever text the program printed and raises an exception if the program exited with a non-zero exit code. The raised exception has an attribute called output that contains the text output of the program. So, even if the program exited with an error code, we could still get the output if we want it.

 

Conclusion

In this tutorial we learned about different functions available with python subprocess module and their usage with different examples. As a Linux administrator coming from shell background, I was using mostly os module which now I have switched to subprocess module as this is the preferred solution to execute system commands and child processes.

In most cases you will end up using subprocess.Popen() or subprocess.run() as they tend to cover most of the basic scenarios related to execution and checking return status but I have tried to give you a comprehensive overview of possible use cases and the recommended function so you can make an informed decision.

Lastly I hope this tutorial on python subprocess module in our programming language section was helpful. So, let me know your suggestions and feedback using the comment section.

 

References

I have used below external references for this tutorial guide
Complete Python Scripting for Automation
Hands-On Enterprise Automation with Python.
docs.python.org: subprocess module