Table of Contents
Running and spawning a new system process can be useful to system administrators who want to automate specific operating system tasks or execute a few commands within their scripts. Python provides many libraries to call external system utilities, and it interacts with the data produced. In this tutorial we will learn about one such python subprocess()
module
The first library that was created is the OS module, which provides some useful tools to invoke external processes, such as os.system
, os.spwan
, and os.popen*
. It lacks some essential functions, however, so Python developers have introduced the subprocess
module which is intended to replace functions such as os.system()
, os.spawnv()
, the variations of popen()
in the os
, popen2
modules, and the commands
module.
Content of python subprocess module
Access contents of python subprocess()
module
#!/usr/bin/env python3 import subprocess # Access contents of subprocess module print(dir(subprocess))
Output from this script:
# python3 exec_system_commands.py
['CalledProcessError', 'CompletedProcess', 'DEVNULL', 'PIPE', 'Popen', 'STDOUT', 'SubprocessError', 'TimeoutExpired', '_PIPE_BUF', '_PLATFORM_DEFAULT_CLOSE_FDS', '_PopenSelector', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_active', '_args_from_interpreter_flags', '_cleanup', '_mswindows', '_optim_args_from_interpreter_flags', '_posixsubprocess', '_time', 'builtins', 'call', 'check_call', 'check_output', 'errno', 'getoutput', 'getstatusoutput', 'io', 'list2cmdline', 'os', 'run', 'select', 'selectors', 'signal', 'sys', 'threading', 'time', 'warnings']
Using python subprocess.Popen() function
- Your Python program can start other programs on your computer with the
Popen()
function in the built-insubprocess
module. - The P in the name of the
Popen()
function stands for process. - If you have multiple instances of an application open, each of those instances is a separate process of the same program.
- For example, if you open multiple windows of your web browser at the same time, each of those windows is a different process of the web browser program
Arguments | Meaning |
---|---|
args | A string, or a sequence of program arguments. |
bufsize | It is supplied as the buffering argument to the open() function when creating the stdin/stdout/stderr pipe file objects. |
executable | A replacement program to execute. |
stdin, stdout, stderr | These specify the executed program's standard input, standard output, and standard error file handles, respectively. |
shell | If True , the command will be executed through the shell (the default is False ). In Linux, this means calling the /bin/sh before running the child process. |
cwd | Sets the current directory before the child is executed. |
env | Defines the environmental variables for the new process. |
The general syntax to use subprocess.Popen
subprocess.Popen(cmd,shell=True/False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
In this syntax we are storing the command output (stdout
) and command error (stderr
) in the same variable i.e. sp
This is a very basic example where we execute "ls -ltr
" using python subprocess, similar to the way one would execute it on a shell terminal
#!/usr/bin/env python3 import subprocess # Define command as string cmd = 'ls -ltr' # Use shell to execute the command and store it in sp variable sp = subprocess.Popen(cmd,shell=True) # Store the return code in rc variable rc=sp.wait() # Print the content of sp variable print(sp)
The output from this script:
# python3 exec_system_commands.py
total 308256
-rw-r--r--. 1 root root 315632268 Jan 1 2020 large_file
-rw-r--r--. 1 root root 577 Apr 1 00:00 my-own-rsa-key.pub
-rw-r--r--. 1 root root 2610 Apr 1 00:00 my-own-rsa-key
-rwxr--r-- 1 root root 428 Jun 8 22:04 create_enum.py
-rwxr--r-- 1 root root 176 Jun 11 06:33 check_string.py
-rw-r--r-- 1 root root 525 Jul 11 19:29 exec_system_commands.py
<subprocess.Popen object at 0x7ff99a7842e8>
Did you observe the last line "<subprocess.Popen object at 0x7ff99a7842e8>
", this is because we are not storing the output from the system command and instead just printing it on the console.
Reading stdin, stdout, and stderr with python subprocess.communicate()
In subprocess
, Popen()
can interact with the three channels and redirect each stream to an external file, or to a special value called PIPE
. An additional method, called communicate()
, is used to read from the stdout
and write on the stdin
.
The spawned processes can communicate with the operating system in three channels:
- Standard input (
stdin
) - Standard output (
stdout
) - Standard error (
stderr
)
The communicate()
method can take input from the user and return both the standard output and the standard error, as shown in the following code snippet:
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'ping -c 5 google.com' # Use shell to execute the command, store the stdout and stderr in sp variable sp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # Store the return code in rc variable rc=sp.wait() # Separate the output and error by communicating with sp variable. # This is similar to Tuple where we store two values to two different variables out,err=sp.communicate() print('Return Code:',rc,'\n') print('output is: \n', out) print('error is: \n', err)
In this code if you observe we are storing the STDOUT
and STDERR
into the sp
variable and later using communicate()
method, we separate the output
and error
individually into two different variables
The output from this script:
# python3 exec_system_commands.py
Return Code: 0
output is:
b b'PING google.com (142.250.67.142) 56(84) bytes of data.\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=1 ttl=115 time=70.8 ms\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=2 ttl=115 time=108 ms\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=3 ttl=115 time=103 ms\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=4 ttl=115 time=106 ms\n64 bytes from bom12s06-in-f14.1e100.net (142.250.67.142): icmp_seq=5 ttl=115 time=71.4 ms\n\n--- google.com ping statistics ---\n5 packets transmitted, 5 received, 0% packet loss, time 24ms\nrtt min/avg/max/mdev = 70.791/91.788/107.585/16.948 ms\n'
error is:
b''
- As you observe, the return code is 0 which means the command was executed successfully
- But the output is not clear, because by default file objects are opened in binary mode. Observe the
'b
' in the starting of output which means the output is in byte code, we will get to this later. - The error code is also empty, this is again because our command was successful.
Convert bytes to string
There are a couple of methods you can use to convert the byte into string format for subprocess.Popen
output:
- decode("utf-8")
- universal_newlines
- text (This is supported with Python 3.7+, text was added as a more readable alias for
universal_newlines
)
We will use universal_newlines=True
in our script
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'ping -c 5 google.com' # Use shell to execute the command sp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) # Separate the output and error. # This is similar to Tuple where we store two values to two different variables out,err=sp.communicate() # Store the return code in rc variable rc=sp.wait() print('Return Code:',rc,'\n') print('output is: \n', out) print('error is: \n', err)
and now the script output is more readable:
# python3 exec_system_commands.py
Return Code: 0
output is:
PING google.com (172.217.26.238) 56(84) bytes of data.
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=1 ttl=115 time=90.8 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=2 ttl=115 time=89.9 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=3 ttl=115 time=79.10 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=4 ttl=115 time=127 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=5 ttl=115 time=127 ms
--- google.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 170ms
rtt min/avg/max/mdev = 79.954/103.012/127.346/20.123 ms
error is:
Difference between shell=True or shell=False, which one to use?
Using subprocess.Popen with shell=True
- Whenever we use
shell=True
, Python is going to open a new shell to execute the provided command. - The benefit of using this is that you can give the command in plain text format and Python will execute the same in the provided format.
- This can be useful for Linux system commands
subprocess
is /bin/sh
. If you're using other shells, like tch
or csh
, you can define them in the executable argument.In this python code, I am just trying to list the content of current directory using "ls -lrt
" with shell=True
#!/usr/bin/env python3 import subprocess cmd = "ls -lrt" # Use shell to execute the command sp = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True) # Store the return code in rc variable rc=sp.wait() # Separate the output and error. # This is similar to Tuple where we store two values to two different variables out,err=sp.communicate() print('Return Code: ',rc) print('output is: \n', out) print('error is: \n', err)
Now if we execute our script:
# python3 exec_system_commands.py
Return Code: 0
output is:
total 308256
-rw-r--r--. 1 root root 315632268 Jan 1 2020 large_file
-rw-r--r--. 1 root root 577 Apr 1 00:00 my-own-rsa-key.pub
-rw-r--r--. 1 root root 2610 Apr 1 00:00 my-own-rsa-key
-rwxr--r-- 1 root root 428 Jun 8 22:04 create_enum.py
-rwxr--r-- 1 root root 176 Jun 11 06:33 check_string.py
-rw-r--r-- 1 root root 475 Jul 11 16:52 exec_system_commands.py
error is:
The output is similar to what we get when we manually execute "ls -lrt
" from the shell terminal. Now let me intentionally fail this script by giving some wrong command, and then the output from our script:
# python3 exec_system_commands.py
Return Code: 127
output is:
b''
error is:
b'/bin/sh: lsss: command not found\n'
- In our script I changed '
ls
' to 'lsss
' to intentionally fail the script. - Now the script has an empty output under "
output is:
" - The return code is non zero which means error
- While the error contains the error output from the provided command
Using subprocess.Popen with shell=False
In this section we will use shell=False
with python subprocess.Popen
to understand the difference with shell=True
So if you define shell=True
, you are asking Python to execute your command under a new shell but with shell=False
you must provide the command in List format instead of string format as we did earlier.
So for example we used below string for shell=True
cmd = 'ls -lrt'
Now to be able to use this command with shell=False
, we must convert into List
format, this can be done manually:
cmd = ['ls', '-lrt']
Or if it is too complex for you, use split()
method (I am little lazy) which should convert the string
into list
cmd = 'ls -lrt'.split()
and this should convert your command into string which can be further used with shell=False
So let us take the same example, and convert the command into list
format to be able to use with Python subprocess
and shell=False
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'ls -ltr'.split() # Check the list value of cmd print('command in list format:',cmd,'\n') # Use shell=False to execute the command sp = subprocess.Popen(cmd,shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True) # Store the return code in rc variable rc=sp.wait() # Separate the output and error. # This is similar to Tuple where we store two values to two different variables out,err=sp.communicate() print('Return Code:',rc,'\n') print('output is: \n', out) print('error is: \n', err)
The output from this script:
# python3 exec_system_commands.py
command in list format: ['ls', '-ltr']
Return Code: 0
output is:
total 308256
-rw-r--r--. 1 root root 315632268 Jan 1 2020 large_file
-rw-r--r--. 1 root root 577 Apr 1 00:00 my-own-rsa-key.pub
-rw-r--r--. 1 root root 2610 Apr 1 00:00 my-own-rsa-key
-rwxr--r-- 1 root root 428 Jun 8 22:04 create_enum.py
-rwxr--r-- 1 root root 176 Jun 11 06:33 check_string.py
-rw-r--r-- 1 root root 623 Jul 11 17:10 exec_system_commands.py
error is:
When should I use shell=True or shell=False?
Now you must be wondering, when should I use which method? As ultimately both seem to be doing the same thing, one way or the other. We will understand this in our next example.
In this example script, we will try to use our system environment variable with shell=True
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'echo $PATH' # Check the list value of cmd print('command in list format:',cmd,'\n') # Use shell to execute the command sp = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True) # Store the return code in rc variable rc=sp.wait() # Separate the output and error. # This is similar to Tuple where we store two values to two different variables out,err=sp.communicate() print('Return Code:',rc,'\n') print('output is: \n', out) print('error is: \n', err)
Output from this script is as expected, it is printing our PATH
variable content
# python3 exec_system_commands.py
command in list format: echo $PATH
Return Code: 0
output is:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
error is:
Now let us try to get the same using shell=False
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'echo $PATH'.split() # Check the list value of cmd print('command in list format:',cmd,'\n') # Use shell=False to execute the command sp = subprocess.Popen(cmd,shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True) # Store the return code in rc variable rc=sp.wait() # Separate the output and error. # This is similar to Tuple where we store two values to two different variables out,err=sp.communicate() print('Return Code:',rc,'\n') print('output is: \n', out) print('error is: \n', err)
Here the script output will just print $PATH
variable as a string instead of the content of $PATH
variable.
# python3 exec_system_commands.py
command in list format: ['echo', '$PATH']
Return Code: 0
output is:
$PATH
error is:
- So whenever you are planning to use any system variable, you must use
shell=True
asshell=False
will not be able to interpret the system variable and would consider it as just another string. - Also one of the demerit of
shell=True
method is that, every time you use this method, Python needs to open a new shell terminal to execute the command, get the status and then close the terminal which takes up some additional time while withshell=False
, you can get comparatively faster results.
shell=True
is strongly discouraged in some specific scenariosSo what did we learned? To summarise:
- If
shell=True
then your command is considered as a string (similar to youros
command) - If
shell=False
then your command is considered as aList
- To convert a string with
shell=False
, you can usesplit()
method for easier operation shell=False
doesn't recognise OS system environment variables
Practical Example
Let us take a practical example from real time scenario. In this python script we aim to get the list of failed services. In RHEL 7/8 we use "systemctl --failed
" to get the list of failed services
# systemctl --failed UNIT LOAD ACTIVE SUB DESCRIPTION ● kdump.service loaded failed failed Crash recovery kernel arming ● nfs-server.service loaded failed failed NFS server and services LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 2 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'.
Now here I only wish to get the service name, instead of complete output.
os.system(your_command)
but here since we want to manipulate the output, we are using python subprocess
Below is our sample script:
1 #!/usr/bin/env python3 2 3 import subprocess 4 5 # Define command as string and then split() into list format 6 cmd = 'systemctl --failed'.split() 7 8 # Check the list value of cmd 9 print('command in list format:',cmd) 10 11 # Use shell=False to execute the command 12 sp = subprocess.Popen(cmd,shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE,universal_newlines=True) 13 14 # Store the return code in rc variable 15 rc=sp.wait() 16 17 # Separate the output and error by communicating with sp variable. 18 # This is similar to Tuple where we store two values to two different variables 19 out,err=sp.communicate() 20 21 if rc == 0: 22 for line in out.splitlines(): 23 if "failed" in line: 24 print(line.split()[1]) 25 else: 26 print('The command returned an error: ',err)
Here,
Line 3:
We import subprocess
module
Line 6:
We define the command variable and use split()
to use it as a List
Line 9:
Print the command in list
format, just to be sure that split()
worked as expected
Line 12:
The subprocess.Popen
command to execute the command with shell=False
. Store the output and error, both into the same variable. Return the output with newline char
instead of byte code
Line 15:
This may not be required here but it is a good practice to use wait()
as sometimes the subprocess
may take some time to execute a command for example some SSH process, in such case wait()
will make sure the subprocess
command is executed successfully and the return code is stored in wait()
Line 19:
We need communicate()
to get the output and error value from subprocess.Popen
and store it in out and err variable respectively
Line 21:
If return code is 0
, i.e. if the command execution was success
Line 22:
Use for
loop and run on each line. Here we will use python splitlines()
method which splits the string based on the lines. It breaks the string at line boundaries and returns a list of splitted strings.
Line 23:
Use "in
" operator to search for "failed
" string in "line
"
Line 24:
If "failed
" is found in the "line
"
Line 25:
The split the found line into list and then we print the content of string with "1
" index number
Line 25:
In case the command fails to execute
Line 26:
Print the provided error message from err
variable which we stored using communicate()
Output from this script:
# python3 exec_system_commands.py
command in list format: ['systemctl', '--failed']
kdump.service
nfs-server.service
So we were able to print only the failed service using python subprocess
module
Using python subprocess.call() function
- The
subprocess
module provides another function that makes process spawning a safer operation than usingPopen()
. - The
subprocess.call()
function waits for the called command/program to finish reading the output. - It supports the same arguments as the
Popen()
constructor, such as shell, executable, andcwd
, but this time, your script will wait for the program to complete and populate the return code without the need tocommunicate()
. - You can fetch the exit status using
Popen.returncode
- To suppress
stdout
orstderr
, supply a value ofsubprocess.DEVNULL
which indicates that the special fileos.devnull
will be used.
stdout=PIPE
or stderr=PIPE
with this function as that can deadlock based on the child process output volume. Use Popen
with the communicate()
method when you need pipes.- In this sample python code, we will check the availability of
eth0
interface using "ip link show eth0
" - The output of the command will be stored in
/tmp/dataFile
and ifeth0
is available we print "Yes, eth0 is available on this server
" - For error condition also, the output of "
ip link show eth0
" will be written in/tmp/dataFile
, you can choose to use another file for error output.
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'ip link show eth0'.split() # Check the list value of cmd print('command in list format:',cmd) # Open the /tmp/dataFile and use "w" to write into the file myfile = open("/tmp/dataFile", "w") # Use shell=False to execute the command is_eth0_present = subprocess.call(cmd, shell=False, stdout=myfile, stderr=subprocess.STDOUT) if is_eth0_present == 0: print('Yes, eth0 is available on this server') else: print('No, eth0 is not available on this server') myfile.close()
The output from this script (when eth0
is available)
# python3 exec_system_commands.py
command in list format: ['ip', 'link', 'show', 'eth0']
Yes, eth0 is available on this server
Content of /tmp/dataFile
# cat /tmp/dataFile
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:5a:d3:83 brd ff:ff:ff:ff:ff:ff
The output from this script (when eth0
is NOT available)
# python3 exec_system_commands.py
command in list format: ['ip', 'link', 'show', 'eth0']
No, eth0 is not available on this server
Content of /tmp/dataFile
# cat /tmp/dataFile
Device "eth0" does not exist.
Using python subprocess.check_call() function
- This is another function which is part of
subprocess
module which can run command with arguments. check_call
will wait for command execution to complete.- If the execution is successful then the function will return zero then return, otherwise raise
CalledProcessError
. - The
CalledProcessError
object will have the return code in thereturncode
attribute.
I will try to use subprocess.check_now
just to print the command execution output:
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'ping -c2 google.com'.split() # Check the list value of cmd print('command in list format:',cmd) sp = subprocess.check_call(cmd, shell=False) print(sp)
The output from this script (when returncode
is zero):
# python3 exec_system_commands.py
command in list format: ['ping', '-c2', 'google.com']
PING google.com (172.217.26.238) 56(84) bytes of data.
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=1 ttl=115 time=579 ms
64 bytes from bom05s09-in-f14.1e100.net (172.217.26.238): icmp_seq=2 ttl=115 time=90.1 ms
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 68ms
rtt min/avg/max/mdev = 90.125/334.576/579.028/244.452 ms
0
The output from this script (when returncode
is non-zero):
# python3 exec_system_commands.py
command in list format: ['ping', '-c2', 'google.co12m']
ping: google.co12m: Name or service not known
Traceback (most recent call last):
File "exec_system_commands.py", line 11, in <module>
sp = subprocess.check_call(cmd, shell=False)
File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ping', '-c2', 'google.co12m']' returned non-zero exit status 2.
As you see we get subprocess.CalledProcessError
for non-zero return code.
So we should use try and except for subprocess.check_now
as used in the below code:
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'ping -c2 google.c12om'.split() # Check the list value of cmd print('command in list format:',cmd) try: sp = subprocess.check_call(cmd, shell=False) except subprocess.CalledProcessError: print('Host is not reachable')
The output from this script (when returncode is non-zero):
# python3 exec_system_commands.py
command in list format: ['ping', '-c2', 'google.c12om']
ping: google.c12om: Name or service not known
Host is not reachable
So now this time we don't get CalledProcessError
, instead the proper stderr
output along with out print statement is printed on the console
Using subprocess.run() function
- The
subprocess.run()
function was added in Python 3.5 - Wait for command to complete, then return a
subprocess.CompletedProcess
instance. - The full function signature is largely the same as that of the
Popen
constructor - most of the arguments to this function are passed through to that interface. (timeout
,input
,check
, andcapture_output
are not.) - If
capture_output
is true,stdout
andstderr
will be captured. - When used, the internal
Popen
object is automatically created withstdout=PIPE
andstderr=PIPE
. - The
stdout
andstderr
arguments may not be supplied at the same time ascapture_output
. - If you wish to capture and combine both streams into one, use
stdout=PIPE
andstderr=STDOUT
instead ofcapture_output
.
In this sample python code we will try to check internet connectivity using subprocess.run()
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'ping -c2 google.c12om'.split() # Check the list value of cmd print('command in list format:',cmd) sp = subprocess.run(cmd, shell=False, check=True, capture_output=True, text=True) print("stdout: ", sp.stdout) print("stderr: ", sp.stderr)
For successful output:
# python3 /tmp/script.py
command in list format: ['ping', '-c2', 'google.com']
stdout: PING google.com (172.217.160.142) 56(84) bytes of data.
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=1 ttl=115 time=199 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=2 ttl=115 time=80.8 ms
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 80.756/139.980/199.204/59.224 ms
stderr:
For failed output i.e. when the command returns non-zero exit code:
# python3 /tmp/script.py
command in list format: ['ping', '-c2', 'google.c12om']
Traceback (most recent call last):
File "/tmp/script.py", line 11, in <module>
sp = subprocess.run(cmd, shell=False, check=True, capture_output=True, text=True)
File "/usr/local/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ping', '-c2', 'google.c12om']' returned non-zero exit status 2.
capture_output
is supported only with Python 3.7, if used with earlier version then you will get TypeError: __init__() got an unexpected keyword argument 'capture_output'
. For older releases you can continue to use stdout=PIPE
and stderr=PIPE
You can use check=false
if you don't want to print any ERROR
on the console, in such case the output will be:
# python3 exec_system_commands.py
command in list format: ['ping', '-c2', 'google.c12om']
ping: google.c12om: Name or service not known
stdout: b''
stderr: None
Using python subprocess.check_output() function
- The
subprocess.check_output()
is similar tosubprocess.run(check=True)
- By default, this function will return the data as encoded bytes so you can use
text=True
oruniversal_newlines=True
to get string value as output - This function will run command with arguments and return its output.
- If the return code was non-zero it raises a
CalledProcessError
. TheCalledProcessError
object will have the return code in thereturncode
attribute and any output in the output attribute.
#!/usr/bin/env python3 import subprocess # Define command as string and then split() into list format cmd = 'ping -c2 google.c2om'.split() # Check the list value of cmd print('command in list format:',cmd) try: output = subprocess.check_output(cmd, shell=False, stderr=subprocess.STDOUT, universal_newlines=True) except subprocess.CalledProcessError as e: print('output before error: ',e.output) print('Return Code: ',e.returncode)
Output from the script for non-zero exit code:
# python3 exec_system_commands.py command in list format: ['ping', '-c2', 'google.c2om'] output before error: ping: google.c2om: Name or service not known Return Code: 2
Which subprocess module function should I use?
- The functions
run()
,call()
,check_call()
, andcheck_output()
are wrappers around thePopen
class. - Using
Popen
directly gives more control over how the command is run, and how its input and output streams are processed.
Wait for the command to complete
- Use
subprocess.call
orsubprocess.run
to run the command described by args. Wait for command to complete, then return thereturncode
attribute. - Use
subprocess.Popen
withwait()
to wait for the command to complete
Here we use subprocess.call
to check internet connectivity and then print "Something
"
#!/usr/bin/env python3 import subprocess cmd='ping -c5 google.com'.split() sp = subprocess.call(cmd, shell=False) print('Something')
Output from this script:
# python3 exec_system_commands.py PING google.com (172.217.160.142) 56(84) bytes of data. 64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=1 ttl=115 time=102 ms 64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=2 ttl=115 time=325 ms 64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=3 ttl=115 time=85.4 ms 64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=4 ttl=115 time=249 ms 64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=5 ttl=115 time=81.0 ms --- google.com ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 94ms rtt min/avg/max/mdev = 81.022/168.509/324.751/99.872 ms Something
So we know subprocess.call
is blocking the execution of the code until cmd
is executed. Now we do the same execution using Popen
(without wait())
.
#!/usr/bin/env python3 import subprocess cmd='ping -c5 google.com'.split() sp = subprocess.Popen(cmd, shell=False) print('Something')
If you observe, "Something
" was printed immediately while ping was still in process, so call()
and run()
are non-blocking function.
# python3 exec_system_commands.py
Something
[root@client scripts]# PING google.com (172.217.160.142) 56(84) bytes of data.
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=1 ttl=115 time=107 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=2 ttl=115 time=136 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=4 ttl=115 time=122 ms
64 bytes from maa03s29-in-f14.1e100.net (172.217.160.142): icmp_seq=5 ttl=115 time=121 ms
--- google.com ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 396ms
rtt min/avg/max/mdev = 106.956/121.745/136.476/10.442 ms
subprocess.run()
is equivalent to: run(...).returncode
(except that the input and check parameters are not supported). So you can choose to use run()
or call()
function based on your requirement.
Capture output from command
- The standard input and output channels for the process started by
call()
are bound to the parent’s input and output. - That means the calling program cannot capture the output of the command. Use
check_output()
to capture the output for later processing. - To prevent error messages from commands run through
check_output()
from being written to the console, set thestderr
parameter to the constantSTDOUT
.
Error Handling
- The
call()
function returns the program's exit code, which is an integer that has a program-defined meaning, which is usually used to know whether the program has succeeded or failed. - The
check_call()
function works likecall()
, except that the exit code is checked, and if it indicates an error happened, then aCalledProcessError
exception is raised. - The
check_output()
function returns whatever text the program printed and raises an exception if the program exited with a non-zero exit code. The raised exception has an attribute called output that contains the text output of the program. So, even if the program exited with an error code, we could still get the output if we want it.
Conclusion
In this tutorial we learned about different functions available with python subprocess
module and their usage with different examples. As a Linux administrator coming from shell background, I was using mostly os module which now I have switched to subprocess
module as this is the preferred solution to execute system commands and child processes.
In most cases you will end up using subprocess.Popen()
or subprocess.run()
as they tend to cover most of the basic scenarios related to execution and checking return status but I have tried to give you a comprehensive overview of possible use cases and the recommended function so you can make an informed decision.
Lastly I hope this tutorial on python subprocess
module in our programming language section was helpful. So, let me know your suggestions and feedback using the comment section.
References
I have used below external references for this tutorial guide
Complete Python Scripting for Automation
Hands-On Enterprise Automation with Python.
docs.python.org: subprocess module
Really enjoyed reading this fantastic blog article. It is everything I enjoy and also very well researched and referenced. Thank you for taking the time to create a good looking an enjoyable technical appraisal.