Python requests library - explained with examples

Introduction to Python requests library

Python requests library is a powerful tool and a simple way to make HTTP requests to APIs. Python requests are very important for REST APIs and web scraping. Python provides built-in functions for making requests and responding to the request.  We will start this tutorial from the very beginning.

In this article we will cover

• Install requests library
• How to make requests using different HTTP methods such as GET, POST, PUT, HEAD, PATCH
• Customize the requests' by inserting cookies and headers
• Perform authentication using requests

Moreover, this tutorial will also cover inspecting data, authenticating and responding to requests

 

Usage of Python requests module

Requests is a very powerful Python library having many built-in functions. It allows us to send HTTP/1.1 requests using Python. With it, we can add contents like headers, form data, multipart files, and parameters via simple Python libraries. Moreover, It also allows us to access the response data of Python in a similar way. It plays an important role while dealing with REST APIs and web scraping.

 

Install python requests library

Installation of the request depends on the type of operating system. But the basic command in python for installing any library is using pip. Use the following commands to install requests on your system.

>>> pip install requests

OR alternatively you can clone the public repository by using the following command.

>>> git clone git://github.com/psf/requests.git

Once we have a copy of the source, we can then embed it in our own Python package, or install it into our site packages easily using the following commands.

$ cd requests && pip install

Once it is installed successfully into our system, we can use keyword import to get access to its functionalities.

import requests 

 

Making request in Python

The Python request module has many powerful built-in functions to make HTTP requests to specific URLs. For example GET, POST, PUT, HEAD and PATCH requests.  Requesting HTTP means either to get data from a specified URL or push data to the server. It is simply working as a request and response between client and server. In the following sections, we will discuss each type of request in more detail.

GET request in Python

As the name suggests, GET method is used to retrieve data from a given URL. The GET method sends the encoded user information appended to the page request. The page and the encoded information are separated by the ‘?’ character.  request.get() function is used, see the following example to understand the syntax.

requests.get(“URL”)

 

The POST request in Python

POST method is a request supported by HTTP. This method requests the webserver to accept the data enclosed in the body of the request usually to store it. This method is often used when uploading a file or image.

requests module in python has a built-in function post() to make POST requests to specific URLs.

See the example below to see how post() in python works. We can check the status of our POST request by using r.status_code and if it returns 200 it means our POST request is successful.

import requests 

Ourdata = {'somekey': 'somevalue'}

<i># Making a POST request</i>
r  = requests.post("http://httpbin.org/post", data=Ourdata)

# checking the status of our post request
print(r.status_code)

Output:

200

To view the content of request, use r.json() method which returns the content of request. See the example below:

import requests  <i>

# Making a POST request</i>
Ourdata = {'somekey': 'somevalue'}

r  = requests.post("http://httpbin.org/post", data=Ourdata) 

<i>## print content of request</i>
print(r.json())

Output:

 

Here is a list of few parameters which are commonly used in the POST request.

• url :this is the required parameter and is the URL of the request.
• json: the optional parameter which returns the content of the request.
• data : options parameter which returns a list of tuples, dictionary, or bytes of object sends to specific URL
• cookies: optional parameter which returns cookies sent to URL. By default, the value is none.
• proxies : the optional parameter which returns a dictionary of protocols to the proxy URL. By default the value is none

 

The PUT request in Python

An HTTP PUT request is used to create or update a specified server.  If the URL refers to an already existing resource, it is modified and if the URL does not point to an existing resource, then the server can create the resource with that URL.

Python requests module have a built-in function put() which is to make put request to a specified URL.In this example we will make a PUT request and check the status of our request.

import requests

Ourdata = {'somekey': 'somevalue'}

<i># Making a PUT request</i>
r  = requests.put("http://httpbin.org/post", data=Ourdata) 

<i>## printing the status</i>
print(r.status_code)

Output:

405

This means the following operation is NOT ALLOWED on the given URL.

 

The HEAD request in Python

The HEAD method asks for a response in a similar way to that of a GET request, but without the response body. This is useful for retrieving meta-information written in response headers, without having to transport the entire content.

Python requests module has a built-in function head() to make a request to the specified URL.

See the example below. We can check the status of our request and if we get 200 that means our request was successful.

import requests  <i># Making a HEAD request</i>

r = requests.head('https://httpbin.org/', data ={'key':'value'})

<i># check status code for response recieved</i>
<i># success code - 200</i>
print(r.status_code)

Output:

200

 

The PATCH request in Python

The PATCH request only needs to contain the changes to the resource, not the complete resource. This resembles PUT, but the body contains a set of instructions describing how a resource currently residing on the server should be modified to produce a new version.

Python requests module has a built-in function patch() send a request to the specified URL. See the example below.
We can check the status of our request.

import requests

<i># Making a PATCH request</i>
r = requests.patch('https://httpbin.org/patch', data ={'key':'value'})

<i># check status code for response received </i>
<i># success code - 200</i>
print(r.status_code)

Output:

200

 

The Response method

A response object contains all the data sent by a server in response to our request. Response is a very powerful object having many built-in functions which helps us to play with the data. See the following examples which show some of the functionalities of response objects. We had already touched this topic making requests in the section above. In this section we will look more closely.

Let’s make a request and store that in a variable known as response so that we can more information about the request using response.

import requests

<i># Making a get request</i>
response = requests.get('https://api.github.com/')

 

Check the response code status

The very first and basic information that we can get from response is the status of code. This will tell us whether our request was successful or not.

For example, if it returns 200 that means the request was successful, 404 means the resource that we are looking for was not found. And there can be many other scenarios as well.

Let's take an example of a request and see if the request was successful or not.

import requests

<i># Making a GET request</i>
response = requests.get('https://api.github.com/')

<i># printing the status of the request</i>
print(response.status_code)

Output:

200

Which means the request was successful.

Sometimes we might want to return something else instead of 200 when the request is successful. We can achieve this by using if else statements. See the example below:

import requests

<i># Making a get request</i>
response = requests.get('https://api.github.com/')

if response.status_code == 200:
   print('Request was successful!')
else:
   print('Request was not successful ')

Output:

Request was successful!

Now keep in mind that the range of status codes for 200-400 also shows the success of requests in a sense that they also return useful info. For example 304 means NOT MODIFIED and 204 means NO CONTENT, although the request is successful but there is no content to return.

So, make sure you use this convenient shorthand only if you want to know if the request was generally successful and then, if necessary, handle the response appropriately based on the status code.

Here is another way to handle unsuccessful request using exception handling. we can do this using .raise_for_status(). See the example below:

import requests

from requests.exceptions import HTTPError

<i># here we use two urls and used for loop to go thourgh over them</i>
<i># used for in loop</i>
for url in ['https://api.github.com', 'https://api.github.com/invalid']: 

   <i># try and except method</i>
   try:
       <i># Stores request in reponse</i>
       <i># get the url from the list above</i>
       response = requests.get(url) 

       <i># If the response was successful, no Exception will be raised</i>
       response.raise_for_status()

   <i># if the request is not successful, it will go to the exceptions</i>
   except HTTPError as http_err:

       <i># prints http error if the link has error in HTTP</i>
       print(f'HTTP error: {http_err}')  <i># Python 3.6</i>

   except Exception as err:
       <i># print other errors apart from HTTP error if any</i>
       print(f'Other error : {err}')  <i># Python 3.6</i>

   <i># this code will be executed if none of the above errors accurs</i>
   else:
       print('Request was Successful!')

Output:

Request was Successful!
HTTP error: 404 Client Error: Not Found for url: https://api.github.com/invalid

See that the sending request to the first URL was successful and the very last statement (else) was executed while there was an error in sending request to the second URL as shown in the output.

 

Content in Response

The methods that we used for requests ( usually the get() ), often contain valuable content. We can always get access to the content using different methods of response. There are many different formats to see the content.

For example see the example below which prints the content in bytes format.

import requests

<i># Making a get request</i>
response = requests.get('https://api.github.com/')

print(response.content)

Output:

 

the  response.content gives us access to the content in bytes. We can convert this into string characters. See the example which prints out the content in string format.

import requests

<i># Making a get request</i>
response = requests.get('https://api.github.com/')

print(response.text)

Output:

 

Headers in response

The response headers give us useful information, such as the content type of the response payload and a time limit on how long to cache the response. We use response.headers to view these headers.

See example below:

import requests

<i># Making a get request and storing in response</i>
response = requests.get('https://api.github.com/')

print(response.headers)

Output:

response.headers returns a dictionary-like object, allowing us to access header values by key. There is something special about this dictionary-like headers object. The HTTP spec defines headers to be case-insensitive, which means we are able to access these headers without worrying about their capitalization.

 

The message body

According to the HTTP specification, POST, GET  and  PATCH requests pass their data through the message body rather than through parameters in the query string. We can use requests to pass payload to the corresponding function’s data parameter. And data takes a dictionary, a list or tuples , a file like object or bytes.

For Example, we can send the content in the form of a dictionary. See the example below:

import requests

<i># sending content in the form of dictionary</i>
response = requests.post('https://httpbin.org/post', data={'key':'value'})

print(response.status_code)

Output:

200

We can also send the same data in the form of a list containing tuples.

For example see this example.

import requests

<i># sending content in the form of list</i>
response = requests.post('https://httpbin.org/post', data=[('key', 'value')])

print(response.status_code)

Output:

200

 

Customize requests in Python

We can access headers and cookies that servers send back to us r.cookies and r.headers. At the same time we can send our own custom cookies and headers using requests in Python. In the following sections we will discuss how we can customize sending requests in python.

 

Inserting cookies into requests

To add HTTP cookies to a request, we can simply pass them in a dict to the cookies parameter.

See the example below which demonstrates how to send cookies and headers. We can print out cookies using r.content.

import requests

# creating dictionary of cookies
cookies_dict = {"my_cookie": "cookie_value"}

r = requests.get("http://httpbin.org/cookies", cookies=cookies_dict)

print(r.content)

Output:

b'{\n "cookies": {\n "my_cookie": "cookie_value"\n }\n}\n'

 

Inserting headers into requests

We can send cookies in a similar way as we did for cookies. To add HTTP cookies to a request, we can simply pass them in a dict to the cookies parameter.

See the example below. We can use r.content to print headers.

import requests

<i># creating header dictionary</i>
headers_dict = {"my headers": "headers values"}

<i># using get method to request and add headers</i>
r = requests.get("http://httpbin.org/headers", headers=headers_dict)

print(r.content)

Output:

b'{\n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate", \n    "Host": "httpbin.org", \n    "My-Headers": "headers_values", \n    "User-Agent": "python-requests/2.20.0", \n    "X-Amzn-Trace-Id": "Root=1-6103a3d3-106a9e8d135974dd6b963b0a"\n  }\n}\n'

 

Inspecting request

The requests library prepares the request before actually sending it to the destination server.  These preparations include  validating headers and serializing JSON content.

We can view these preparations using requests. See the example below:

import requests

<i># sending content in the form of json</i>
response = requests.post('https://httpbin.org/post', json={'key':'value'})

<i># this prints the type of content headers</i>
print(response.request.headers["content-type"])

Output:

application/json

We can also inspect the URL as well.

import requests

<i># sending content in the form of json</i>
response = requests.post('https://httpbin.org/post', json={'key':'value'})

<i># this prints out the url</i>
print(response.request.url)

Output:

https://httpbin.org/post

Moreover, we can also access the message body of the content as well.

import requests

<i># sending content in the form of json</i>
response = requests.post('https://httpbin.org/post', json={'key':'value'})

<i># this prints out the messge body</i>
print(response.request.body)

Output:

b'{"key": "value"}'

 

Authentication using requests in Python

Authentication means giving access to a particular resource. Since everyone can’t be allowed to access data from every URL, one would require authentication primarily. We can achieve this by providing authentication data through an authorization header or a custom header defined by the server.

Python module requests provide many types of authentication. We will discuss two different types of authentications using requests in Python.

 

Basic authentication in requests

This is the simplest form of authentication to the server. We need to import the HTTPBasicAuth class from the requests library. See the following example which demonstrates how it works.

import requests
from requests.auth import HTTPBasicAuth    

r = requests.get('https://api.github.com/user',
auth=HTTPBasicAuth('userName', 'password'))

print(r.text)

Output:

 

Digest Authentication in requests

This is another way of authentication on request. We have to import HTTPDigAuth class from requests to get access to this authentication.

Here is simple syntax:

import requests
from requests.api import request
from requests.auth import HTTPDigestAuth

r = requests.get('https://postman-echo.com/digest-auth', auth=HTTPDigestAuth('postman','password'))

print(r.text)

Output:

{"authenticated":true}

 

Performance

We can check the performance of our request through accessing different methods. For example by checking the time out, session object or retry limits. In this section we will discuss timeout and session objects.

 

Timeouts in request

When we make a request to an external server, our system waits for the response from the external specified server. If our application waits too long for that response, requests to our service could back up, user experience could suffer, or our background jobs could hang. By default, requests will wait indefinitely on the response, so we have to specify a timeout duration to prevent these things from happening. To set the request’s timeout, we use the timeout parameter. timeout can be an integer or float representing the number of seconds to wait on a response before timing out.

See example below:

import requests

<i># specifing the timout</i>
response = requests.post('https://httpbin.org/post', timeout=2)

<i># printing the status</i>
print(response.status_code)

Output:

200

We can also pass tuples, where the first element specifies the time needed to create a connection and the second element specifies the time to wait until it gets a response from the server.

See example below.

import requests

<i># specifying the timeout</i>
response = requests.post('https://httpbin.org/post', timeout=(2, 5))

<i># printing the status</i>
print(response.status_code)

Output:

200

 

Session Object

The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance. For example, if we want to use the same authentication across multiple requests, we can use a session. Each time we make a request with session, once it has been initialized with authentication credentials, the credentials will be persisted.

See example below:

import requests
from getpass import getpass

<i># By using a context manager, we can ensure the resources used by</i>
<i># the session will be released after use</i>
with requests.Session() as session:
   session.auth = ('username', getpass())

   <i># Instead of requests.get(), we'll use session.get()</i>
   <i># and store it in response</i>
   response = session.get('https://api.github.com/user')

<i># we can inspect the response just like you did before</i>
print(response.headers)

Output:

 

Summary

In this tutorial, we learned about the requests module in python which is a very powerful tool for REST APIs and web scraping. We learned about the installation process at the beginning of the tutorials. We also cover different types of requests that we can make using the requests module including GET, POST, HEAD, and many more. Moreover, we also learned about the authentication process using the requests module.

 

Further readings

Installation of Python requests
requests module in python
Authentication in requests