Set up proxy using http_proxy & https_proxy environment variable in Linux?

Introduction to Proxies

A proxy server acts as an intermediary between your computer and the internet. It allows you to make requests to the internet on behalf of your computer, hiding the actual origin of the request.

Different Types of Proxies

• HTTP Proxies: Primarily used for web browsing, these handle web traffic and are the most common type of proxies used.
• SOCKS Proxies: More versatile than HTTP proxies, they can handle various types of traffic, including email, FTP, and torrents.
• SSL Proxies: Specifically designed to handle HTTPS traffic, offering a secure connection between the client and the proxy server.
• Transparent Proxies: These are mainly used for caching purposes and do not provide anonymity as they forward the user’s original IP address.

 

Setting up http_proxy and https_proxy using Environment Variables

Environment variables such as http_proxy, https_proxy, ftp_proxy, and no_proxy can be configured to define proxies in a Linux environment. Here’s how you can set up these variables with examples:

 

Without Username and Password

You can set up HTTP and HTTPS proxies by exporting them as environment variables. Open a terminal and input the following commands:

export http_proxy=http://proxy_server_address:port
export https_proxy=https://proxy_server_address:port

Example:

export http_proxy=http://192.168.1.1:8080
export https_proxy=https://192.168.1.1:8080

 

With Username and Password

When authentication is required, include the username and password in the environment variable.

export http_proxy=http://username:password@proxy_server_address:port
export https_proxy=https://username:password@proxy_server_address:port

Example:

export http_proxy=http://user123:pass123@192.168.1.1:8080
export https_proxy=https://user123:pass123@192.168.1.1:8080

 

With Domain, Username, and Password

If you need to include a domain along with the username and password, the format would look like this:

export http_proxy=http://domain\\username:password@proxy_server_address:port
export https_proxy=https://domain\\username:password@proxy_server_address:port

Example:

export http_proxy=http://mydomain\\user123:pass123@192.168.1.1:8080
export https_proxy=https://mydomain\\user123:pass123@192.168.1.1:8080

 

Special character handling in http_proxy and https_proxy

When setting the http_proxy and https_proxy environment variables in Linux, you might encounter situations where the username or password includes special characters. Special characters, such as !, @, #, $, %, ^, &, *, (, ), {, }, [, ], :, ;, ", ', <, >, ?, /, and \, need to be handled carefully because they can be interpreted differently by the shell or as part of the URL.

With more complex and robust handling of special characters in username or password follow How to setup http or https proxy with special characters in username and password

Special characters should be URL-encoded to ensure that they are interpreted correctly. URL encoding replaces special characters with a "%" followed by two hexadecimal digits corresponding to the ASCII code of the character.

Examples of Encoding Special Characters:

• @ would become %40
• # would become %23
• $ would become %24
• : would become %3A
• / would become %2F
• & would become %26

If either the username or password contains special characters, they need to be URL-encoded.

export http_proxy=http://username:URL-encoded-password@proxy_server_address:port
export https_proxy=https://username:URL-encoded-password@proxy_server_address:port

Example:

export http_proxy=http://user123:%24%26%40pass@192.168.1.1:8080
export https_proxy=https://user123:%24%26%40pass@192.168.1.1:8080

Here, the password "$&@pass" is URL-encoded as "%24%26%40pass".

 

Setting Environment Variables Permanently

The previous examples I shared covers setting http_proxy and https_proxy temporarily i.e. they are non persistent and will not survive a reboot. Setting environment variables permanently can be crucial to ensure that your proxy configurations persist across system reboots and terminal sessions.

1. Individual User:

Open .bashrc or .bash_profile in your home directory.

vi ~/.bashrc

Add the export lines at the end of the file.

export http_proxy=http://username:password@proxy_server_address:port
export https_proxy=https://username:password@proxy_server_address:port

Save and exit the editor, then source the file to apply the changes immediately.

source ~/.bashrc

2. System-wide:

Open /etc/environment using a text editor.

sudo vi /etc/environment

Add the proxy configurations without the export command.

http_proxy="http://username:password@proxy_server_address:port"
https_proxy="https://username:password@proxy_server_address:port"

Save and exit.

Verification

After editing the necessary files and sourcing them, you can use the echo command to verify if the environment variables are set correctly.

echo $http_proxy
echo $https_proxy

 

Accessing URLs using Curl/Wget with Proxies

Accessing URLs using curl or wget with proxies involves passing the proxy details as part of the command. Both curl and wget have options that allow you to specify a proxy to be used for the request.

 

Using Curl with Proxies

Basic Usage without Authentication:

curl -x http://proxy_server_address:port http://example.com

With Username and Password Authentication:

curl -U username:password -x http://proxy_server_address:port http://example.com

If you have Environment Variables Set:

curl -x $http_proxy http://example.com

Here, curl uses the http_proxy environment variable that has been set previously.

 

Using Wget with Proxies

Basic Usage without Authentication:

wget -e use_proxy=yes -e http_proxy=http://proxy_server_address:port http://example.com

With Username and Password Authentication:

You can include the username and password in the URL.

wget -e use_proxy=yes -e http_proxy=http://username:password@proxy_server_address:port http://example.com

If you have Environment Variables Set:

wget automatically uses the environment variables. So, you just need to execute a regular wget command, and it will use the proxy settings from the environment variables.

wget http://example.com

Ensure that the http_proxy, https_proxy, and other related environment variables are correctly set.

 

Bypassing Proxy for Specific Domains/Addresses (NO_PROXY)

The no_proxy environment variable allows for exceptions to be made in proxy configurations, letting you bypass the proxy for specific domains or IP addresses. This can be essential for intranet resources or local development environments that should not be accessed via a proxy.

• The no_proxy variable is a comma-separated list of domain names or IP addresses that should bypass the proxy.
• Wildcards can be used for broader matches. For instance, .local will match all hosts in the .local domain.
• It's also possible to specify ports along with domains or IP addresses.

1. Bypassing for Specific Domains

export no_proxy=".example.com,.local,localhost"

This setting bypasses the proxy for all subdomains of example.com, all domains ending in .local, and localhost.

2. Bypassing for Specific IP Addresses

export no_proxy="192.168.1.1,10.0.0.1"

This configuration bypasses the proxy for traffic going to the specified IP addresses.

3. Bypassing for Localhost and Local Network

export no_proxy="localhost,127.0.0.1,10.0.0.0/8"

Here, all traffic to localhost, 127.0.0.1, and the 10.x.x.x network bypasses the proxy.

4. Bypassing for Specific Ports

export no_proxy="localhost:8080,example.com:9090"

In this example, traffic to localhost on port 8080 and example.com on port 9090 will bypass the proxy.

You might want to automate the no_proxy configuration by adding the export command to a startup script like ~/.bashrc or ~/.bash_profile.

echo 'export no_proxy=".example.com,.local,localhost"' >> ~/.bashrc
source ~/.bashrc

You can verify the no_proxy configuration by echoing the variable:

echo $no_proxy

 

Summary

In this comprehensive guide, we delved into the multifaceted process of setting up and managing proxies in a Linux environment, focusing on the utilization of http_proxy and https_proxy environment variables. Beginning with a foundational understanding, the guide elucidated the definitions, purposes, and various types of proxies. It meticulously navigated through the steps of configuring proxies, handling special characters in credentials, and establishing a permanent proxy setup for persistent configurations across different shells such as bash, zsh, and csh.

The tutorial also explored advanced usage scenarios like accessing URLs via command-line tools, employing scripts for automated setups, and implementing specific configurations to bypass proxies for designated domains and addresses. The guide accentuated the customization and flexibility that environment variables offer in tailoring proxy configurations to meet specific network requirements and preferences. For a deeper dive and continuous learning, it also provided a reservoir of essential further reading resources, including official documentation, man pages, and community forums, aiming to foster a well-rounded and up-to-date understanding of proxy management in Linux.

 

Further Reading

• Environment Variables in Unix
• Proxy Environment Variables
• URL Encoding: RFC 3986