What is SSH Multiplexing and How to use Securely?

This tutorial covers SSH Multiplexing, explaining its benefits, setup, and use cases. We'll guide you through configuration steps, security best practices, and optimization techniques to enhance your SSH operations efficiently and securely.

SSH Multiplexing is a powerful feature of the Secure Shell (SSH) protocol, enabling multiple SSH connections to share a single network channel. This not only accelerates the process of establishing connections to a remote server but also reduces the overall system and network load. By leveraging a single authenticated SSH session, users can initiate additional sessions without the need for re-authentication, making tasks like file transfers, remote command execution, and port forwarding more efficient and faster.

 

How SSH Multiplexing Works?

SSH Multiplexing is a feature that allows a single SSH connection to carry multiple sessions, such as terminal, SCP, or SFTP sessions, over a single network connection. This is achieved by reusing the first established connection for subsequent connections to the same host, thus reducing the overhead of setting up new TCP connections and performing SSH handshakes for each session.

Let's understand in a step by step approach:

1. The first SSH connection to a remote server establishes a TCP connection and performs the necessary SSH handshake to authenticate the user. This connection becomes the "master" connection.
2. Along with the master connection, a control socket is created on the client's filesystem. This socket is used by subsequent SSH sessions to communicate with the master connection.
3. The SSH client has to be configured to enable multiplexing by using configuration parameters such as ControlMaster, ControlPath (the path to the control socket), and optionally, ControlPersist (how long to keep the master connection open after the last session closes). These client configurations can be passed to ssh client using -o <options> or can be added in a client configuration file under ~/.ssh/config.
4. When a new SSH session is initiated to the same host, instead of establishing a new TCP connection and performing another handshake, the SSH client checks if a master connection (and its control socket) is available. If so, it communicates through the control socket to the master connection, which then multiplexes (channels) the new session over the existing TCP connection.
5. Data for each session is encapsulated in its own channel within the single TCP connection. SSH ensures that data from one session does not interfere with another by maintaining separate channels for each session's data stream.
6. Sessions can be closed independently of each other. If ControlPersist is not set, the master connection closes when the last multiplexed session closes. If ControlPersist is set, the master connection can stay open for a specified duration, waiting for new sessions to multiplex.

 

Configuring SSH Multiplexing on Server

We don't need to configure anything explicitly for SSH Multiplexing on server but for SSH multiplexing to work effectively, the SSH server (sshd) configuration can play a crucial role, especially when handling multiple concurrent connections. Two important settings in the server's SSH configuration file (/etc/ssh/sshd_config) that can impact multiplexing are MaxSessions and MaxStartups. Adjusting these settings can help optimize the server for handling multiple simultaneous SSH sessions, including those used in multiplexing scenarios.

 

1. MaxSessions

MaxSessions specifies the maximum number of open sessions permitted per network connection. When using SSH multiplexing, multiple sessions can be established over a single network connection. If you're using multiplexing to open many sessions over a single connection, increasing this limit may be necessary. The default is usually 10, which means 10 sessions can be opened over each network connection.

MaxSessions 20

 

2. MaxStartups

MaxStartups specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Beyond this threshold, additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for some connections. The parameter can be a single number or a series of three colon-separated numbers specifying the start, rate of increase, and full values.

The default is typically 10:30:100, but it can vary. This means up to 10 connections can start immediately, but once there are 10 ongoing connections, only 30% of new connection attempts will be processed until the number of unauthenticated connections reaches 100, at which point all new connections will be dropped.

MaxStartups 10:50:200

Restart SSHD service after you have updated the configuration parameters

systemctl restart sshd

 

Configuring SSH Multiplexing on Client

The client is the Linux node which will initiate the SSH connection towards a Linux server. Our most configuration would be done on the client side. I will initiate connection to 10.39.251.204 which is my SSHD server.

 

1. Edit the SSH Client Configuration

Open the SSH client configuration file ~/.ssh/config with your preferred text editor. If the file doesn't exist, you can create it.

vi ~/.ssh/config

 

2. Configure Multiplexing Settings

Add the following configuration to the file. This configuration enables multiplexing for all hosts you connect to. You can also restrict it to specific hosts by replacing the Host * line with something more specific, like Host 192.168.0.100.

Host *
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 600

• ControlMaster auto: This tells SSH to automatically check if a master connection exists and use it. If not, it creates a new one.
• ControlPath ~/.ssh/sockets/%r@%h-%p: This defines the path to the control socket used for multiplexing. %r, %h, and %p are placeholders for the remote user, hostname, and port, respectively. Ensure the directory (~/.ssh/sockets/) exists or change the path to an existing one.
• ControlPersist 600: This keeps the master connection open for 600 seconds (10 minutes) after the last multiplexed session is closed, allowing new sessions to be initiated without re-authentication within this period.

The ControlPath option in SSH configuration supports various placeholders that can be used to dynamically specify the path to the control socket file used for connection multiplexing. Here's a table listing all the supported placeholders and their meanings:

 

3. Create the Socket Directory

If you used a custom directory for the ControlPath, make sure it exists. If it doesn't, create it with:

mkdir -p ~/.ssh/sockets

 

4. Establish the Master Connection

Now, when you SSH to a server, the configuration automatically sets up a master connection if it doesn't already exist.

ssh root@10.39.251.204

Running the same command in a new terminal window will connect through the existing master connection, utilizing the multiplexing setup.

 

5. Verify Multiplexing

You can verify that multiplexing is working by checking for the control socket file.

[root@secvm-2 ~]# ll .ssh/sockets/
total 0
srw-------. 1 root root 0 Feb 22 22:53 root@10.39.251.204-22

If multiplexing is correctly configured, you'll see the control socket file listed, indicating an active master connection.

Now if I attempt to connect to the same host then it should happen without any authentication:

[root@secvm-2 ~]# ssh -vvv root@10.39.251.204
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 3087
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
Last login: Thu Feb 22 23:01:36 2024 from 10.39.251.200
[root@secvm-1 ~]# 

As you can see, I was able to connect to 10.39.251.204 without any password authentication request.

But I attempt to authenticate after ControlPersist time value i.e. 10 minutes after the last connection then I will have to re-authenticate and the socket will be automatically deleted.

[root@secvm-2 ~]# debug1: ControlPersist timeout expired
debug3: send packet: type 1
debug1: channel 0: free: /root/.ssh/sockets/a9a65826aafce43688db4f53d55e45693ab113a9, nchannels 1
debug3: channel 0: status: The following connections are open:

debug3: fd 0 is not O_NONBLOCK
debug3: fd 1 is not O_NONBLOCK
Transferred: sent 3140, received 2908 bytes, in 601.6 seconds
Bytes per second: sent 5.2, received 4.8
debug1: Exit status -1

 

Performing SSH Multiplexing using Shell Script

Here's a shell script that demonstrates how to use SSH multiplexing by specifying options directly in the command line instead of relying on configurations in ~/.ssh/config. This approach can be particularly useful for scripts where you want to control multiplexing behavior on a per-invocation basis without altering global or user-specific SSH settings.

The script establishes a master connection to a remote host, then uses that master connection for subsequent SSH and SCP commands. It checks for an existing control socket to decide whether to initiate a new master connection or use an existing one.

#!/bin/bash

# Remote host information
REMOTE_USER="root"
REMOTE_HOST="10.39.251.200"
REMOTE_PORT="22"

# Control socket for SSH multiplexing
CONTROL_SOCKET="/tmp/ssh_mux_${REMOTE_USER}_${REMOTE_HOST}_${REMOTE_PORT}.sock"

# Function to establish a master connection
establish_master_connection() {
    # Check if the control socket already exists
    if [ ! -S "$CONTROL_SOCKET" ]; then
        echo "Establishing new master connection..."
        # -f: Requests ssh to go to background just before command execution
        # -N: Do not execute a remote command
        # -M: Places the ssh client into “master” mode for connection sharing
        # -o ControlPath: Specifies the path to the control socket
        ssh -fN -M -o ControlPath="$CONTROL_SOCKET" -p "$REMOTE_PORT" "$REMOTE_USER@$REMOTE_HOST"
        if [ $? -eq 0 ]; then
            echo "Master connection established."
        else
            echo "Failed to establish master connection."
            exit 1
        fi
    else
        echo "Using existing master connection."
    fi
}

# Function to run a command on the remote host using the master connection
run_remote_command() {
    local command="$1"
    echo "Executing command on remote host: $command"
    ssh -o ControlPath="$CONTROL_SOCKET" -p "$REMOTE_PORT" "$REMOTE_USER@$REMOTE_HOST" "$command"
}

# Function to copy a file to the remote host using the master connection
copy_file_to_remote() {
    local local_path="$1"
    local remote_path="$2"
    echo "Copying file to remote host: $local_path -> $remote_path"
    scp -o ControlPath="$CONTROL_SOCKET" -P "$REMOTE_PORT" "$local_path" "$REMOTE_USER@$REMOTE_HOST":"$remote_path"
}

# Main script logic

# Establish or reuse a master connection
establish_master_connection

# Run a command on the remote host
run_remote_command "hostname"

# Copy a file to the remote host (specify the local and remote paths)
copy_file_to_remote "/path/to/local/file" "/path/to/remote/destination"

# Close the master connection when done
echo "Closing master connection..."
ssh -O exit -o ControlPath="$CONTROL_SOCKET" -p "$REMOTE_PORT" "$REMOTE_USER@$REMOTE_HOST"

 

Best Practices for Secure Multiplexing

1. Use Strong Authentication Methods: Always use strong authentication methods for SSH, such as public key authentication with a passphrase-protected private key. Avoid using weaker methods like password authentication, especially for accounts with elevated privileges.
2. Limit Multiplexing to Trusted Networks: Only enable multiplexing for connections over trusted networks. If you're connecting over untrusted or public networks, consider the potential risks and use VPNs or other secure tunnels to encapsulate your SSH connections.
3. Secure Control Socket Files: The control socket files used by SSH Multiplexing should be securely stored. Ensure that:
   • The directory containing control socket files (ControlPath) has restricted permissions, ideally only accessible by the user initiating the SSH connections.
   • Use the %C placeholder in ControlPath to ensure unique socket filenames, reducing the risk of socket file collisions or unauthorized access.
4. Implement Connection Timeouts: Use the ControlPersist option judiciously to automatically close master connections after a period of inactivity. This limits the window of opportunity for unauthorized use of an existing multiplexed connection.
5. Regularly Review and Prune SSH Configurations: Regularly audit your ~/.ssh/config and /etc/ssh/sshd_config files for unnecessary entries or overly permissive settings. Remove configurations for hosts you no longer connect to or do not require multiplexing.

 

Potential Risks and Mitigation Strategies

1. Unauthorized Access Through Control Sockets:
   • Risk: If an attacker gains access to a user's control socket file, they could piggyback on an existing authenticated session.
   • Mitigation: Store control socket files in a secure, non-writable directory. Use the ssh -O exit command to explicitly close master connections when not in use.
2. Compromised SSH Agent:
   • Risk: If the SSH agent is compromised, an attacker could use stored keys to establish new sessions via multiplexing.
   • Mitigation: Use ssh-add with the -t option to set a time limit for keys stored in the agent. Regularly purge keys from the agent when not in use.
3. Man-in-the-Middle Attacks:
   • Risk: An attacker could intercept SSH connections, particularly during the initial handshake before a multiplexed session is established.
   • Mitigation: Always verify host keys by checking ~/.ssh/known_hosts against known good values. Consider using SSH certificates instead of raw public keys for host authentication.
4. Session Hijacking:
   • Risk: An active multiplexed session could be hijacked if an attacker gains access to the client machine.
   • Mitigation: Use tmux or screen to add an additional layer of authentication and encryption to your terminal sessions. Enable SSH's built-in options for client and server to request checks on the integrity of the connection, preventing unauthorized session takeover.
5. Denial of Service (DoS):
   • Risk: Excessive multiplexed connections could overwhelm system resources, leading to a denial of service.
   • Mitigation: Configure MaxSessions and MaxStartups judiciously on the server side to limit the number of concurrent connections and sessions.

 

Summary

SSH Multiplexing stands out as a powerful feature within the SSH protocol suite, offering significant benefits in terms of efficiency, speed, and resource management for repeated SSH connections to the same server. By allowing multiple SSH sessions to share a single TCP/IP connection, it reduces the overhead associated with establishing new connections and performing multiple authentications. This capability is especially beneficial in scenarios involving frequent file transfers, remote command executions, and automated tasks that require SSH access. For more information on supported options you can refer man page of ssh command or we have also written a cheat sheet on SSH command.