SOLVED: SSH fails with postponed publickey error

The other day I was getting this error "Postponed publickey" in my /var/log/sshd file on RHEL 7. It took me some time to debug the error and understand the root cause. Let me tell you if you are looking for a one stop solution, you should go away as this would need some analysis on your environment.

There is no such single solution available for this error but I am hopeful that you should be able to fix it with the DEBUG steps I share here

 

ERROR: SSH connection fails with postponed publickey

Below is the snippet from my /var/log/sshd. Depending upon distribution you may also get this error in /var/log/secure or I personally prefer using journalctl to check all the logs so that I don't have to go through different logs

May 22 12:46:59 server1 sshd[28720]: Connection from 10.43.138.1 port 46668 on 10.43.138.8 port 22
May 22 12:46:59 server1 sshd[28720]: Postponed publickey for root from 10.43.138.1 port 46668 ssh2 [preauth]
May 22 12:47:02 server1 sshd[28720]: ROOT LOGIN REFUSED FROM 10.43.138.1 port 46668
May 22 12:47:02 server1 sshd[28720]: Failed publickey for root from 10.43.138.1 port 46668 ssh2: RSA SHA256:mTcxALlpalgvTOaASZ3xbykAoWLrPQk356cd/YhKgg8
May 22 12:47:02 server1 sshd[28720]: ROOT LOGIN REFUSED FROM 10.43.138.1 port 46668 [preauth]

I hope you are familiar with how logging works with systemd-journald and rsyslog which is completely different than traditional syslog

Any ways, let's come back to the topic.

 

DEBUG STEP 1: Understand your environment

  • It is very important that you are familiar with your environment.
  • You must know what kind of SSH authentication is allowed and expected. There are 6 different types of SSH authentication methods so which one do you use?
  • As if you are getting this error means some configuration has been changed on the server side in /etc/ssh/sshd_config
  • But this can also be caused due to client SSH configuration i.e. ssh_config which is ideally under /etc/ssh or one can also keep it under user's home directory
  • We create such client SSH configuration to parse additional arguments every time we do SSH to the server to avoid repeating the long arguments in the commands

 

DEBUG STEP 2: Check the logs

  • The next thing we need to do is check the logs for hints and clues related to the SSH connection failure.
  • The SSH log file location would vary based on your Linux or Unix distribution but I would recommend to use journalctl (if supported)
  • Now Postponed publickey may not always mean that you have an error.
  • It can also mean that you have enabled different mode of SSH communication and pubkey auth method is ignored and SSH looks for different auth method
  • For example, if we are getting this message in the provided order then it would mean that SSH was actually successful so it is important you check the complete message
Postponed publickey for user from 10.43.138.9 port 45514 ssh2 [preauth]
Accepted publickey for user from 10.43.138.9 port 45514 ssh2

 

DEBUG STEP 3: Execute SSH with verbose

Now we know the logs and error message from server but it is not much helpful in our case.
Next we will capture the SSH Client logs using script tool

# script /tmp/SSHclient
Script started, file is /tmp/SSHclient

This will start a new session and will capture everything from your screen and store it in the file /tmp/SSHclient which we will use later for getting RCA
Next we execute SSH from the client using -vvv to enable verbose

# ssh -vvv root@server1

I will only put the important messages to look out for in the verbose output

debug1: Authenticating to server1:22 as 'root'
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug3: authmethod_lookup gssapi-keyex

debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context

debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1007)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1007)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/deepak/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug2: input_userauth_pk_ok: fp SHA256:mTcxALlpalgvTOaASZ3xbykAoWLrPQk356cd/YhKgg8
debug3: sign_and_send_pubkey: RSA SHA256:mTcxALlpalgvTOaASZ3xbykAoWLrPQk356cd/YhKgg8
Enter passphrase for key '/home/deepak/.ssh/id_rsa':

debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
Password:
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:

 

Steps involved

  • The supported list of authentication were publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
  • The first preferred auth mode was gssapi-keyex which failed due to No valid Key exchange context
  • Next SSH client used gssapi-with-mic which again failed due to "No Kerberos credentials available"
  • Next preferred auth mode was pubkey. Now this gets interesting, SSH was able to find the public key and asked for passphrase but for some reason the SSH didn't went through
  • Next SSH tried keyboard-interactive which again didn't worked
  • Lastly SSH tries to connect via password and that also fails

 

Solution:

Now we come to the last part of getting the Root Cause Analysis and fixing the error:

In my environment I know that we only allow SSH Public Key based authentication so the first thing we should do is disable all other types of SSH authentication methods from the server side i.e. /etc/ssh/sshd_config and the client side /etc/ssh/ssh_config
Disable these in sshd_config file (or if you have client configuration then disable in ssh_config file also)

GSSAPIAuthentication no
ChallengeResponseAuthentication no

Since we use Public key based authentication, this is enabled

PubkeyAuthentication yes

Next check for AllowUsers and AllowGroups section (if added) and make sure your user is allowed

Lastly in this case we know that public key authentication was not working so, I checked and found

PermitRootLogin no
PasswordAuthentication yes

So we were getting the Password prompt because Password based authentication was also enabled and the Public Key Authentication was failing for user root because the PermitRootLogin was set to "no"

With PubkeyAuthentication if you also expect to login as root then PermitRootLogin must be set to without-password

With this change I restarted my sshd

# systemctl restart sshd

and I was able to login to my server. But you can also check the next chapter if you are still facing SSH connection failure.

 

Check SSH Permissions

It is also possible you have some permission issues with your SSH configuration, so make sure you look out for these messages on server and client side in the log files. These are just a few possible permission related error messages from SSH, you may get any other possible message as well:

Load key "/home/deepak/.ssh/id_rsa": Permission denied
Authentication refused: bad ownership or modes for file /root/.ssh/authorized_keys
User amit from server10 not allowed because none of user's groups are listed in AllowGroups
User root from 10.43.138.1 not allowed because none of user's groups are listed in AllowGroups

 

Recommended permission values for SSH related keys and files

  • ~/.ssh/authorized_keys must have 644 permission
  • Private key file i.e. ~/.ssh/id_rsa should be 600, the name of the key may be different as per user environment
  • Public key file i.e. ~/.ssh/id_rsa.pub should be 644, the name of the key may be different as per user environment
  • The ~/.ssh directory must not be world readable/writable so you can keep it with 700 permission

 

Conclusion

Debugging the SSH connection failures are always tricky but there are different methods to enable verbose logging and with the additional debug messages it becomes little easier to find the root cause. It is always recommended to disable unwanted SSH authentication methods to reduce auth lookup time and also to avoid such errors.

It is also possible that an intruder can use such unused auth methods to gain access to your system. So from security point of view also you should consider to disable unused authentication methods.

Lastly I hope the steps from the article to debug and solve Postponed publickey error on Linux was helpful. So, let me know your suggestions and feedback using the comment section.

 

References:

I have used below external references for this tutorial guide
ssh postponed publickey error in /var/log/secure: "Accepted publickey for <user>"
6 ssh authentication methods to secure connection (sshd_config)
How to configure ssh host based authentication per user (CentOS/RHEL 7/8)

Leave a Comment

Please use shortcodes <pre class=comments>your code</pre> for syntax highlighting when adding code.