The other day I was getting this error "Postponed publickey
" in my /var/log/sshd
file on RHEL 7. It took me some time to debug the error and understand the root cause. Let me tell you if you are looking for a one stop solution, you should go away as this would need some analysis on your environment.
There is no such single solution available for this error but I am hopeful that you should be able to fix it with the DEBUG steps I share here
ERROR: SSH connection fails with postponed publickey
Below is the snippet from my /var/log/sshd
. Depending upon distribution you may also get this error in /var/log/secure
or I personally prefer using journalctl
to check all the logs so that I don't have to go through different logs
May 22 12:46:59 server1 sshd[28720]: Connection from 10.43.138.1 port 46668 on 10.43.138.8 port 22 May 22 12:46:59 server1 sshd[28720]: Postponed publickey for root from 10.43.138.1 port 46668 ssh2 [preauth] May 22 12:47:02 server1 sshd[28720]: ROOT LOGIN REFUSED FROM 10.43.138.1 port 46668 May 22 12:47:02 server1 sshd[28720]: Failed publickey for root from 10.43.138.1 port 46668 ssh2: RSA SHA256:mTcxALlpalgvTOaASZ3xbykAoWLrPQk356cd/YhKgg8 May 22 12:47:02 server1 sshd[28720]: ROOT LOGIN REFUSED FROM 10.43.138.1 port 46668 [preauth]
I hope you are familiar with how logging works with systemd-journald and rsyslog which is completely different than traditional syslog
Any ways, let's come back to the topic.
DEBUG STEP 1: Understand your environment
- It is very important that you are familiar with your environment.
- You must know what kind of SSH authentication is allowed and expected. There are 6 different types of SSH authentication methods so which one do you use?
- As if you are getting this error means some configuration has been changed on the server side in
/etc/ssh/sshd_config
- But this can also be caused due to client SSH configuration i.e.
ssh_config
which is ideally under/etc/ssh
or one can also keep it underuser's home directory
- We create such client SSH configuration to parse additional arguments every time we do SSH to the server to avoid repeating the long arguments in the commands
DEBUG STEP 2: Check the logs
- The next thing we need to do is check the logs for hints and clues related to the SSH connection failure.
- The SSH log file location would vary based on your Linux or Unix distribution but I would recommend to use journalctl (if supported)
- Now
Postponed publickey
may not always mean that you have an error. - It can also mean that you have enabled different mode of SSH communication and
pubkey
auth
method is ignored and SSH looks for differentauth
method - For example, if we are getting this message in the provided order then it would mean that SSH was actually successful so it is important you check the complete message
Postponed publickey for user from 10.43.138.9 port 45514 ssh2 [preauth] Accepted publickey for user from 10.43.138.9 port 45514 ssh2
DEBUG STEP 3: Execute SSH with verbose
Now we know the logs and error message from server but it is not much helpful in our case.
Next we will capture the SSH Client logs using script
tool
# script /tmp/SSHclient
Script started, file is /tmp/SSHclient
This will start a new session and will capture everything from your screen and store it in the file /tmp/SSHclient
which we will use later for getting RCA
Next we execute SSH from the client using -vvv
to enable verbose
# ssh -vvv root@server1
I will only put the important messages to look out for in the verbose output
debug1: Authenticating to server1:22 as 'root' debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive debug3: authmethod_lookup gssapi-keyex debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug3: authmethod_is_enabled gssapi-with-mic debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1007) debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1007) debug2: we did not send a packet, disable method debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/deepak/.ssh/id_rsa debug3: send_pubkey_test debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 60 debug1: Server accepts key: pkalg rsa-sha2-512 blen 279 debug2: input_userauth_pk_ok: fp SHA256:mTcxALlpalgvTOaASZ3xbykAoWLrPQk356cd/YhKgg8 debug3: sign_and_send_pubkey: RSA SHA256:mTcxALlpalgvTOaASZ3xbykAoWLrPQk356cd/YhKgg8 Enter passphrase for key '/home/deepak/.ssh/id_rsa': debug3: authmethod_lookup keyboard-interactive debug3: remaining preferred: password debug3: authmethod_is_enabled keyboard-interactive debug1: Next authentication method: keyboard-interactive debug2: userauth_kbdint debug3: send packet: type 50 debug2: we sent a keyboard-interactive packet, wait for reply Password: debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive debug2: userauth_kbdint debug2: we sent a keyboard-interactive packet, wait for reply debug2: input_userauth_info_req debug2: input_userauth_info_req: num_prompts 1 Password:
Steps involved
- The supported list of authentication were
publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
- The first preferred
auth
mode wasgssapi-keyex
which failed due toNo valid Key exchange context
- Next SSH client used
gssapi-with-mic
which again failed due to "No Kerberos credentials available
" - Next preferred
auth
mode waspubkey
. Now this gets interesting, SSH was able to find the public key and asked for passphrase but for some reason the SSH didn't went through - Next SSH tried
keyboard-interactive
which again didn't worked - Lastly SSH tries to connect via password and that also fails
Solution:
Now we come to the last part of getting the Root Cause Analysis and fixing the error:
In my environment I know that we only allow SSH Public Key based authentication so the first thing we should do is disable all other types of SSH authentication methods from the server side i.e. /etc/ssh/sshd_config
and the client side /etc/ssh/ssh_config
Disable these in sshd_config
file (or if you have client configuration then disable in ssh_config
file also)
GSSAPIAuthentication no ChallengeResponseAuthentication no
Since we use Public key based authentication, this is enabled
PubkeyAuthentication yes
Next check for AllowUsers
and AllowGroups
section (if added) and make sure your user is allowed
Lastly in this case we know that public key authentication was not working so, I checked and found
PermitRootLogin no PasswordAuthentication yes
So we were getting the Password prompt because Password based authentication was also enabled and the Public Key Authentication was failing for user root because the PermitRootLogin
was set to "no
"
With PubkeyAuthentication
if you also expect to login as root then PermitRootLogin
must be set to without-password
With this change I restarted my sshd
# systemctl restart sshd
and I was able to login to my server. But you can also check the next chapter if you are still facing SSH connection failure.
Check SSH Permissions
It is also possible you have some permission issues with your SSH configuration, so make sure you look out for these messages on server and client side in the log files. These are just a few possible permission related error messages from SSH, you may get any other possible message as well:
Load key "/home/deepak/.ssh/id_rsa": Permission denied
Authentication refused: bad ownership or modes for file /root/.ssh/authorized_keys
User amit from server10 not allowed because none of user's groups are listed in AllowGroups
User root from 10.43.138.1 not allowed because none of user's groups are listed in AllowGroups
Recommended permission values for SSH related keys and files
~/.ssh/authorized_keys
must have644
permission- Private key file i.e.
~/.ssh/id_rsa
should be600
, the name of the key may be different as per user environment - Public key file i.e.
~/.ssh/id_rsa.pub
should be644
, the name of the key may be different as per user environment - The
~/.ssh
directory must not be world readable/writable so you can keep it with700
permission
Conclusion
Debugging the SSH connection failures are always tricky but there are different methods to enable verbose logging and with the additional debug messages it becomes little easier to find the root cause. It is always recommended to disable unwanted SSH authentication methods to reduce auth lookup time and also to avoid such errors.
It is also possible that an intruder can use such unused auth methods to gain access to your system. So from security point of view also you should consider to disable unused authentication methods.
Lastly I hope the steps from the article to debug and solve Postponed publickey error on Linux was helpful. So, let me know your suggestions and feedback using the comment section.
References:
I have used below external references for this tutorial guide
ssh postponed publickey error in /var/log/secure: "Accepted publickey for <user>"
6 ssh authentication methods to secure connection (sshd_config)
How to configure ssh host based authentication per user (CentOS/RHEL 7/8)