Why was SSHD refusing my key?

I was contacted by a colleague about a problem he was having. “I’m trying to set up something simple which I’ve done millions of times, but it’s not working. I might be missing something obvious.”

The issue was that the SSH public key authentication didn’t work. The environment was running a virtualized Oracle Enterprise Linux 6.4 operating system (similar to Red Hat Enterprise Linux RHEL or Centos 6.) We’ll call this box Badboy for the purpose of this post.

I logged onto Badboy and attempted to do it myself, following the basic steps to set up public key authentication on Linux:

[[email protected] ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/knecht/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/knecht/.ssh/id_rsa.
Your public key has been saved in /home/knecht/.ssh/id_rsa.pub.
The key fingerprint is:
0c:32:d6:9e:4f:82:7d:f6:ff:c7:f3:0b:c6:a4:88:29 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| |
| . |
| + o |
| . * + . . |
| . = S . ..+ |
| *o...+. o |
| E o.... + .|
| . . . .|
| . |
+-----------------+

I then copied that public key file to Badboy and added the key to the ~/.ssh/authorized_keys file.

Let’s verify that the public key authentication has not been disabled in sshd_config:

[[email protected] ~]# grep Pubkey /etc/ssh/sshd_config
#PubkeyAuthentication yes

The value isn’t set explicitly, so the default setting comes into play which is yes.

I then switched back to my random-client and attempted to log in using SSH with the public key.

[[email protected] ~]$ ssh [email protected]
[email protected]'s password:

Hmm, it’s asking for a password. That’s not what you would expect. Let’s disable password authentication entirely in the client:

[[email protected] ~]$ ssh -o "PasswordAuthentication no" [email protected]
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[[email protected] ~]$

Interesting.

This box was really not behaving nicely. Badboy it is then.

I checked the logs to see if anything shows up by doing a tail -f /var/log/secure while logging on from a second session, trying again with password authentication disabled.

The only entry that showed up was:

Nov 21 20:14:21 badboy sshd[4299]: Connection closed by xx.xx.xxx.xxx

That’s helpful. Moving on.

Hoping to get a little bit more insight, I increased the logging verbosity of sshd, by changing LogLevel in sshd_config

LogLevel VERBOSE

Retrying the previous exercise, we now have a wee bit more information:

Nov 21 20:21:59 badboy sshd[4980]: Failed publickey for knecht from xx.xx.xxx.xxx port 36189 ssh2
Nov 21 20:21:59 badboy sshd[4981]: Connection closed by xx.xx.xxx.xxx

Well, you’re not very talkative Mr. Badboy, are you? I went again and triple-checked all of the possible options, read the main pages, and couldn’t determine what was wrong with Badboy’s configuration. As my colleague mentioned, it’s a simple exercise. We’ve all done it countless times… So what was the problem now?

It dawned on me that I had a similar experience in the past, when things just wouldn’t work without any clear reason. The new suspect was now Mr. Badboy’s big brother, SELinux.

[[email protected] ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: enforcing
Mode from config file: enforcing
Policy version: 26
Policy from config file: targeted


Well, there you have it. A quick peek in /var/log/audit/audit.log showed several actions being denied during an attempted connection.

How to configure SELinux to get this to work is beyond the scope of this document. In our environment, we are not using SELinux so we disabled it by setting SELINUX=disabled
in /etc/selinux/config and rebooted the system.

As soon as it was back up:

[[email protected] ~]# ssh [email protected]
[[email protected] ~]#

Good boy.

Want to talk with an expert? Schedule a call with our team to get the conversation started.