HDFS Authentication Puzzle

If you are interested in understanding HDFS permissions model, you would google something like “hdfs adming guide”, which will probably bring you to this page. If you start reading it you will soone see this:

In this release of Hadoop the identity of a client process is just whatever the host operating system says it is. For Unix-like systems,

• The user name is the equivalent of `whoami`;
• The group list is the equivalent of `bash -c groups`.

In the future there will be other ways of establishing user identity (think Kerberos, LDAP, and others). There is no expectation that this first method is secure in protecting one user from impersonating another. This user identity mechanism combined with the permissions model allows a cooperative community to share file system resources in an organized fashion.

This is as scary as it looks: If somebody substitutes `whoami` command, he will be able to fake any user when connecting to HDFS.
But wait, the documentation is for version 0.19.1, which is an old one. Indeed,  look at the code for org.apache.hadoop.security in this version you will see this:

static String getUnixUserName() throws IOException {
String[] result = executeShellCommand(
new String[]{Shell.USER_NAME_COMMAND}); <-- this is not good!
if (result.length!=1) {
throw new IOException("Expect one token as the result of " +
Shell.USER_NAME_COMMAND + ": " + toString(result));
}
return result[0];
}


Okay, let’s now take a look at the same code in a more recent 0.20 version:

static Subject getCurrentUser() {
return Subject.getSubject(AccessController.getContext());
}


The whole user authentication module was rewritten to use Java’s SecurityManager API, which is not as easy to trick into thinking you are someone else. So, the code was changed. Now let’s check documentation for the latest versions. This is latest documentation version on Apache Hadoop page. I hope you will be as surprised and amazed as I was when I found out, but docs were not updated and are still referring to the old implementation of security module, which is very confusing and can lead many people to think Hadoop is using an even more primitive authentication model than it does.

Discover more about our expertise in Hadoop