KDC Kerberized Yarn Services Fail to Start on Isilon OneFS 8.0.1 with Ambari via WebHDFS curl calls

An issue has been seen when a KDC kerberized Hortonworks cluster attempts to start Yarn services or any other services that leverage WebHDFS to start with OneFS 8.0.1. The incorrect generation of the the krb5.conf can leave the file without a READ permission for the services handling WebHDFS calls and authentication cannot occur leading to the services failing to start.


On starting the Yarn services Ambari makes a kerberized WebHDFS call to Isilon, this fails with a general http 401 error and no yarn service can start. (any service using a Kerberized WebHDFS call will show similar behavior)

The yarn service log looks similar to this:

File “/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py”, line 305, in _get_file_status

list_status = self.util.run_command(target, ‘GETFILESTATUS’, method=’GET’, ignore_status_codes=[‘404’], assertable_result=False)

File “/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py”, line 210, in run_command

raise Fail(err_msg)

resource_management.core.exceptions.Fail: Execution of ‘curl -sS -L -w ‘%{http_code}’ -X GET –negotiate -u : ‘http://rip2-horton1.foo.com:8082/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs” returned status_code=401.



<title>401 Authorization Required</title>


<h1>Authorization Required</h1>

<p>This server could not verify that you

are authorized to access the document

requested.  Either you supplied the wrong

credentials (e.g., bad password), or your

browser doesn’t understand how to supply

the credentials required.</p>


If the Kerberized curl command is run interactively it still fails with the same http 401 error.

No error are logged in the Isilon hdfs log for this issue, in fact the only log entries are located in the Apache log in the form of.

From the apache error log:

2016-11-28T14:00:19-08:00 <18.3> rip2-horton1.foo.com httpd[76700]: [error] [client] gss_accept_sec_context() failed: Unspecified GSS failure. Minor code may provide more information (Permission denied)

2016-11-28T14:00:20-08:00 <18.3>rip2-horton1.foo.com httpd[76700]: [error] [client] gss_accept_sec_context() failed: Unspecified GSS failure. Minor code may provide more information (Permission denied)

The issue is caused by the permissions being incorrectly set on the /etc/krb5.conf files on cluster nodes. Since the WebHDFS calls are initially handled by Apache, which is running as a daemon user this user cannot read the /etc/krb5.conf and authentication fails before it has even started.


Taking a look at the permissions on the /etc/krb5.conf file show that no permission exist on Other, with no read on Other the Apache user cannot read the file and authentication fails.


rip2-1# ls -le /etc/krb5.conf

-rw-r—–    1 root  wheel  400 Aug 23 16:43 /etc/krb5.conf


In order to resolve this issue just add a READ permission on other for the krb5.conf on all nodes.

#isi_for_array chmod o+r /etc/krb5.conf


Review the updated permissions:

rip2-1# ls -le /etc/krb5.conf

-rw-r–r–    1 root  wheel  400 Aug 23 16:43 /etc/krb5.conf


Restart the Yarn Services, now the Apache user can read the krb5.conf authentication can occur and service will start, you will now see Kerberos entries in the hdfs.log for these service start calls(you may need to increase logging level and view on the node the call is made to)

This issue has also been seen on 8.0.1 clusters that have upgrade from 8.0.0.x to, other configurations may also cause http 401 errors on service starts outside of this krb5.conf file permission issues. Other likely causes are incorrect permission on the hdfs root system directories but that is a topic for another blog.


KDC Kerberized Yarn Service Fail to Start on 8.0.1 with Ambari via WebHDFS curl calls

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s