Common issues after Disabling kerberos via Ambari

These issues I have faced when I enabled spnego for hadoop services. And following are my env details:

Ambari version with Postgress db ,HDP2.5.3

Disabled Kerberos and getting following issues:

After trying to disable kerberos via the UI the following issues you may encounter so here I have tried to cover them with resolutions.

Issue 1. After disable kerberos ambari-server failing with following error

ERROR [main]

KerberosChecker:115 – Unable to obtain password from user

27 Feb 2017 08:55:14,166 WARN [main] KerberosChecker:81 – /etc/security/keytabs/ambari.server.keytab doesn’t exist.

27 Feb 2017 08:55:14,174 ERROR [main] KerberosChecker:115 – Unable to obtain password from user

27 Feb 2017 08:55:14,175 ERROR [main] AmbariServer:927 – Failed to run the Ambari Server

org.apache.ambari.server.AmbariException: Ambari Server Kerberos credentials check failed.

Check KDC availability and JAAS configuration in /etc/ambari-server/conf/krb5JAASLogin.conf

at org.apache.ambari.server.controller.utilities.KerberosChecker.checkJaasConfiguration(

at org.apache.ambari.server.controller.AmbariServer.main(

RootCause:This issue was because you may have enabled ambari security to kerberos and by default ambari does not change few kerberos properties.

Solutions: To solve this issue you have to follwo the given steps: 

Step 1: Edit /etc/ambari-server/conf/ to Ensure the following is set: kerberos.check.jaas.configuration=false

Step 2: Further disable kerberos for ambari server by removing:

delete values from AMBARI_JVM_ARGS in /var/lib/ambari-server/


Step 3: Restart Ambari server and agent on all the worker nodes.


Issue 2. NameNode does not start and failed with following error in ambari

resource_management.core.exceptions.Fail: Execution of ‘ su hdfs -l -s /bin/bash -c ‘ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/ –config /usr/hdp/current/hadoop-client/conf start namenode” returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-m1.hdp22.out

RootCause: If you have enabled SPNEGO Authentication for hadoop components then you may encounter following namenode issue.

And following error in namenode logs:

2017-02-27 07:49:17,333 WARN mortbay.log ( – failed authentication: javax.servlet.ServletException: javax.servlet.ServletException: Keytab does not exist: /etc/security/keytabs/spnego.service.keytab

2017-02-27 07:49:17,334 WARN mortbay.log ( – Failed startup of context org.mortbay.jetty.webapp.WebAppContext@37912c1a{/,file:/usr/hdp/}

javax.servlet.ServletException: javax.servlet.ServletException: Keytab does not exist: /etc/security/keytabs/spnego.service.keytab       at Caused by: javax.servlet.ServletException: Keytab does not exist: /etc/security/keytabs/spnego.service.keytab


… 24 more

Solutions: Remove following properties from Advanced core-site.




Restart HDFS restart and now NN will start without any issue.


Issue 3: Ambari not able identify active and standby node and failing to HA. Also other hadoop services like oozie,hive not getting start.

2017-02-28 02:11:21,360 – Getting jmx metrics from NN failed. URL: http://m1.hdp22:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem

Traceback (most recent call last):

File “/usr/lib/python2.6/site-packages/resource_management/libraries/functions/”, line 41, in get_value_from_jmx

data_dict = json.loads(data)

File “/usr/lib/python2.6/site-packages/ambari_simplejson/”, line 307, in loads

return _default_decoder.decode(s)

File “/usr/lib/python2.6/site-packages/ambari_simplejson/”, line 335, in decode

obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File “/usr/lib/python2.6/site-packages/ambari_simplejson/”, line 353, in raw_decode

raise ValueError(“No JSON object could be decoded”)

ValueError: No JSON object could be decoded

2017-02-28 02:11:27,465 – Getting jmx metrics from NN failed. URL: http://m2.hdp22:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem

Traceback (most recent call last):

RootCause:This issue was also because of SPNEGO Authentication for Hadoop.

Try to open http://m1.hdp22:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem in browser and you will get Authentication issue 401 error.


Solutions: Apply following solution to resolve it. 

Change hadoop.http.authentication.simple.anonymous.allowed to true from false.

and remove following properties from ambari hdfs.