Common issues after Disabling kerberos via Ambari

These issues I have faced when I enabled spnego for hadoop services. And following are my env details:

Ambari version 2.4.1.0 with Postgress db ,HDP2.5.3

Disabled Kerberos and getting following issues:

After trying to disable kerberos via the UI the following issues you may encounter so here I have tried to cover them with resolutions.

Issue 1. After disable kerberos ambari-server failing with following error

ERROR [main]

KerberosChecker:115 – Unable to obtain password from user

27 Feb 2017 08:55:14,166 WARN [main] KerberosChecker:81 – /etc/security/keytabs/ambari.server.keytab doesn’t exist.

27 Feb 2017 08:55:14,174 ERROR [main] KerberosChecker:115 – Unable to obtain password from user

27 Feb 2017 08:55:14,175 ERROR [main] AmbariServer:927 – Failed to run the Ambari Server

org.apache.ambari.server.AmbariException: Ambari Server Kerberos credentials check failed.

Check KDC availability and JAAS configuration in /etc/ambari-server/conf/krb5JAASLogin.conf

at org.apache.ambari.server.controller.utilities.KerberosChecker.checkJaasConfiguration(KerberosChecker.java:116)

at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:922)

RootCause:This issue was because you may have enabled ambari security to kerberos and by default ambari does not change few kerberos properties.

Solutions: To solve this issue you have to follwo the given steps: 

Step 1: Edit /etc/ambari-server/conf/ambari.properties to Ensure the following is set: kerberos.check.jaas.configuration=false

Step 2: Further disable kerberos for ambari server by removing:

delete values from AMBARI_JVM_ARGS in /var/lib/ambari-server/ambari-env.sh

“-Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf

-Djava.security.krb5.conf=/etc/krb5.conf

-Djavax.security.auth.useSubjectCredsOnly=false”

Step 3: Restart Ambari server and agent on all the worker nodes.

 

Issue 2. NameNode does not start and failed with following error in ambari

resource_management.core.exceptions.Fail: Execution of ‘ambari-sudo.sh su hdfs -l -s /bin/bash -c ‘ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh –config /usr/hdp/current/hadoop-client/conf start namenode” returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-m1.hdp22.out

RootCause: If you have enabled SPNEGO Authentication for hadoop components then you may encounter following namenode issue.

And following error in namenode logs:

2017-02-27 07:49:17,333 WARN mortbay.log (Slf4jLog.java:warn(76)) – failed authentication: javax.servlet.ServletException: javax.servlet.ServletException: Keytab does not exist: /etc/security/keytabs/spnego.service.keytab

2017-02-27 07:49:17,334 WARN mortbay.log (Slf4jLog.java:warn(89)) – Failed startup of context org.mortbay.jetty.webapp.WebAppContext@37912c1a{/,file:/usr/hdp/2.5.3.0-37/hadoop-hdfs/webapps/hdfs}

javax.servlet.ServletException: javax.servlet.ServletException: Keytab does not exist: /etc/security/keytabs/spnego.service.keytab       at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:241 Caused by: javax.servlet.ServletException: Keytab does not exist: /etc/security/keytabs/spnego.service.keytab

at rg.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:193)

… 24 more

Solutions: Remove following properties from Advanced core-site.

hadoop.http.authentication.kerberos.keytab

hadoop.http.authentication.kerberos.principal

hadoop.http.authentication.type

Restart HDFS restart and now NN will start without any issue.

 

Issue 3: Ambari not able identify active and standby node and failing to HA. Also other hadoop services like oozie,hive not getting start.

2017-02-28 02:11:21,360 – Getting jmx metrics from NN failed. URL: http://m1.hdp22:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem

Traceback (most recent call last):

File “/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py”, line 41, in get_value_from_jmx

data_dict = json.loads(data)

File “/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py”, line 307, in loads

return _default_decoder.decode(s)

File “/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py”, line 335, in decode

obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File “/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py”, line 353, in raw_decode

raise ValueError(“No JSON object could be decoded”)

ValueError: No JSON object could be decoded

2017-02-28 02:11:27,465 – Getting jmx metrics from NN failed. URL: http://m2.hdp22:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem

Traceback (most recent call last):

RootCause:This issue was also because of SPNEGO Authentication for Hadoop.

Try to open http://m1.hdp22:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem in browser and you will get Authentication issue 401 error.

 

Solutions: Apply following solution to resolve it. 

Change hadoop.http.authentication.simple.anonymous.allowed to true from false.

and remove following properties from ambari hdfs.

hadoop.http.authentication.cookie.domain

hadoop.http.authentication.signature.secret.file

hadoop.http.filter.initializers

hadoop.http.staticuser.user