These issues I have faced when I enabled spnego for hadoop services. And following are my env details:
Ambari version 2.4.1.0 with Postgress db ,HDP2.5.3
Disabled Kerberos and getting following issues:
After trying to disable kerberos via the UI the following issues you may encounter so here I have tried to cover them with resolutions.
Issue 1. After disable kerberos ambari-server failing with following error
ERROR [main]
KerberosChecker:115 – Unable to obtain password from user
27 Feb 2017 08:55:14,166 WARN [main] KerberosChecker:81 – /etc/security/keytabs/ambari.server.keytab doesn’t exist.
27 Feb 2017 08:55:14,174 ERROR [main] KerberosChecker:115 – Unable to obtain password from user
27 Feb 2017 08:55:14,175 ERROR [main] AmbariServer:927 – Failed to run the Ambari Server
org.apache.ambari.server.AmbariException: Ambari Server Kerberos credentials check failed.
Check KDC availability and JAAS configuration in /etc/ambari-server/conf/krb5JAASLogin.conf
at org.apache.ambari.server.controller.utilities.KerberosChecker.checkJaasConfiguration(KerberosChecker.java:116)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:922)
RootCause:This issue was because you may have enabled ambari security to kerberos and by default ambari does not change few kerberos properties.
Solutions: To solve this issue you have to follwo the given steps:
Step 1: Edit /etc/ambari-server/conf/ambari.properties to Ensure the following is set: kerberos.check.jaas.configuration=false
Step 2: Further disable kerberos for ambari server by removing:
delete values from AMBARI_JVM_ARGS in /var/lib/ambari-server/ambari-env.sh
“-Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf
-Djava.security.krb5.conf=/etc/krb5.conf
-Djavax.security.auth.useSubjectCredsOnly=false”
Step 3: Restart Ambari server and agent on all the worker nodes.
Issue 2. NameNode does not start and failed with following error in ambari
resource_management.core.exceptions.Fail: Execution of ‘ambari-sudo.sh su hdfs -l -s /bin/bash -c ‘ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh –config /usr/hdp/current/hadoop-client/conf start namenode” returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-m1.hdp22.out
RootCause: If you have enabled SPNEGO Authentication for hadoop components then you may encounter following namenode issue.
And following error in namenode logs:
2017-02-27 07:49:17,333 WARN mortbay.log (Slf4jLog.java:warn(76)) – failed authentication: javax.servlet.ServletException: javax.servlet.ServletException: Keytab does not exist: /etc/security/keytabs/spnego.service.keytab
2017-02-27 07:49:17,334 WARN mortbay.log (Slf4jLog.java:warn(89)) – Failed startup of context org.mortbay.jetty.webapp.WebAppContext@37912c1a{/,file:/usr/hdp/2.5.3.0-37/hadoop-hdfs/webapps/hdfs}
javax.servlet.ServletException: javax.servlet.ServletException: Keytab does not exist: /etc/security/keytabs/spnego.service.keytab at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:241 Caused by: javax.servlet.ServletException: Keytab does not exist: /etc/security/keytabs/spnego.service.keytab
at rg.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:193)
… 24 more
Solutions: Remove following properties from Advanced core-site.
hadoop.http.authentication.kerberos.keytab
hadoop.http.authentication.kerberos.principal
hadoop.http.authentication.type
Restart HDFS restart and now NN will start without any issue.
Issue 3: Ambari not able identify active and standby node and failing to HA. Also other hadoop services like oozie,hive not getting start.
2017-02-28 02:11:21,360 – Getting jmx metrics from NN failed. URL: http://m1.hdp22:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
File “/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py”, line 41, in get_value_from_jmx
data_dict = json.loads(data)
File “/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py”, line 307, in loads
return _default_decoder.decode(s)
File “/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py”, line 335, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File “/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py”, line 353, in raw_decode
raise ValueError(“No JSON object could be decoded”)
ValueError: No JSON object could be decoded
2017-02-28 02:11:27,465 – Getting jmx metrics from NN failed. URL: http://m2.hdp22:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
RootCause:This issue was also because of SPNEGO Authentication for Hadoop.
Try to open http://m1.hdp22:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem in browser and you will get Authentication issue 401 error.
Solutions: Apply following solution to resolve it.
Change hadoop.http.authentication.simple.anonymous.allowed to true from false.
and remove following properties from ambari hdfs.
hadoop.http.authentication.cookie.domain
hadoop.http.authentication.signature.secret.file
hadoop.http.filter.initializers
hadoop.http.staticuser.user