After upgrading ambari it is not coming up (hostcomponentdesiredstate.admin_state)
Category : Ambari
If you upgrade ambari and in case if you see following error then you should not worry, following steps will help you to bring your cluster into running state.
Issue: Once you upgrade your cluster and after restarting you don’t see any service or their metrics on ambari then you need following given steps.
You also can see following error in your ambari-server logs.
23 Sep 2016 05:08:13,966 ERROR [main] ViewRegistry:1695 – Caught exception loading view TEZ{0.7.0.2.5.0.0-1}
Local Exception Stack:
Exception [EclipseLink-116] (Eclipse Persistence Services – 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DescriptorException
Exception Description: No conversion value provided for the value [NULL] in field [hostcomponentdesiredstate.admin_state].
Mapping: org.eclipse.persistence.mappings.DirectToFieldMapping[adminState–>hostcomponentdesiredstate.admin_state]
Descriptor: RelationalDescriptor(org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity –> [DatabaseTable(hostcomponentdesiredstate)])
at org.eclipse.persistence.exceptions.DescriptorException.noFieldValueConversionToAttributeValueProvided(DescriptorException.java:1066)
at org.eclipse.persistence.mappings.converters.ObjectTypeConverter.convertDataValueToObjectValue(ObjectTypeConverter.java:226)
at org.eclipse.persistence.mappings.converters.EnumTypeConverter.convertDataValueToObjectValue(EnumTypeConverter.java:141)
at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.getObjectValue(AbstractDirectMapping.java:616)
at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.valueFromRow(AbstractDirectMapping.java:1220)
at org.eclipse.persistence.mappings.DatabaseMapping.readFromRowIntoObject(DatabaseMapping.java:1539)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:462)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1005)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:899)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:852)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:735)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:689)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:805)
at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:895)
at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:562)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1175)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134)
at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:441)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1222)
Resolutions :
Step 1: Check all installed services admin_status in hostcomponentdesiredstate table. You should not have any NULL value in admin_state for any service.
mysql> select * from hostcomponentdesiredstate;
+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+
| cluster_id | component_name | desired_state | service_name | admin_state | maintenance_state | restart_required | security_state | host_id | desired_stack_id |
+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+
| 2 | APP_TIMELINE_SERVER | STARTED | YARN | NULL | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | ATLAS_SERVER | STARTED | ATLAS | NULL | OFF | 0 | UNSECURED | 1 | 1 |
| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |
| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |
| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |
| 2 | DRPC_SERVER | STARTED | STORM | NULL | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | FALCON_CLIENT | INSTALLED | FALCON | NULL | OFF | 0 | UNSECURED | 1 | 1 |
| 2 | FALCON_CLIENT | INSTALLED | FALCON | NULL | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | FALCON_SERVER | STARTED | FALCON | NULL | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |
| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |
| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |
| 2 | HBASE_CLIENT | INSTALLED | HBASE | NULL | OFF | 0 | UNSECURED | 1 | 1 |
| 2 | HBASE_CLIENT | INSTALLED | HBASE | NULL | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | HBASE_MASTER | STARTED | HBASE | NULL | OFF | 0 | UNSECURED | 1 | 1 |
| 2 | HBASE_REGIONSERVER | STARTED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |
| 2 | HBASE_REGIONSERVER | STARTED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |
| 2 | HBASE_REGIONSERVER | STARTED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |
| 2 | HCAT | INSTALLED | HIVE | NULL | OFF | 0 | UNSECURED | 1 | 1 |
| 2 | HCAT | INSTALLED | HIVE | NULL | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | HDFS_CLIENT | INSTALLED | HDFS | NULL | OFF | 0 | UNSECURED | 1 | 1 |
| 2 | HDFS_CLIENT | INSTALLED | HDFS | NULL | OFF | 0 | UNSECURED | 2 | 1 |
Step2: In that case you have to update this table manually :
mysql> update hostcomponentdesiredstate set admin_state=’INSERVICE’;
Query OK, 58 rows affected (0.01 sec)
Rows matched: 92 Changed: 58 Warnings: 0
Step 3: Now you can check once again the admin_state to have all inservice:
mysql> select * from hostcomponentdesiredstate;
+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+
| cluster_id | component_name | desired_state | service_name | admin_state | maintenance_state | restart_required | security_state | host_id | desired_stack_id |
+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+
| 2 | APP_TIMELINE_SERVER | STARTED | YARN | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | ATLAS_SERVER | STARTED | ATLAS | INSERVICE | OFF | 0 | UNSECURED | 1 | 1 |
| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |
| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |
| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |
| 2 | DRPC_SERVER | STARTED | STORM | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | FALCON_CLIENT | INSTALLED | FALCON | INSERVICE | OFF | 0 | UNSECURED | 1 | 1 |
| 2 | FALCON_CLIENT | INSTALLED | FALCON | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | FALCON_SERVER | STARTED | FALCON | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |
| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |
| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |
| 2 | HBASE_CLIENT | INSTALLED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 1 | 1 |
| 2 | HBASE_CLIENT | INSTALLED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |
| 2 | HBASE_MASTER | STARTED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 1 | 1 |
Step 4: Now restart your ambari and you should be good with your cluster.
[ambari@server1 ~]$ ambari-server restart
Using python /usr/bin/python
Restarting ambari-server
Using python /usr/bin/python
Stopping ambari-server
Ambari Server stopped
Using python /usr/bin/python
Starting ambari-server
Organizing resource files at /var/lib/ambari-server/resources…
Unable to check firewall status when starting without root privileges.
Please do not forget to disable or adjust firewall if needed
Ambari database consistency check started…
No errors were found.
Ambari database consistency check finished
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start………………..
Ambari Server ‘start’ completed successfully.