After upgrading ambari it is not coming up (hostcomponentdesiredstate.admin_state)

  • 0

After upgrading ambari it is not coming up (hostcomponentdesiredstate.admin_state)

If you upgrade ambari and in case if you see following error then you should not worry, following steps will help you to bring your cluster into running state.

Issue: Once you upgrade your cluster and after restarting you don’t see any service or their metrics on ambari then you need following given steps.

You also can see following error in your ambari-server logs.

23 Sep 2016 05:08:13,966 ERROR [main] ViewRegistry:1695 – Caught exception loading view TEZ{0.7.0.2.5.0.0-1}
Local Exception Stack:
Exception [EclipseLink-116] (Eclipse Persistence Services – 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DescriptorException
Exception Description: No conversion value provided for the value [NULL] in field [hostcomponentdesiredstate.admin_state].
Mapping: org.eclipse.persistence.mappings.DirectToFieldMapping[adminState–>hostcomponentdesiredstate.admin_state]
Descriptor: RelationalDescriptor(org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity –> [DatabaseTable(hostcomponentdesiredstate)])
at org.eclipse.persistence.exceptions.DescriptorException.noFieldValueConversionToAttributeValueProvided(DescriptorException.java:1066)
at org.eclipse.persistence.mappings.converters.ObjectTypeConverter.convertDataValueToObjectValue(ObjectTypeConverter.java:226)
at org.eclipse.persistence.mappings.converters.EnumTypeConverter.convertDataValueToObjectValue(EnumTypeConverter.java:141)
at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.getObjectValue(AbstractDirectMapping.java:616)
at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.valueFromRow(AbstractDirectMapping.java:1220)
at org.eclipse.persistence.mappings.DatabaseMapping.readFromRowIntoObject(DatabaseMapping.java:1539)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:462)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1005)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:899)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:852)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:735)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:689)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:805)
at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:895)
at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:562)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1175)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134)
at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:441)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1222)

Resolutions : 

Step 1: Check all installed services admin_status in hostcomponentdesiredstate table. You should not have any NULL value in admin_state for any service. 

mysql> select * from hostcomponentdesiredstate;

+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+

| cluster_id | component_name         | desired_state | service_name   | admin_state | maintenance_state | restart_required | security_state | host_id | desired_stack_id |

+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+

|          2 | APP_TIMELINE_SERVER    | STARTED       | YARN           | NULL        | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | ATLAS_SERVER           | STARTED       | ATLAS          | NULL        | OFF               |                0 | UNSECURED      |       1 |                1 |

|          2 | DATANODE               | STARTED       | HDFS           | INSERVICE   | OFF               |                0 | UNSECURED      |       3 |                1 |

|          2 | DATANODE               | STARTED       | HDFS           | INSERVICE   | OFF               |                0 | UNSECURED      |       4 |                1 |

|          2 | DATANODE               | STARTED       | HDFS           | INSERVICE   | OFF               |                0 | UNSECURED      |       5 |                1 |

|          2 | DRPC_SERVER            | STARTED       | STORM          | NULL        | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | FALCON_CLIENT          | INSTALLED     | FALCON         | NULL        | OFF               |                0 | UNSECURED      |       1 |                1 |

|          2 | FALCON_CLIENT          | INSTALLED     | FALCON         | NULL        | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | FALCON_SERVER          | STARTED       | FALCON         | NULL        | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | FLUME_HANDLER          | STARTED       | FLUME          | INSERVICE   | OFF               |                0 | UNSECURED      |       3 |                1 |

|          2 | FLUME_HANDLER          | STARTED       | FLUME          | INSERVICE   | OFF               |                0 | UNSECURED      |       4 |                1 |

|          2 | FLUME_HANDLER          | STARTED       | FLUME          | INSERVICE   | OFF               |                0 | UNSECURED      |       5 |                1 |

|          2 | HBASE_CLIENT           | INSTALLED     | HBASE          | NULL        | OFF               |                0 | UNSECURED      |       1 |                1 |

|          2 | HBASE_CLIENT           | INSTALLED     | HBASE          | NULL        | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | HBASE_MASTER           | STARTED       | HBASE          | NULL        | OFF               |                0 | UNSECURED      |       1 |                1 |

|          2 | HBASE_REGIONSERVER     | STARTED       | HBASE          | INSERVICE   | OFF               |                0 | UNSECURED      |       3 |                1 |

|          2 | HBASE_REGIONSERVER     | STARTED       | HBASE          | INSERVICE   | OFF               |                0 | UNSECURED      |       4 |                1 |

|          2 | HBASE_REGIONSERVER     | STARTED       | HBASE          | INSERVICE   | OFF               |                0 | UNSECURED      |       5 |                1 |

|          2 | HCAT                   | INSTALLED     | HIVE           | NULL        | OFF               |                0 | UNSECURED      |       1 |                1 |

|          2 | HCAT                   | INSTALLED     | HIVE           | NULL        | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | HDFS_CLIENT            | INSTALLED     | HDFS           | NULL        | OFF               |                0 | UNSECURED      |       1 |                1 |

|          2 | HDFS_CLIENT            | INSTALLED     | HDFS           | NULL        | OFF               |                0 | UNSECURED      |       2 |                1 |

Step2: In that case you have to update this table manually :

mysql> update hostcomponentdesiredstate set admin_state=’INSERVICE’;

Query OK, 58 rows affected (0.01 sec)

Rows matched: 92  Changed: 58  Warnings: 0

Step 3: Now you can check once again the  admin_state to have all inservice: 

mysql> select * from hostcomponentdesiredstate;

+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+

| cluster_id | component_name         | desired_state | service_name   | admin_state | maintenance_state | restart_required | security_state | host_id | desired_stack_id |

+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+

|          2 | APP_TIMELINE_SERVER    | STARTED       | YARN           | INSERVICE   | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | ATLAS_SERVER           | STARTED       | ATLAS          | INSERVICE   | OFF               |                0 | UNSECURED      |       1 |                1 |

|          2 | DATANODE               | STARTED       | HDFS           | INSERVICE   | OFF               |                0 | UNSECURED      |       3 |                1 |

|          2 | DATANODE               | STARTED       | HDFS           | INSERVICE   | OFF               |                0 | UNSECURED      |       4 |                1 |

|          2 | DATANODE               | STARTED       | HDFS           | INSERVICE   | OFF               |                0 | UNSECURED      |       5 |                1 |

|          2 | DRPC_SERVER            | STARTED       | STORM          | INSERVICE   | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | FALCON_CLIENT          | INSTALLED     | FALCON         | INSERVICE   | OFF               |                0 | UNSECURED      |       1 |                1 |

|          2 | FALCON_CLIENT          | INSTALLED     | FALCON         | INSERVICE   | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | FALCON_SERVER          | STARTED       | FALCON         | INSERVICE   | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | FLUME_HANDLER          | STARTED       | FLUME          | INSERVICE   | OFF               |                0 | UNSECURED      |       3 |                1 |

|          2 | FLUME_HANDLER          | STARTED       | FLUME          | INSERVICE   | OFF               |                0 | UNSECURED      |       4 |                1 |

|          2 | FLUME_HANDLER          | STARTED       | FLUME          | INSERVICE   | OFF               |                0 | UNSECURED      |       5 |                1 |

|          2 | HBASE_CLIENT           | INSTALLED     | HBASE          | INSERVICE   | OFF               |                0 | UNSECURED      |       1 |                1 |

|          2 | HBASE_CLIENT           | INSTALLED     | HBASE          | INSERVICE   | OFF               |                0 | UNSECURED      |       2 |                1 |

|          2 | HBASE_MASTER           | STARTED       | HBASE          | INSERVICE   | OFF               |                0 | UNSECURED      |       1 |                1 |

Step 4: Now restart your ambari and you should be good with your cluster. 

[ambari@server1 ~]$ ambari-server restart

Using python  /usr/bin/python

Restarting ambari-server

Using python  /usr/bin/python

Stopping ambari-server

Ambari Server stopped

Using python  /usr/bin/python

Starting ambari-server

Organizing resource files at /var/lib/ambari-server/resources…

Unable to check firewall status when starting without root privileges.

Please do not forget to disable or adjust firewall if needed

Ambari database consistency check started…

No errors were found.

Ambari database consistency check finished

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Waiting for server start………………..

Ambari Server ‘start’ completed successfully.