Ambari – BigData

July 4, 2017
2

Unable to view OS Host information in the Ambari Dashboard(No data Available)

Tags : ambari ambari-metrics-collector No data Available

Category : Ambari

On the Ambari dashboard, the memory usage, Network Usage, CPU usage and Cluster Load information are missing.The dashboard displays the following error:

No data Available

Root Cause :
This issue occurs when there are some temporary files present in the AMS collector folder.

Solution:

You need to stop ams service vi ambari and then remove all temp files.

mv /var/lib/ambari-metrics-collector /tmp/ambari-metrics-collector_OLD

Now you can restart ams service again and now you should be good with Ambari dashboard, the memory usage, Network Usage, CPU usage and Cluster Load information.

May 31, 2017
0

Run all service checks in bulk

Tags : _SERVICE_CHECK 8080/api/v1/clusters AMBARI_HOST bulk service checks Run all service checks in bulk run_all_service_checks running_services X-Requested-By:X-Requested-By" -X POST

Category : Ambari

In this blogs I tried to explain that how you can use ambari API to trigger all Service Checks with a single command.

In order to check the status and stability of any service in your cluster you need to run the service checks that are included in Ambari. Usually each Service provides its own service check in ambari and to run a service check you have to select the service (e.g. HDFS) in Ambari and click “Run Service Check” in the “Actions” dropdown menu.

But its a tedious job to run every service check one by one in case if we have many services. So I created this script by using Ambari API to start all available service checks via single script. Only thing you need to pass required parameters according to your env.

Example:

[s0998dnz@m1.hdp22 ~]$ ./run_all_service_checks.sh
Enter Ambari server name : m1.hdp22
Enter Ambari admin's User Name: saurkuma
Enter Password for saurkuma : 
Your cluster name is: HDPTST
There are following running services :
FALCON
FLUME
HBASE
HDFS
HIVE
KAFKA
KNOX
MAHOUT
MAPREDUCE2
OOZIE
PIG
RANGER
RANGER_KMS
SLIDER
SPARK
SQOOP
STORM
TEZ
YARN
ZOOKEEPER
{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1315",
  "Requests" : {
    "id" : 1315,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1316",
  "Requests" : {
    "id" : 1316,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1317",
  "Requests" : {
    "id" : 1317,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1318",
  "Requests" : {
    "id" : 1318,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1319",
  "Requests" : {
    "id" : 1319,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1320",
  "Requests" : {
    "id" : 1320,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1321",
  "Requests" : {
    "id" : 1321,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1322",
  "Requests" : {
    "id" : 1322,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1323",
  "Requests" : {
    "id" : 1323,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1324",
  "Requests" : {
    "id" : 1324,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1325",
  "Requests" : {
    "id" : 1325,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1326",
  "Requests" : {
    "id" : 1326,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1327",
  "Requests" : {
    "id" : 1327,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1328",
  "Requests" : {
    "id" : 1328,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1329",
  "Requests" : {
    "id" : 1329,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1330",
  "Requests" : {
    "id" : 1330,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1331",
  "Requests" : {
    "id" : 1331,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1332",
  "Requests" : {
    "id" : 1332,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1333",
  "Requests" : {
    "id" : 1333,
    "status" : "Accepted"
  }
}{
  "href" : "http://m1.hdp22:8080/api/v1/clusters/HDPTST/requests/1334",
  "Requests" : {
    "id" : 1334,
    "status" : "Accepted"
  }
}

Now if you will login to your ambari server you will see all service checks are running. So now you will be thinking which script is doing this magic,so don’t worry here is script. You use it and enjoy.

[s0998dnz@m1.hdp22 ~]$ cat run_all_service_checks.sh 
#!/usr/bin/env bash
###########################
## Saurabh Singh ###
### Version 1.0 ####
###########################
echo -n "Enter Ambari server name : "
read "server"
AMBARI_HOST=$server
echo -n "Enter Ambari admin's User Name: "
read "user"
echo -n "Enter Password for $user : "
read -s "pwd"
LOGIN=$user
PASSWORD=$pwd
if [ -e "~/.ambari_login" ]; then
    . ~/.ambari_login
fi

cluster_name=$(curl -s -u $LOGIN:$PASSWORD --insecure "http://$AMBARI_HOST:8080/api/v1/clusters"  | python -mjson.tool | perl -ne '/"cluster_name":.*?"(.*?)"/ && print "$1\n"')
if [ -z "$cluster_name" ]; then
    exit
fi
echo -e "\nYour cluster name is: $cluster_name"

running_services=$(curl -s -u $LOGIN:$PASSWORD --insecure "http://$AMBARI_HOST:8080/api/v1/clusters/$cluster_name/services?fields=ServiceInfo/service_name&ServiceInfo/maintenance_state=OFF" | python -mjson.tool | perl -ne '/"service_name":.*?"(.*?)"/ && print "$1\n"')
if [ -z "$running_services" ]; then
    exit
fi
echo "There are following running services :
$running_services"

post_body=
for s in $running_services; do
    if [ "$s" == "ZOOKEEPER" ]; then
        post_body="{\"RequestInfo\":{\"context\":\"$s Service Check\",\"command\":\"${s}_QUORUM_SERVICE_CHECK\"},\"Requests/resource_filters\":[{\"service_name\":\"$s\"}]}"

    else
        post_body="{\"RequestInfo\":{\"context\":\"$s Service Check\",\"command\":\"${s}_SERVICE_CHECK\"},\"Requests/resource_filters\":[{\"service_name\":\"$s\"}]}"
    fi
    curl -s -u $LOGIN:$PASSWORD --insecure -H "X-Requested-By:X-Requested-By" -X POST --data "$post_body"  "http://$AMBARI_HOST:8080/api/v1/clusters/$cluster_name/requests"
done

As always I welcome your valuable feedback or any suggestion.

April 10, 2017
0

Ambari is showing “Add Service Wizard in Progress” or “Move Master Wizard In Progress”

Tags : Add service wizard Add Service Wizard in Progress addServiceController Move Master Wizard In Progress moveMasterController

Category : Ambari

If you are using ambari 2.4.1 or 2.4.2 then you may see following message in your ambari page and you will not get any option to “Service Action” to restart or doing anything to any services.

Root Cause : If there are more than one Ambari Admin users present. Then if one of the admin user say “admin1” click on the “Actions” => “Add Service” in the ambari UI and does nothing. Then other logged in admin users will keep seeing the “Add Service Wizard in Progress” blinking bar in the ambari UI and the “service action” dropdown button will not be visible to them.It is captured by following jira also.

https://issues.apache.org/jira/browse/AMBARI-18932

Solution: Take ambari server backup and run below command from Ambari node.

curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d 
'{"wizard-data":"{\"userName\":\"admin\",\"controllerName\":\"addServiceController\"}"}' 
http://<ambari_server_name>.lowes.com:8080/api/v1/persist

or for “Move Master Wizard In Progress” message do the following :

curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d 
'{"wizard-data":"{\"userName\":\"admin\",\"controllerName\":\"moveMasterController\"}"}' 
http://&lt;ambari_server_name&gt;.lowes.com:8080/api/v1/persist

Note : Replace the following value as per your cluster config.
username = the user for which you are facing issue
Ambari_host = hostname of ambari node
controller_name = name of the controller for which you are making request.

There are few things to be aware of

1. This will resolve issue for the current logged in user but other users will have same issue of seeing “Add service wizard” in progress label unless the current logged in user logs off and logs in back at which user will be directed to add service wizard and then cancels add service wizard. After that this issue will go away for all other ambari users.

2. As said above after the execution of the API if current user signs out and signs back in then user will be directly redirected to add service wizard.

Please feel free to give your valuable feedback or suggestion.

February 9, 2017
0

Encrypt Database and LDAP Passwords for Ambari-Server

Tags : ambari ldap AMBARI_SECURITY_MASTER_KEY

Category : Ambari

By default the passwords to access the Ambari database and the LDAP server are stored in a plain text configuration file. To have those passwords encrypted, you need to run a special setup command.

[root@m1 ~]# cd /etc/ambari-server/conf/

[root@m1 conf]# ls -ltrh

total 52K

-rw-r–r– 1 root root 2.8K Mar 31 2015 ambari.properties.rpmsave.20161004015858

-rwxrwxrwx 1 root root 286 Sep 15 19:53 krb5JAASLogin.conf

-rw-r–r– 1 root root 3.9K Oct 4 01:58 ambari.properties.rpmsave.20161005033229

-rw-r–r– 1 root root 4.7K Oct 5 03:32 ambari.properties.rpmsave.20161005065356

-rw-r–r– 1 root root 286 Oct 5 06:45 krb5JAASLogin.conf.rpmsave

-rwxrwxrwx 1 root root 4.9K Oct 5 06:45 log4j.properties

-rw-r–r– 1 root root 9 Jan 31 05:35 users.txt

-rw-r—– 1 root root 7 Feb 8 11:51 password.dat

-rw-rw—- 1 root root 15 Feb 8 11:53 ldap-password.dat

-rwxrwxrwx 1 root root 7.8K Feb 8 11:53 ambari.properties

So to protect these you need to run ambari security setup. Ambari Server should not be running when you do this: either make the edits before you start Ambari Server the first time or bring the server down to make the edits.

Step 1: On the Ambari Server, run the special setup command and answer the prompts:

[root@m1 ~]# ambari-server setup-security

Using python /usr/bin/python

Security setup options…

===========================================================================

Choose one of the following options:

[1] Enable HTTPS for Ambari server.

[2] Encrypt passwords stored in ambari.properties file.

[3] Setup Ambari kerberos JAAS configuration.

[4] Setup truststore.

[5] Import certificate to truststore.

===========================================================================

Enter choice, (1-5): 2 (Select Option 2: Choose one of the following options:)

Please provide master key for locking the credential store: ******** (Provide a master key for encrypting the passwords. You are prompted to enter the key twice for accuracy.)

Re-enter master key:********

Do you want to persist master key. If you choose not to persist, you need to provide the Master Key while starting the ambari server as an env variable named AMBARI_SECURITY_MASTER_KEY or the start will prompt for the master key. Persist [y/n] (y)? n

Adjusting ambari-server permissions and ownership…

Ambari Server ‘setup-security’ completed successfully.

Note : Now you need to restart ambari server and provide the same master key when it prompt during restart. You can avoid it by setting up environment variable in ambari-env.sh file.

Option 1: Provide key during restart.

root@m1 ~]# ambari-server restart

Using python /usr/bin/python

Restarting ambari-server

Using python /usr/bin/python

Stopping ambari-server

Ambari Server stopped

Using python /usr/bin/python

Starting ambari-server

Ambari Server running with administrator privileges.

Organizing resource files at /var/lib/ambari-server/resources…

Enter current Master Key: ********

Ambari database consistency check started…

No errors were found.

Ambari database consistency check finished

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Waiting for server start………………..

Ambari Server ‘start’ completed successfully.

Option 2: set AMBARI_SECURITY_MASTER_KEY in ambari-env.sh

[root@m1 ~]# vi /var/lib/ambari-server/ambari-env.sh

[root@m1 ~]# grep -C4 AMBARI_SECURITY_MASTER_KEY /var/lib/ambari-server/ambari-env.sh

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

export AMBARI_SECURITY_MASTER_KEY=hadoop

AMBARI_PASSHPHRASE=”DEV”

export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS’ -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false’

export PATH=$PATH:/var/lib/ambari-server

export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.6/site-packages

Now restart Ambari-Server

[root@m1 ~]# ambari-server restart

Using python /usr/bin/python

Restarting ambari-server

Using python /usr/bin/python

Stopping ambari-server

Ambari Server stopped

Using python /usr/bin/python

Starting ambari-server

Ambari Server running with administrator privileges.

Organizing resource files at /var/lib/ambari-server/resources…

Ambari database consistency check started…

No errors were found.

Ambari database consistency check finished

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Waiting for server start………………..

Ambari Server ‘start’ completed successfully.

[root@m1 conf]# ls -ltrh

total 52K

-rw-r–r– 1 root root 2.8K Mar 31 2015 ambari.properties.rpmsave.20161004015858

-rwxrwxrwx 1 root root 286 Sep 15 19:53 krb5JAASLogin.conf

-rw-r–r– 1 root root 3.9K Oct 4 01:58 ambari.properties.rpmsave.20161005033229

-rw-r–r– 1 root root 4.7K Oct 5 03:32 ambari.properties.rpmsave.20161005065356

-rw-r–r– 1 root root 286 Oct 5 06:45 krb5JAASLogin.conf.rpmsave

-rwxrwxrwx 1 root root 4.9K Oct 5 06:45 log4j.properties

-rw-r–r– 1 root root 9 Jan 31 05:35 users.txt

-rwxrwxrwx 1 root root 7.8K Feb 8 11:53 ambari.properties

Remove Encryption Entirely

To reset Ambari database and LDAP passwords to a completely unencrypted state:

On the Ambari host, open /etc/ambari-server/conf/ambari.properties with a text editor and set this propertysecurity.passwords.encryption.enabled=false
Delete /var/lib/ambari-server/keys/credentials.jceks
Delete /var/lib/ambari-server/keys/master
You must now reset the database password and, if necessary, the LDAP password. Run ambari-server setup and ambari-server setup-ldap again.

[root@m1 ~]# vi /etc/ambari-server/conf/ambari.properties

[root@m1 ~]# ls -ltrh /var/lib/ambari-server/keys/credentials.jceks

-rw-r—– 1 root root 992 Feb 8 11:35 /var/lib/ambari-server/keys/credentials.jceks

[root@m1 ~]# rm /var/lib/ambari-server/keys/credentials.jceks

rm: remove regular file `/var/lib/ambari-server/keys/credentials.jceks’? y

[root@m1 ~]# ambari-server setup

Using python /usr/bin/python

Setup ambari-server

Checking SELinux…

SELinux status is ‘disabled’

Customize user account for ambari-server daemon [y/n] (n)? n

Adjusting ambari-server permissions and ownership…

Checking firewall status…

Checking JDK…

Do you want to change Oracle JDK [y/n] (n)? n

Completing setup…

Configuring database…

Enter advanced database configuration [y/n] (n)? y

Configuring database…

==============================================================================

Choose one of the following options:

[1] – PostgreSQL (Embedded)

[2] – Oracle

[3] – MySQL / MariaDB

[4] – PostgreSQL

[5] – Microsoft SQL Server (Tech Preview)

[6] – SQL Anywhere

[7] – BDB

==============================================================================

Enter choice (1): 1

Database name (ambari):

Postgres schema (ambari):

Username (ambari):

Enter Database Password (ambari.db.password):

Re-enter password:

Default properties detected. Using built-in database.

Configuring ambari database…

Checking PostgreSQL…

Configuring local database…

Connecting to local database…done.

Configuring PostgreSQL…

Backup for pg_hba found, reconfiguration not required

Extracting system views…

…………

Adjusting ambari-server permissions and ownership…

Ambari Server ‘setup’ completed successfully.

[root@m1 ~]# ambari-server setup-ldap

Using python /usr/bin/python

Setting up LDAP properties…

Primary URL* {host:port} (ad.lowes.com:389):

Secondary URL {host:port} :

Use SSL* [true/false] (false):

User object class* (user):

User name attribute* (sAMAccountName):

Group object class* (group):

Group name attribute* (cn):

Group member attribute* (memberOf):

Distinguished name attribute* (dn):

Base DN* (dc=lowes,dc=com):

Referral method [follow/ignore] (ignore):

Bind anonymously* [true/false] (false):

Manager DN* (cn=ambariaddev,cn=users,dc=lowes,dc=com):

Enter Manager Password* :

Re-enter password:

====================

Review Settings

====================

authentication.ldap.managerDn: cn=ambariaddev,cn=users,dc=lowes,dc=com

authentication.ldap.managerPassword: *****

Save settings [y/n] (y)? y

Saving…done

Ambari Server ‘setup-ldap’ completed successfully.

[root@m1 ~]# ambari-server restart

Using python /usr/bin/python

Restarting ambari-server

Using python /usr/bin/python

Stopping ambari-server

Ambari Server stopped

Using python /usr/bin/python

Starting ambari-server

Ambari Server running with administrator privileges.

Organizing resource files at /var/lib/ambari-server/resources…

Ambari database consistency check started…

No errors were found.

Ambari database consistency check finished

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Waiting for server start………………..

Ambari Server ‘start’ completed successfully.

Change the Current Master Key

To change the master key:

If you know the current master key or if the current master key has been persisted:
1. Re-run the encryption setup command and follow the prompts.ambari-server setup-security
  1. Select Option 2: Choose one of the following options:
    
    [1] Enable HTTPS for Ambari server.
    
    [2] Encrypt passwords stored in ambari.properties file.
    
    [3] Setup Ambari kerberos JAAS configuration.
  2. Enter the current master key when prompted if necessary (if it is not persisted or set as an environment variable).
  3. At the Do you want to reset Master Key prompt, enter yes.
  4. At the prompt, enter the new master key and confirm.
If you do not know the current master key:
- Remove encryption entirely, as described here.
- Re-run ambari-server setup-security as described here.
- Start or restart the Ambari Server.ambari-server restart

Please feel free to give your suggestion or feedback.

December 8, 2016
1

Datanode doesn’t start with error “java.net.BindException: Address already in use”

Tags : 50010 is already in use datanode java.net.BindException: Address already in use port is in use

Category : Ambari , HDFS

In many real time scenario we have seen a error “java.net.BindException: Address already in use” with datanode when we start datanode.

You can observe following things during that issue.

1. Datanode doesn’t start with error saying “address already in use”.
2. “netstat -anp | grep 50010” shows no result.

ROOT CAUSE:
There are 3 ports needed when datanode starts and each has a different error message when address already in use.

1. Port 50010 is already in use
2016-12-02 00:01:14,056 ERROR datanode.DataNode (DataNode.java:secureMain(2630)) – Exception in secureMain
java.net.BindException: Problem binding to [0.0.0.0:50010] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException

2. Port 50075 is already in use
2016-12-01 23:57:57,298 ERROR datanode.DataNode (DataNode.java:secureMain(2630)) – Exception in secureMain
java.net.BindException: Address already in use

3. Port 8010 is already in use
2016-12-02 00:09:40,422 ERROR datanode.DataNode (DataNode.java:secureMain(2630)) – Exception in secureMain
java.net.BindException: Problem binding to [0.0.0.0:8010] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException

Note that there is no port information within datanode.log when Port 50075 is already in use.If it’s required to set a different port for datanode service, review the following properties:

dfs.datanode.address : default 50010
dfs.datanode.http.address : default 50075
dfs.datanode.ipc.address : default 8010

RESOLUTION :
Stop/kill the process which uses port 50010/50075/8010.

Please feel free to give your valuable feedback.

November 24, 2016
0

Installing grafana and it is failing with resource_management.core.exceptions.Fail: Ambari Metrics Grafana data source creation failed. POST request status: 401 Unauthorized

Tags : ambari-metrics ams grafana

Category : Ambari

When we do fresh install for grafana in ambari 2.4 and when you start it then it may be fail with following error.

stderr: /var/lib/ambari-agent/data/errors-14517.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/
metrics_grafana.py", line 67, in <module>
    AmsGrafana().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", 
line 280, in execute method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 725, in restart self.start(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/
metrics_grafana.py", line 49, in start   create_ams_datasource()
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts
/metrics_grafana_util.py", line 261, in create_ams_datasource
    (response.status, response.reason, data))
resource_management.core.exceptions.Fail: Ambari Metrics Grafana data source creation failed. 
POST request status: 401 Unauthorized {"message":"Invalid username or password"}

Root Cause: It could be because first time it use default username and password (i.e admin, admin) 
and you may be used wrong username & password or something went wrong during installation. 

Resolutions: Grafana username and password is stored in a sqlite3 database. One of the way is to
reset the password back to admin first and then can be changed in Grafana Dashboard. To do the same, 
following steps could be used:

1. Logon to the node where Grafana is installed and invoke Grafana sqlite3 database as follows and 
sudo to ams user.

[s0998dnz@server1 ~]$ sudo su – ams

Last login: Thu Nov 24 04:11:25 EST 2016

[ams@server1 ~]$ sqlite3 /var/lib/ambari-metrics-grafana/grafana.db

SQLite version 3.6.20

Enter “.help” for instructions

Enter SQL statements terminated with a “;”

sqlite> select salt, password from user;

GZtvpYh3e0|56053ec1580b26a14b339f3c95c1e51117f7ce730c5400955c0288c650deba14a3dbeb70ba4a65464d822a9fa47fc7f7c6ba

sqlite> update user set password = ’59acf18b94d7eb0694c61e60ce44c110c7a683ac6a8f09580d626f90f4a242000746579358d77dd9e570e83fa24faa88a8a6′, salt = ‘F3FAxVm33R’ where login = ‘admin’;

sqlite> .exit

2. Once done, edit Ambari Metrics Server-Configs and update Grafana Password to admin

3. Restart the Ambari Metrics Server

4. Access Grafana page using the Quick Links under Ambari Metric Server Dashboard

5. Click on the Grafana Symbol in the top left corner of the screen and Sign-in as admin user

September 23, 2016
0

After upgrading ambari it is not coming up (hostcomponentdesiredstate.admin_state)

Tags : ambari hostcomponentdesiredstate hostcomponentdesiredstate.admin_state INSERVICE

Category : Ambari

If you upgrade ambari and in case if you see following error then you should not worry, following steps will help you to bring your cluster into running state.

Issue: Once you upgrade your cluster and after restarting you don’t see any service or their metrics on ambari then you need following given steps.

You also can see following error in your ambari-server logs.

23 Sep 2016 05:08:13,966 ERROR [main] ViewRegistry:1695 – Caught exception loading view TEZ{0.7.0.2.5.0.0-1}
Local Exception Stack:
Exception [EclipseLink-116] (Eclipse Persistence Services – 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DescriptorException
Exception Description: No conversion value provided for the value [NULL] in field [hostcomponentdesiredstate.admin_state].
Mapping: org.eclipse.persistence.mappings.DirectToFieldMapping[adminState–>hostcomponentdesiredstate.admin_state]
Descriptor: RelationalDescriptor(org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity –> [DatabaseTable(hostcomponentdesiredstate)])
at org.eclipse.persistence.exceptions.DescriptorException.noFieldValueConversionToAttributeValueProvided(DescriptorException.java:1066)
at org.eclipse.persistence.mappings.converters.ObjectTypeConverter.convertDataValueToObjectValue(ObjectTypeConverter.java:226)
at org.eclipse.persistence.mappings.converters.EnumTypeConverter.convertDataValueToObjectValue(EnumTypeConverter.java:141)
at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.getObjectValue(AbstractDirectMapping.java:616)
at org.eclipse.persistence.mappings.foundation.AbstractDirectMapping.valueFromRow(AbstractDirectMapping.java:1220)
at org.eclipse.persistence.mappings.DatabaseMapping.readFromRowIntoObject(DatabaseMapping.java:1539)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:462)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:1005)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:899)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:852)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:735)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:689)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:805)
at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:895)
at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:562)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1175)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134)
at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:441)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1222)

Resolutions :

Step 1: Check all installed services admin_status in hostcomponentdesiredstate table. You should not have any NULL value in admin_state for any service.

mysql> select * from hostcomponentdesiredstate;

+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+

+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+

| 2 | APP_TIMELINE_SERVER | STARTED | YARN | NULL | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | ATLAS_SERVER | STARTED | ATLAS | NULL | OFF | 0 | UNSECURED | 1 | 1 |

| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |

| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |

| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |

| 2 | DRPC_SERVER | STARTED | STORM | NULL | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | FALCON_CLIENT | INSTALLED | FALCON | NULL | OFF | 0 | UNSECURED | 1 | 1 |

| 2 | FALCON_CLIENT | INSTALLED | FALCON | NULL | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | FALCON_SERVER | STARTED | FALCON | NULL | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |

| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |

| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |

| 2 | HBASE_CLIENT | INSTALLED | HBASE | NULL | OFF | 0 | UNSECURED | 1 | 1 |

| 2 | HBASE_CLIENT | INSTALLED | HBASE | NULL | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | HBASE_MASTER | STARTED | HBASE | NULL | OFF | 0 | UNSECURED | 1 | 1 |

| 2 | HBASE_REGIONSERVER | STARTED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |

| 2 | HBASE_REGIONSERVER | STARTED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |

| 2 | HBASE_REGIONSERVER | STARTED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |

| 2 | HCAT | INSTALLED | HIVE | NULL | OFF | 0 | UNSECURED | 1 | 1 |

| 2 | HCAT | INSTALLED | HIVE | NULL | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | HDFS_CLIENT | INSTALLED | HDFS | NULL | OFF | 0 | UNSECURED | 1 | 1 |

| 2 | HDFS_CLIENT | INSTALLED | HDFS | NULL | OFF | 0 | UNSECURED | 2 | 1 |

Step2: In that case you have to update this table manually :

mysql> update hostcomponentdesiredstate set admin_state=’INSERVICE’;

Query OK, 58 rows affected (0.01 sec)

Rows matched: 92 Changed: 58 Warnings: 0

Step 3: Now you can check once again the admin_state to have all inservice:

mysql> select * from hostcomponentdesiredstate;

+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+

+————+————————+—————+—————-+————-+——————-+——————+—————-+———+——————+

| 2 | APP_TIMELINE_SERVER | STARTED | YARN | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | ATLAS_SERVER | STARTED | ATLAS | INSERVICE | OFF | 0 | UNSECURED | 1 | 1 |

| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |

| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |

| 2 | DATANODE | STARTED | HDFS | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |

| 2 | DRPC_SERVER | STARTED | STORM | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | FALCON_CLIENT | INSTALLED | FALCON | INSERVICE | OFF | 0 | UNSECURED | 1 | 1 |

| 2 | FALCON_CLIENT | INSTALLED | FALCON | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | FALCON_SERVER | STARTED | FALCON | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 3 | 1 |

| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 4 | 1 |

| 2 | FLUME_HANDLER | STARTED | FLUME | INSERVICE | OFF | 0 | UNSECURED | 5 | 1 |

| 2 | HBASE_CLIENT | INSTALLED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 1 | 1 |

| 2 | HBASE_CLIENT | INSTALLED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 2 | 1 |

| 2 | HBASE_MASTER | STARTED | HBASE | INSERVICE | OFF | 0 | UNSECURED | 1 | 1 |

Step 4: Now restart your ambari and you should be good with your cluster.

[ambari@server1 ~]$ ambari-server restart

Using python /usr/bin/python

Restarting ambari-server

Using python /usr/bin/python

Stopping ambari-server

Ambari Server stopped

Using python /usr/bin/python

Starting ambari-server

Organizing resource files at /var/lib/ambari-server/resources…

Unable to check firewall status when starting without root privileges.

Please do not forget to disable or adjust firewall if needed

Ambari database consistency check started…

No errors were found.

Ambari database consistency check finished

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Waiting for server start………………..

Ambari Server ‘start’ completed successfully.

August 31, 2016
3

How to integrate Ambari with ldap

Tags : ambari ldap

Category : Ambari

By default, Ambari uses an internal database as the user store for authentication and authorization. If you wish to add LDAP external authentication in addition for Ambari Web, you need to make some edits to the Ambari properties file.

Collect following information :

ldap.primaryUrl=<ldap_server_name>:389
ldap.useSSL=false
ldap.usernameAttribute=sAMAccountName
ldap.baseDn=cn=Users,dc=<sreach_dir>,dc=com
ldap.bindAnonymously=false
ldap.managerDn=cn=ambari,cn=users,dc=<sreach_dir>,dc=com
ldap.managerPassword=/etc/ambari-server/conf/ldap-password.dat
ldap.userObjectClass=user
ldap.groupObjectClass=group
ldap.groupMembershipAttr=memberOf
ldap.groupNamingAttr=cn
ldap.referral=ignore
ldap.isConfigured=true
security=ldap
ldap.dnAttribute=dn

Create ldap-password.dat file with ldap password under =/etc/ambari-server/conf :

[root@m1 ~]# vi /etc/ambari-server/conf/ldap-password.dat

Run Ambari ldap integration command on Ambari server and provide all required information, which you have collected above:

[root@m1 ~]# ambari-server setup-ldap

Once it is done then restart your Ambari server:

[root@m1 ~]# ambari-server restart

There is a problem in Ambari which does not sync users automatically so for that we have create a file users.txt(put user’s windows id in this file) under home dir and then usersync command manually:

[root@m1 ~]# vi users.txt

[root@m1 ~]# sudo /usr/sbin/ambari-server sync-ldap –users /root/users.txt

Using python /usr/bin/python2

Syncing with LDAP…

Enter Ambari Admin login: admin

Enter Ambari Admin password:

Syncing specified users and groups….

Completed LDAP Sync.

Summary:

memberships:

removed = 0

created = 0

users:

updated = 0

removed = 0

created = 1

groups:

updated = 0

removed = 0

created = 0

Ambari Server ‘sync-ldap’ completed successfully.

When you have made the necessary edits to the properties file. Initially the users you have enabled will all have User privileges. Users can read metrics, view service status and configuration, and browse job information.

June 18, 2016
7

How do I change an existing Ambari DB Postgres to MySQL?

Tags : ambari db migrate mysql

Category : Ambari , Bigdata

By default when you configure your ambari server then it runs on postgres database. And if after sometime we need to change it to our comfortable and your org lovable db(like mysql) then you need to use following steps.

Step 1: Please stop your ambari server and then take back of postgres ambari db(the default password is ‘bigdata’):

$ ambari-server stop

$ pg_dump -U ambari ambari > /temp/ambari.sql

Step 2: Now you need to setup mysql on any of the node with the help of following command :

$ yum install mysql-connector-java

Step 2: Now confirm that .jar is in the Java share directory and Make sure the .jar file has the appropriate permissions – 644

$ ls /usr/share/java/mysql-connector-java.jar

Step 3: Create a user for Ambari and grant it permissions.

For example, using the MySQL database admin utility:

# mysql -u root -p

CREATE USER ‘ambari’@’%’ IDENTIFIED BY ‘bigdata’;

GRANT ALL PRIVILEGES ON *.* TO ‘ambari’@’%’;

CREATE USER ‘ambari’@’localhost’ IDENTIFIED BY ‘bigdata’;

GRANT ALL PRIVILEGES ON *.* TO ‘ambari’@’localhost’;

CREATE USER ‘ambari’@’hdpm1.com>’ IDENTIFIED BY ‘bigdata’;

GRANT ALL PRIVILEGES ON *.* TO ‘ambari’@’hdpm1.com’;

FLUSH PRIVILEGES;

Step 4: Now you need load/restore the Ambari Server database schema.

mysql -u ambari -p

CREATE DATABASE ambaridb;

USE ambaridb;

SOURCE temp/ambari.sql; (the backup from postgres);

Step 5: Now finally update the ambari-server configuration to reference the MySQL instance:

On the ambari-server node you need to run ambari setup:

$ ambari-server setup

Enter advanced database configuration [y/n] (n)?

choose “y”, and follow the steps for setting it up as MySQL (option 3) using the guide mentioned above. Once that is done, don’t change any other settings after the db change.

Once setup is complete for the MySQL instance then you can start ambari:

$ ambari-server start

So now you have successfully migrated ambari postgres db to mysql.

January 9, 2016
3

How to start learning hadoop

Tags : hadoop learn hadoop learningg

Category : Ambari

The easiest way to get started with Hadoop is Sandbox with VM Player or Virtual Box. It is a personal, portable Hadoop environment that comes with a dozen interactive Hadoop tutorials. Sandbox includes many of the most exciting developments from the latest CDH/HDP distribution, packaged up in a virtual environment. You can start working on hadoop environment within 10 minutes.

Hadoop Sandbox provides:

A virtual machine with Hadoop preconfigured.
A set of hands-on tutorials to get you started with Hadoop.
An environment to help you explore related projects in the Hadoop ecosystem like Apache Pig, Apache Hive, Apache HCatalog and Apache HBase.

Lets start downloading and installing hadoop into our windows machine in 10 mins

System requirements to run VMplayer:

RAM: at least 8 GB (For a 2-Node Virtual Cluster). Processor: i3 or above. at least 20 GB free disk space.

Step 1. Download and install VMPlayer from the following websites

http://www.vmware.com/in/products/player

http://filehippo.com/download_vmware_player/

Or download and install Virtual Box from the below link.

www.virtualbox.org/wiki/Downloads

http://filehippo.com/download_virtualbox/

Step 2. Now download the Sandbox from Hortonworks website or Cloudera. I will explain here Hortonwork sandbox process:

http://hortonworks.com/products/hortonworks-sandbox/

The Sandbox download is available for both VirtualBox and VMware Fusion/Player environments. Just follow the instruction to import the Sandbox into your environment.

1. Open the Oracle VM VirtualBox Manager
You can do so by double clicking the icon:

2. Open the Preferences dialog window.
Select File‐>Preferences… within the Oracle VM VirtualBox Manager

3.Uncheck Auto‐Capture Keyboard within the Preferences dialog window.
Select the Input icon button from the left hand pane of the window first
to get to the following window.

Click the OK button once done. This will close the Preferences window.

4. Open the Import Appliance window.

Select File‐>Import Appliance… within the Oracle VM VirtualBox Manager

A separate dialog window is put in front of the VM VirtualBox Manager
window:

5. Click on the folder icon that will open a file dialog window. Select the virtual
appliance file that you downloaded as a prerequisite. After selecting the file click
the Open button.

NOTE: The name of the file you have downloaded depends on the version of the
Hortonworks Sandbox you have chosen to download. The above pictures are referencing
Sandbox HDP version 2.2

Application settings are now displayed.

On Windows after you select the virtual appliance file, you are brought back to this
window.

After clicking on Next, the Appliance Settings are displayed.

6. Modify Appliance Settings as needed.
Within the Appliance Settings section you may wish to allocate more RAM to the
virtual appliance.  Setting 8GB of RAM to the Hortonworks Sandbox virtual appliance
will improve the performance.  Make sure you have enough physical RAM on the
host machine to make this change. To make the change, click on the specific value to
modify and make your edits.  Once finished configuring, click Import.

Progress of the Import

7. Once the import finishes, you are brought to the main Oracle VM VirtualBox
Manager screen. From the left hand pane, select the appliance you just imported
and click the green Start arrow.

A console window opens and displaying the boot up information.

Once the virtual machine fully boots up, the console displays the login instructions.

8. Use one of the supported browsers mentioned in the prerequisites section of this
document within your host machine. Enter the URL displayed in the console. By
default it should be http://127.0.0.1:8888.

Category Archives: Ambari

2

Unable to view OS Host information in the Ambari Dashboard(No data Available)

0

Run all service checks in bulk

0

Ambari is showing “Add Service Wizard in Progress” or “Move Master Wizard In Progress”

0

Encrypt Database and LDAP Passwords for Ambari-Server

Remove Encryption Entirely

Change the Current Master Key

1

Datanode doesn’t start with error “java.net.BindException: Address already in use”

0

Installing grafana and it is failing with resource_management.core.exceptions.Fail: Ambari Metrics Grafana data source creation failed. POST request status: 401 Unauthorized

0

After upgrading ambari it is not coming up (hostcomponentdesiredstate.admin_state)

3

How to integrate Ambari with ldap

7

How do I change an existing Ambari DB Postgres to MySQL?

3

How to start learning hadoop

Recent Posts

Recent Comments

Archives