Analyze your jobs running on top of Tez

  • 0

Analyze your jobs running on top of Tez

Category : Bigdata

Sometime we have to analyze our jobs to tune our jobs or to prepare some reports. We can use following method to get running time for each and every steps for your job in tez execution engine.

By setting up hive.tez.exec.print.summary=true property you can achieve it.

hive> select count(*) from cars_beeline;

Query ID = s0998dnz_20160711080520_e282c377-5607-4cf4-bcda-bd7010918f9c

Total jobs = 1

Launching Job 1 out of 1

Status: Running (Executing on YARN cluster with App id application_1468229364042_0003)

——————————————————————————–

        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

——————————————————————————–

Map 1 ……….   SUCCEEDED      1          1        0        0       0       0

Reducer 2 ……   SUCCEEDED      1          1        0        0       0       0

——————————————————————————–

VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 6.65 s     

——————————————————————————–

OK

6

Time taken: 11.027 seconds, Fetched: 1 row(s)

hive> set hive.tez.exec.print.summary=true;

hive> select count(*) from cars_beeline;

Query ID = s0998dnz_20160711080557_28453c83-9e17-4874-852d-c5e13dd97f82

Total jobs = 1

Launching Job 1 out of 1

Status: Running (Executing on YARN cluster with App id application_1468229364042_0003)

——————————————————————————–

        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

——————————————————————————–

Map 1 ……….   SUCCEEDED      1          1        0        0       0       0

Reducer 2 ……   SUCCEEDED      1          1        0        0       0       0

——————————————————————————–

VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 15.36 s    

——————————————————————————–

Status: DAG finished successfully in 15.36 seconds

METHOD                         DURATION(ms)

parse                                    2

semanticAnalyze                        130

TezBuildDag                            229

TezSubmitToRunningDag                   13

TotalPrepTime                          979

VERTICES         TOTAL_TASKS  FAILED_ATTEMPTS KILLED_TASKS DURATION_SECONDS    CPU_TIME_MILLIS     GC_TIME_MILLIS  INPUT_RECORDS   OUTPUT_RECORDS

Map 1                      1                0            0            10.64              9,350                299              6                1

Reducer 2                  1                0            0             0.41                760                  0              1                0

OK

6

Time taken: 16.478 seconds, Fetched: 1 row(s)