Analyze your jobs running on top of Tez
Category : Bigdata
Sometime we have to analyze our jobs to tune our jobs or to prepare some reports. We can use following method to get running time for each and every steps for your job in tez execution engine.
By setting up hive.tez.exec.print.summary=true property you can achieve it.
hive> select count(*) from cars_beeline;
Query ID = s0998dnz_20160711080520_e282c377-5607-4cf4-bcda-bd7010918f9c
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1468229364042_0003)
——————————————————————————–
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
——————————————————————————–
Map 1 ………. SUCCEEDED 1 1 0 0 0 0
Reducer 2 …… SUCCEEDED 1 1 0 0 0 0
——————————————————————————–
VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 6.65 s
——————————————————————————–
OK
6
Time taken: 11.027 seconds, Fetched: 1 row(s)
hive> set hive.tez.exec.print.summary=true;
hive> select count(*) from cars_beeline;
Query ID = s0998dnz_20160711080557_28453c83-9e17-4874-852d-c5e13dd97f82
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1468229364042_0003)
——————————————————————————–
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
——————————————————————————–
Map 1 ………. SUCCEEDED 1 1 0 0 0 0
Reducer 2 …… SUCCEEDED 1 1 0 0 0 0
——————————————————————————–
VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 15.36 s
——————————————————————————–
Status: DAG finished successfully in 15.36 seconds
METHOD DURATION(ms)
parse 2
semanticAnalyze 130
TezBuildDag 229
TezSubmitToRunningDag 13
TotalPrepTime 979
VERTICES TOTAL_TASKS FAILED_ATTEMPTS KILLED_TASKS DURATION_SECONDS CPU_TIME_MILLIS GC_TIME_MILLIS INPUT_RECORDS OUTPUT_RECORDS
Map 1 1 0 0 10.64 9,350 299 6 1
Reducer 2 1 0 0 0.41 760 0 1 0
OK
6
Time taken: 16.478 seconds, Fetched: 1 row(s)