Error: java.io.IOException: java.lang.RuntimeException: serious problem (state=,code=0)

  • 1

Error: java.io.IOException: java.lang.RuntimeException: serious problem (state=,code=0)

Category : Bigdata

If you run your hive query on ORC tables in hdp 2.3.4 then you may encounter this issue and it is because ORC split generation running on a global threadpool and doAs not being propagated to that threadpool. Threads in the threadpool are created on demand at execute time and thus execute as random users that were active at that time.

It is known issue and fixed by hitting: https://issues.apache.org/jira/browse/HIVE-13120 jira.

Intermittently ODBC users get error that another user doesn’t have permissions on the table. It seems hiveserver2 is checking on wrong user. For example, say you run a job as user ‘user1’, then the error message you will get is something like:

WARN [HiveServer2-Handler-Pool: Thread-587]: thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(681)) – Error fetching results:
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: serious problem
Caused by: java.io.IOException: java.lang.RuntimeException: serious problem

Caused by: java.lang.RuntimeException: serious problem
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1059)

Caused by: java.util.concurrent.ExecutionException: org.apache.hadoop.security.AccessControlException: Permission denied: user=haha, access=READ_EXECUTE, inode=”/apps/hive/warehouse/xixitb”:xixi:hdfs:drwx——
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=haha, access=READ_EXECUTE, inode=”/apps/hive/warehouse/xixitb”:xixi:hdfs:drwx——
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219)
Note that user ‘haha’ is not querying on ‘xixitb’ at all.

Resolution:
For this issue we need to set following property at run time as a workaround which is to turn off local fetch task for hiveserver2.

set hive.fetch.task.conversion=none

0: jdbc:hive2://localhost:8443/default> set hive.fetch.task.conversion=none;

No rows affected (0.033 seconds)

0: jdbc:hive2://localhost:8443/default> select * from database1.table1 where lct_nbr=2413 and ivo_nbr in (17469,18630);

INFO  : Tez session hasn’t been created yet. Opening session

INFO  : Dag name: select * from ldatabase1.table1…(17469,18630)(Stage-1)

INFO  :

INFO  : Status: Running (Executing on YARN cluster with App id application_1462173172032_65644)

INFO  : Map 1: -/-

INFO  : Map 1: 0/44

INFO  : Map 1: 0(+1)/44

Please feel free to give any feedback or suggestion for any improvements.


1 Comment

Pratik Debnath

November 28, 2019 at 1:36 pm

After trying this we are getting Vertex Failure error.
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2)

Leave a Reply to Pratik Debnath Cancel reply