azkaban-hive作业失败

8ehkhllq  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(392)

我想在阿兹卡班做Hive工作
我可以上载配置单元作业,但配置单元作业未正确执行。

ERROR [hive-demo] [Azkaban] Failed to build job executor for job hive-demoJob type     'hive' is unrecognized. Could not construct job[{hive.query.01: drop table words;, hive.query.03: describe words;, hive.query.02: create table words (freq int, word string) row format delimited fields terminated by ' ' stored as textfile;, working.dir: /home/hduser/Technology/azkaban/azkaban-executor/executions/44, azk.hive.action: execute.query, azkaban.job.attempt: 0, type: hive, hive.query.05: select * from words limit   10;, hive.query.04: load data local inpath "res/input" into table words;, user.to.proxy:   azkaban, hive.query.06: select freq, count(1) as f2 from words group by freq sort by f2 desc  limit 10;, parent = {azkaban.flow.flowid: hive-demo, azkaban.flow.execid: 44,  azkaban.flow.start.timezone: America/Los_Angeles, azkaban.flow.start.hour: 22,  azkaban.flow.start.second: 17, azkaban.flow.start.year: 2014,  azkaban.flow.start.milliseconds: 111, azkaban.flow.start.minute: 54,  azkaban.flow.start.timestamp: 2014-01-31T22:54:17.111-08:00, azkaban.flow.start.month: 01,  azkaban.flow.projectversion: 2, azkaban.flow.projectid: 4, azkaban.flow.uuid: 29a46293-cc95- 4c66-b2b1-51014205ce3b, azkaban.flow.start.day: 31, parent = {}}}] of type[hive]. 2014/02/01  12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Failed to build job type, skipping this job    2014/02/01 12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Job run failed! 2014/02/01 12:24:17.295 +0530 INFO [hive-demo] [Azkaban] Finishing job hive-demo at 1391237657295  2014/02/01 12:24:17.382 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished  2014/02/01 12:24:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution 43 from  recently finished flows list. 2014/02/01 12:24:17.408 +0530 INFO [hive-demo] [Azkaban] Job  Finished hive-demo with status FAILED 2014/02/01 12:24:17.463 +0530 INFO [hive-demo]  [Azkaban] Finishing up flow. Awaiting Termination 2014/02/01 12:24:17.463 +0530 INFO [hive- demo] [Azkaban] Setting flow status to Failed. 2014/02/01 12:24:17.464 +0530 INFO [hive- demo] [Azkaban] Flow is set to FAILED 2014/02/01 12:24:17.464 +0530 INFO [hive-demo]  [Azkaban] Setting end time for flow 44 to 1391237657464 2014/02/01 12:24:17.552 +0530 INFO  [FlowRunnerManager] [Azkaban] Flow 44 is finished. Adding it to recently finished flows  list. 2014/02/01 12:26:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently  finished 2014/02/01 12:26:17.384 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution  44 from recently finished flows list.
 ========================================================================

这是我的plugin.properties中的内容

job.class=azkaban.jobtype.hiveutils.azkaban.HiveViaAzkaban

hive.home=HIVE_HOME

hive.aux.jars.path=/usr/lib/hive/lib
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib

# hive.jvm.args=-Dhive.querylog.location=. -Dhive.exec.scratchdir=/tmp/hive-    ${user.to.proxy} -Dhive.aux.jars.path=${hive.aux.jars.path}

jobtype.global.jvm.args=${hive.jvm.args}

这是我的private.properties jobtype.classpath=$hadoop\u home/conf,$hadoop\u home/lib/,$hive\u home/lib/,$hive\u home/conf,/usr/lib/hive/lib jobtype.class=azkaban.jobtype.hadoopjavajob

hive.aux.jars.path=/usr/lib/hive/lib
env.HADOOP_HOME=$HADOOP_HOME
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib

hive.home=$HIVE_HOME

jobtype.global.classpath=${hive.classpath.items}
  hive.classpath.items=./*,${hive.home}/lib/*,${hive.home}/conf/,${hive.aux.jars.path}/*,${had    oop.home}/*,${hadoop.home}/lib/*,${hadoop.home}/conf/

我只在executor文件夹中添加了插件文件夹。
为什么不检测作业类型。请帮忙。

c86crjj0

c86crjj01#

很难说是怎么回事,因为你既不共享作业本身也不共享日志,但请尝试下面的方法,让我们知道这是否有效。创建一个文件 test.job 包括以下内容:

type=hive
azk.hive.action=execute.query
hive.query=CREATE TABLE test(id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
user.to.proxy=hadoop

并按照“创建流”部分的描述提交。有关azkaban(2.1)支持开箱即用的不同类型作业的更多详细信息,请参阅作业类型/插件文档。

相关问题