我已经编写了一个sqoop导入脚本来将数据从teradata导入到hive。当我从命令行运行它时,它工作得很好,但是当我将它放入shell脚本并尝试通过oozie工作流执行时,我得到了以下错误
`[0000069-150114201015959-oozie-oozi-W@sqoop-shell]**Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain],**exit code [1]
2015-04-02 08:50:55,440 INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@sqoop-shell] end executor for wf action 0000069-150114201015959-oozie-oozi-W with wf job 0000069-150114201015959-oozie-oozi-W
2015-04-02 08:50:55,459 INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@sqoop-shell] ERROR is considered as FAILED for SLA
2015-04-02 08:50:55,505 INFO ActionStartXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail] Start action [0000069-150114201015959-oozie-oozi-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2015-04-02 08:50:55,505 WARN ActionStartXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail] [***0000069-150114201015959-oozie-oozi-W@fail***]Action status=DONE
2015-04-02 08:50:55,505 WARN ActionStartXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail] [***0000069-150114201015959-oozie-oozi-W@fail***]Action updated in DB!
2015-04-02 08:50:55,522 INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail] end executor for wf action 0000069-150114201015959-oozie-oozi-W with wf job 0000069-150114201015959-oozie-oozi-W
2015-04-02 08:50:55,556 WARN CoordActionUpdateXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[-]**E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100**`
下面是我的workflow.xml
`<workflow-app xmlns='uri:oozie:workflow:0.3' name='sqoop-shell-wf'>
<start to='sqoop-shell' />
<action name='sqoop-shell'>
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${resourceManager}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>Sqoopscript.sh</exec>
<file>Sqoopscript.sh#script.sh</file>
</shell>
<ok to="end" />
<error to="fail" />
</action>
<kill name="fail">
<message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
Map减少日志
error: error reading /usr/lib/hadoop/lib/smore.jar; /usr/lib/hadoop/lib/smore.jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/janusclient.jar; /usr/lib/hadoop/lib/janusclient .jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/aster- networking.jar; /usr/lib/hadoop/lib/aster-networking.jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/adfs-api-loader.jar; /usr/lib/hadoop/lib/adfs-api-loader.jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/aftp.jar; /usr/lib/hadoop/lib/aftp.jar (Permission denied)
error: error reading /usr/lib/hadoop/lib/adfs.jar; /usr/lib/hadoop/lib/adfs.jar (Permission denied)
Note: /tmp/sqoop yarn/compile/f3aca99f37fa19e505b47bab1499bf5d/EXT_EVNT_ERR_ARC.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
che.sqoop.hive.HiveImport -**Caused by: java.io.FileNotFoundException: File does not exist: hdfs:/user/yarn**`
1条答案
按热度按时间lqfhib0f1#
看起来您正在使用shell操作,而您可以/应该使用sqoop操作。运行sqoop作业时需要包含几个jar。其中大部分都包含在oozie共享库中。下面是一个示例工作流:
我建议遵循关于如何使用oozie和sqoop的指南。