包含加载数据inpath的配置单元脚本在oozie中不起作用

ki0zmccv  于 2021-06-29  发布在  Hive
关注(0)|答案(1)|浏览(228)

我的任务是创建一个oozie工作流,以便每小时将数据加载到配置单元表。
我在virtualbox中使用cdh5.7
当我在customer表中运行包含load data inpath“/sqoop\u import\u increment”的配置单元脚本时;它工作得很好,数据被加载到配置单元表中。
但当我在oozie工作流上运行相同的脚本时,作业将以66%的速度终止,错误消息是main class[org.apache.oozie.action.hadoop.hivemain],退出代码[10001]
注意:但是用于create table的配置单元脚本可以与oozie工作流完美配合。请帮忙。
配置单元脚本:

use test;

create external table if not exists customer(customer_id int,name string,address string)row format delimited fields terminated by ',';

load data inpath /sqoop_import_increment into table customer;

工作流.xml:

<workflow-app name="hive_script" xmlns="uri:oozie:workflow:0.5">
    <start to="hive-4327"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="hive-4327" cred="hcat">
        <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
              <job-xml>lib/hive-config.xml</job-xml>
            <script>lib/impala-script.hql</script>
        </hive>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

作业属性:

oozie.use.system.libpath=True
security_enabled=False
dryrun=False
jobTracker=localhost:8032
nameNode=hdfs://quickstart.cloudera:8020

配置文件:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

  <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files  -->
  <!-- that are implied by Hadoop setup variables.                                                -->
  <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive    -->
  <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
  <!-- resource).                                                                                 -->

  <!-- Hive Execution Parameters -->

  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://127.0.0.1/metastore?createDatabaseIfNotExist=true</value>
    <description>JDBC connect string for a JDBC metastore</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>cloudera</value>
  </property>

  <property>
    <name>hive.hwi.war.file</name>
    <value>/usr/lib/hive/lib/hive-hwi-0.8.1-cdh4.0.0.jar</value>
    <description>This is the WAR file with the jsp content for Hive Web Interface</description>
  </property>

  <property>
    <name>datanucleus.fixedDatastore</name>
    <value>true</value>
  </property>

  <property>
    <name>datanucleus.autoCreateSchema</name>
    <value>false</value>
  </property>

  <property>
    <name>hive.metastore.uris</name>
    <value>thrift://127.0.0.1:9083</value>
    <description>IP address (or fully-qualified domain name) and port of the metastore host</description>
  </property>
</configuration>
unguejic

unguejic1#

上次我遇到这个问题时,发现并非所有数据节点上都安装了配置单元客户端。
当您手动运行配置单元查询时,可能是从安装了配置单元客户端的节点执行查询。但是当要求oozie运行查询时,它将从随机数据节点执行查询。因此,您需要在所有数据节点上设置配置单元客户端。
这假设您一般不能让oozie运行配置单元查询(并且这个特定命令没有任何特定问题)。

相关问题