没有在hive中创建表,但是数据在hdfs中被crerated

iswrvxsc  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(347)

我是hdfs的新手,我正在尝试从我的oracle12cdb导入数据。我有一个表emp,它需要导入到hdfs以及hive表中。
我的数据正在hdfs中创建('/user/hdfs'创建一个文件夹'emp'。但当我打开配置单元查询编辑器并键入“showtables”时,我在这里看不到任何表。我也需要在Hive里创建表
我正在运行以下命令。

1. Since am running sqoop as root user
    usermod -a -G supergroup hardik

    2.
    export SQOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop
    export HIVE_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive

    export HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ojdbc7.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive/lib/*
    export HADOOP_USER_NAME=hdfs

    3.
    export PATH=$PATH:$HIVE_HOME/bin

现在我运行sqoop import命令,并在控制台上获得以下内容

4.

    sqoop import --connect jdbc:oracle:thin:@bigdatadev2:1521/orcl --username BDD1 --password oracle123 --table EMP --hive-import -m 1 --create-hive-table --hive-table EMP

    [root@bigdatadev1 ~]# sqoop import --connect jdbc:oracle:thin:@bigdatadev2:1521/orcl --username BDD1 --password oracle123 --table EMP --hive-import -m 1 --create-hive-table --hive-table EMP
    Warning: /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
    Please set $ACCUMULO_HOME to the root of your Accumulo installation.
    16/04/07 22:15:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.1
    16/04/07 22:15:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
    16/04/07 22:15:23 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
    16/04/07 22:15:23 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
    16/04/07 22:15:23 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
    16/04/07 22:15:23 INFO manager.SqlManager: Using default fetchSize of 1000
    16/04/07 22:15:23 INFO tool.CodeGenTool: Beginning code generation
    16/04/07 22:15:24 INFO manager.OracleManager: Time zone has been set to GMT
    16/04/07 22:15:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EMP t WHERE 1=0
    16/04/07 22:15:24 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
    Note: /tmp/sqoop-root/compile/fcb6484db042a7b4295d911956145a4e/EMP.java uses or overrides a deprecated API.
    Note: Recompile with -Xlint:deprecation for details.
    16/04/07 22:15:25 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/fcb6484db042a7b4295d911956145a4e/EMP.jar
    16/04/07 22:15:25 INFO manager.OracleManager: Time zone has been set to GMT
    16/04/07 22:15:25 INFO manager.OracleManager: Time zone has been set to GMT
    16/04/07 22:15:25 INFO mapreduce.ImportJobBase: Beginning import of EMP
    16/04/07 22:15:25 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
    16/04/07 22:15:25 INFO manager.OracleManager: Time zone has been set to GMT
    16/04/07 22:15:26 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    16/04/07 22:15:26 INFO client.RMProxy: Connecting to ResourceManager at bigdata/10.103.25.39:8032
    16/04/07 22:15:30 INFO db.DBInputFormat: Using read commited transaction isolation
    16/04/07 22:15:30 INFO mapreduce.JobSubmitter: number of splits:1
    16/04/07 22:15:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460040138373_0007
    16/04/07 22:15:31 INFO impl.YarnClientImpl: Submitted application application_1460040138373_0007
    16/04/07 22:15:31 INFO mapreduce.Job: The url to track the job: http://bigdata:8088/proxy/application_1460040138373_0007/
    16/04/07 22:15:31 INFO mapreduce.Job: Running job: job_1460040138373_0007
    16/04/07 22:15:37 INFO mapreduce.Job: Job job_1460040138373_0007 running in uber mode : false
    16/04/07 22:15:37 INFO mapreduce.Job:  map 0% reduce 0%
    16/04/07 22:15:43 INFO mapreduce.Job: Task Id : attempt_1460040138373_0007_m_000000_0, Status : FAILED
    Error: EMP : Unsupported major.minor version 52.0
    16/04/07 22:15:56 INFO mapreduce.Job: Task Id : attempt_1460040138373_0007_m_000000_1, Status : FAILED
    Error: EMP : Unsupported major.minor version 52.0
    16/04/07 22:16:03 INFO mapreduce.Job:  map 100% reduce 0%
    16/04/07 22:16:04 INFO mapreduce.Job: Job job_1460040138373_0007 completed successfully
    16/04/07 22:16:04 INFO mapreduce.Job: Counters: 31
        File System Counters
            FILE: Number of bytes read=0
            FILE: Number of bytes written=137942
            FILE: Number of read operations=0
            FILE: Number of large read operations=0
            FILE: Number of write operations=0
            HDFS: Number of bytes read=87
            HDFS: Number of bytes written=12
            HDFS: Number of read operations=4
            HDFS: Number of large read operations=0
            HDFS: Number of write operations=2
        Job Counters 
            Failed map tasks=2
            Launched map tasks=3
            Other local map tasks=3
            Total time spent by all maps in occupied slots (ms)=20742
            Total time spent by all reduces in occupied slots (ms)=0
            Total time spent by all map tasks (ms)=20742
            Total vcore-seconds taken by all map tasks=20742
            Total megabyte-seconds taken by all map tasks=10619904
        Map-Reduce Framework
            Map input records=3
            Map output records=3
            Input split bytes=87
            Spilled Records=0
            Failed Shuffles=0
            Merged Map outputs=0
            GC time elapsed (ms)=53
            CPU time spent (ms)=2090
            Physical memory (bytes) snapshot=207478784
            Virtual memory (bytes) snapshot=2169630720
            Total committed heap usage (bytes)=134217728
        File Input Format Counters 
            Bytes Read=0
        File Output Format Counters 
            Bytes Written=12
    16/04/07 22:16:04 INFO mapreduce.ImportJobBase: Transferred 12 bytes in 38.6207 seconds (0.3107 bytes/sec)
    16/04/07 22:16:04 INFO mapreduce.ImportJobBase: Retrieved 3 records.
    16/04/07 22:16:05 INFO manager.OracleManager: Time zone has been set to GMT
    16/04/07 22:16:05 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EMP t WHERE 1=0
    16/04/07 22:16:05 INFO hive.HiveImport: Loading uploaded data into Hive

    Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/hive-common-1.1.0-cdh5.5.1.jar!/hive-log4j.properties
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. AlreadyExistsException(message:Table EMP already exists)

我已经尝试了sqoop导入命令的所有变体,但没有一个成功。我今天更糊涂了。请帮忙。请不要将此标记为重复。

cvxl0en2

cvxl0en21#

从你的日志中,我发现了两个错误:
错误:emp:不支持的主版本。次版本52.0
当您试图将使用java 1.8编译器编译的类运行到较低的jre版本(如jre 1.7或jre 1.6)时,会出现不支持的major.minor版本52.0。在这里查看更多。
失败:执行错误,从org.apache.hadoop.hive.ql.exec.ddltask返回代码1。已存在异常(message:table emp 已存在)
你的工作一直到把数据送到hdfs。您必须再次尝试相同的命令而不删除此命令 /user/hdfs/EMP 目录。所以你才会犯这个错误。
请检查此相关答案。

相关问题