无法在cloudera的配置单元中创建存储桶

qco9c6ql  于 2021-06-26  发布在  Hive
关注(0)|答案(1)|浏览(347)

我正试图在cloudera的Hive中创建一个带扣的table。但是,创建的普通表没有任何存储桶。
首先,我使用hivecli创建了一个普通表,它使用了named marks\u temp

CREATE  TABLE marks_temp(
id INT,
Name string,
mark int
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',';

我已经从一个文本文件“desktop/data/littlebigdata.txt”将以下数据加载到marks\ temp表中

101,Firdaus,88
102,Pranav,78
103,Rahul,65
104,Sanjoy,65
105,Firdaus,88
106,Pranav,78
107,Rahul,65
108,Sanjoy,65
109,Amar,54
110,Sahil,34
111,Rahul,45
112,Rajnish,67
113,Ranjeet,56
114,Sanjoy,34

我已经用下面的命令加载了上面的数据

LOAD DATA LOCAL INPATH 'Desktop/Data/littlebigdata.txt'
INTO TABLE  marks_temp;

在成功加载数据之后,我将创建一个名为marks\u temp的bucked表

CREATE TABLE marks_bucketed(
id INT,
Name string,
mark int
)
CLUSTERED BY (id) INTO 4 BUCKETS;

现在,我将数据从marks\u temp表插入marks\u bucketed表。

INSERT INTO marks_bucketed
SELECT id,Name, mark FROM marks_temp;

在此之后,一些作业开始运行。什么,我在作业日志中看到它说“由于没有reduce操作符,reduce任务的数量被设置为0”

hive> insert into marks_bucketed
        > select id,Name,mark from marks_temp;
    Query ID = cloudera_20180601035353_29b25ffe-541e-491e-aea6-b36ede88ed79
    Total jobs = 3
    Launching Job 1 out of 3
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_1527668582032_0004, Tracking URL = http://quickstart.cloudera:8088/proxy/application_1527668582032_0004/
    Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1527668582032_0004
    Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
    2018-06-01 03:54:01,328 Stage-1 map = 0%,  reduce = 0%
    2018-06-01 03:54:14,444 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.21 sec
    MapReduce Total cumulative CPU time: 2 seconds 210 msec
    Ended Job = job_1527668582032_0004
    Stage-4 is selected by condition resolver.
    Stage-3 is filtered out by condition resolver.
    Stage-5 is filtered out by condition resolver.
    Moving data to: hdfs://quickstart.cloudera:8020/user/hive/warehouse/marks_bucketed/.hive-staging_hive_2018-06-01_03-53-45_726_2788383119636056364-1/-ext-10000
    Loading data to table default.marks_bucketed
    Table default.marks_bucketed stats: [numFiles=1, numRows=14, totalSize=194, rawDataSize=180]
    MapReduce Jobs Launched: 
    Stage-Stage-1: Map: 1   Cumulative CPU: 2.21 sec   HDFS Read: 3937 HDFS Write: 273 SUCCESS
    Total MapReduce CPU Time Spent: 2 seconds 210 msec
    OK
    Time taken: 31.307 seconds

甚至,hue文件浏览器也只显示一个文件。附上截图。色相文件浏览器屏幕截图为马克琰扣表

nkcskrwz

nkcskrwz1#

从配置单元文档
仅版本0.x和1.x
命令set hive.enforce.bucketing=true;允许根据表自动选择正确的还原数和按列分类。否则,您需要将reducer的数量设置为与set mapred.reduce.tasks=256中的bucket数量相同;有一个群集。。。select中的子句。
所以您需要设置属性来强制bucketing,或者选择manual选项并像

set mapred.reduce.tasks = 4;
select id,Name,mark from marks_temp cluster by id;

相关问题