从现有snapy文件创建配置单元表

xwbd5t1u  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(334)

我是hadoop和hive的新手。我和你一起工作 hadoop 2.7.0 以及 hive 1.1.1 .
我有一些快速格式的文件。我已经在我的hdfs中复制了它们。我想知道:1。如何在hdfs中使用这些文件创建配置单元表。2如何配置hadoop以使用snappy文件。
编辑:我尝试了以下方法:
1.修改 core-site.xml :

<property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.GzipCodec,
           org.apache.hadoop.io.compress.DefaultCodec,
           org.apache.hadoop.io.compress.SnappyCodec
    </value>
</property>

2.修改 mapred-site.xml :

<property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
</property>

<property>
    <name>mapred.map.output.compress.codec</name>
    <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

<property>
    <name>mapreduce.admin.user.env</name>
    <value>LD_LIBRARY_PATH=/Users/8417/repos/hadoop_ecosystem/hadoop-2.7.0/lib/native</value>
</property>

安装snappy brew install snappy .
cp /usr/local/Cellar/snappy//lib/snappy.* to $HADOOP_HOME/lib/snappy cp /usr/local/Cellar/snappy/<version>/lib/snappy.* /Library/Java/Home/jre/lib .
6.增加了以下环境变量:

export JAVA_LIBRARY_PATH="$HADOOP_HOME/lib/native"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

但我没法让它工作。我得到以下错误:

Machine:native user$ hadoop fs -text hive/part-00000.snappy 
15/09/24 19:25:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
    at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
    at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
    at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:192)
    at org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:178)
    at org.apache.hadoop.io.compress.CompressionCodec$Util.createInputStreamWithCodecPool(CompressionCodec.java:157)
    at org.apache.hadoop.io.compress.SnappyCodec.createInputStream(SnappyCodec.java:162)
    at org.apache.hadoop.fs.shell.Display$Text.getInputStream(Display.java:163)
    at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:102)
    at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
    at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
    at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
    at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
    at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
    at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
    at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

Machine:native user$ hadoop checknative
15/09/24 19:41:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop:  false 
zlib:    false 
snappy:  false 
lz4:     false 
bzip2:   false 
openssl: false 
15/09/24 19:41:16 INFO util.ExitUtil: Exiting with status 1
k4ymrczo

k4ymrczo1#

确保已安装snappy,并且hadoop能够“看到”它:

hadoop checknative
...
snappy:  true /usr/lib64/libsnappy.so.1
...

还要确保文件的扩展名是“.snappy”。差不多了。
启动hive或beeline并创建表:

create external table t1(f1 int, ...) location '/path/to/dir/with/snappy/files';

相关问题