使用java在hdfs中编写文件

2fjabf4q  于 2021-06-03  发布在  Hadoop
关注(0)|答案(3)|浏览(327)

我想在hdfs中创建一个文件并在其中写入数据。我用了这个代码:

Configuration config = new Configuration();     
FileSystem fs = FileSystem.get(config); 
Path filenamePath = new Path("input.txt");  
try {
    if (fs.exists(filenamePath)) {
        fs.delete(filenamePath, true);
    }

    FSDataOutputStream fin = fs.create(filenamePath);
    fin.writeUTF("hello");
    fin.close();
}

它创建文件,但不在其中写入任何内容。我找了很多,但什么也没找到。我有什么问题?我用hdfs写东西需要许可吗?
谢谢。

suzh9iv8

suzh9iv81#

请尝试以下方法。

FileSystem fs = path.getFileSystem(conf);
SequenceFile.Writer inputWriter = new SequenceFile.Writer(fs, conf, path, LongWritable.class, MyWritable.class);
inputWriter.append(new LongWritable(uniqueId++), new MyWritable(data));
inputWriter.close();
tjvv9vkg

tjvv9vkg2#

作为@tariq的asnwer的替代方法,您可以在获取文件系统时传递uri

import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.conf.Configuration
import java.net.URI
import org.apache.hadoop.fs.Path
import org.apache.hadoop.util.Progressable
import java.io.BufferedWriter
import java.io.OutputStreamWriter

Configuration configuration = new Configuration();
FileSystem hdfs = FileSystem.get( new URI( "hdfs://localhost:54310" ), configuration );
Path file = new Path("hdfs://localhost:54310/s2013/batch/table.html");
if ( hdfs.exists( file )) { hdfs.delete( file, true ); } 
OutputStream os = hdfs.create( file,
    new Progressable() {
        public void progress() {
            out.println("...bytes written: [ "+bytesWritten+" ]");
        } });
BufferedWriter br = new BufferedWriter( new OutputStreamWriter( os, "UTF-8" ) );
br.write("Hello World");
br.close();
hdfs.close();

参见示例:https://github.com/mpereira-dev/mapreduce-with-accumulo/blob/master/src/main/java/batch/batchscan2html.java

3j86kqsm

3j86kqsm3#

要么定义 HADOOP_CONF_DIR 将环境变量添加到hadoop配置文件夹或在代码中添加以下两行:

config.addResource(new Path("/HADOOP_HOME/conf/core-site.xml"));
config.addResource(new Path("/HADOOP_HOME/conf/hdfs-site.xml"));

如果您不添加这个,您的客户机将尝试写入本地fs,从而导致权限拒绝异常。

相关问题