当试图运行Flume作业时,我得到下面给出的错误。我在cloudera上运行这个。
Kafka是源头
morphline用作拦截器,从中创建avro记录。
Flume为hdfs
测试环境中完全相同的文件(morpline、avro schema等、flume config)。但在另一个环境中,它抛出了这个错误。
2019-07-15 14:24:17,669 WARN org.apache.flume.sink.hdfs.BucketWriter: Caught IOException writing to HDFSWriter (no protocol: value). Closing file (hdfs://8.8.8.8:8020/user/hive/warehouse/folder/folder/FlumeData.1563162656585.tmp) and rethrowing exception.
2019-07-15 14:24:17,670 INFO org.apache.flume.sink.hdfs.BucketWriter: Closing hdfs://8.8.8.8:8020/user/hive/warehouse/folder/folder/FlumeData.1563162656585.tmp
2019-07-15 14:24:17,670 ERROR org.apache.flume.sink.hdfs.HDFSEventSink: process failed
java.lang.NullPointerException
at org.apache.flume.sink.hdfs.AvroEventSerializer.flush(AvroEventSerializer.java:187)
at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:131)
at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:327)
at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:323)
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:698)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2019-07-15 14:24:17,671 ERROR org.apache.flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.NullPointerException
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.flume.sink.hdfs.AvroEventSerializer.flush(AvroEventSerializer.java:187)
at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:131)
at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:327)
at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:323)
at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701)
at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:698)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
我在flume上找到了相关代码:https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/bucketwriter.java (line:602)
// write the event
try {
sinkCounter.incrementEventDrainAttemptCount();
callWithTimeout(new CallRunner<Void>() {
@Override
public Void call() throws Exception {
writer.append(event); // could block
return null;
}
});
} catch (IOException e) {
LOG.warn("Caught IOException writing to HDFSWriter ({}). Closing file (" +
bucketPath + ") and rethrowing exception.",
e.getMessage());
close(true);
throw e;
}
错误: Caught IOException writing to HDFSWriter (no protocol: value). Closing file
我不能锻炼出什么错误 no protocol: value
手段。
在与flume和hdfs相关的任何上下文中,我都找不到对此错误的任何引用。
1条答案
按热度按时间1sbrub3j1#
配置中缺少incerceptor协议--在修复问题的flume配置文件中添加了“file:/”。
类似问题参考:https://community.cloudera.com/t5/data-ingestion-integration/flume-hdfs-sink-error-quot-unknown-protocol-hdfs-quot/td-p/19344