flume传输gzip文件

ddarikpa  于 2021-06-03  发布在  Flume
关注(0)|答案(1)|浏览(356)

我想把gzip文件传输到hdfs。我正在探索的选择是spooldir。我在指定spooldir的远程位置时遇到问题。任何建议请:

agent1.sources = src-1
agent1.channels = c1
agent1.sinks = k1

agent1.sources.src-1.type = spooldir
agent1.sources.src-1.channels = c1
agent1.sources.src-1.hostname = ipaddress
agent1.sources.src-1.spoolDir = /var/logs/webserver
agent1.channels.c1.type = memory

agent1.sinks.k1.type = hdfs
agent1.sinks.k1.channel = c1
agent1.sinks.k1.hdfs.path = hdfspath
agent1.sinks.k1.hdfs.fileType = CompressedStream
agent1.sinks.k1.hdfs.round  = true
agent1.sinks.k1.hdfs.roundValue = 10
agent1.sinks.k1.hdfs.roundUnit = minute
pexxcrt2

pexxcrt21#

假脱机目录源需要指向本地系统上运行flume的目录。此行不是假脱机目录源的有效选项:

agent1.sources.src-1.hostname = ipaddress

相关问题