docker—使用fluent和flume无限增加的文件描述符数

00jrzges  于 2021-06-04  发布在  Flume
关注(0)|答案(1)|浏览(674)

我用fluent插件flume在docker容器中运行fluent代理。一段时间后,容器中的某些程序失败,出现下一个错误:

Check failed: _s.ok() Bad status: Runtime error: Could not create thread: Resource temporarily unavailable (error 11)

我发现docker容器中的文件描述符数量无限增加:

sudo lsof| grep flume | wc -l
469292

一段时间后,描述符的数量变得比文件描述符的限制还要大。看起来thrift连接是以错误的方式处理的:文件描述符在事务结束后仍然存在。
fluend配置:

<source>
  @type forward
  tag forward_1
  bind 0.0.0.0
  port 24224
  linger_timeout 0
</source>

<match forward_1>
      @type copy
      <store>
              @type flume
              timeout 15
              host localhost
             port 33333
      </store>
</match>

烟雾配置:

forward_1.sources  = source1
forward_1.channels = channel1
forward_1.sinks = sink1

# 

forward_1.sources.source1.type = thrift
forward_1.sources.source1.bind = localhost
forward_1.sources.source1.port = 33333
forward_1.sources.source1.channels = channel1

# 

forward_1.channels.channel1.type = memory
forward_1.channels.channel1.capacity = 10000
forward_1.channels.channel1.transactionCapacity = 1000

# 

forward_1.sinks.sink1.type = org.apache.kudu.flume.sink.KuduSink
forward_1.sinks.sink1.channel = channel1
forward_1.sinks.sink1.masterAddresses = 10.1.0.1:7051
forward_1.sinks.sink1.tableName = shop_logs
forward_1.sinks.sink1.batchSize = 50
forward_1.sinks.sink1.producer = KuduJsonProducer

docker版本:17.03 fluentd版本:0.14.19 flume版本:1.7.0
有没有办法限制文件描述符的数量?

taor4pac

taor4pac1#

尝试将源代码中的“线程”添加到flume配置中。这样地:

forward_1.sources.source1.threads = number_of_threads

阅读更多https://flume.apache.org/flumeuserguide.html#thrift-来源

相关问题