如何配置flink群集内存管理

p8ekf7hl  于 2021-06-21  发布在  Flink
关注(0)|答案(0)|浏览(229)

我有一个flink集群,并用它来运行批处理作业。问题是每次作业结束后,taskmanager的内存使用仍然繁忙。
集群运行在docker swarm中,有两台250gb内存的机器。
我的flink配置如下:

by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.

by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.

# In high availability mode, if you use the bin/start-cluster.sh script and setup

# the conf/masters file, this will be taken care of automatically. Yarn/Mesos

# automatically configure the host name based on the hostname of the node where the

# JobManager runs.

jobmanager.rpc.address: jobmanager17

# The RPC port where the JobManager is reachable.

jobmanager.rpc.port: 6123

# The heap size for the JobManager JVM

jobmanager.heap.mb: 204800

# The heap size for the TaskManager JVM

taskmanager.heap.mb: 204800

# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.

taskmanager.numberOfTaskSlots: 32

# Specify whether TaskManager memory should be allocated when starting up (true) or when

# memory is required in the memory manager (false)

# Important Note: For pure streaming setups, we highly recommend to set this value to `false`

# as the default state backends currently do not use the managed memory.

taskmanager.memory.preallocate: false

# The parallelism used for programs that did not specify and other parallelism.

parallelism.default: 1

集群工作得很好,但它在一个作业上花费内存,最后使用交换和磁盘结束。
我希望这是一个配置问题,但如果你需要更多的细节码头,请告诉我

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题