apache beam:初始化分区“topic-1”时超时kafka客户端可能无法连接到服务器

kadbb459  于 2021-06-06  发布在  Kafka
关注(0)|答案(1)|浏览(381)

当我的apachebeam应用程序连接到kafka集群并启用acl时,出现了这个错误。请帮我解决这个问题。

Caused by: java.io.IOException: Reader-4: Timeout while initializing partition 'test-1'. Kafka client may not be able to connect to servers.
    org.apache.beam.sdk.io.kafka.KafkaUnboundedReader.start(KafkaUnboundedReader.java:128)
    org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:779)
    org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:361)
    org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:194)
    org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
    org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76)
    org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1228)
    org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:143)
    org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:967)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    java.lang.Thread.run(Thread.java:745)

我有一个Kafka集群,在gke上有3个节点。我用复制因子3和分区5创建了一个主题。

kafka-topics --create --zookeeper zookeeper:2181 \
  --replication-factor 3 --partitions 5 --topic topic

我为test\u consumer\u group consumer group设置了主题测试的读取权限。

kafka-acls --authorizer-properties zookeeper.connect=zookeeper:2181 \
  --add --allow-principal User:CN=myuser.test.io --consumer \
  --topic test --group 'test_consumer_group'

在我的apachebeam应用程序中,我设置configuration group.id=test\u consumer\u group。
也测试控制台消费者和它不工作。

$ docker run --rm   -v `pwd`:/cert   confluentinc/cp-kafka:5.1.0 \
  kafka-console-consumer   --bootstrap-server kafka.xx.xx:19092 \
  --topic topic --consumer.config /cert/client-ssl.properties
[2019-03-08 05:43:07,246] WARN [Consumer clientId=consumer-1, groupId=test_consumer_group]
Received unknown topic or partition error in ListOffset request for
partition test-3 (org.apache.kafka.clients.consumer.internals.Fetcher)
gupuwyp2

gupuwyp21#

对你的Kafka读者来说似乎是个沟通问题 Kafka client may not be able to connect to servers

相关问题