spark kafka流作业在dc/os(mesos)中部署时不会发现协调器

ao218c7q  于 2021-06-07  发布在  Kafka
关注(0)|答案(1)|浏览(254)

我在java中按照https://spark.apache.org/docs/2.1.0/streaming-kafka-0-10-integration.html. 当kafka和cassandra服务器是独立的时,它可以完美地工作。原木提取

...
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

18/02/04 20:40:35 INFO AppInfoParser: Kafka version : 0.11.0.0
18/02/04 20:40:35 INFO AppInfoParser: Kafka commitId : cb8625948210849f
18/02/04 20:40:35 INFO AbstractCoordinator: Discovered coordinator spark-vm:9092 (id: 2147483647 rack: null) for group group_1.
18/02/04 20:40:35 INFO ConsumerCoordinator: Revoking previously assigned partitions [] for group group_1
18/02/04 20:40:35 INFO AbstractCoordinator: (Re-)joining group group_1
[2018-02-04 20:40:35,929] INFO [GroupCoordinator 0]: Preparing to rebalance group group_1 with old generation 6 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
[2018-02-04 20:40:35,946] INFO [GroupCoordinator 0]: Stabilized group group_1 generation 7 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
...

当我在mesos集群中部署时,问题出现了,因为作业连接到kafka,但它没有发现任何协调器

metrics.sample.window.ms = 30000
auto.offset.reset = latest

18/02/04 19:27:41 INFO AppInfoParser: Kafka version : 0.10.0.1
18/02/04 19:27:41 INFO AppInfoParser: Kafka commitId : a7a17cdec9eaa6c5
18/02/04 19:27:42 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 1 is now TASK_RUNNING
18/02/04 19:27:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.65.111:50842) with ID 1
18/02/04 19:27:45 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.65.111:39485 with 366.3 MB RAM, BlockManagerId(1, 192.168.65.111, 39485, None)
18/02/04 19:27:47 INFO CassandraConnector: Disconnected from Cassandra cluster: cassandra
2018-02-04 19:32:19,161:7(0x7fb4fd02b700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 12ms

Kafka使用的zk是master。mesos:2181/dcos-service-kafka 但我不知道如何配置它在Spark作业,我甚至不知道这是问题,或者如果它只是一个Kafka版本的问题。

2izufjch

2izufjch1#

问题是连通性。spark job无法使用dc/os提供的vip端点连接到kafka,但是日志没有引发任何异常,正如这些技术的新手所期望的那样。

相关问题