spark流kafka消费者不喜欢dstream

tzdcorbm  于 2021-06-07  发布在  Kafka
关注(0)|答案(1)|浏览(378)

我使用的是spark shell(scala 2.10和spark streaming) org.apache.spark:spark-streaming-kafka-0-10_2.10:2.0.1 )要测试spark/kafka消费者:

import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.spark.streaming.kafka010._
import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent
import org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribe
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.dstream.DStream

val kafkaParams = Map[String, Object](
  "bootstrap.servers" -> "mykafka01.example.com:9092",
  "key.deserializer" -> classOf[StringDeserializer],
  "value.deserializer" -> classOf[StringDeserializer],
  "group.id" -> "mykafka",
  "auto.offset.reset" -> "latest",
  "enable.auto.commit" -> (false: java.lang.Boolean)
)

val topics = Array("mytopic")

def createKafkaStream(ssc: StreamingContext, topics: Array[String], kafkaParams: Map[String,Object]) : DStream[(String, String)] = {
    KafkaUtils.createDirectStream[String, String](ssc, PreferConsistent, Subscribe[String, String](topics, kafkaParams))
}

def messageConsumer(): StreamingContext = {
    val ssc = new StreamingContext(SparkContext.getOrCreate(), Seconds(10))

    createKafkaStream(ssc, topics, kafkaParams).foreachRDD(rdd => {
        rdd.collect().foreach { msg =>
            try {
                println("Received message: " + msg._2)
            } catch {
                case e @ (_: Exception | _: Error | _: Throwable) => {
                println("Exception: " + e.getMessage)
                e.printStackTrace()
            }
          }
        }
    })

    ssc
}

val ssc = StreamingContext.getActiveOrCreate(messageConsumer)
ssc.start()
ssc.awaitTermination()

运行此命令时,出现以下异常:

<console>:60: error: type mismatch;
 found   : org.apache.spark.streaming.dstream.InputDStream[org.apache.kafka.clients.consumer.ConsumerRecord[String,String]]
 required: org.apache.spark.streaming.dstream.DStream[(String, String)]
                  KafkaUtils.createDirectStream[String, String](ssc, PreferConsistent, Subscribe[String, String](topics, kafkaParams))
                                                               ^

我反复检查scala/api文档,这段代码看起来应该正确执行。你知道我哪里出错了吗?

ttygqcqt

ttygqcqt1#

Subscribetopics 参数为 Array[String] ,您正在按照传递单个字符串 def createKafkaStream(ssc: StreamingContext, topics: String, . 将参数类型更改为 Array[String] (并恰当地称之为)将解决问题。

相关问题