spark核心依赖中的冲突它是如何工作的?

oyt4ldly  于 2021-05-24  发布在  Spark
关注(0)|答案(1)|浏览(534)

我的问题的切入点是一个例外:

[info]   Cause: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.8.4
[info]   at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:64)
[info]   at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:51)
[info]   at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
[info]   at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:745)
[info]   at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
[info]   at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
[info]   at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
[info]   at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
[info]   at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
[info]   at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)

它发生在spark升级之后。我设法克服了这个问题,但我仍然不明白根本原因。看起来spark相关性有问题,请查看:
https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/2.4.7
它需要 com.fasterxml.jackson.core » jackson-databind 2.6.7.3 . 没问题。此外,它还需要:https://mvnrepository.com/artifact/org.json4s/json4s-jackson_2.12/3.5.3 没问题。然而,https://mvnrepository.com/artifact/org.json4s/json4s-jackson_2.12/3.5.3 要求https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind/2.8.4 现在,我们可以看到 spark-2.4.7 引入了依赖项冲突 jackson-databind 它自己。我的推理哪里错了?

c90pui9n

c90pui9n1#

正如你在父pom中看到的, com.fasterxml.jackson.core.* 被排除在 json4s-jackson :
https://github.com/apache/spark/blob/v2.4.7/pom.xml#l755
所以,毕竟没有冲突

相关问题