apache-flink-kryo序列化程序-classnotfoundexception

qojgxg4l  于 2021-06-21  发布在  Flink
关注(0)|答案(1)|浏览(470)

我在ApacheFlink1.8.1中有一个项目,使用Scala2.11和Java8。我以前使用maven进行编译和所有的依赖关系管理,但改用gradle。。。这就引出了下面这个问题:

j.l.ClassNotFoundException: om.tinker.my.project.ProjectPayload
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    ... 3 frames excluded
    at c.e.k.u.DefaultClassResolver.readName(DefaultClassResolver.java:172)
    ... 15 common frames omitted
    Wrapped by: c.e.kryo.KryoException: Unable to find class: om.tinker.my.project.ProjectPayload
    Serialization trace:
        eventOutputTag (com.my.project.contexts.ProjectContext)
        at c.e.k.u.DefaultClassResolver.readName(DefaultClassResolver.java:178)
        at c.e.k.u.DefaultClassResolver.readClass(DefaultClassResolver.java:147)
        at c.e.kryo.Kryo.readClass(Kryo.java:674)
        at c.e.k.s.ReflectField.read(ReflectField.java:107)
        at c.e.k.s.FieldSerializer.read(FieldSerializer.java:122)
        at c.e.kryo.Kryo.readClassAndObject(Kryo.java:793)
        at o.a.f.a.j.t.r.k.KryoSerializer.deserialize(KryoSerializer.java:346)
        at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:202)
        at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
        at o.a.f.r.p.NonReusingDeserializationDelegate.read(NonReusin...

首先,错误消息缺少“c”。类路径应该是“com.tinker.my.project.projectpayload”。。。我用那个代码检查了文件,我的import语句中没有丢失的c。。。
我还编辑了flink conf文件以使用父级优先策略。。。
进一步的背景信息:我有另一个文件叫做 ProjectContext 它有一个 ArrayList<ProjectPayload> . 它还有 eventOutputTag (如序列化跟踪中所述)。。。当我发表评论时 ArrayList<ProjectPayload> 它的创始者/创始者,一切正常!
当我把示例变量及其getters/setters放回 ProjectContext ,那么 ClassNotFoundException 发生。。。
此外,我还添加了大量的print语句,创建了projectpayload的示例,并很好地将其注销。

编辑(2020年6月30日)###

鉴于此序列化问题,我添加了以下代码: env.getConfig.registerTypeWithKryoSerializer(classOf[ProjectPayload], classOf[JavaSerializer[ProjectPayload]]) 现在我有一个尴尬的(但类似的)错误:

"j.l.ClassNotFoundException: \u0005sr\u00008com.tinker.my.project.ProjectPayload+\"v
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    ... 3 frames excluded
    at c.e.k.u.DefaultClassResolver.readName(DefaultClassResolver.java:172)
    ... 15 common frames omitted
    Wrapped by: c.e.kryo.KryoException: Unable to find class: \u0005sr\u00008com.tinker.my.project.ProjectPayload+\"v
    Serialization trace:
    allMyPayloads (com.tinker.my.project.ProjectContext)
    at c.e.k.u.DefaultClassResolver.readName(DefaultClassResolver.java:178)
    at c.e.k.u.DefaultClassResolver.readClass(DefaultClassResolver.java:147)
    at c.e.kryo.Kryo.readClass(Kryo.java:674)
    at c.e.k.s.ReflectField.read(ReflectField.java:107)
    at c.e.k.s.FieldSerializer.read(FieldSerializer.java:122)
    at c.e.kryo.Kryo.readClassAndObject(Kryo.java:793)
    at o.a.f.a.j.t.r.k.KryoSerializer.deserialize(KryoSerializer.java:346)
    at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:202)
    at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
    at o.a.f.r.p.NonReusingDeserializationDelegate....

结果是 \u0005 是unicode字符“inquiry”。以及 \u00008 导致谷歌搜索结果乱七八糟。。。稍后再汇报

编辑(2020年7月1日)####一些进展:我正在初始化 ArrayList<ProjectPayload> 内部 ProjectContext . 当我删除了初始化,将其移到外部,然后设置arraylist值时,我的代码走得更远了。然后它抱怨 HashMap<String, String> 示例变量——我最终删除了它,因为它没有被使用。

这让我想到 IndexOutOfBoundsException :

j.l.IndexOutOfBoundsException: Index: 93, Size: 9
    at java.util.ArrayList.rangeCheck(ArrayList.java:657)
    at java.util.ArrayList.get(ArrayList.java:433)
    at c.e.k.u.MapReferenceResolver.getReadObject(MapReferenceResolver.java:62)
    at c.e.kryo.Kryo.readReferenceOrNull(Kryo.java:838)
    at c.e.kryo.Kryo.readObjectOrNull(Kryo.java:761)
    at c.e.k.s.ReflectField.read(ReflectField.java:120)
    ... 12 common frames omitted
    Wrapped by: c.e.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 93, Size: 9
    Serialization trace:
        fooBarStr (com.tinker.my.project.contexts.ProjectContext)
        at c.e.k.s.ReflectField.read(ReflectField.java:133)
        at c.e.k.s.FieldSerializer.read(FieldSerializer.java:122)
        at c.e.kryo.Kryo.readClassAndObject(Kryo.java:793)
        at o.a.f.a.j.t.r.k.KryoSerializer.deserialize(KryoSerializer.java:346)
        at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:202)
        at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
        at o.a.f.r.p.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:55)
        at o.a.f.r.i.n.a.s.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRec...

关于kryo的github问题:https://github.com/esotericsoftware/kryo/issues/456

6psbrbz9

6psbrbz91#

试试这个:

env.getConfig.registerTypeWithKryoSerializer(classOf[ProjectPayload], classOf[JavaSerializer[ProjectPayload]])
env.getConfig.registerTypeWithKryoSerializer(classOf[ProjectContext], classOf[JavaSerializer[ProjectContext]])

并确保您正在导入 org.apache.flink.api.java.typeutils.runtime.kryo.JavaSerializer https://ci.apache.org/projects/flink/flink-docs-stable/dev/custom_serializers.html#issue-使用kryos javaserializer

相关问题