如何解决self.\u java\u obj=\u jvm().ml.combust.mleap.spark.simplesparkserializer()类型错误:“javapackage”对象不可调用

5hcedyr0  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(209)

我尝试在emr群集上使用mleap,但当我尝试使用它时,收到以下错误:

self._java_obj = _jvm().ml.combust.mleap.spark.SimpleSparkSerializer()
 TypeError: 'JavaPackage' object is not callable

我从代码中加载jar文件
启动作业的脚本:

CLUSTER_ID=XXXXX
JOB_NAME=mleap_sample
SCRIPT_PATH=s3://XXXX/mleap_model.py

aws emr --profile xxx add-steps --cluster-id $CLUSTER_ID \
--steps Name=$JOB_NAME,Jar=command-runner.jar,\
Args=[spark-submit,--deploy-mode,client,\
--conf,spark.yarn.submit.waitAppCompletion=true,\
$SCRIPT_PATH],ActionOnFailure=CONTINUE

在我的代码里:

sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-base_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-core_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-runtime_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-spark-base_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-spark-extension_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
    "s3a://xxxx/mleap_script/mleap-tensor_2.11-0.16.0-sources.jar")

在加载期间的emr日志中,我看到:

20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-base_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-base_2.11-0.16.0-sources.jar with timestamp 1594130507784
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-core_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-core_2.11-0.16.0-sources.jar with timestamp 1594130507786
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-runtime_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-runtime_2.11-0.16.0-sources.jar with timestamp 1594130507788
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-spark-base_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-spark-base_2.11-0.16.0-sources.jar with timestamp 1594130507790
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-spark-extension_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-spark-extension_2.11-0.16.0-sources.jar with timestamp 1594130507793
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-tensor_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-tensor_2.11-0.16.0-sources.jar with timestamp 1594130507796

我用的是spark 2.4.5版
知道我为什么要面对这个问题吗?
ps:如果我在sagemaker笔记本上使用Pypark,我会收到相同的错误消息。
谢谢

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题