aws glue失败,exitcode:15,用户类引发异常:java.lang.runtimeexception:未使用外部依赖项初始化

zzwlnbp8  于 2021-05-24  发布在  Spark
关注(0)|答案(0)|浏览(402)

我正在尝试使用aws glue scala脚本处理awss3文件夹中的几个文件。在那之前我需要检查文件的数量和名字。为此,我编写了一个java类,它包含检查文件数量和名称的所有操作,并打包到一个jar文件中。我在glue配置中将这个jar文件作为一个依赖jars路径。当我在本地运行这个java类时,它运行得很好。但用于涂胶时,其刮刀误差如下。

ERROR ApplicationMaster: User class threw exception: java.lang.RuntimeException: Not initialized
java.lang.RuntimeException: Not initialized
    at com.amazonaws.services.glue.util.JobBookmark$.preCheck(JobBookmarkUtils.scala:22)
    at com.amazonaws.services.glue.util.JobBookmark$.commit(JobBookmarkUtils.scala:84)
    at com.amazonaws.services.glue.util.Job$.commit(Job.scala:94)
    at GlueApp$.runJob(script_2020-10-04-04-10-24.scala:117)
    at GlueApp$.main(script_2020-10-04-04-10-24.scala:62)

我的java jar文件pom.xml依赖项

<dependencies>
         <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.11.875</version>
        </dependency>

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-core</artifactId>
            <version>1.11.875</version>
        </dependency>
    </dependencies>

java类方法

public ArrayList<String> getFiles() {
        BasicAWSCredentials credit = new BasicAWSCredentials("key",
                "secreat");
        AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.US_EAST_2)
                .withCredentials(new AWSStaticCredentialsProvider(credit)).build();
        ArrayList<String> records = new ArrayList<>();
        ListObjectsV2Request req = new ListObjectsV2Request().withBucketName("my-inputpath")
                .withPrefix("monthly/data/");

        ListObjectsV2Result listing = s3Client.listObjectsV2(req);  

        for (S3ObjectSummary summary : listing.getObjectSummaries()) {

                records.add(summary.getKey().replace("monthly/", "/"));
        }

        return records;
    }

我的胶水作业版本0.9
我不明白是什么问题?在日志中发现异常后,作业中止

INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.RuntimeException: Not initialized)

提前谢谢。。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题