jvm 使用IOUtils.copyLarge时出现Java堆空间不足的错误,我如何解决这个问题?

unftdfkk  于 4个月前  发布在  Java
关注(0)|答案(1)|浏览(61)

我有一个应用程序(Springboot API)运行un kubernetes,我有一个cpu:200mmemory:8Gi的示例。我的jvm参数是:_JAVA_OPTIONS: -Xmx7G -Xms7G我的目标是将一个或多个大小在30Mb1Gb之间的文件从一个s3 bucket复制到另一个带有预签名url的s3 bucket。
我得到java.lang.OutOfMemoryError: Java heap space错误时,我调用我的API超过2次在1GB的文件目录。
下面是我的代码:

public ResponseEntity<String> transferData(Config config) throws IOException {

    var inputUrl = config.getInputdUrl();
    var outputUrl = config.getOutputdUrl();

    var uploadConnection = uploader.getUploadConnection(outputUrl);

    try (var inputStream = downloader.getDownloadConnection(inputUrl); 
         var outputStream = uploadConnection.getOutputStream()) {

        IOUtils.copyLarge(inputStream, outputStream);

        var resp = uploadConnection.getResponseCode();

        return ResponseEntity
                .status(HttpStatus.OK)
                .body("Transfer done with response:" + resp);
    } catch (IOException ioe) {
        return ResponseEntity
                .status(HttpStatus.INTERNAL_SERVER_ERROR)
                .body("An  error occurred while transferring data : " + ioe);
    }

}

字符串
getUploadConnection返回HttpURLConnection

var connection = (HttpURLConnection) new URL(presignedUrl).openConnection();
        connection.setDoOutput(true);
        connection.setRequestMethod(PUT.toString());
        return connection;


getDownloadConnection返回HttpURLConnection inputStream。

HttpURLConnection connection = (HttpURLConnection) new URL(presignedUrl).openConnection();
        connection.setRequestMethod(GET.toString());
        connection.setDoOutput(true);
return connection.getInputStream();


关于我的代码和配置,我认为对内存的影响不太重要,因为我流文件,并没有得到它完全在内存中。
我错过了什么?
编辑1:我添加了setChunkedStreamingMode,所以现在我的getUploadConnection返回:

var connection = (HttpURLConnection) new URL(url).openConnection();
        connection.setDoOutput(true);
        connection.setDoInput(true);
        connection.setChunkedStreamingMode(-1);
        connection.setRequestMethod(PUT.toString());
        return connection;


但是我从我的s3 bucket得到一个411错误

xn1cxnb4

xn1cxnb41#

在我的输出连接中设置了setFixedLengthStreamingMode,我能够传输更多的文件,而不会出现Out of memory error java heap space问题。
我的输出连接:

var connection = (HttpURLConnection) new URL(presignedUrl).openConnection();
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setRequestMethod("PUT");
connection.setFixedLengthStreamingMode(downloadConnection.getContentLengthLong());

字符串

相关问题