OutOfMemoryError on S3 put of large file when max heap is limited

[lread]

Describe the bug

Starting with v2.30.9, when I attempt to s3 put a large file, and my maximum heap is limited, I suffer an OutOfMemoryException. I have not witnessed this with previous versions.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Large file puts successfully when max heap is relatively small.

Current Behavior

When heap is small, put of large file will result in OutOfMemoryException.
When file is large (2gb) will always result in OutOfMemoryException.

See reproduction below.

Reproduction Steps

I'm a clojure developer so forgive my rusty Java skills.
And apologies if I'm doing something goofy and this "just happened to work" in previous versions.

Summary

For versions 2.30.9 to 2.30.11 (latest as of this writing)

• A put of a 1gb file will pass if the heap is big enough but will fail if the heap is smaller (tried 200mb and 1gb) with an OutOfMemoryError: Java heap space
• A put of a 2gb file seems to always fail regardless of heap size with an OutOfMemoryError: Requested array size exceeds VM limit
• I don't see problems with gets.

For versions < 2.30.9

• All seems good

Test Source Code

Given src/main/java/S3Test.java:

import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.ResponseInputStream;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.core.sync.ResponseTransformer;
import software.amazon.awssdk.core.util.VersionInfo;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.net.URI;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.Map;

public class S3Test {

    public static void main(String[] args) throws IOException {
        System.out.printf("aws sdk version: %s%n", VersionInfo.SDK_VERSION);
        System.out.printf("max heap %dmb%n", Runtime.getRuntime().maxMemory() / 1024 / 1024);

        Map<String, String> opts = parseArgs(args);
        int fileMb = Integer.parseInt(opts.getOrDefault("file-mb", "512"));
        String bucket = "foobucket";
        String op = opts.getOrDefault("op", "put");

        S3Client s3 = createS3Client("http://127.0.0.1:9000", "foouser", "foosecret");

        try {
            // Create test bucket if it does not exist
            try {
                s3.headBucket(HeadBucketRequest.builder().bucket(bucket).build());
            } catch (S3Exception e) {
                if (e.statusCode() == 404) {
                    s3.createBucket(CreateBucketRequest.builder().bucket(bucket).build());
                } else {
                    throw e;
                }
            }

            switch (op) {
                case "put":
                    System.out.printf("Putting %dmb file%n", fileMb);
                    createTestFile("testfile", fileMb);
                    s3.putObject(PutObjectRequest.builder()
                                    .bucket(bucket)
                                    .key("testfile")
                                    .build(),
                            RequestBody.fromFile(new File("testfile")));
                    break;

                case "get":
                    System.out.println("Getting file");
                    File destFile = new File("testfile.down");
                    if (destFile.exists()) {
                        destFile.delete();
                    }
                    s3.getObject(GetObjectRequest.builder()
                                    .bucket(bucket)
                                    .key("testfile")
                                    .build(),
                            ResponseTransformer.toFile(Paths.get("testfile.down")));
                    System.out.printf("Downloaded file: %.2fmb%n", destFile.length() / 1024.0 / 1024.0);
                    break;

                default:
                    throw new IllegalArgumentException("Invalid operation: " + op);
            }
        } finally {
            s3.close();
        }
    }

    private static Map<String, String> parseArgs(String[] args) {
        Map<String, String> opts = new HashMap<>();
        for (String arg : args) {
            String[] parts = arg.split("=", 2);
            if (parts.length == 2) {
                opts.put(parts[0], parts[1]);
            }
        }
        return opts;
    }

    private static void createTestFile(String filePath, int sizeMb) throws IOException {
        long sizeBytes = (long) sizeMb * 1024 * 1024;
        try (RandomAccessFile file = new RandomAccessFile(filePath, "rw")) {
            file.setLength(sizeBytes);
        }
    }

    private static S3Client createS3Client(String endpoint, String accessKey, String secretKey) {
        return S3Client.builder()
                .region(Region.AWS_GLOBAL)
                .endpointOverride(URI.create(endpoint))
                .credentialsProvider(StaticCredentialsProvider.create(
                        AwsBasicCredentials.create(accessKey, secretKey)))
                .build();
    }
}

And ./pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>s3-test</artifactId>
  <version>1.0-SNAPSHOT</version>
  <properties>
    <maven.compiler.release>11</maven.compiler.release>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <sdk.version>2.30.8</sdk.version>
    <max.heap>4096m</max.heap>
    <test.op>put</test.op>
    <file.mb>1</file.mb>
  </properties>
  <dependencies>
    <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>s3</artifactId>
      <version>${sdk.version}</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-simple</artifactId>
      <version>2.0.16</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.5.0</version>
        <configuration>
          <executable>java</executable>
          <arguments >
            <argument>-classpath</argument>
            <classpath/>
            <argument>-Xmx${max.heap}</argument>
            <argument>S3Test</argument>
            <argument>op=${test.op}</argument>
            <argument>file-mb=${file.mb}</argument>
          </arguments>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Environment Setup

I went with a local minio server as my S3 target launched like so:

docker run -p 9000:9000 -p 9001:9001 --name minio \
  -e "MINIO_ROOT_USER=foouser" \
  -e "MINIO_ROOT_PASSWORD=foosecret" \
  minio/minio server /data --console-address ":9001"

Java version is:

$ java --version
openjdk 23.0.1 2024-10-15
OpenJDK Runtime Environment Temurin-23.0.1+11 (build 23.0.1+11)
OpenJDK 64-Bit Server VM Temurin-23.0.1+11 (build 23.0.1+11, mixed mode, sharing)

Sanity Test

(I'll leave out the copious mvn INFO output)

Let's put a 1mb file with a max heap of 4gb using v2.30.11

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=1 -Dmax.heap=4096m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 4096mb
Putting 1mb file

Ok now let's fetch that same file

$ mvn clean compile exec:exec -Dtest.op=get -Dmax.heap=4096m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 4096mb
Getting file
Downloaded file: 1.00mb

All looks fine

Reproduction

[fail] v2.30.11 put 2g file 200mb heap

Ok, now let's try to put a 2g file with a max heap of 200mb using v2.30.11

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=200m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 200mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

[pass] v2.30.8 put 2g file 200mb heap

Let's try the same again but with v2.30.8

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=200m -Dsdk.version=2.30.8
aws sdk version: 2.30.8
max heap 200mb
Putting 2048mb file

It works.

[pass] v2.30.8 get 2gb file 200mb heap

And with the same limited heap, lets get the file with v2.30.8

$ mvn clean compile exec:exec -Dtest.op=get -Dmax.heap=200m -Dsdk.version=2.30.8
aws sdk version: 2.30.8
max heap 200mb
Getting file
Downloaded file: 2048.00mb

works fine.

[pass] v2.30.11 get 2gb file 200mb heap

We don't seem to have issues with get on sdk v2.30.11

$ mvn clean compile exec:exec -Dtest.op=get -Dmax.heap=200m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 200mb
Getting file
Downloaded file: 2048.00mb

all is good.

[fail] v2.30.10 put 2gb file 200mb heap

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=200m -Dsdk.version=2.30.10
aws sdk version: 2.30.10
max heap 200mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

[fail] v2.30.9 put 2gb file 200mb heap

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=200m -Dsdk.version=2.30.9
aws sdk version: 2.30.9
max heap 200mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

[fail] v2.30.11 put 2gb file 2gb heap

Maybe 200mb is silly small, let's try a 2gb heap.

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=2048m -Dsdk.version=2.30.9
aws sdk version: 2.30.11
max heap 2048mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

[fail] v2.30.11 put 2gb file 8gb heap

Let's give ourselves some elbow room.

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=2048 -Dmax.heap=8192m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 8192mb
Putting 2048mb file
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

Huh! Notice this time though: Requested array size exceeds VM limit, a different issue maybe.

[pass] v2.30.11 put 1gb file 8gb heap

Let's try a smaller file then:

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=1024 -Dmax.heap=8192m -Dsdk.version=2.30.11 
aws sdk version: 2.30.11
max heap 8192mb
Putting 1024mb file

Ok that worked.

[fail] v2.30.11 put 1gb file 1gb heap

But if we reduce the heap to 1gb, we fail.

$ mvn clean compile exec:exec -Dtest.op=put -Dfile.mb=1024 -Dmax.heap=1024m -Dsdk.version=2.30.11
aws sdk version: 2.30.11
max heap 1024mb
Putting 1024mb file
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:277)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)
	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405)
	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66)
	at software.amazon.awssdk.http.auth.aws.internal.signer.io.ChecksumInputStream.read(ChecksumInputStream.java:52)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:136)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.getChunk(ChunkedEncodedInputStream.java:112)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.currentChunk(ChunkedEncodedInputStream.java:95)
	at software.amazon.awssdk.http.auth.aws.internal.signer.chunkedencoding.ChunkedEncodedInputStream.read(ChunkedEncodedInputStream.java:90)
	at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
	at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:157)
	at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:121)
	at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
	at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:252)
	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:103)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:229)
	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:226)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:355)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:253)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:903)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:863)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:454)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)

Possible Solution

In the 2.30.9 changelog I see:

AWS SDK for Java v2

Features
Buffer input data from ContentStreamProvider in cases where content length is known.

Feels like the new behaviour I am witnessing might be related to this change.

Additional Information/Context

I've been using the aws sdk for a long while without any issues.
Thanks for all the hard work!

AWS Java SDK version used

2.30.8,2.30.9,2.30.10,2.30.11

JDK version used

openjdk 23.0.1 2024-10-15

Operating System and version

Linux pop-os 6.9.3-76060903-generic #202405300957173698068022.04~44ea8a9 SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux

[scarytom]

Just to add some context here: we have been experiencing multiple out-of-memory crashes of our systems since updating to the latest AWS SDK version, and this bug seems to be the cause.

Rolling back to v2.30.8 has got things working again.

[bhoradc]

Hello @lread,

Thank you for providing detailed information about the issue you encountered and the steps you took to reproduce it.

I tested below cases to confirm the OutOfMemoryError pre and post ContentStreamProvider changes released in v2.30.9.

With maximum heap size of 4096MB, the results were as follows for a PUT S3 operation.

Java SDK version 2.30.8 and 2.30.9:

• With a 512MB file, the operation succeeded.

Java SDK version 2.30.9:

• With a 1024MB file, the operation failed with a "Java heap space" OutOfMemoryError.

Java SDK version 2.30.8:

• With a 1024MB file, the operation succeeded.

While the above mentioned buffering changes on v2.30.9 seems related to this issue, we'll investigate this further and provide an update.

Regards,
Chaitanya

[lread]

Thanks for sharing @scarytom!

Thanks for the follow up @bhoradc! You might also want to try with a 2gb file.
For versions >= 2.30.9 it seems to always throw regardless of heap size with the slightly different OutOfMemoryError: Requested array size exceeds VM limit

[lread]

Thanks @zoewangg! One idea to consider (or discard, I don't know if this would fit in with your testing strategies): adding some tests that exercise put big file/under small heap scenario (and maybe put big >= 2gb file under any size heap).

[zoewangg]

Hi @lread, thank you for your suggestion! Will do. We will also explore ways to add more testing coverage for memory usage in our tests. BTW, thank you for providing such thorough analysis and self-contained repro code. It really helped us pin point the issue quickly!

The fix will be included in the next release.

[zoewangg]

Hi all, the fix has been released in 2.30.13. Please try with it and let us know if you continue to see the issue.