Skip to content

Commit d77cb8b

Browse files
L-ApplinL-Applin
authored
Add all streaming operation to s3 checksums tests (#5998)
* add test configs for all streaming type * buildspec fix * buildspec fix * check if tests hand without multipart * add http-auth-aws-crt for crt sigv4a tests * fix request body methods with 'remaining' * test 2MiB instead of 200MiB * fix for 'remaining' test type * fix for fromString, reduce logs * fix for string content length * fix for string content length * test 200MiB large size * use 80MiB for large files * Fix for TM_JAVA * Fix for TM_JAVA * Fix for TM_JAVA * need more memory * more fix for TM * some more dl stuff * logs * fix multi * fix multi bucket * fix multi bucket * part index starts at 1 * complete multipart fix * complete multipart fix * complete multipart fix * complete multipart fix * fix for parts * adding logs * more logs, no crc * improved message for failed assertion * fix crc check for multi and cleanup * fix memory issues * add s3 checksum validation * avoid using AwsChunkedV4aPayloadSigner * avoid using AwsChunkedV4aPayloadSigner * DownloadConfig toString * fix Unmanaged * fix for crc32 * rename tp regression testing seperate data-place, upload and download test avoid running useless tests cases * fix logs * added README, verify crc32 of uploaded object with an additional get object, precalculated crc32 when possible, rename S3ClientFlavor fields, * fix npe * junit parallel execution * multipart with no-length * fix pom * remove bucket accelerate config * remove bucket accelerate config * remove accelerate config from ControlPlaneOperationRegressionTesting remove bundle from dependencies to reduce build time * bring back bundle * bring back bundle-sdk * s3 as test * fix Junit failure to discover tests, due to class not found * remove bundle * fix concurrent bucket creation logs. Fix log4j2 logging. * Add s3-regressions to GitHub ci * use info level debuf logs * remove ci * Lower large file and split codebuilds to reduce build time * fix typo * fix mpu setup object size * include regression tests in codebuild ci jobs * remove BodyType which are duplicated of other body which simply don't copy the input content (to reduce build time) * update readme * parallel execution of test cases * use same CodeBuild for all 3 CI check * edit ci descriptions * split upload tests per client type: sync, async, crt, multipart-java. Lower large non-file content from 80mib to 60mib to attempt lower build time. * fix MethodSource method name * fix for MPU tests failing * add timeout. Add sync http client with higher conn pool manually and close s3 client * remove unused forced path style from upload configs * set timeout to 30 sec for uploads * try to reduce build time * add log to error * fix async upload without length * fix sync upload when missing checksum from response * fix missing checksum from response * fix missing checksum from response * increase connection acquisition timeout to fix "failed to acquire connection" error in sync uploads * skip RequestBody.fromRemainingByteBuffer with RequestChecksumCalculation.WHEN_SUPPORTED because of connection acquire timeout, added TOT for investigation * test cleanup catch exception * use bucket type in upload tests * All bucket still cause connection timeouts
1 parent 14ac5a6 commit d77cb8b

24 files changed

+2904
-1070
lines changed

‎.github/workflows/codebuild-ci.yml

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,3 +167,106 @@ jobs:
167167
uses: aws-actions/aws-codebuild-run-build@v1
168168
with:
169169
project-name: aws-sdk-java-v2-migration-test
170+
171+
s3-regression-tests-download:
172+
if: github.repository == 'aws/aws-sdk-java-v2'
173+
runs-on: ubuntu-latest
174+
env:
175+
REGRESSION_TEST: DownloadStreamingRegressionTesting
176+
steps:
177+
- name: Configure AWS Credentials
178+
uses: aws-actions/configure-aws-credentials@v4
179+
with:
180+
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
181+
aws-region: us-west-2
182+
role-duration-seconds: 7200
183+
- name: Run s3 regression tests for downloads
184+
uses: aws-actions/aws-codebuild-run-build@v1
185+
with:
186+
project-name: aws-sdk-java-v2-s3-regression-tests
187+
env-vars-for-codebuild: REGRESSION_TEST
188+
s3-regression-tests-control-plane:
189+
if: github.repository == 'aws/aws-sdk-java-v2'
190+
runs-on: ubuntu-latest
191+
env:
192+
REGRESSION_TEST: ControlPlaneOperationRegressionTesting
193+
steps:
194+
- name: Configure AWS Credentials
195+
uses: aws-actions/configure-aws-credentials@v4
196+
with:
197+
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
198+
aws-region: us-west-2
199+
role-duration-seconds: 7200
200+
- name: Run s3 regression tests for control plane
201+
uses: aws-actions/aws-codebuild-run-build@v1
202+
with:
203+
project-name: aws-sdk-java-v2-s3-regression-tests
204+
env-vars-for-codebuild: REGRESSION_TEST
205+
s3-regression-tests-upload-sync:
206+
if: github.repository == 'aws/aws-sdk-java-v2'
207+
runs-on: ubuntu-latest
208+
env:
209+
REGRESSION_TEST: UploadSyncRegressionTesting
210+
steps:
211+
- name: Configure AWS Credentials
212+
uses: aws-actions/configure-aws-credentials@v4
213+
with:
214+
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
215+
aws-region: us-west-2
216+
role-duration-seconds: 7200
217+
- name: Run s3 regression tests for uploads
218+
uses: aws-actions/aws-codebuild-run-build@v1
219+
with:
220+
project-name: aws-sdk-java-v2-s3-regression-tests
221+
env-vars-for-codebuild: REGRESSION_TEST
222+
s3-regression-tests-upload-async:
223+
if: github.repository == 'aws/aws-sdk-java-v2'
224+
runs-on: ubuntu-latest
225+
env:
226+
REGRESSION_TEST: UploadAsyncRegressionTesting
227+
steps:
228+
- name: Configure AWS Credentials
229+
uses: aws-actions/configure-aws-credentials@v4
230+
with:
231+
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
232+
aws-region: us-west-2
233+
role-duration-seconds: 7200
234+
- name: Run s3 regression tests for uploads
235+
uses: aws-actions/aws-codebuild-run-build@v1
236+
with:
237+
project-name: aws-sdk-java-v2-s3-regression-tests
238+
env-vars-for-codebuild: REGRESSION_TEST
239+
s3-regression-tests-upload-crt:
240+
if: github.repository == 'aws/aws-sdk-java-v2'
241+
runs-on: ubuntu-latest
242+
env:
243+
REGRESSION_TEST: UploadCrtRegressionTesting
244+
steps:
245+
- name: Configure AWS Credentials
246+
uses: aws-actions/configure-aws-credentials@v4
247+
with:
248+
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
249+
aws-region: us-west-2
250+
role-duration-seconds: 7200
251+
- name: Run s3 regression tests for uploads
252+
uses: aws-actions/aws-codebuild-run-build@v1
253+
with:
254+
project-name: aws-sdk-java-v2-s3-regression-tests
255+
env-vars-for-codebuild: REGRESSION_TEST
256+
s3-regression-tests-upload-multi:
257+
if: github.repository == 'aws/aws-sdk-java-v2'
258+
runs-on: ubuntu-latest
259+
env:
260+
REGRESSION_TEST: UploadTransferManagerRegressionTesting
261+
steps:
262+
- name: Configure AWS Credentials
263+
uses: aws-actions/configure-aws-credentials@v4
264+
with:
265+
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
266+
aws-region: us-west-2
267+
role-duration-seconds: 7200
268+
- name: Run s3 regression tests for uploads
269+
uses: aws-actions/aws-codebuild-run-build@v1
270+
with:
271+
project-name: aws-sdk-java-v2-s3-regression-tests
272+
env-vars-for-codebuild: REGRESSION_TEST

‎buildspecs/resources/ci.cloudformation.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ Resources:
5858
- !Sub arn:aws:codebuild:${ AWS::Region }:${ AWS::AccountId }:project/aws-sdk-java-v2-sonar
5959
- !Sub arn:aws:codebuild:${ AWS::Region }:${ AWS::AccountId }:project/aws-sdk-java-v2-endpoints-test
6060
- !Sub arn:aws:codebuild:${ AWS::Region }:${ AWS::AccountId }:project/aws-sdk-java-v2-migration-test
61+
- !Sub arn:aws:codebuild:${ AWS::Region }:${ AWS::AccountId }:project/aws-sdk-java-v2-s3-regression-tests
6162
- Effect: Allow
6263
Action:
6364
- logs:GetLogEvents
@@ -71,6 +72,7 @@ Resources:
7172
- !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/aws-sdk-java-v2-sonar:*
7273
- !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/aws-sdk-java-v2-endpoints-test:*
7374
- !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/aws-sdk-java-v2-migration-test:*
75+
- !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/aws-sdk-java-v2-s3-regression-tests:*
7476

7577
GithubOidc:
7678
Type: AWS::IAM::OIDCProvider

‎buildspecs/s3-regression-tests.yml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
version: 0.2
2+
3+
phases:
4+
build:
5+
commands:
6+
- mvn clean install -P s3-regression-tests -pl :s3-tests -am -T1C -Dregression.test=$REGRESSION_TEST $MAVEN_OPTIONS
7+
- echo $MAVEN_OPTIONS
8+
finally:
9+
- mkdir -p codebuild-test-reports
10+
- find ./ -name 'TEST-*.xml' -type f -exec cp {} codebuild-test-reports/ \;
11+
reports:
12+
ChecksumsTests:
13+
files:
14+
- 'codebuild-test-reports/**/*'

‎core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncRequestBody.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -421,8 +421,8 @@ static AsyncRequestBody fromInputStream(Consumer<AsyncRequestBodyFromInputStream
421421
* S3AsyncClient s3 = S3AsyncClient.create(); // Use one client for your whole application!
422422
*
423423
* byte[] dataToSend = "Hello".getBytes(StandardCharsets.UTF_8);
424-
* InputStream streamToSend = new ByteArrayInputStream();
425-
* long streamToSendLength = dataToSend.length();
424+
* InputStream streamToSend = new ByteArrayInputStream(dataToSend);
425+
* long streamToSendLength = dataToSend.length;
426426
*
427427
* // Start the operation
428428
* BlockingInputStreamAsyncRequestBody body =

‎pom.xml

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -893,6 +893,55 @@
893893
</build>
894894
</profile>
895895

896+
<profile>
897+
<id>s3-regression-tests</id>
898+
<activation>
899+
<property>
900+
<name>doRelease</name>
901+
</property>
902+
</activation>
903+
904+
<properties>
905+
<checkstyle.skip>true</checkstyle.skip>
906+
<spotbugs.skip>true</spotbugs.skip>
907+
<skip.unit.tests>true</skip.unit.tests>
908+
<mdep.analyze.skip>true</mdep.analyze.skip>
909+
<japicmp.skip>true</japicmp.skip>
910+
<maven.javadoc.skip>true</maven.javadoc.skip>
911+
</properties>
912+
<build>
913+
<plugins>
914+
<plugin>
915+
<groupId>org.apache.maven.plugins</groupId>
916+
<artifactId>maven-failsafe-plugin</artifactId>
917+
<version>${maven-failsafe-plugin.version}</version>
918+
<executions>
919+
<execution>
920+
<phase>integration-test</phase>
921+
<goals>
922+
<goal>integration-test</goal>
923+
<goal>verify</goal>
924+
</goals>
925+
<configuration>
926+
<!-- this test needs a lot of memory, we need to increase java heap size or we get OOM in codebuild -->
927+
<argLine>-Xmx12g -Xms4g</argLine>
928+
<includes>
929+
<include>**/*${regression.test}.java</include>
930+
</includes>
931+
<trimStackTrace>false</trimStackTrace>
932+
<classesDirectory>${project.build.outputDirectory}</classesDirectory>
933+
<testClassesDirectory>${project.build.testOutputDirectory}</testClassesDirectory>
934+
<parallel>methods</parallel>
935+
<threadCount>8</threadCount>
936+
<perCoreThreadCount>false</perCoreThreadCount>
937+
</configuration>
938+
</execution>
939+
</executions>
940+
</plugin>
941+
</plugins>
942+
</build>
943+
</profile>
944+
896945
<profile>
897946
<id>endpoint-tests</id>
898947
<activation>

‎test/s3-tests/README.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# SDK Regression Tests for Amazon S3
2+
3+
## Description
4+
This module contains SDK regression tests for Amazon S3 streaming operations with various SDK configurations.
5+
6+
7+
## How to run
8+
9+
### Credentials
10+
11+
The tests require valid AWS credentials to be available in the default credential file under the `aws-test-account` profile.
12+
13+
### Run the tests
14+
15+
- Run from your IDE
16+
17+
- Run from maven command. Include the class you want to run with the `regression.test` property
18+
19+
```
20+
mvn clean install -P s3-regression-tests -pl :s3-tests -am -T1C -Dregression.test=DownloadStreamingRegressionTesting
21+
```
22+
23+
## Adding New Tests
24+
25+
- The tests are built using [JUnit 5](https://junit.org/junit5/). Make sure you are using the correct APIs and mixing of
26+
Junit 4 and Junit 5 APIs on the same test can have unexpected results.
27+
28+
- All tests should have the suffix of `RegressionTesting`, eg: `DownloadStreamingRegressionTesting`
29+
30+
31+

‎test/s3-tests/pom.xml

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@
8181
<groupId>software.amazon.awssdk</groupId>
8282
<artifactId>s3</artifactId>
8383
<version>${awsjavasdk.version}</version>
84+
<scope>test</scope>
8485
</dependency>
8586
<dependency>
8687
<groupId>software.amazon.awssdk</groupId>
@@ -115,11 +116,6 @@
115116
<artifactId>junit-jupiter</artifactId>
116117
<scope>test</scope>
117118
</dependency>
118-
<dependency>
119-
<groupId>org.junit.vintage</groupId>
120-
<artifactId>junit-vintage-engine</artifactId>
121-
<scope>test</scope>
122-
</dependency>
123119
<dependency>
124120
<groupId>org.apache.logging.log4j</groupId>
125121
<artifactId>log4j-api</artifactId>
@@ -152,6 +148,18 @@
152148
<version>${awsjavasdk.version}</version>
153149
<scope>test</scope>
154150
</dependency>
151+
<dependency>
152+
<groupId>software.amazon.awssdk</groupId>
153+
<artifactId>http-auth-aws-crt</artifactId>
154+
<version>${awsjavasdk.version}</version>
155+
<scope>test</scope>
156+
</dependency>
157+
<dependency>
158+
<groupId>software.amazon.awssdk</groupId>
159+
<artifactId>s3-transfer-manager</artifactId>
160+
<version>${awsjavasdk.version}</version>
161+
<scope>test</scope>
162+
</dependency>
155163
<dependency>
156164
<groupId>org.eclipse.jetty</groupId>
157165
<artifactId>jetty-servlet</artifactId>
@@ -162,6 +170,18 @@
162170
<artifactId>jetty-server</artifactId>
163171
<scope>test</scope>
164172
</dependency>
173+
<dependency>
174+
<groupId>software.amazon.awssdk</groupId>
175+
<artifactId>test-utils</artifactId>
176+
<version>${project.version}</version>
177+
<scope>test</scope>
178+
</dependency>
179+
<dependency>
180+
<groupId>software.amazon.awssdk</groupId>
181+
<artifactId>apache-client</artifactId>
182+
<version>${project.version}</version>
183+
<scope>test</scope>
184+
</dependency>
165185
</dependencies>
166186

167187
<build>

0 commit comments

Comments
 (0)