Skip to content

S3 Multipart API implementation #4224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 319 commits into from

Conversation

L-Applin
Copy link
Contributor

Adds MultipartConfiguration to the S3AsyncClientBuilder to enables Multipart operation on S3AsyncClient.

The following methods were added to S3AsyncClientBuilder (codegen):

    /**
     * Configuration for multipart operation of this client. Calling this method with a non null argument will enable
     * automatic conversion of Get, Put and Copy requests to their multipart equivalent.
     */
    default S3AsyncClientBuilder multipartConfiguration(MultipartConfiguration multipartConfiguration) {
        throw new UnsupportedOperationException();
    }

    /**
     * Configuration for multipart operation of this client. Calling this method with a non null argument will enable
     * automatic conversion of Get, Put and Copy requests to their multipart equivalent.
     */
    default S3AsyncClientBuilder multipartConfiguration(Consumer<Builder> multipartConfiguration) {
        Builder builder = MultipartConfiguration.builder();
        multipartConfiguration.accept(builder);
        return multipartConfiguration(builder.build());
    }

The following method has been added to DefaultS3AsyncClientBuilder (codegen):

    @Override
    public DefaultS3AsyncClientBuilder multipartConfiguration(MultipartConfiguration multipartConfig) {
        clientContextParams.put(MultipartConfiguration.MULTIPART_CONFIGURATION_KEY, multipartConfig);
        return this;
    }

AWS and others added 30 commits July 19, 2023 15:01
#4066)

* Fixed issue with leased connection leaks when threads executing HTTP connections with Apache HttpClient were interrupted while the connection was in progress.

* Added logic in MakeHttpRequestStage to check and abort request if interrupted

* Add test cases for UrlConnectionHttpClient

* Moved the fix to AfterTransmissionExecutionInterceptorsStage to just close the stream instaed of aborting the reqyest in MakeHttpRequestStage

* Removing test cases related to UrlConnectionHttp since adding depenency in protocol-test for urlConnectionClient cause failues since it uses default Client all the places

* Updated after Zoe's comments
…ng DNS (#3990)

* Now it's possible to configure NettyNioAsyncHttpClient in order to use a
non blocking DNS resolver.

* Add package mapping for netty-resolver-dns.

---------

Co-authored-by: Matthew Miller <[email protected]>
…pts, Quick Connects and Hours of Operations, which can be used to search for those resources within a Connect Instance.
…t-only update to refresh CLI documentation for AWS Private CA. No change to the service.
* Use WeakHashMap in IdleConenctionReaper to not prevent connection manager from getting GC'd

* Checkstyle fix
…uracy with user vector in Amazon Rekognition Face Search. Adds new APIs: AssociateFaces, CreateUser, DeleteUser, DisassociateFaces, ListUsers, SearchUsers, SearchUsersByImage. Also adds new face metadata that can be stored: user vector.
… storage virtual machine (SVM) to Active Directory after the SVM has been created.
…on for inferentia2 (ML_INF2) and Trainium1 (ML_TRN1) as available targets. With these devices, you can run your workloads at highest performance with lowest cost. inferentia2 (ML_INF2) is available in CMH and Trainium1 (ML_TRN1) is available in IAD currently
…egen UI, a new feature that enables you to generate your amplify uibuilder components and forms.
…navailable connection property for cross cluster search
* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

---------

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
Co-authored-by: Debora N. Ito <[email protected]>
…ation attempts with the new AWS WAF Fraud Control account creation fraud prevention (ACFP) managed rule group AWSManagedRulesACFPRuleSet.
…rofiles that help customers prioritize which questions to focus on first by providing a list of prioritized questions that are better aligned with their business goals and outcomes.
…NextRun field to more accurately describe the data.
L-Applin and others added 16 commits July 19, 2023 15:02
Allowing configuring the prefetchTime and staleTime on WebIdentityTokenFileCredentialsProvider and session duration
* Adding a generic method invokeOperation for abstract delegating clien… (#3996)

Adding a generic method invokeOperation for abstract delegating client to allow overriding behavior changes for all operations

* Add endpointProvider in ClientConfig and RequestOverrideConfig. Also add crossRegionBucketAccess in S3Configuration (#3978)

* Add endpointProvider in ClientConfig and RequestOverrideConfig. Also add crossRegionBucketAccess in S3Configuration

* updated review comments 1

* updated review comments 2

* Adds S3CrossRegion clients and codegen for retrieving bucket name (#4008)

* Adds S3CrossRegion clients and codegen for retrieving bucket name

* Removes codegen of bucket parameter since getValueForField() is better

* Add back overloads

* Adds cross region client logic for decorating endpoint provider (#4026)

* Adds cross region client logic for decorating endpoint provider

* Paginated methods returning publisher or iterable implement logic in … (#4046)

* Paginated methods returning publisher or iterable implement logic in interface instead of throwing unsupported exception

* Added internal annotation to class

* Wraps s3 client in cross regional client when enabled (#4080)

* Wraps s3 client in cross regional client when enabled

* Adds composer interface

* Add User Agent Api name for Cross region API calls (#4105)

* Add User Agent Api name for Cross region API calls

* Added test case for default client not to have user agent related to cross region

* S3CrossRegion Sync and Async Clients Redirect implementation (#4089)

* S3CrossRegionSyncClient Redirect implementation

* Added implementation for Async client Decorator

* Updated older Cross region test cases

* Added paramterized test

* Async Exception checged to completableException

* Updated test cases and changes the Exception handling when HeadBucket Call fails

* Handled Anna-karin's comments

* Removed async execution of HeadBucket and attached it to the completableFuture of main request

* Handled Zoe's comments

* Added test case when Redirected after the Region is cached

* Changed region constant to Region Type in Tests

* Cross region support for CRT Client (#4129)

* Cross region support for CRT Client

* removing common class

* handled review comments

* Integration test cases for Cross region Async and Sync Clients (#4128)

* Integration test cases for Cross region Sync and Sync Clients

* Renamed files as Integration test

* Handled CR comments to take care of retries thus captureInterceptor is removed

* rebased and added CRT in Integ test

* removed sout from integ

* Attach crossRegionAccessEnabled as CustomClientParams to a client Builder, also mention of Composers are renamed as decorators (#4151)

* Attach crossRegionAccessEnabled as CustomClientParams to a client Builder, also mention of Composers are renamed as decorators

* Removed from S3Configurations

* Move the optional parameter in getter

---------

Co-authored-by: Anna-Karin Salander <[email protected]>
… feature/master/s3multipart

# Conflicts:
#	core/sdk-core/src/main/java/software/amazon/awssdk/core/internal/async/ByteBuffersAsyncRequestBody.java
#	http-clients/netty-nio-client/src/test/java/software/amazon/awssdk/http/nio/netty/NettyNioAsyncHttpClientWireMockTest.java
… feature/master/s3multipart

# Conflicts:
#	services/s3/src/it/java/software/amazon/awssdk/services/s3/multipart/S3MultipartClientPutObjectIntegrationTest.java
@L-Applin L-Applin requested a review from a team as a code owner July 25, 2023 14:32
@L-Applin L-Applin requested a review from millems July 25, 2023 15:38
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 30 Code Smells

83.1% 83.1% Coverage
0.0% 0.0% Duplication

Comment on lines +1 to +6
{
"type": "feature",
"category": "AWS SDK for Java v2",
"contributor": "StephenFlavin",
"description": "Add \"unsafe\" and \"fromRemaining\" AsyncRequestBody constructors for byte arrays and ByteBuffers"
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this in this PR?

Copy link
Contributor Author

@L-Applin L-Applin Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

had to merge master branch to get ClientComposer/DelegateClient related classes

Comment on lines 230 to 239
/**
* For S3AsyncClient only. Fully qualified name of the class to be used for multipart configuration
*/
private String multipartConfigurationClass;

/**
* For S3AsyncClient only. Javadoc associated with the generated method on the S3AsyncClientBuilder interface for multipart
* configuration.
*/
private String multipartMethodJavadoc;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the pros/cons of this compared to just using customClientContextParams or something similar? https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/src/main/resources/codegen-resources/customization.config#L255-L260

Copy link
Contributor Author

@L-Applin L-Applin Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried that first, but a custom client context param also adds the method to the sync client builder, but we only want multipart for the async builder. Maybe I am unaware of a way to limit customClientContextParams to only only the sync/async interface?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems fine, since this is just an internal API. We can always refactor it later if we wanted to. E.g. make customClientContextParams have an async-only flag.

Comment on lines +130 to +131
log.info(() -> String.format("Configured partSize is %d, but using %d to prevent reaching maximum number of parts "
+ "allowed", partSizeInBytes, optimalPartSize));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Info is a pretty loud log level. Is this necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only happens when total size / part size > 10,000 which will result in not using the configured part size set by the user (to not hit the 10,000 part limit imposed by s3), I thought being a little loud about it was a good idea.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have a flag that the user opts-in to allow this to happen, so that we don't have to warn when it does? (They've consented to it, after all!)

Logging above debug for anything that isn't an error scares me, since we don't know what the user's throughput is like, and we frequently get people upset about us filling their logs.

Copy link
Contributor Author

@L-Applin L-Applin Jul 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I will lower to debug then, dont want customers to be mad at us 😅

Comment on lines +124 to +125
log.info(() -> String.format("Configured partSize is %d, but using %d to prevent reaching maximum number of parts "
+ "allowed", partSizeInBytes, optimalPartSize));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see other comment

L-Applin added 4 commits July 25, 2023 17:49
- Move multipartEnabled to S3AsyncClientBuilder
- thresholdInByte javadoc, only copy and put
- document edge case related to minimumPartSizeInBytes and thresholdInBytes
- remove MultipartDownloadType
- removed MultipartConfiguration.create()
- getObject not supported exception message improvement
- use s3Multipart as user-agent version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.