-
Notifications
You must be signed in to change notification settings - Fork 916
S3 Multipart API implementation #4224
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
#4066) * Fixed issue with leased connection leaks when threads executing HTTP connections with Apache HttpClient were interrupted while the connection was in progress. * Added logic in MakeHttpRequestStage to check and abort request if interrupted * Add test cases for UrlConnectionHttpClient * Moved the fix to AfterTransmissionExecutionInterceptorsStage to just close the stream instaed of aborting the reqyest in MakeHttpRequestStage * Removing test cases related to UrlConnectionHttp since adding depenency in protocol-test for urlConnectionClient cause failues since it uses default Client all the places * Updated after Zoe's comments
…ng DNS (#3990) * Now it's possible to configure NettyNioAsyncHttpClient in order to use a non blocking DNS resolver. * Add package mapping for netty-resolver-dns. --------- Co-authored-by: Matthew Miller <[email protected]>
…pts, Quick Connects and Hours of Operations, which can be used to search for those resources within a Connect Instance.
…t-only update to refresh CLI documentation for AWS Private CA. No change to the service.
* Use WeakHashMap in IdleConenctionReaper to not prevent connection manager from getting GC'd * Checkstyle fix
…uracy with user vector in Amazon Rekognition Face Search. Adds new APIs: AssociateFaces, CreateUser, DeleteUser, DisassociateFaces, ListUsers, SearchUsers, SearchUsersByImage. Also adds new face metadata that can be stored: user vector.
… storage virtual machine (SVM) to Active Directory after the SVM has been created.
…on for inferentia2 (ML_INF2) and Trainium1 (ML_TRN1) as available targets. With these devices, you can run your workloads at highest performance with lowest cost. inferentia2 (ML_INF2) is available in CMH and Trainium1 (ML_TRN1) is available in IAD currently
…egen UI, a new feature that enables you to generate your amplify uibuilder components and forms.
…navailable connection property for cross cluster search
* docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] --------- Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com> Co-authored-by: Debora N. Ito <[email protected]>
…or CloudTrail Lake event data stores.
…ation attempts with the new AWS WAF Fraud Control account creation fraud prevention (ACFP) managed rule group AWSManagedRulesACFPRuleSet.
…rofiles that help customers prioritize which questions to focus on first by providing a list of prioritized questions that are better aligned with their business goals and outcomes.
…tificates API operation.
…NextRun field to more accurately describe the data.
Allowing configuring the prefetchTime and staleTime on WebIdentityTokenFileCredentialsProvider and session duration
* Adding a generic method invokeOperation for abstract delegating clien… (#3996) Adding a generic method invokeOperation for abstract delegating client to allow overriding behavior changes for all operations * Add endpointProvider in ClientConfig and RequestOverrideConfig. Also add crossRegionBucketAccess in S3Configuration (#3978) * Add endpointProvider in ClientConfig and RequestOverrideConfig. Also add crossRegionBucketAccess in S3Configuration * updated review comments 1 * updated review comments 2 * Adds S3CrossRegion clients and codegen for retrieving bucket name (#4008) * Adds S3CrossRegion clients and codegen for retrieving bucket name * Removes codegen of bucket parameter since getValueForField() is better * Add back overloads * Adds cross region client logic for decorating endpoint provider (#4026) * Adds cross region client logic for decorating endpoint provider * Paginated methods returning publisher or iterable implement logic in … (#4046) * Paginated methods returning publisher or iterable implement logic in interface instead of throwing unsupported exception * Added internal annotation to class * Wraps s3 client in cross regional client when enabled (#4080) * Wraps s3 client in cross regional client when enabled * Adds composer interface * Add User Agent Api name for Cross region API calls (#4105) * Add User Agent Api name for Cross region API calls * Added test case for default client not to have user agent related to cross region * S3CrossRegion Sync and Async Clients Redirect implementation (#4089) * S3CrossRegionSyncClient Redirect implementation * Added implementation for Async client Decorator * Updated older Cross region test cases * Added paramterized test * Async Exception checged to completableException * Updated test cases and changes the Exception handling when HeadBucket Call fails * Handled Anna-karin's comments * Removed async execution of HeadBucket and attached it to the completableFuture of main request * Handled Zoe's comments * Added test case when Redirected after the Region is cached * Changed region constant to Region Type in Tests * Cross region support for CRT Client (#4129) * Cross region support for CRT Client * removing common class * handled review comments * Integration test cases for Cross region Async and Sync Clients (#4128) * Integration test cases for Cross region Sync and Sync Clients * Renamed files as Integration test * Handled CR comments to take care of retries thus captureInterceptor is removed * rebased and added CRT in Integ test * removed sout from integ * Attach crossRegionAccessEnabled as CustomClientParams to a client Builder, also mention of Composers are renamed as decorators (#4151) * Attach crossRegionAccessEnabled as CustomClientParams to a client Builder, also mention of Composers are renamed as decorators * Removed from S3Configurations * Move the optional parameter in getter --------- Co-authored-by: Anna-Karin Salander <[email protected]>
… feature/master/s3multipart # Conflicts: # core/sdk-core/src/main/java/software/amazon/awssdk/core/internal/async/ByteBuffersAsyncRequestBody.java # http-clients/netty-nio-client/src/test/java/software/amazon/awssdk/http/nio/netty/NettyNioAsyncHttpClientWireMockTest.java
… feature/master/s3multipart # Conflicts: # services/s3/src/it/java/software/amazon/awssdk/services/s3/multipart/S3MultipartClientPutObjectIntegrationTest.java
Kudos, SonarCloud Quality Gate passed! |
{ | ||
"type": "feature", | ||
"category": "AWS SDK for Java v2", | ||
"contributor": "StephenFlavin", | ||
"description": "Add \"unsafe\" and \"fromRemaining\" AsyncRequestBody constructors for byte arrays and ByteBuffers" | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this in this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
had to merge master branch to get ClientComposer/DelegateClient related classes
/** | ||
* For S3AsyncClient only. Fully qualified name of the class to be used for multipart configuration | ||
*/ | ||
private String multipartConfigurationClass; | ||
|
||
/** | ||
* For S3AsyncClient only. Javadoc associated with the generated method on the S3AsyncClientBuilder interface for multipart | ||
* configuration. | ||
*/ | ||
private String multipartMethodJavadoc; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are the pros/cons of this compared to just using customClientContextParams or something similar? https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/src/main/resources/codegen-resources/customization.config#L255-L260
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried that first, but a custom client context param also adds the method to the sync client builder, but we only want multipart for the async builder. Maybe I am unaware of a way to limit customClientContextParams to only only the sync/async interface?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems fine, since this is just an internal API. We can always refactor it later if we wanted to. E.g. make customClientContextParams have an async-only flag.
...n/src/main/java/software/amazon/awssdk/codegen/poet/builder/AsyncClientBuilderInterface.java
Outdated
Show resolved
Hide resolved
log.info(() -> String.format("Configured partSize is %d, but using %d to prevent reaching maximum number of parts " | ||
+ "allowed", partSizeInBytes, optimalPartSize)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Info is a pretty loud log level. Is this necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is only happens when total size / part size > 10,000
which will result in not using the configured part size set by the user (to not hit the 10,000 part limit imposed by s3), I thought being a little loud about it was a good idea.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we have a flag that the user opts-in to allow this to happen, so that we don't have to warn when it does? (They've consented to it, after all!)
Logging above debug for anything that isn't an error scares me, since we don't know what the user's throughput is like, and we frequently get people upset about us filling their logs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, I will lower to debug then, dont want customers to be mad at us 😅
log.info(() -> String.format("Configured partSize is %d, but using %d to prevent reaching maximum number of parts " | ||
+ "allowed", partSizeInBytes, optimalPartSize)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see other comment
...ces/s3/src/main/java/software/amazon/awssdk/services/s3/multipart/MultipartDownloadType.java
Outdated
Show resolved
Hide resolved
- Move multipartEnabled to S3AsyncClientBuilder - thresholdInByte javadoc, only copy and put - document edge case related to minimumPartSizeInBytes and thresholdInBytes - remove MultipartDownloadType - removed MultipartConfiguration.create() - getObject not supported exception message improvement - use s3Multipart as user-agent version
Adds
MultipartConfiguration
to theS3AsyncClientBuilder
to enables Multipart operation on S3AsyncClient.The following methods were added to
S3AsyncClientBuilder
(codegen):The following method has been added to
DefaultS3AsyncClientBuilder
(codegen):