S3 Cross Region Client #4132

joviegas · 2023-06-24T05:52:05Z

Motivation and Context

When working with AWS S3 buckets, one common challenge arises: buckets can exist in different regions, unbeknownst to users. As a result, utilizing a single client to access buckets across regions becomes complicated. This new capability aims to resolve the difficulties faced by users, enabling seamless access to buckets across different regions. In this concise blog post, we will explore the significance of this feature and how it simplifies cross-region bucket access.

New Public APIs

API to enable the S3 Cross region feature

S3Client/S3AsyncClient

S3Client s3Client = S3Client.builder().crossRegionAccessEnabled(true).build();

CrtClient

S3AsyncClient crtClient = S3AsyncClient.crtBuilder().crossRegionAccessEnabled(true).build();

API for getting EndpointProviders in SdkServiceClientConfiguration

S3Client build = S3Client.builder().build();
// New API to get Optional endpointProvider
Optional<EndpointProvider> endpointProvider =
s3Client.serviceClientConfiguration().endpointProvider();

API for setting EndpointProviders in RequestOverrideConfiguration

 
PutObjectRequest putObjectRequest = 
    PutObjectRequest.builder()
    .overrideConfiguration( o -> o.endpointProvider(new CustomProvider()))
    .build()

Testing

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

I have read the CONTRIBUTING document
Local run of mvn install succeeds
My code follows the code style of this project
My change requires a change to the Javadoc documentation
I have updated the Javadoc documentation accordingly
I have added tests to cover my changes
All new and existing tests passed
I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

I confirm that this pull request can be released under the Apache 2 license

#3996) Adding a generic method invokeOperation for abstract delegating client to allow overriding behavior changes for all operations

…add crossRegionBucketAccess in S3Configuration (#3978) * Add endpointProvider in ClientConfig and RequestOverrideConfig. Also add crossRegionBucketAccess in S3Configuration * updated review comments 1 * updated review comments 2

) * Adds S3CrossRegion clients and codegen for retrieving bucket name * Removes codegen of bucket parameter since getValueForField() is better * Add back overloads

* Adds cross region client logic for decorating endpoint provider

#4046) * Paginated methods returning publisher or iterable implement logic in interface instead of throwing unsupported exception * Added internal annotation to class

* Wraps s3 client in cross regional client when enabled * Adds composer interface

* Add User Agent Api name for Cross region API calls * Added test case for default client not to have user agent related to cross region

* S3CrossRegionSyncClient Redirect implementation * Added implementation for Async client Decorator * Updated older Cross region test cases * Added paramterized test * Async Exception checged to completableException * Updated test cases and changes the Exception handling when HeadBucket Call fails * Handled Anna-karin's comments * Removed async execution of HeadBucket and attached it to the completableFuture of main request * Handled Zoe's comments * Added test case when Redirected after the Region is cached * Changed region constant to Region Type in Tests

* Cross region support for CRT Client * removing common class * handled review comments

* Integration test cases for Cross region Sync and Sync Clients * Renamed files as Integration test * Handled CR comments to take care of retries thus captureInterceptor is removed * rebased and added CRT in Integ test

dagnir · 2023-06-27T16:37:21Z

Can we add a changelog entry?

…lder, also mention of Composers are renamed as decorators (#4151) * Attach crossRegionAccessEnabled as CustomClientParams to a client Builder, also mention of Composers are renamed as decorators * Removed from S3Configurations * Move the optional parameter in getter

sonarqubecloud · 2023-07-05T17:18:12Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
50 Code Smells

87.5% Coverage
0.0% Duplication

* Adding a generic method invokeOperation for abstract delegating clien… (#3996) Adding a generic method invokeOperation for abstract delegating client to allow overriding behavior changes for all operations * Add endpointProvider in ClientConfig and RequestOverrideConfig. Also add crossRegionBucketAccess in S3Configuration (#3978) * Add endpointProvider in ClientConfig and RequestOverrideConfig. Also add crossRegionBucketAccess in S3Configuration * updated review comments 1 * updated review comments 2 * Adds S3CrossRegion clients and codegen for retrieving bucket name (#4008) * Adds S3CrossRegion clients and codegen for retrieving bucket name * Removes codegen of bucket parameter since getValueForField() is better * Add back overloads * Adds cross region client logic for decorating endpoint provider (#4026) * Adds cross region client logic for decorating endpoint provider * Paginated methods returning publisher or iterable implement logic in … (#4046) * Paginated methods returning publisher or iterable implement logic in interface instead of throwing unsupported exception * Added internal annotation to class * Wraps s3 client in cross regional client when enabled (#4080) * Wraps s3 client in cross regional client when enabled * Adds composer interface * Add User Agent Api name for Cross region API calls (#4105) * Add User Agent Api name for Cross region API calls * Added test case for default client not to have user agent related to cross region * S3CrossRegion Sync and Async Clients Redirect implementation (#4089) * S3CrossRegionSyncClient Redirect implementation * Added implementation for Async client Decorator * Updated older Cross region test cases * Added paramterized test * Async Exception checged to completableException * Updated test cases and changes the Exception handling when HeadBucket Call fails * Handled Anna-karin's comments * Removed async execution of HeadBucket and attached it to the completableFuture of main request * Handled Zoe's comments * Added test case when Redirected after the Region is cached * Changed region constant to Region Type in Tests * Cross region support for CRT Client (#4129) * Cross region support for CRT Client * removing common class * handled review comments * Integration test cases for Cross region Async and Sync Clients (#4128) * Integration test cases for Cross region Sync and Sync Clients * Renamed files as Integration test * Handled CR comments to take care of retries thus captureInterceptor is removed * rebased and added CRT in Integ test * removed sout from integ * Attach crossRegionAccessEnabled as CustomClientParams to a client Builder, also mention of Composers are renamed as decorators (#4151) * Attach crossRegionAccessEnabled as CustomClientParams to a client Builder, also mention of Composers are renamed as decorators * Removed from S3Configurations * Move the optional parameter in getter --------- Co-authored-by: Anna-Karin Salander <[email protected]>

cenedhryn and others added 11 commits May 12, 2023 12:27

Adding a generic method invokeOperation for abstract delegating clien… (

7a28087

#3996) Adding a generic method invokeOperation for abstract delegating client to allow overriding behavior changes for all operations

Adds S3CrossRegion clients and codegen for retrieving bucket name (#4008

8fcef4e

) * Adds S3CrossRegion clients and codegen for retrieving bucket name * Removes codegen of bucket parameter since getValueForField() is better * Add back overloads

Adds cross region client logic for decorating endpoint provider (#4026)

2a15452

* Adds cross region client logic for decorating endpoint provider

Paginated methods returning publisher or iterable implement logic in … (

a87b09a

#4046) * Paginated methods returning publisher or iterable implement logic in interface instead of throwing unsupported exception * Added internal annotation to class

Wraps s3 client in cross regional client when enabled (#4080)

578cc0d

* Wraps s3 client in cross regional client when enabled * Adds composer interface

Add User Agent Api name for Cross region API calls (#4105)

f7e4188

* Add User Agent Api name for Cross region API calls * Added test case for default client not to have user agent related to cross region

Cross region support for CRT Client (#4129)

4039f39

* Cross region support for CRT Client * removing common class * handled review comments

removed sout from integ

6aa4bc8

joviegas requested a review from a team as a code owner June 24, 2023 05:52

joviegas force-pushed the cross_region_bucket_access/project_branch branch 2 times, most recently from 2d03cbf to fe546ec Compare June 26, 2023 19:00

Rebased from Master

6c63aa9

joviegas force-pushed the cross_region_bucket_access/project_branch branch from fe546ec to 6c63aa9 Compare June 27, 2023 16:39

dagnir approved these changes Jun 27, 2023

View reviewed changes

joviegas and others added 2 commits June 30, 2023 17:07

Rebase and merge from master

13bd2c6

joviegas force-pushed the cross_region_bucket_access/project_branch branch from dd20e2d to 13bd2c6 Compare July 5, 2023 15:20

Merge branch 'master' into cross_region_bucket_access/project_branch

2bd4c2d

dagnir approved these changes Jul 5, 2023

View reviewed changes

joviegas enabled auto-merge (squash) July 5, 2023 17:09

joviegas merged commit 9798b4f into master Jul 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

S3 Cross Region Client #4132

S3 Cross Region Client #4132

Uh oh!

joviegas commented Jun 24, 2023 •

edited

Loading

Uh oh!

dagnir commented Jun 27, 2023

Uh oh!

sonarqubecloud bot commented Jul 5, 2023

Uh oh!

Uh oh!

S3 Cross Region Client #4132

S3 Cross Region Client #4132

Uh oh!

Conversation

joviegas commented Jun 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

New Public APIs

API to enable the S3 Cross region feature

API for getting EndpointProviders in SdkServiceClientConfiguration

API for setting EndpointProviders in RequestOverrideConfiguration

Testing

Screenshots (if appropriate)

Types of changes

Checklist

License

Uh oh!

dagnir commented Jun 27, 2023

Uh oh!

sonarqubecloud bot commented Jul 5, 2023

Uh oh!

Uh oh!

joviegas commented Jun 24, 2023 •

edited

Loading