Skip to content

[Hackathon] Add method to S3Client: deleteBucketAndAllContents #3070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@

/**
* {@link SdkExtensionMethod} indicates a method that has been explicitly added to a service client interface on behalf of the
* SDK. While most service clients are automatically and fully generated based upon the corresponding service's API model, the SDK
* may sometimes choose to extend a service's logical API in order to provide improved abstractions or convenience functions. As
* such, {@link SdkExtensionMethod} implementations may invoke one or more service requests to fulfil their behavior.
* SDK. While most service clients are automatically and fully generated from the corresponding service's API model, the SDK may
* sometimes choose to extend a service's logical API in order to provide improved abstractions or convenience functions. {@link
* SdkExtensionMethod} implementations may invoke one or more service requests to fulfil their behavior.
*/
@Target(ElementType.METHOD)
@SdkProtectedApi
Expand Down
4 changes: 2 additions & 2 deletions docs/LaunchChangelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,8 +374,8 @@ The S3 client in 2.0 is drastically different from the client in 1.11, because i
| `deleteObjects` | `deleteObjects` |
| `deleteVersion` | `deleteObject` |
| `disableRequesterPays` | `putBucketRequestPayment` |
| `doesBucketExist` | `headBucket` |
| `doesBucketExistV2` | `headBucket` |
| `doesBucketExist` | `doesBucketExist` or `headBucket` |
| `doesBucketExistV2` | `doesBucketExist` or `headBucket` |
| `doesObjectExist` | `headObject` |
| `enableRequesterPays` | `putBucketRequestPayment` |
| `generatePresignedUrl` | ~~Not Supported~~ [S3Presigner](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-s3-presign.html) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,7 @@
import software.amazon.awssdk.services.s3.model.BucketLocationConstraint;
import software.amazon.awssdk.services.s3.model.CreateBucketConfiguration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.DeleteBucketRequest;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsResponse;
import software.amazon.awssdk.services.s3.model.ListObjectsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectsResponse;
import software.amazon.awssdk.services.s3.model.ObjectVersion;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.model.S3Object;
import software.amazon.awssdk.testutils.service.AwsTestBase;

/**
Expand Down Expand Up @@ -116,36 +108,7 @@ private static void createBucket(String bucketName, int retryCount) {

protected static void deleteBucketAndAllContents(String bucketName) {
System.out.println("Deleting S3 bucket: " + bucketName);
ListObjectsResponse response = s3.listObjects(ListObjectsRequest.builder().bucket(bucketName).build());

while (true) {
if (response.contents() == null) {
break;
}
for (S3Object objectSummary : response.contents()) {
s3.deleteObject(DeleteObjectRequest.builder().bucket(bucketName).key(objectSummary.key()).build());
}

if (response.isTruncated()) {
response = s3.listObjects(ListObjectsRequest.builder().marker(response.nextMarker()).build());
} else {
break;
}
}

ListObjectVersionsResponse versionsResponse = s3
.listObjectVersions(ListObjectVersionsRequest.builder().bucket(bucketName).build());
if (versionsResponse.versions() != null) {
for (ObjectVersion s : versionsResponse.versions()) {
s3.deleteObject(DeleteObjectRequest.builder()
.bucket(bucketName)
.key(s.key())
.versionId(s.versionId())
.build());
}
}

s3.deleteBucket(DeleteBucketRequest.builder().bucket(bucketName).build());
s3.deleteBucketAndAllContents(bucketName);
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
import static software.amazon.awssdk.testutils.service.S3BucketUtils.temporaryBucketName;

import java.io.IOException;
import java.util.Iterator;
import org.apache.commons.io.IOUtils;
import org.junit.AfterClass;
import org.junit.BeforeClass;
Expand All @@ -39,15 +38,7 @@
import software.amazon.awssdk.services.cloudtrail.model.UpdateTrailResponse;
import software.amazon.awssdk.services.s3.model.CreateBucketConfiguration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.DeleteBucketRequest;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsResponse;
import software.amazon.awssdk.services.s3.model.ListObjectsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectsResponse;
import software.amazon.awssdk.services.s3.model.ObjectVersion;
import software.amazon.awssdk.services.s3.model.PutBucketPolicyRequest;
import software.amazon.awssdk.services.s3.model.S3Object;

public class CloudTrailIntegrationTest extends IntegrationTestBase {
private static final String BUCKET_NAME = temporaryBucketName("aws-java-cloudtrail-integ");
Expand Down Expand Up @@ -84,38 +75,7 @@ public static void tearDown() {

public static void deleteBucketAndAllContents(String bucketName) {
System.out.println("Deleting S3 bucket: " + bucketName);
ListObjectsResponse response = s3.listObjects(ListObjectsRequest.builder().bucket(bucketName).build());

while (true) {
if (response.contents() == null) {
break;
}
for (Iterator<?> iterator = response.contents().iterator(); iterator
.hasNext(); ) {
S3Object objectSummary = (S3Object) iterator.next();
s3.deleteObject(DeleteObjectRequest.builder().bucket(bucketName).key(objectSummary.key()).build());
}

if (response.isTruncated()) {
response = s3.listObjects(ListObjectsRequest.builder().marker(response.nextMarker()).build());
} else {
break;
}
}

ListObjectVersionsResponse versionsResponse = s3
.listObjectVersions(ListObjectVersionsRequest.builder().bucket(bucketName).build());
if (versionsResponse.versions() != null) {
for (ObjectVersion s : versionsResponse.versions()) {
s3.deleteObject(DeleteObjectRequest.builder()
.bucket(bucketName)
.key(s.key())
.versionId(s.versionId())
.build());
}
}

s3.deleteBucket(DeleteBucketRequest.builder().bucket(bucketName).build());
s3.deleteBucketAndAllContents(bucketName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Red code is the best code!

}

@Test
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/

package software.amazon.awssdk.services.s3;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static software.amazon.awssdk.testutils.service.S3BucketUtils.temporaryBucketName;

import java.nio.charset.StandardCharsets;
import java.util.UUID;
import java.util.stream.IntStream;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.internal.extensions.DeleteBucketAndAllContents;
import software.amazon.awssdk.services.s3.model.BucketVersioningStatus;

/**
* Tests the {@link S3Client#deleteBucketAndAllContents(String)} extension method.
*
* @see DeleteBucketAndAllContents
*/
public class DeleteBucketAndAllContentsIntegrationTest extends S3IntegrationTestBase {

private static final String BUCKET = temporaryBucketName(DeleteBucketAndAllContentsIntegrationTest.class);

@BeforeClass
public static void initializeTestData() {
createBucket(BUCKET);
s3.putBucketVersioning(r -> r
.bucket(BUCKET)
.versioningConfiguration(v -> v.status(BucketVersioningStatus.ENABLED)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have a test that sets up a more complicated bucket where objects have different versions?

Copy link
Contributor Author

@Bennett-Lynch Bennett-Lynch Mar 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point. I could rewrite the same first ~50 key names. Would that be sufficient?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

}

@AfterClass
public static void tearDown() {
if (s3.doesBucketExist(BUCKET)) {
deleteBucketAndAllContents(BUCKET);
}
}

@Test
public void deleteBucketAndAllContents_WithVersioning_DeletesBucket() {
// Populate the bucket with >1000 objects in order to exercise pagination behavior.
int maxDeleteObjectsSize = 1_000;
int numObjectsToCreate = maxDeleteObjectsSize + 50;
IntStream.range(0, numObjectsToCreate).parallel().forEach(this::putObject);
// Overwrite some keys to create multiple versions of objects
int numKeysToOverwrite = 50;
IntStream.range(0, numKeysToOverwrite).parallel().forEach(this::putObject);
// Test deleting the bucket
s3.deleteBucketAndAllContents(BUCKET);
assertThat(s3.doesBucketExist(BUCKET), is(false));
}

private void putObject(int i) {
s3.putObject(r -> r.bucket(BUCKET)
.key(String.valueOf(i)),
RequestBody.fromString(UUID.randomUUID().toString(), StandardCharsets.UTF_8));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,15 @@

import java.rmi.NoSuchObjectException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.Bucket;
import software.amazon.awssdk.services.s3.model.DeleteBucketRequest;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.ExpirationStatus;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsResponse;
import software.amazon.awssdk.services.s3.model.ListObjectsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectsResponse;
import software.amazon.awssdk.services.s3.model.NoSuchBucketException;
import software.amazon.awssdk.services.s3.model.S3Object;
import software.amazon.awssdk.testutils.Waiter;
import software.amazon.awssdk.utils.Logger;

Expand Down Expand Up @@ -117,51 +109,7 @@ public static void runCleanupTasks(Class<?> testClass) {
public static void deleteBucketAndAllContents(S3Client s3, String bucketName) {
try {
System.out.println("Deleting S3 bucket: " + bucketName);
ListObjectsResponse response = Waiter.run(() -> s3.listObjects(r -> r.bucket(bucketName)))
.ignoringException(NoSuchBucketException.class)
.orFail();
List<S3Object> objectListing = response.contents();

if (objectListing != null) {
while (true) {
for (Iterator<?> iterator = objectListing.iterator(); iterator.hasNext(); ) {
S3Object objectSummary = (S3Object) iterator.next();
s3.deleteObject(DeleteObjectRequest.builder().bucket(bucketName).key(objectSummary.key()).build());
}

if (response.isTruncated()) {
objectListing = s3.listObjects(ListObjectsRequest.builder()
.bucket(bucketName)
.marker(response.marker())
.build())
.contents();
} else {
break;
}
}
}


ListObjectVersionsResponse versions = s3
.listObjectVersions(ListObjectVersionsRequest.builder().bucket(bucketName).build());

if (versions.deleteMarkers() != null) {
versions.deleteMarkers().forEach(v -> s3.deleteObject(DeleteObjectRequest.builder()
.versionId(v.versionId())
.bucket(bucketName)
.key(v.key())
.build()));
}

if (versions.versions() != null) {
versions.versions().forEach(v -> s3.deleteObject(DeleteObjectRequest.builder()
.versionId(v.versionId())
.bucket(bucketName)
.key(v.key())
.build()));
}

s3.deleteBucket(DeleteBucketRequest.builder().bucket(bucketName).build());
s3.deleteBucketAndAllContents(bucketName);
} catch (Exception e) {
System.err.println("Failed to delete bucket: " + bucketName);
e.printStackTrace();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,14 @@

import software.amazon.awssdk.annotations.SdkExtensionMethod;
import software.amazon.awssdk.annotations.SdkPublicApi;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.internal.extensions.DefaultS3ClientSdkExtension;
import software.amazon.awssdk.services.s3.internal.extensions.DeleteBucketAndAllContents;
import software.amazon.awssdk.services.s3.model.DeleteBucketRequest;
import software.amazon.awssdk.services.s3.model.DeleteObjectsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.S3Exception;

/**
Expand All @@ -32,12 +38,36 @@ public interface S3ClientSdkExtension {
* not accessible (e.g., due to access being denied or the bucket existing in another region), an {@link S3Exception} will be
* thrown.
*
* @param bucketName the bucket to check
* @param bucket the bucket to check
* @return true if the bucket exists and you have permission to access it; false if the bucket does not exist
* @throws S3Exception if the bucket exists but is not accessible
*/
@SdkExtensionMethod
default boolean doesBucketExist(String bucketName) {
return new DefaultS3ClientSdkExtension((S3Client) this).doesBucketExist(bucketName);
default boolean doesBucketExist(String bucket) {
return new DefaultS3ClientSdkExtension((S3Client) this).doesBucketExist(bucket);
}

/**
* Permanently delete a bucket and all of its content, including any versioned objects and delete markers.
* <p>
* Internally this method will use the {@link S3Client#listObjectsV2(ListObjectsV2Request)} and {@link
* S3Client#listObjectVersions(ListObjectVersionsRequest)} APIs to list a bucket's content, buffer keys that are eligible for
* deletion into batches of 1000, and delete them in bulk with the {@link S3Client#deleteObjects(DeleteObjectsRequest)} API.
* <p>
* While this method is optimized to use batch APIs for both listing and deleting, it may not be suitable for buckets
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may also be useful to direct people to TransferManager for their more complicated needs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, but I don't think we currently plan to support "delete directory" in TM, but it makes sense to support it there.

* containing a very large number of objects (i.e., hundreds of thousands). For such use cases, it is usually preferable to
* either create an <i>S3 Lifecycle configuration</i> to delete the objects, or to leverage <i>S3 Batch Operations</i> to
* perform large-scale deletes.
* <p>
* Note that this method does not attempt to protect against concurrent writes or modifications to a bucket. It will iterate
* and delete the entire contents of the bucket once. If a new key is created during or after the iteration, then the final
* call to {@link S3Client#deleteBucket(DeleteBucketRequest)} may fail.
*
* @param bucket the bucket to delete
* @throws SdkException if an error occurs
*/
@SdkExtensionMethod
default void deleteBucketAndAllContents(String bucket) {
new DeleteBucketAndAllContents((S3Client) this).deleteBucketAndAllContents(bucket);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delegation has again proven to be more flexible. Good choice using a separate class.

}
}
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ public DefaultS3ClientSdkExtension(S3Client s3) {

@Override
public boolean doesBucketExist(String bucket) {
Validate.notNull(bucket, "bucket");
Validate.notEmpty(bucket, "bucket");
try {
s3.headBucket(r -> r.bucket(bucket));
return true;
Expand Down
Loading