-
Notifications
You must be signed in to change notification settings - Fork 1.2k
fix: allow download_folder to download file even if bucket is more restricted #1295
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
src/sagemaker/utils.py
Outdated
|
||
# the prefix points to an s3 'directory' download the whole thing | ||
# Assume the prefix points to an S3 'directory' and download the whole thing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggestion: Could moving it into a function and calling it from inside the except when object is not a file make it better?
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Issue #, if available:
#1283
Description of changes:
S3 doesn't provide a way of definitively determining if a prefix is a folder, so originally we used
ListObjects
to try and gauge that. However,ListObjects
requires extra permissions on the bucket, and so download a public file from a private bucket fails. In this PR, I've changed the logic to try and download the file first, and then if that incurs a 404 because it's a folder (no idea why that's how S3 chooses to implement this...), then the code will proceed with as if the prefix points to a folder.Side note - I noticed that pylint was requiring me to put
botocore
in the wrong order for the imports. I'll fix that in a separate PR.Testing done:
tried the example S3 path from #1283:
(also ran
tox tests/unit
)Merge Checklist
Put an
x
in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.General
Tests
unique_name_from_base
to create resource names in integ tests (if appropriate)By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.