feature: support training inputs from EFS and FSx #991

laurenyu · 2019-08-21T22:19:48Z

Issue #, if available:

Description of changes:
Amazon SageMaker now supports Amazon EFS and Amazon FSx for Lustre file systems as data sources for training machine learning models. This is work from @caxiaohu to support that here.

NB: the resources don't necessarily get cleaned up properly if things fail during setup, but we can track that as a separate issue in our backlog

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have updated any necessary documentation (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…ation tests

jesterhazy · 2019-08-21T22:22:27Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-21T22:36:59Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-21T22:48:55Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-21T22:49:10Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-21T23:13:40Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-21T23:27:57Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-21T23:38:31Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-21T23:39:47Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T00:06:49Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…pytest

jesterhazy · 2019-08-22T00:19:10Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T00:31:29Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T00:33:38Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T02:34:22Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T03:39:59Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T03:49:34Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T06:59:55Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T07:14:00Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T07:24:23Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T07:45:51Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T09:14:24Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T09:28:10Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T09:38:51Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-22T10:14:34Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T00:29:15Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T00:44:18Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T00:53:36Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

knakad · 2019-08-23T00:46:13Z

src/sagemaker/inputs.py

+            }
+        }
+
+        if compression is not None:


[Minor] Can these be written as one-liners with "or"?

not exactly sure what you're suggesting here. are you thinking self.config["CompressionType"] = compression or None?

That's what I was thinking. Am I missing something that would make that incorrect?

boto3 may complain about parameters being set to None. The error will be something like "incorrect parameter type <NoneType>, should be one of: <str>"

src/sagemaker/session.py

tests/integ/file_system_input_utils.py

tests/integ/s3_utils.py

knakad · 2019-08-23T00:56:38Z

tests/integ/vpc_test_utils.py

@@ -21,33 +21,107 @@
 LOCK_PATH = os.path.join(tempfile.gettempdir(), "sagemaker_test_vpc_lock")


-def _get_subnet_ids_by_name(ec2_client, name):
+def _subnet_ids_by_name(ec2_client, name):


[Minor] Not gonna lie, I think I preferred the previous name. It was more obvious what it did.

"get" isn't idiomatic in Python. since this is just a private method that wasn't actually modified in this PR, though, I can change it back.

Do you have any readings on this that I could take advantage of? I wasn't able to find anything useful on the interwebs.

I forget where I originally read/heard it from, though I see it mentioned a lot in StackOverflow answers. iirc the idea is that "get" is redundant to what's happening. @mvsusp, do you have any references handy on this one?

tests/integ/vpc_test_utils.py

jesterhazy · 2019-08-23T01:28:31Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T05:38:18Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T05:50:53Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T06:02:52Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T06:32:08Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T06:43:20Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T06:55:47Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T07:08:09Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jesterhazy · 2019-08-23T07:16:48Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

tests/integ/file_system_input_utils.py

caxiaohu and others added 10 commits August 20, 2019 12:56

feature: add FileSystemInput class and unit tests

1efd696

Feature: Add FileSystemRecordSet class and a unit test

5b60c0c

change: fix 1p algorithm HPO with two channel data sets issue

1d769ff

feature: add documentation for efs and fsx

60bd406

Feature: add utility functions for file system input setup and integr…

52fda98

…ation tests

Feature: add kmeans with efs and fsx data sources

b6d65e9

Merge branch 'master' into file-system-support

d9ae029

fix doc spacing

ed95870

fix license headers

c59cca3

Merge branch 'master' into file-system-support

ad18484

Address pylint errors and update boto3 version

82c9694

fix vpc util functions

2fc6177

fix vpc util functions

7bf0794

remove "test_" from util file name so that it doesn't interfere with …

6603093

…pytest

fix efs path

176d3b5

fix error handling

d4503c2

laurenyu force-pushed the file-system-support branch from 1731f24 to d4503c2 Compare August 22, 2019 09:10

undo vpc name experiment

62c1a97

knakad previously approved these changes Aug 23, 2019

View reviewed changes

address PR comments and OSError for reading stdin

8b42f13

laurenyu dismissed knakad’s stale review via 8b42f13 August 23, 2019 05:34

fix kmeans efs path

89a8f90

caxiaohu approved these changes Aug 23, 2019

View reviewed changes

tests/integ/file_system_input_utils.py Show resolved Hide resolved

knakad merged commit b33d1a2 into aws:master Aug 23, 2019

laurenyu deleted the file-system-support branch August 23, 2019 16:49

feature: support training inputs from EFS and FSx #991

feature: support training inputs from EFS and FSx #991

Uh oh!

Conversation

laurenyu commented Aug 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

Uh oh!

jesterhazy commented Aug 21, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 21, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 21, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 21, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 21, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 21, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 21, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 21, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 22, 2019

AWS CodeBuild CI Report

Uh oh!

jesterhazy commented Aug 23, 2019

AWS CodeBuild CI Report

Uh oh!

laurenyu commented Aug 21, 2019 •

edited

Loading

laurenyu Aug 23, 2019 •

edited

Loading