fix: change AMI ids in tests to be dynamic based on regions #1004

caxiaohu · 2019-08-27T08:39:51Z

Issue #, if available:

Description of changes:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have updated any necessary documentation (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…to Absolute or normalized path

sagemaker-bot · 2019-08-27T08:44:06Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T08:57:22Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T09:08:47Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T09:14:27Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

laurenyu · 2019-08-27T15:47:03Z

tests/scripts/fs_mount_setup.sh

@@ -18,14 +18,14 @@ FILE_SYSTEM_EFS_ID=$1
 FILE_SYSTEM_FSX_ID=$2

 echo "Mounting EFS File Systems"
-sudo yum install -y amazon-efs-utils.noarch 0:1.10-1.amzn2
+sudo yum install -y amazon-efs-utils


do different regions have different versions released?

The list for the AMI ids are Amazon Linux AMI 2018, while the old hard-coded one is Amazon Linux2 AMI.
The new list needs amazon-efs-utils.noarch 0:1.10-1.amzn1 version. I just don't want specify specific version in case some of them need amazon-efs-utils.noarch 0:1.10-1.amzn2 version.
But it will automatically select the correct version if i don't specify

laurenyu · 2019-08-27T15:47:28Z

tests/integ/file_system_input_utils.py

@@ -31,7 +31,6 @@
 PREFIX = "ec2_fs_key_"
 KEY_NAME = PREFIX + str(uuid.uuid4().hex.upper()[0:8])
 ROLE_NAME = "SageMakerRole"
-REGION = "us-west-2"
 EC2_INSTANCE_TYPE = "t2.micro"
 AMI_ID = "ami-082b5a644766e0e6f"


I think you can remove this

Forgot to remove it

laurenyu · 2019-08-27T15:48:13Z

tests/integ/file_system_input_utils.py

@@ -49,6 +48,28 @@
 KEY_PATH = os.path.join(tempfile.gettempdir(), FILE_NAME)
 STORAGE_CAPACITY_IN_BYTES = 3600

+AWSRegionArch2AMI = {


rename this to REGION_TO_AMI_MAP - we still want to honor Python capitalization/naming convention

knakad · 2019-08-27T16:07:01Z

tests/integ/file_system_input_utils.py

+    "cn-north-1": "ami-0a4eaf6c4454eda75",
+    "cn-northwest-1": "ami-6b6a7d09",
+    "us-gov-west-1": "ami-906cf0f1",
+}


@caxiaohu @laurenyu Is there any way we could make this dynamic based on the AMI name or type? Since we only need a basic AL AMI?
i.e.: Some dynamic logic that searches for an AMI by name and uses whatever ID is appropriate in that region. Eliminating the need for a map that we need to maintain.

I'm concerned that we'll need to add to this list for all future regions. This is likely another time-bomb that'll trigger a page on region expansion (or the dev will catch it during region expansion).

good catch. I'd forgotten that I found this yesterday; I think it should do what we want: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/finding-an-ami.html#finding-quick-start-ami

The AMI ids search results:

Region AMI id

us-west-1 ami-0fcdcdb074d2bac5f

us-west-2 ami-0f2176987ee50226e

us-east-1 ami-035b3c7efe6d061d5

us-east-2 ami-02f706d959cedf892

eu-west-1 ami-0862aabda3fb488b5

eu-west-2 ami-0bdfa1adc3878cd23

eu-west-3 ami-05b93cd5a1b552734

eu-central-1 ami-026d3b3672c6e7b66

ap-northeast-1 ami-04b2d1589ab1d972c

ap-northeast-2 ami-0be3e6f84d3b968cd

ap-northeast-3 ami-0639e0e4c18187b30

ap-southeast-1 ami-0fb6b6f9e81056553

ap-southeast-2 ami-075caa3491def750b

ap-south-1 ami-0b99c7725b9484f9e

ca-central-1 ami-0a67d15f2858e33cb

sa-east-1 ami-0bb96001cf2299257

cn-north-1 AWS was not able to validate the provided access credentials

cn-northwest-1 AWS was not able to validate the provided access credentials

us-gov-west-1 AWS was not able to validate the provided access credentials

All of them are Amazon Linux AMI 2018.03.0

…hing Amazon Linux AMI

sagemaker-bot · 2019-08-27T20:43:48Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T20:46:06Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

laurenyu · 2019-08-27T20:49:47Z

tests/integ/file_system_input_utils.py

    except Exception:
        tear_down(sagemaker_session, fs_resources)
        raise

    return fs_resources


+def _dynamic_ami_id(sagemaker_session):


let's rename this to _ami_id_for_region

laurenyu · 2019-08-27T20:50:33Z

tests/integ/file_system_input_utils.py

+def _dynamic_ami_id(sagemaker_session):
+    ec2_client = sagemaker_session.boto_session.client("ec2")
+    filters = [
+        {"Name": "name", "Values": ["amzn-ami-hvm-????.??.?.????????-x86_64-gp2"]},


let's make this a constant. that'll both make it easier to find if it needs to be changed later for whatever reason and help describe what the string is.

laurenyu · 2019-08-27T20:51:13Z

tests/integ/file_system_input_utils.py

+    image_details = sorted(response["Images"], key=itemgetter("CreationDate"), reverse=True)
+    if len(image_details) > 0:
+        ami_id = image_details[0]["ImageId"]
+        return ami_id


nit: you can combine ll. 126-127

sagemaker-bot · 2019-08-27T20:59:31Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T20:59:39Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:08:38Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:10:26Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:10:52Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:29:14Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:33:41Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:42:24Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:49:10Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:54:03Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:58:32Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T21:59:04Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T22:46:26Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

laurenyu · 2019-08-27T23:13:08Z

tests/integ/file_system_input_utils.py

@@ -49,6 +48,11 @@
 KEY_PATH = os.path.join(tempfile.gettempdir(), FILE_NAME)
 STORAGE_CAPACITY_IN_BYTES = 3600

+AMI_FILTERS = [
+    {"Name": "name", "Values": ["amzn-ami-hvm-????.??.?.????????-x86_64-gp2"]},


sorry for the confusion. I meant specifically "amzn-ami-hvm-????.??.?.????????-x86_64-gp2" - it's not really obvious what this string means

I see what you mean now

sagemaker-bot · 2019-08-27T23:22:41Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T23:42:22Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T23:51:52Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-27T23:57:51Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-28T00:09:14Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-28T00:35:43Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-28T00:42:18Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-28T00:57:59Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-28T01:07:15Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2019-08-28T01:54:30Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws#1004)

* feature: method to build pipeline parameters from existing execution … (#951) * feature: method to build pipeline parameters from existing execution with optional value overrides * fix style check * assert error message in unit test * feature: allow opt out from referencing latest execution in the selec… (#1004) * fix: Update pipeline.py and selective_execution_config.py with small fixes (#1099) --------- Co-authored-by: stacicho <[email protected]> Co-authored-by: Zuoyuan Huang <[email protected]>

caxiaohu added 4 commits August 23, 2019 11:48

update: change directory_path for EFS usage example in overview doc

a877ca9

update: changed EFS directory path instructions in doc and docstring …

2ca2c9e

…to Absolute or normalized path

update: changed AMI ids to be dynamic based on regions

a4ec855

fix: change Amazon AMI ids to be dynamic based on regions

1bdf0ad

caxiaohu requested review from chuyang-deng and laurenyu August 27, 2019 08:40

laurenyu reviewed Aug 27, 2019

View reviewed changes

knakad reviewed Aug 27, 2019

View reviewed changes

caxiaohu added 2 commits August 27, 2019 13:39

fix: change Amazon AMI ids to be dynamic based on regions using searc…

c43ff38

…hing Amazon Linux AMI

update: move region closer to its usage

f41d3c5

laurenyu reviewed Aug 27, 2019

View reviewed changes

update: make filter as constant variable

0393384

update: remove else

6b6e070

laurenyu reviewed Aug 27, 2019

View reviewed changes

update: add constant variable for search AMI name

b80ccbc

laurenyu approved these changes Aug 27, 2019

View reviewed changes

laurenyu changed the title ~~fix: changed AMI ids to be dynamic based on regions~~ fix: change AMI ids in tests to be dynamic based on regions Aug 27, 2019

chuyang-deng approved these changes Aug 28, 2019

View reviewed changes

Merge branch 'master' into fix_region_issue

6a95b89

chuyang-deng merged commit f233dc0 into aws:master Aug 28, 2019

chuyang-deng mentioned this pull request Aug 28, 2019

Undo skip #1005

Closed

4 tasks

shenlongtang added a commit to shenlongtang/sagemaker-python-sdk that referenced this pull request Oct 3, 2023

feature: allow opt out from referencing latest execution in the selec… (

b1d588c

aws#1004)

Region	AMI id
us-west-1	ami-0fcdcdb074d2bac5f
us-west-2	ami-0f2176987ee50226e
us-east-1	ami-035b3c7efe6d061d5
us-east-2	ami-02f706d959cedf892
eu-west-1	ami-0862aabda3fb488b5
eu-west-2	ami-0bdfa1adc3878cd23
eu-west-3	ami-05b93cd5a1b552734
eu-central-1	ami-026d3b3672c6e7b66
ap-northeast-1	ami-04b2d1589ab1d972c
ap-northeast-2	ami-0be3e6f84d3b968cd
ap-northeast-3	ami-0639e0e4c18187b30
ap-southeast-1	ami-0fb6b6f9e81056553
ap-southeast-2	ami-075caa3491def750b
ap-south-1	ami-0b99c7725b9484f9e
ca-central-1	ami-0a67d15f2858e33cb
sa-east-1	ami-0bb96001cf2299257
cn-north-1	AWS was not able to validate the provided access credentials
cn-northwest-1	AWS was not able to validate the provided access credentials
us-gov-west-1	AWS was not able to validate the provided access credentials

fix: change AMI ids in tests to be dynamic based on regions #1004

fix: change AMI ids in tests to be dynamic based on regions #1004

Uh oh!

Conversation

caxiaohu commented Aug 27, 2019 • edited by laurenyu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caxiaohu Aug 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Aug 27, 2019

caxiaohu commented Aug 27, 2019 •

edited by laurenyu

Loading

caxiaohu Aug 27, 2019 •

edited

Loading

caxiaohu Aug 27, 2019 •

edited

Loading