doc: Support for generation of Jumpstart model table on build #2888

bencrabtree · 2022-02-01T20:14:52Z

Issue #, if available:

Description of changes:
This PR augments the sphinx build process to add a script that generates an up-to-date table of available JumpStart model_ids in documentation. This is done by adding a setup function to docs/conf.py. This script will run each time the documentation is built and will pull the current models from JumpStarts S3 bucket (which is publicly readable).

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-02-01T20:23:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4d0bfed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-01T20:43:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4d0bfed
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-01T20:45:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4d0bfed
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

evakravi · 2022-02-01T21:57:21Z

doc/conf.py

@@ -15,6 +15,10 @@

 import pkg_resources
 from datetime import datetime
+import sys
+import os
+sys.path.append(os.path.join(os.path.dirname(__file__), '/doc/'))


why do we need this?

sphinx has issues importing modules from the file system. You have to explicitly define where it should look sphinx-doc/sphinx#2390

sagemaker-bot · 2022-02-01T22:43:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1f9b88f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-01T22:44:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 1f9b88f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-01T23:06:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 1f9b88f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

JGuinegagne

non-blocking: a fast follow, could you add unit tests for doc/jumpstart/utils.py ?

JGuinegagne · 2022-02-02T13:46:15Z

doc/jumpstart/utils.py

+def get_jumpstart_sdk_manifest():
+    s3_client = boto3.client("s3")
+    studio_metadata = (
+        s3_client.get_object(Bucket=JUMPSTART_BUCKET, Key=SDK_MANIFEST_FILE)["Body"]


This is a text-book case of:

[PM/customer]: "it doesn't work"

[engineer]: "it works in my local environment"

Taking a step back: boto3 relies on botocore to make the http calls to the AWS service endpoints. botocore automatically adds identification headers to these requests. To populate these headers, botocore needs to access AWS credentials, generally from environment variables. Then the AWS services will use these headers to (1) authenticate and (2) authorize, ie check the IAM permissions the principals authenticated in (1) has against the permissions required by the request being made.

So here, you are implicitly making the assumption that (a) the execution environment of readthedocs where this code will be executed has AWS credentials, and (b) that those AWS credentials have S3:GetObject permission. If (a) is false, then the authentication will fail, most likely botocore will raise an exception even before sending the request. If (b) is false, the authorization will fail, and you will get a 403 error.

Luckily, our distribution buckets are world-readable, and for S3 you can use the public URL of the object, instead of going through the service endpoint. You could use wget, but the Pythonic way of doing this is to rely on a fetching library, most likely urllib. It would look something like:

from urllib import request url = 'https://jumpstart-cache-prod-us-west-2.s3.us-west-2.amazonaws.com/models_manifest.json' with request.openurl(url) as f: models_manifest = f.read().decode("utf-8")

Gotcha, this makes plenty of sense. Unfortunately there's not a great way to test in the production environment but I should've caught this.

JGuinegagne · 2022-02-02T13:47:20Z

doc/jumpstart/utils.py

+
+def get_jumpstart_sdk_manifest():
+    s3_client = boto3.client("s3")
+    studio_metadata = (


Well this is a fine misnomer, since this file is not the models metadata used in Studio.

Could you rename models_manifest?

doc/conf.py

evakravi · 2022-02-02T14:24:43Z

doc/jumpstart/utils.py

@@ -0,0 +1,77 @@
+from __future__ import absolute_import


Question: Do we need the legal comment at the top?

evakravi · 2022-02-02T14:32:38Z

doc/jumpstart/utils.py

+
+    f = open("jumpstart/jumpstart.rst", "w")
+
+    f.write("==================================\n")


Should we write to the file each time, or can we instead build a list of strings, and do something like

lines_of_file = [] lines_of_file.append("==================================") lines_of_file.append("JumpStart Available Model Table") ... ... ... file_content = '\n'.join(lines_of_file) f.write(file_content)

This may not be necessary, the http calls to download the metadata are probably orders of magnitude longer to execute.

This is personal preference- I'm not positive what the pythonic best practice here would be

sagemaker-bot · 2022-02-02T15:33:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f94245c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

JGuinegagne

non-blocking comments

JGuinegagne · 2022-02-02T15:34:37Z

doc/doc_utils/jumpstart_doc_utils.py

+import json
+from packaging.version import Version
+
+JUMPSTART_REGION = "eu-west-2"


non-blocking: if you have time, please add a comment as to why we use DUB so that someone doesn't just change it back to us-west-2. Also, could you rename JUMPSTART_METADATA_REGION?

JGuinegagne · 2022-02-02T15:34:52Z

doc/doc_utils/jumpstart_doc_utils.py

+
+JUMPSTART_REGION = "eu-west-2"
+SDK_MANIFEST_FILE = "models_manifest.json"
+JUMPSTART_BUCKET_BASE_URL = 'https://jumpstart-cache-prod-{}.s3.{}.amazonaws.com'.format(JUMPSTART_REGION, JUMPSTART_REGION)


non-blocking: f-string please

sagemaker-bot · 2022-02-02T15:51:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f94245c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-02T15:53:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f94245c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Mufaddal Rohawala <[email protected]>

sagemaker-bot · 2022-02-03T01:02:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 858a0dc
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-03T01:09:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 858a0dc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-03T01:10:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 858a0dc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-03T01:43:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 858a0dc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Basil Beirouti <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Ahsan Khan <[email protected]> Co-authored-by: Shreya Pandit <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Basil Beirouti <[email protected]> Co-authored-by: Payton Staub <[email protected]> Co-authored-by: Mohamed Ali Jamaoui <[email protected]> Co-authored-by: ci <ci> Co-authored-by: Jeniya Tabassum <[email protected]> Co-authored-by: sreedes <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Miyoung <[email protected]> Co-authored-by: Ameen Khan <[email protected]> Co-authored-by: Zhankui Lu <[email protected]> Co-authored-by: Xiaoguang Chen <[email protected]> Co-authored-by: Jonathan Guinegagne <[email protected]> Co-authored-by: Zhankui Lu <[email protected]> Co-authored-by: Yifei Zhu <[email protected]> Co-authored-by: Qingzi-Lan <[email protected]> Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Xinghan Chen <[email protected]> Co-authored-by: Ivy Bazan <[email protected]> Co-authored-by: IvyBazan <[email protected]>

…o feat/jumpstart-docs-model-table

sagemaker-bot · 2022-02-04T15:21:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f05b04e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-04T15:28:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f05b04e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-04T15:28:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f05b04e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T14:46:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7932372
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T15:04:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 11aea41
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T15:28:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 44ad05e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T15:47:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 11aea41
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T15:47:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7932372
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T15:54:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 7932372
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T16:07:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 44ad05e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T16:33:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 11aea41
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T16:33:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 44ad05e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T16:39:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 11aea41
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2022-02-07T23:46:00Z

doc/doc_utils/jumpstart_doc_utils.py

+        if model["model_id"] not in sdk_manifest_top_versions_for_models:
+            sdk_manifest_top_versions_for_models[model["model_id"]] = model
+        else:
+            if Version(


nit: can use elif

mufaddal-rohawala · 2022-02-07T23:53:59Z

doc/doc_utils/jumpstart_doc_utils.py

+    sdk_manifest = get_jumpstart_sdk_manifest()
+    sdk_manifest_top_versions_for_models = {}
+
+    for model in sdk_manifest:


Future improvement (non-blocking):
Do we know how much time is added to read and update read the docs for all the jumpstart models (I heard no of models are 300+)?
If it takes a significant time, alternatively we could have checked for jumpstart manifest hash (md5 or something) and compared against one stored in sagemaker defalt bucket and only update if hash mismatches.

I've seen this take a max of 5 minutes as it stands. I agree we should definitely look to optimize this call, but we wanted to get P0 into production ASAP so customers could access the model table. We can discuss the implementation further at our next sync.

keerthanvasist and others added 4 commits January 29, 2022 16:25

Add deprecation warning in Clarify DataConfig (aws#2847)

ed7dce3

feature: Update instance types for integ test (aws#2881)

7a347c7

feature: Adds support for async inference (aws#2846)

34b07c0

doc: added support for generation of jumpstart model table on build

4d0bfed

evakravi reviewed Feb 1, 2022

View reviewed changes

test: update linting

1f9b88f

JGuinegagne approved these changes Feb 1, 2022

View reviewed changes

JGuinegagne requested changes Feb 2, 2022

View reviewed changes

evakravi reviewed Feb 2, 2022

View reviewed changes

fix: use request url so that we dont have to pass aws creds

f94245c

bencrabtree requested review from JGuinegagne and evakravi February 2, 2022 15:23

JGuinegagne approved these changes Feb 2, 2022

View reviewed changes

easyj2j added the type: documentation label Feb 2, 2022

jeniyat and others added 2 commits February 2, 2022 16:24

fix: update to incorporate black v22, pin tox versions (aws#2889)

90b0b0f

Co-authored-by: Mufaddal Rohawala <[email protected]>

Merge branch 'dev' into feat/jumpstart-docs-model-table

858a0dc

mufaddal-rohawala and others added 3 commits February 3, 2022 20:39

fix: pin pytest-xdist to avoid release failures (aws#2896)

0b6fb78

fix: jumpstart typo (aws#2895)

57ade33

mufaddal-rohawala force-pushed the dev branch from 4fd04f1 to e3398d9 Compare February 4, 2022 04:44

Merge branch 'dev' of https://github.com/aws/sagemaker-python-sdk int…

f05b04e

…o feat/jumpstart-docs-model-table

evakravi approved these changes Feb 4, 2022

View reviewed changes

mufaddal-rohawala force-pushed the dev branch 2 times, most recently from 96c01d7 to c437191 Compare February 5, 2022 00:08

fix: update link and some logic

7932372

fix: linting issue

11aea41

fix: lint

44ad05e

jeniyat force-pushed the dev branch from 9d612a7 to bb7563f Compare February 7, 2022 23:40

mufaddal-rohawala reviewed Feb 7, 2022

View reviewed changes

mufaddal-rohawala approved these changes Feb 7, 2022

View reviewed changes

shreyapandit approved these changes Feb 8, 2022

View reviewed changes

shreyapandit changed the title ~~doc: added support for generation of jumpstart model table on build~~ doc: Support for generation of Jumpstart model table on build Feb 8, 2022

shreyapandit merged commit 7ea4b80 into aws:dev Feb 8, 2022


		f = open("jumpstart/jumpstart.rst", "w")

		f.write("==================================\n")

doc: Support for generation of Jumpstart model table on build #2888

doc: Support for generation of Jumpstart model table on build #2888

Uh oh!

Conversation

bencrabtree commented Feb 1, 2022

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Feb 1, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 1, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 1, 2022

AWS CodeBuild CI Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Feb 1, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 1, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 1, 2022

AWS CodeBuild CI Report

Uh oh!

JGuinegagne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Feb 2, 2022

AWS CodeBuild CI Report

Uh oh!

JGuinegagne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Feb 2, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 2, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 3, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 3, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 3, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 3, 2022

AWS CodeBuild CI Report