feature: Support for SageMaker Intelligent Defaults (#3757)

* feature: Added Config parser for SageMaker Python SDK (#840)

Co-authored-by: Balaji Sankar <[email protected]>

* intelligent defaults - tags and encryption (#842)

* feature: sagemaker config - support tags for all APIs

* feature: sagemaker config - support EnableInterContainerTrafficEncryption for relevant APIs

---------

Co-authored-by: Ruban Hussain <[email protected]>

* intelligent defaults - custom parameters and small fixes (#845)

* fix: sagemaker-config - S3 session, tuning tags, config schema test side-effects

* feature: sagemaker-config - support for custom parameters in config schema

---------

Co-authored-by: Ruban Hussain <[email protected]>

* feature: Added support for VPC Config, EnableNetworkIsolation, KMS Key ID, Volume KMS Key ID, IAM role to be fetched from Config (#846)

Co-authored-by: Balaji Sankar <[email protected]>

* fix: Make Key, Value as required fields for each "Tags" entry in the config file.

* fix: Make 'role' as Optional for ModelQualityMonitor and DefaultModelMonitor, and fixed PROCESSING_CONFIG_PATH (#849)

Co-authored-by: Balaji Sankar <[email protected]>

* Fix: Certain unit tests aren't passing sagemaker_session. Modify the logic to accommodate that case (#850)

Co-authored-by: Balaji Sankar <[email protected]>

* fix: Sagemaker Config - KeyError: 'MonitoringJobDefinition' in model_monitoring

* change: Sagemaker Config - improved readability of print statements and simplified its code

* fix: Sagemaker Config - Reduce duplicate and misleading config-related print statements

* fix: Sagemaker Config - add function description

* fix: Sagemaker Config - Fix failing Integ tests, fix backwards incompatible behavior, and improved some unit tests

* change: new integ test for sagemaker_config

* fix: Sagemaker Config - fleshed out unit tests and fixed bugs

* fix: Sagemaker Config - Removed hard coded config values in the unit tests

* fix: inject from config into existing ProductionVariants inside create_endpoint_config_from_existing

* change: added unit test for verifying yaml safe_load method

* change: addressed PR comments for SageMaker Config

* change: Sagemaker Config - minor clarification

* change: ModelMonitoring and Processing now use helper methods for updating NetworkConfig

* change: Refactoring session.py and added additional schema validation for ValidationProfiles

* update: expand one unit test

* update: new integ test for cross context injection

* change: remove unwanted method and replace it with a different method for config injection

* fix: Address documentation errors and removed unnecessary properties and setters

* fix: moving certain config file helper methods to utils.py

* change: Add a separate helper to merge list of objects

* fix: Documentation updates for SageMakerConfig

* fix: bubble up exceptions from S3 while fetching the Config

* fix: Added additional test cases for config helper methods. Also made minor documentation updates.

* fix: small bug fix to print statements for update_list_of_dicts_with_values_from_config

* fix: Replace SageMakerConfig class with just method invocations

* fix: fix broken unit tests due to refactoring

* fix: bug where a user-provided sagemaker_config wasnt set

* change: rename fetch_sagemaker_config to load_sagemaker_config

* fix: update Schema to match exactly with APIs

* add documentation for default configuration support

* fix linting errors

* fix link lint

* fix lint

---------

Co-authored-by: Balaji Sankar <[email protected]>
Co-authored-by: Balaji Sankar <[email protected]>
Co-authored-by: Ruban Hussain <[email protected]>
Co-authored-by: Ivy Bazan <[email protected]>

Author: 5 people

doc/api/utility/config.rst

@@ -0,0 +1,7 @@
+Config
+-------
+
+.. automodule:: sagemaker.config.config
+    :members:
+    :undoc-members:
+    :show-inheritance:

doc/overview.rst

@@ -59,6 +59,9 @@ def read_requirements(filename):
     "pandas",
     "pathos",
     "schema",
+    "PyYAML==5.4.1",
+    "jsonschema",
+    "platformdirs",
 ]
 
 # Specific use case dependencies

setup.py

@@ -46,7 +46,7 @@ class AlgorithmEstimator(EstimatorBase):
     def __init__(
         self,
         algorithm_arn: str,
-        role: str,
+        role: str = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         volume_size: Union[int, PipelineVariable] = 30,

src/sagemaker/algorithm.py

@@ -50,7 +50,7 @@ class AmazonAlgorithmEstimatorBase(EstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         data_location: Optional[str] = None,

src/sagemaker/amazon/amazon_estimator.py

@@ -87,7 +87,7 @@ class FactorizationMachines(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         num_factors: Optional[int] = None,
@@ -326,7 +326,7 @@ class FactorizationMachinesModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/factorization_machines.py

@@ -63,7 +63,7 @@ class IPInsights(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         num_entity_vectors: Optional[int] = None,
@@ -229,7 +229,7 @@ class IPInsightsModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/ipinsights.py

@@ -62,7 +62,7 @@ class KMeans(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         k: Optional[int] = None,
@@ -255,7 +255,7 @@ class KMeansModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/kmeans.py

@@ -73,7 +73,7 @@ class KNN(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         k: Optional[int] = None,
@@ -246,7 +246,7 @@ class KNNModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/knn.py

@@ -52,7 +52,7 @@ class LDA(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         num_topics: Optional[int] = None,
         alpha0: Optional[float] = None,
@@ -230,7 +230,7 @@ class LDAModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/lda.py

@@ -145,7 +145,7 @@ class LinearLearner(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         predictor_type: Optional[str] = None,
@@ -499,7 +499,7 @@ class LinearLearnerModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/linear_learner.py

@@ -74,7 +74,7 @@ class NTM(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         num_topics: Optional[int] = None,
@@ -263,7 +263,7 @@ class NTMModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/ntm.py

@@ -153,7 +153,7 @@ class Object2Vec(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         epochs: Optional[int] = None,
@@ -361,7 +361,7 @@ class Object2VecModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/object2vec.py

@@ -60,7 +60,7 @@ class PCA(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         num_components: Optional[int] = None,
@@ -243,7 +243,7 @@ class PCAModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/pca.py

@@ -54,7 +54,7 @@ class RandomCutForest(AmazonAlgorithmEstimatorBase):
 
     def __init__(
         self,
-        role: str,
+        role: Optional[Union[str, PipelineVariable]] = None,
         instance_count: Optional[Union[int, PipelineVariable]] = None,
         instance_type: Optional[Union[str, PipelineVariable]] = None,
         num_samples_per_tree: Optional[int] = None,
@@ -216,7 +216,7 @@ class RandomCutForestModel(Model):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
+        role: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         **kwargs
     ):

src/sagemaker/amazon/randomcutforest.py

@@ -19,9 +19,16 @@
 
 from sagemaker import Model, PipelineModel
 from sagemaker.automl.candidate_estimator import CandidateEstimator
+from sagemaker.config import (
+    AUTO_ML_ROLE_ARN_PATH,
+    AUTO_ML_KMS_KEY_ID_PATH,
+    AUTO_ML_VPC_CONFIG_PATH,
+    AUTO_ML_VOLUME_KMS_KEY_ID_PATH,
+    AUTO_ML_INTER_CONTAINER_ENCRYPTION_PATH,
+)
 from sagemaker.job import _Job
 from sagemaker.session import Session
-from sagemaker.utils import name_from_base
+from sagemaker.utils import name_from_base, resolve_value_from_config
 from sagemaker.workflow.entities import PipelineVariable
 from sagemaker.workflow.pipeline_context import runnable_by_pipeline
 
@@ -98,15 +105,15 @@ class AutoML(object):
 
     def __init__(
         self,
-        role: str,
-        target_attribute_name: str,
+        role: Optional[str] = None,
+        target_attribute_name: str = None,
         output_kms_key: Optional[str] = None,
         output_path: Optional[str] = None,
         base_job_name: Optional[str] = None,
         compression_type: Optional[str] = None,
         sagemaker_session: Optional[Session] = None,
         volume_kms_key: Optional[str] = None,
-        encrypt_inter_container_traffic: Optional[bool] = False,
+        encrypt_inter_container_traffic: Optional[bool] = None,
         vpc_config: Optional[Dict[str, List]] = None,
         problem_type: Optional[str] = None,
         max_candidates: Optional[int] = None,
@@ -176,14 +183,10 @@ def __init__(
         Returns:
             AutoML object.
         """
-        self.role = role
-        self.output_kms_key = output_kms_key
         self.output_path = output_path
         self.base_job_name = base_job_name
         self.compression_type = compression_type
-        self.volume_kms_key = volume_kms_key
         self.encrypt_inter_container_traffic = encrypt_inter_container_traffic
-        self.vpc_config = vpc_config
         self.problem_type = problem_type
         self.max_candidate = max_candidates
         self.max_runtime_per_training_job_in_seconds = max_runtime_per_training_job_in_seconds
@@ -204,6 +207,31 @@ def __init__(
         self._auto_ml_job_desc = None
         self._best_candidate = None
         self.sagemaker_session = sagemaker_session or Session()
+        self.vpc_config = resolve_value_from_config(
+            vpc_config, AUTO_ML_VPC_CONFIG_PATH, sagemaker_session=self.sagemaker_session
+        )
+        self.volume_kms_key = resolve_value_from_config(
+            volume_kms_key, AUTO_ML_VOLUME_KMS_KEY_ID_PATH, sagemaker_session=self.sagemaker_session
+        )
+        self.output_kms_key = resolve_value_from_config(
+            output_kms_key, AUTO_ML_KMS_KEY_ID_PATH, sagemaker_session=self.sagemaker_session
+        )
+        self.role = resolve_value_from_config(
+            role, AUTO_ML_ROLE_ARN_PATH, sagemaker_session=self.sagemaker_session
+        )
+        if not self.role:
+            # Originally IAM role was a required parameter.
+            # Now we marked that as Optional because we can fetch it from SageMakerConfig
+            # Because of marking that parameter as optional, we should validate if it is None, even
+            # after fetching the config.
+            raise ValueError("An AWS IAM role is required to create an AutoML job.")
+
+        self.encrypt_inter_container_traffic = resolve_value_from_config(
+            direct_input=encrypt_inter_container_traffic,
+            config_path=AUTO_ML_INTER_CONTAINER_ENCRYPTION_PATH,
+            default_value=False,
+            sagemaker_session=self.sagemaker_session,
+        )
 
         self._check_problem_type_and_job_objective(self.problem_type, self.job_objective)
 
@@ -276,6 +304,8 @@ def attach(cls, auto_ml_job_name, sagemaker_session=None):
             volume_kms_key=auto_ml_job_desc.get("AutoMLJobConfig", {})
             .get("SecurityConfig", {})
             .get("VolumeKmsKeyId"),
+            # Do not override encrypt_inter_container_traffic from config because this info
+            # is pulled from an existing automl job
             encrypt_inter_container_traffic=auto_ml_job_desc.get("AutoMLJobConfig", {})
             .get("SecurityConfig", {})
             .get("EnableInterContainerTrafficEncryption", False),

src/sagemaker/automl/automl.py

@@ -14,10 +14,14 @@
 from __future__ import absolute_import
 
 from six import string_types
-
-from sagemaker import Session
+from sagemaker.config import (
+    TRAINING_JOB_VPC_CONFIG_PATH,
+    TRAINING_JOB_VOLUME_KMS_KEY_ID_PATH,
+    TRAINING_JOB_INTER_CONTAINER_ENCRYPTION_PATH,
+)
+from sagemaker.session import Session
 from sagemaker.job import _Job
-from sagemaker.utils import name_from_base
+from sagemaker.utils import name_from_base, resolve_value_from_config
 
 
 class CandidateEstimator(object):
@@ -72,7 +76,8 @@ def fit(
         inputs,
         candidate_name=None,
         volume_kms_key=None,
-        encrypt_inter_container_traffic=False,
+        # default of False for training job, checked inside function
+        encrypt_inter_container_traffic=None,
         vpc_config=None,
         wait=True,
         logs=True,
@@ -87,7 +92,8 @@ def fit(
             volume_kms_key (str): The KMS key id to encrypt data on the storage volume attached to
                 the ML compute instance(s).
             encrypt_inter_container_traffic (bool): To encrypt all communications between ML compute
-                instances in distributed training. Default: False.
+                instances in distributed training. If not passed, will be fetched from
+                sagemaker_config if a value is defined there. Default: False.
             vpc_config (dict): Specifies a VPC that jobs and hosted models have access to.
                 Control access to and from training and model containers by configuring the VPC
             wait (bool): Whether the call should wait until all jobs completes (default: True).
@@ -99,7 +105,14 @@ def fit(
                 """Logs can only be shown if wait is set to True.
                 Please either set wait to True or set logs to False."""
             )
-
+        vpc_config = resolve_value_from_config(
+            vpc_config, TRAINING_JOB_VPC_CONFIG_PATH, sagemaker_session=self.sagemaker_session
+        )
+        volume_kms_key = resolve_value_from_config(
+            volume_kms_key,
+            TRAINING_JOB_VOLUME_KMS_KEY_ID_PATH,
+            sagemaker_session=self.sagemaker_session,
+        )
         self.name = candidate_name or self.name
         running_jobs = {}
 
@@ -131,12 +144,22 @@ def fit(
                 base_name = "sagemaker-automl-training-rerun"
                 step_name = name_from_base(base_name)
                 step["name"] = step_name
+
+                # Check training_job config not auto_ml_job config because this function calls
+                # training job API
+                _encrypt_inter_container_traffic = resolve_value_from_config(
+                    direct_input=encrypt_inter_container_traffic,
+                    config_path=TRAINING_JOB_INTER_CONTAINER_ENCRYPTION_PATH,
+                    default_value=False,
+                    sagemaker_session=self.sagemaker_session,
+                )
+
                 train_args = self._get_train_args(
                     desc,
                     channels,
                     step_name,
                     volume_kms_key,
-                    encrypt_inter_container_traffic,
+                    _encrypt_inter_container_traffic,
                     vpc_config,
                 )
                 self.sagemaker_session.train(**train_args)

src/sagemaker/automl/candidate_estimator.py

@@ -82,8 +82,8 @@ class ChainerModel(FrameworkModel):
     def __init__(
         self,
         model_data: Union[str, PipelineVariable],
-        role: str,
-        entry_point: str,
+        role: Optional[str] = None,
+        entry_point: Optional[str] = None,
         image_uri: Optional[Union[str, PipelineVariable]] = None,
         framework_version: Optional[str] = None,
         py_version: Optional[str] = None,