Disable Debugger When Checkpointing Is Enabled #2260

NihalHarish · 2021-03-31T22:00:05Z

Issue #, if available:

Description of changes:

We do not want debugger to run in the background if customers enable checkpointing.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2021-03-31T22:02:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 336475f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ndodda-amazon

Changes look good, left a few minor comments.

ndodda-amazon · 2021-03-31T22:19:53Z

src/sagemaker/estimator.py

@@ -2147,6 +2147,11 @@ def __init__(
        self._hyperparameters = hyperparameters or {}
        self.checkpoint_s3_uri = checkpoint_s3_uri
        self.checkpoint_local_path = checkpoint_local_path
+
+        # Disable debugger if checkpointing is enabled by the customer
+        self.debugger_hook_config = \


If a user specifies a custom hook config while checkpointing is enabled, should we log or print a message saying that debugger is disabled?

ndodda-amazon · 2021-03-31T22:27:49Z

tests/integ/test_debugger.py

+            path=os.path.join(data_path, "test"), key_prefix="integ-test-data/mxnet_mnist/test"
+        )
+
+        mx.fit({"train": train_input, "test": test_input})


Do we actually need to train to test the behavior? Can't we simply inspect the training params after the estimator is created? Something like

assert "DebugHookConfig" not in mx.describe()`

Updated the test to not run an end-to-end training job

sagemaker-bot · 2021-03-31T22:35:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 336475f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T22:41:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 336475f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T22:45:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 336475f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T22:45:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 336475f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T23:16:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 347d6b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ndodda-amazon

LGTM, thanks for doing this!

sagemaker-bot · 2021-03-31T23:18:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dc67287
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T23:34:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ce002f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T23:48:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 347d6b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T23:53:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: dc67287
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T23:53:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 347d6b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T23:55:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 347d6b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T23:55:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dc67287
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-31T23:58:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dc67287
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T00:04:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: ce002f4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T00:05:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: ce002f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T00:05:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: ce002f4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T00:58:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: ce002f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/estimator.py

sagemaker-bot · 2021-04-02T00:05:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 28b4073
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T00:31:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 28b4073
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T00:36:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 28b4073
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T00:37:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 28b4073
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T00:45:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 28b4073
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ajaykarpur · 2021-04-02T19:05:00Z

src/sagemaker/estimator.py

+        # Disable debugger if checkpointing is enabled by the customer
+        if self.checkpoint_s3_uri and self.checkpoint_local_path and self.debugger_hook_config:
+            logger.info(
+                "SM Debug Does Not Currently Support Training Jobs With Checkpointing Enabled"


nit: Can you make this sentence case instead of capitalizing each word?

NihalHarish · 2021-04-09T20:32:50Z

Closing this since the duplicate was already merged with #2264

disable debugger when checkpointing is enabled

336475f

ndodda-amazon suggested changes Mar 31, 2021

View reviewed changes

NihalHarish changed the title ~~Disable Debugger When Checkpointing Is Enabled~~ [Draft] Disable Debugger When Checkpointing Is Enabled Mar 31, 2021

NihalHarish added 2 commits March 31, 2021 16:14

refactor + add log + update test

347d6b2

remove whitespace changes

dc67287

ndodda-amazon previously approved these changes Mar 31, 2021

View reviewed changes

lint

ce002f4

NihalHarish dismissed ndodda-amazon’s stale review via ce002f4 March 31, 2021 23:24

NihalHarish changed the title ~~[Draft] Disable Debugger When Checkpointing Is Enabled~~ Disable Debugger When Checkpointing Is Enabled Mar 31, 2021

NihalHarish changed the title ~~Disable Debugger When Checkpointing Is Enabled~~ [WIP] Disable Debugger When Checkpointing Is Enabled Apr 1, 2021

ajaykarpur reviewed Apr 1, 2021

View reviewed changes

src/sagemaker/estimator.py Show resolved Hide resolved

NihalHarish changed the title ~~[WIP] Disable Debugger When Checkpointing Is Enabled~~ Disable Debugger When Checkpointing Is Enabled Apr 1, 2021

Merge branch 'master' into debug_disable_with_checkpointing

28b4073

ajaykarpur approved these changes Apr 2, 2021

View reviewed changes

NihalHarish closed this Apr 9, 2021

Disable Debugger When Checkpointing Is Enabled #2260

Disable Debugger When Checkpointing Is Enabled #2260

Uh oh!

Conversation

NihalHarish commented Mar 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

ndodda-amazon left a comment

Choose a reason for hiding this comment

Uh oh!

ndodda-amazon Mar 31, 2021

Choose a reason for hiding this comment

Uh oh!

NihalHarish Mar 31, 2021

Choose a reason for hiding this comment

Uh oh!

ndodda-amazon Mar 31, 2021

Choose a reason for hiding this comment

Uh oh!

NihalHarish Mar 31, 2021

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

ndodda-amazon left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Mar 31, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

NihalHarish commented Mar 31, 2021 •

edited

Loading