fix: Disable debugger when checkpointing is enabled with distributed training #2264

NihalHarish · 2021-04-01T23:02:55Z

Issue #, if available:

Description of changes:

update _validate_and_set_debugger_configs to disable debugger when checkpointing is enabled with distributed training

Testing done:

Added new unit test

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…ributed_trainig

sagemaker-bot · 2021-04-01T23:05:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: fd9dd85
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T23:07:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 58d6913
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ndodda-amazon

Changes look good, left some comments

src/sagemaker/estimator.py

ndodda-amazon · 2021-04-01T23:10:41Z

src/sagemaker/estimator.py

@@ -2219,7 +2219,20 @@ def _validate_and_set_debugger_configs(self):
        ):
            self.debugger_hook_config = DebuggerHookConfig(s3_output_path=self.output_path)
        elif not self.debugger_hook_config:
-            self.debugger_hook_config = None
+            self.debugger_hook_config = False


Why are we changing this variable to False? Is this intentional? If so, why?

The intent of the line above is to disable the debug hook config.
False is a more explicit indicator of that than None

None is already a falsy value in Python.

It doesn't make sense to change this to a boolean, since the debugger_hook_config variable can hold a DebuggerHookConfig type. You can simply check if debugger_hook_config is None instead of checking if it's False.

Documentation however indicates to customers that they need to pass False to opt-out of debugger.

None is the default value of this variable before we reach this code-block.

Debugger is initialized on behalf of the customer it is None

I'm making sure that when we check the value of post the _validate function call, we're indicative that we want debugger off instead of None which might come to mean that it has not yet been initialized or explicitly set by the customer.

Yes, you're right:

sagemaker-python-sdk/src/sagemaker/tensorflow/estimator.py

Lines 361 to 365 in b66cb98

elif self.debugger_hook_config is None and fw._region_supports_debugger(

self.sagemaker_session.boto_session.region_name

):

# Set defaults for debugging.

self.debugger_hook_config = DebuggerHookConfig(s3_output_path=self.output_path)

Thank you for clarifying.

sagemaker-bot · 2021-04-01T23:37:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: fd9dd85
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T23:41:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: fd9dd85
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T23:42:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 58d6913
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T23:43:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: fd9dd85
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T23:44:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 58d6913
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-01T23:45:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 58d6913
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…ig' of https://github.com/NihalHarish/sagemaker-python-sdk into debug_disable_with_checkpointing_with_distributed_trainig

sagemaker-bot · 2021-04-02T00:14:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 58d6913
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T00:31:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2fa1856
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T01:03:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2fa1856
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T01:11:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2fa1856
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T01:15:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2fa1856
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T01:25:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2b0944e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ndodda-amazon · 2021-04-02T01:26:52Z

tests/integ/test_debugger.py

+        # Debug Hook should be disabled
+        assert mx.debugger_hook_config is False
+
+        # Estimator with checkpointing enabled and Model Parallel Enabled


nit: comment is wrong, this is Data Parallel

Nice catch, fixed.

ndodda-amazon

LGTM! Left a minor comment but otherwise changes look good.

sagemaker-bot · 2021-04-02T01:41:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 582b64d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T01:52:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2b0944e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T01:57:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2b0944e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T01:57:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2b0944e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ahsan-z-khan · 2021-04-02T02:09:41Z

src/sagemaker/estimator.py

@@ -2219,7 +2219,20 @@ def _validate_and_set_debugger_configs(self):
        ):
            self.debugger_hook_config = DebuggerHookConfig(s3_output_path=self.output_path)
        elif not self.debugger_hook_config:


Just to understand it, so it's possible for self.debugger_hook_config to be None or False. And you're changing it to False intentionally to disable it?

The default value of self.debugger_hook_config is None.
The if-statement above checks if the debugger is supported in the region in which it is invoked and if not it should be set to False instead of None in my opinion.

sagemaker-bot · 2021-04-02T02:11:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 582b64d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T02:13:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 582b64d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T03:45:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4ab76aa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T04:08:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 4ab76aa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T04:13:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4ab76aa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T04:15:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4ab76aa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T04:18:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4ab76aa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

tests/integ/test_debugger.py

…ributed_trainig

sagemaker-bot · 2021-04-02T19:05:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c9cd18e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T19:06:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 3b2338d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T19:29:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c9cd18e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T19:31:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 3b2338d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T19:34:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 3b2338d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T19:36:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 3b2338d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T19:37:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c9cd18e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T19:38:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c9cd18e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T20:18:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 3b2338d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-04-02T21:34:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 3b2338d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…training (aws#2264)

NihalHarish added 2 commits April 1, 2021 16:01

disable debugger when checkpointing is enabled

fd9dd85

Merge branch 'master' into debug_disable_with_checkpointing_with_dist…

58d6913

…ributed_trainig

ndodda-amazon suggested changes Apr 1, 2021

View reviewed changes

NihalHarish added 2 commits April 1, 2021 16:51

lint

651b662

Merge branch 'debug_disable_with_checkpointing_with_distributed_train…

2fa1856

…ig' of https://github.com/NihalHarish/sagemaker-python-sdk into debug_disable_with_checkpointing_with_distributed_trainig

NihalHarish changed the title ~~[WIP] Disable debugger when checkpointing is enabled with distributed training~~ Disable debugger when checkpointing is enabled with distributed training Apr 2, 2021

ignore xgboost estimator

2b0944e

ndodda-amazon reviewed Apr 2, 2021

View reviewed changes

ndodda-amazon previously approved these changes Apr 2, 2021

View reviewed changes

fix comment

582b64d

NihalHarish dismissed ndodda-amazon’s stale review via 582b64d April 2, 2021 01:30

ndodda-amazon previously approved these changes Apr 2, 2021

View reviewed changes

ahsan-z-khan reviewed Apr 2, 2021

View reviewed changes

update TF child function

4ab76aa

NihalHarish dismissed ndodda-amazon’s stale review via 4ab76aa April 2, 2021 03:35

ahsan-z-khan reviewed Apr 2, 2021

View reviewed changes

tests/integ/test_debugger.py Outdated Show resolved Hide resolved

NihalHarish changed the title ~~Disable debugger when checkpointing is enabled with distributed training~~ [fix] Disable debugger when checkpointing is enabled with distributed training Apr 2, 2021

NihalHarish added 2 commits April 2, 2021 11:55

use latest tf version

c9cd18e

Merge branch 'master' into debug_disable_with_checkpointing_with_dist…

3b2338d

…ributed_trainig

ajaykarpur changed the title ~~[fix] Disable debugger when checkpointing is enabled with distributed training~~ fix: Disable debugger when checkpointing is enabled with distributed training Apr 2, 2021

ajaykarpur approved these changes Apr 2, 2021

View reviewed changes

ahsan-z-khan approved these changes Apr 2, 2021

View reviewed changes

ahsan-z-khan merged commit 709bb18 into aws:master Apr 2, 2021

TEChopra1000 pushed a commit to TEChopra1000/sagemaker-python-sdk that referenced this pull request Apr 6, 2021

fix: Disable debugger when checkpointing is enabled with distributed …

157f9b3

…training (aws#2264)

NihalHarish mentioned this pull request Apr 9, 2021

Disable Debugger When Checkpointing Is Enabled #2260

Closed

7 tasks

	elif self.debugger_hook_config is None and fw._region_supports_debugger(
	self.sagemaker_session.boto_session.region_name
	):
	# Set defaults for debugging.
	self.debugger_hook_config = DebuggerHookConfig(s3_output_path=self.output_path)

fix: Disable debugger when checkpointing is enabled with distributed training #2264

fix: Disable debugger when checkpointing is enabled with distributed training #2264

Uh oh!

Conversation

NihalHarish commented Apr 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

ndodda-amazon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ndodda-amazon Apr 1, 2021

Choose a reason for hiding this comment

Uh oh!

NihalHarish Apr 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajaykarpur Apr 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NihalHarish Apr 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajaykarpur Apr 2, 2021

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 1, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 2, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 2, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 2, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 2, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 2, 2021

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 2, 2021

AWS CodeBuild CI Report

Uh oh!

ndodda-amazon Apr 2, 2021

Choose a reason for hiding this comment

Uh oh!

NihalHarish Apr 2, 2021

Choose a reason for hiding this comment

Uh oh!

NihalHarish commented Apr 1, 2021 •

edited

Loading

NihalHarish Apr 1, 2021 •

edited

Loading

ajaykarpur Apr 2, 2021 •

edited

Loading

NihalHarish Apr 2, 2021 •

edited

Loading