Implement common hooks in core (aws#212)

* Implement common hooks in core

* combine abstract and static methods

* rename method

* fix variable name

* fix test

* fix test

* Fix loss decreasing test

* remove data

* Reduce length of tests, also removed unnecessary s3 test

* Fix test

* Delete test.py

* Address review by Edward

* Call super constructor immediately

* Reduce logging in CI and fix tests

* Fix bug and reduce verbosity in CI

* Trying out CI with loss test disabled

* Update hook.py

* Disable unreliable test

* use new collection file name

* fix args for method

* create coll by default

* fix pytorch tests

* Add xgboost tests

* remove exception catching

* remove extra import

* Fix all names of collections to use CollectionKeys, and create collection in a safe way for when json parser creates collections

* Cleanup

* Rename input, and add xgboost to CI evn

* fix bug of tuple name of dict

* Add couple of docstrings

* Address renamed cleanup method

* Rename bias to biases

* cleanup removal of dir code

* Remove save manager

* docs for loss not decreasing

* rename reduction method

* pass out dir to test

* fix test

* use coll manager to add tensors

* change a loop and some imports

Author: rahul003

docs/mxnet/README.md

@@ -269,7 +269,7 @@ tm.get_collection("ReluActivation").set_reduction_config(ReductionConfig(reducti
 tm.get_collection("flatten").include(["flatten*"])
 tm.get_collection("flatten").set_save_config(SaveConfig(save_steps=[4,5,6]))
 tm.get_collection("flatten").set_reduction_config(ReductionConfig(norms=["l1"], abs_norms=["l2"]))
-hook = TornasoleHook(out_dir=out_dir, include_collections=['weights', 'bias','gradients',
+hook = TornasoleHook(out_dir=out_dir, include_collections=['weights', 'biases','gradients',
                                                     'default', 'ReluActivation', 'flatten'])
 ```
 
@@ -280,7 +280,7 @@ Refer [API](api.md) for a list of the reductions available as well as examples.
 
 There are different ways to save tensors when using Tornasole.
 Tornasole provides easy ways to save certain standard tensors by way of default collections (a Collection represents a group of tensors).
-Examples of such collections are 'weights', 'gradients', 'bias' and 'default'.
+Examples of such collections are 'weights', 'gradients', 'biases' and 'default'.
 Besides the tensors in above default collections, you can save tensors by name or regex patterns on those names.
 Users can also specify a certain block in the model to save the inputs and outputs of that block.
 This section will take you through these ways in more detail.
@@ -289,7 +289,7 @@ This section will take you through these ways in more detail.
 The TornasoleHook API supports _include\_regex_ parameter. The users can specify a regex pattern with this pattern. The TornasoleHook will store the tensors that match with the specified regex pattern. With this approach, users can store the tensors without explicitly creating a Collection object. The specified regex pattern will be associated with 'default' Collection and the SaveConfig object that is associated with the 'default' collection.
 
 #### Default Collections
-Currently, the tornasole\_mxnet hook creates Collection objects for 'weights', 'gradients', 'bias' and 'default'. These collections contain the regex pattern that match with tensors of type weights, gradient and bias. The regex pattern for the 'default' collection is set when user specifies _include\_regex_ with TornasoleHook or sets the _SaveAll=True_.  These collections use the SaveConfig parameter provided with the TornasoleHook initialization. The TornasoleHook will store the related tensors, if user does not specify any special collection with _include\_collections_ parameter. If user specifies a collection with _include\_collections_ the above default collections will not be in effect.
+Currently, the tornasole\_mxnet hook creates Collection objects for 'weights', 'gradients', 'biases' and 'default'. These collections contain the regex pattern that match with tensors of type weights, gradient and bias. The regex pattern for the 'default' collection is set when user specifies _include\_regex_ with TornasoleHook or sets the _SaveAll=True_.  These collections use the SaveConfig parameter provided with the TornasoleHook initialization. The TornasoleHook will store the related tensors, if user does not specify any special collection with _include\_collections_ parameter. If user specifies a collection with _include\_collections_ the above default collections will not be in effect.
 
 #### Custom Collections
 You can also create any other customized collection yourself.
@@ -424,7 +424,7 @@ def create_tornasole_hook(output_s3_uri, block):
     # In order to log the inputs and output of a model, we will create a collection as follows:
     tm.get_collection('TopBlock').add_block_tensors(block, inputs=True, outputs=True)
     # Create a hook that logs weights, biases, gradients and inputs outputs of model while training.
-    hook = TornasoleHook(out_dir=output_s3_uri, save_config=save_config, include_collections=['weights', 'gradients', 'bias','TopBlock'])
+    hook = TornasoleHook(out_dir=output_s3_uri, save_config=save_config, include_collections=['weights', 'gradients', 'biases','TopBlock'])
     return hook
 ```
 
@@ -503,7 +503,7 @@ def create_tornasole_hook(output_s3_uri, block):
     tm.get_collection(block.name).add_block_tensors(block, inputs=True, outputs=True)
     # Create a hook that logs weights, biases, gradients and inputs outputs of model while training.
     hook = TornasoleHook(out_dir=output_s3_uri, save_config=save_config, include_collections=[
-        'weights', 'gradients', 'bias', block.name])
+        'weights', 'gradients', 'biases', block.name])
     return hook
 ```
 

docs/mxnet/api.md

@@ -56,7 +56,7 @@ TornasoleHook is the entry point for Tornasole into your program.
 
     include_collections: list of str
         takes as input the names of collections which should be saved.
-        by default, ['weights','gradients', 'bias', 'default'] are passed to include_collections.
+        by default, ['weights','gradients', 'biases', 'default'] are passed to include_collections.
 
     save_all: bool
         a shortcut for saving all tensors in the model.
@@ -69,7 +69,7 @@ TornasoleHook is the entry point for Tornasole into your program.
         reduction_config=None,
         save_config=SaveConfig(save_interval=100),
         include_regex=None,
-        include_collections=['weights', 'gradients', 'bias', 'default'],
+        include_collections=['weights', 'gradients', 'biases', 'default'],
         save_all=False,
         ):
 ```

docs/pytorch/README.md

@@ -401,7 +401,7 @@ def create_tornasole_hook(output_dir, module):
 
     # Create a hook that logs weights, biases, gradients and inputs outputs of model while training.
     hook = TornasoleHook(out_dir=output_dir, save_config=SaveConfig(save_steps=[i * 10 for i in range(5)]),
-			    include_collections=['weights', 'gradients', 'bias','l_mod'])
+			    include_collections=['weights', 'gradients', 'biases','l_mod'])
 ```
 
 Here is how to register the above hook.

docs/pytorch/api.md

@@ -73,7 +73,7 @@ A class used to represent the hook which gets attached to the
                  reduction_config=None,
                  save_config=default_save_config(),
                  include_regex=None,
-                 include_collections=['weights', 'bias', 'gradients', 'default'],
+                 include_collections=['weights', 'biases', 'gradients', 'default'],
                  save_all=False):
 ```
 ### Collection

docs/rules/FirstPartyRules.md

@@ -73,7 +73,7 @@ For this rule, users must specify either the `collection_names` or `tensor_regex
 
 ```
 from tornasole.rules.generic import AllZero
-collections = ['weights', 'bias']
+collections = ['weights', 'biases']
 tensor_regex = ['input*']
 allzero = AllZero(base_trial=trial_obj, collection_names=collections, tensor_regex=tensor_regex)
 ```
@@ -133,6 +133,10 @@ for step 21 is compared with the loss for step 9. The next step where loss is ch
 since 10 steps after 21 is 31, and at 31 and 32 loss is not being saved.
 - `diff_percent`: float (default is 0.0)  (between 0.0 and 100.0)
 The minimum difference in percentage that loss should be lower by. By default, the rule just checks if loss is going down. If you want to specify a stricter check that loss is going down fast enough, you might want to pass diff_percent.
+- `mode`: string
+The name of tornasole mode to query tensor values for rule checking. 
+If this is not passed, the rule checks for eval mode, then training mode and then global mode in this order.
+        
 
 ```
 from tornasole.rules.generic import LossNotDecreasing

examples/mxnet/scripts/mnist_gluon_all_zero_demo.py

@@ -113,7 +113,7 @@ def create_tornasole_hook(output_s3_uri):
 
     # Create a hook that logs weights, biases and gradients while training the model.
     hook = TornasoleHook(out_dir=output_s3_uri, save_config=save_config,
-                         include_collections=['ReluActivation','weights', 'bias','gradients'])
+                         include_collections=['ReluActivation','weights', 'biases','gradients'])
     return hook
 
 

examples/mxnet/scripts/mnist_gluon_basic_hook_demo.py

@@ -111,7 +111,7 @@ def create_tornasole_hook(output_s3_uri):
 
     # Create a hook that logs weights, biases and gradients while training the model.
     hook = TornasoleHook(out_dir=output_s3_uri, save_config=save_config, include_collections=['weights', 'gradients',
-                                                                                              'bias'])
+                                                                                              'biases'])
     return hook
 
 

examples/mxnet/scripts/mnist_gluon_block_input_output_demo.py

@@ -119,7 +119,7 @@ def create_tornasole_hook(output_s3_uri, block):
 
     # Create a hook that logs weights, biases, gradients and inputs outputs of model while training.
     hook = TornasoleHook(out_dir=output_s3_uri, save_config=save_config, include_collections=[
-        'weights', 'gradients', 'bias', block.name])
+        'weights', 'gradients', 'biases', block.name])
     return hook
 
 

examples/mxnet/scripts/mnist_gluon_model_input_output_demo.py

@@ -106,7 +106,7 @@ def create_tornasole_hook(output_s3_uri, block):
     tm.get_collection('TopBlock').add_block_tensors(block, inputs=True, outputs=True)
 
     # Create a hook that logs weights, biases, gradients and inputs outputs of model while training.
-    hook = TornasoleHook(out_dir=output_s3_uri, save_config=save_config, include_collections=['weights', 'gradients', 'bias','TopBlock'])
+    hook = TornasoleHook(out_dir=output_s3_uri, save_config=save_config, include_collections=['weights', 'gradients', 'biases','TopBlock'])
     return hook
 
 

examples/mxnet/scripts/mnist_gluon_vg_demo.py

@@ -114,7 +114,7 @@ def create_tornasole_hook(output_uri, tornasole_frequency):
 
     # Create a hook that logs weights, biases and gradients while training the model.
     hook = TornasoleHook(out_dir=output_uri, save_config=save_config, include_collections=['weights', 'gradients',
-                                                                                              'bias'])
+                                                                                            'biases'])
     return hook
 
 

examples/mxnet/scripts/mnist_mxnet.py

@@ -103,7 +103,7 @@ def create_tornasole_hook(output_uri):
     # Create a hook that logs weights, biases and gradients while training the model.
     hook = TornasoleHook(out_dir=output_uri,
                          save_config=save_config,
-                         include_collections=['weights', 'gradients', 'bias'])
+                         include_collections=['weights', 'gradients', 'biases'])
     return hook
 
 

examples/pytorch/scripts/pytorch_hook_demos.py

@@ -91,7 +91,7 @@ def create_tornasole_hook(output_dir, module=None, hook_type='saveall'):
 
         # Create a hook that logs weights, biases, gradients and inputs/outputs of model every 5 steps from steps 0-100 while training.
         hook = TornasoleHook(out_dir=output_dir, save_config=SaveConfig(save_steps=[i * 5 for i in range(20)]),
-                                include_collections=['weights', 'gradients', 'bias','l_mod'])
+                                include_collections=['weights', 'gradients', 'biases','l_mod'])
     elif hook_type == 'weights-bias-gradients':
         save_config = SaveConfig(save_steps=[i * 5 for i in range(20)])
         # Create a hook that logs ONLY weights, biases, and gradients every 5 steps (from steps 0-100) while training the model.

examples/pytorch/scripts/simple.py

@@ -58,7 +58,7 @@ def create_tornasole_hook(output_dir, module=None, hook_type='saveall',
         hook = TornasoleHook(out_dir=output_dir,
                              save_config=SaveConfig(save_steps=save_steps),
                              include_collections=['weights', 'gradients',
-                                                  'bias', 'l_mod'])
+                                                  'biases', 'l_mod'])
     elif hook_type == 'weights-bias-gradients':
         save_config = SaveConfig(save_steps=save_steps)
         # Create a hook that logs ONLY weights, biases, and gradients

examples/tensorflow/scripts/mnist.py

@@ -17,6 +17,9 @@
 parser.add_argument('--num_steps', type=int,
                     help="Number of steps to train for. If this" \
                          "is passed, it overrides num_epochs")
+parser.add_argument('--num_eval_steps', type=int,
+                    help="Number of steps to evaluate for. If this" \
+                         "is passed, it doesnt evaluate over the full eval set")
 parser.add_argument('--model_dir', type=str, default='/tmp/mnist_model')
 args = parser.parse_args()
 
@@ -134,4 +137,6 @@ def cnn_model_fn(features, labels, mode):
   hooks=[hook])
 
 hook.set_mode(ts.modes.EVAL)
-mnist_classifier.evaluate(input_fn=eval_input_fn, hooks=[hook])
+mnist_classifier.evaluate(input_fn=eval_input_fn,
+                          steps=args.num_eval_steps,
+                          hooks=[hook])

sagemaker-docs/DeveloperGuide_MXNet.md

@@ -191,7 +191,7 @@ tm.get_collection("ReluActivation").set_reduction_config(ReductionConfig(reducti
 tm.get_collection("flatten").include(["flatten*"])
 tm.get_collection("flatten").set_save_config(SaveConfig(save_steps=[4,5,6]))
 tm.get_collection("flatten").set_reduction_config(ReductionConfig(norms=["l1"], abs_norms=["l2"]))
-hook = TornasoleHook(out_dir=out_dir, include_collections=['weights', 'bias','gradients',
+hook = TornasoleHook(out_dir=out_dir, include_collections=['weights', 'biases','gradients',
                                                     'default', 'ReluActivation', 'flatten'])
 ```
 
@@ -202,7 +202,7 @@ Refer [API](api.md) for a list of the reductions available as well as examples.
 
 There are different ways to save tensors when using Tornasole.
 Tornasole provides easy ways to save certain standard tensors by way of default collections (a Collection represents a group of tensors).
-Examples of such collections are 'weights', 'gradients', 'bias' and 'default'.
+Examples of such collections are 'weights', 'gradients', 'biases' and 'default'.
 Besides the tensors in above default collections, you can save tensors by name or regex patterns on those names.
 Users can also specify a certain block in the model to save the inputs and outputs of that block.
 This section will take you through these ways in more detail.
@@ -211,7 +211,7 @@ This section will take you through these ways in more detail.
 The TornasoleHook API supports _include\_regex_ parameter. The users can specify a regex pattern with this pattern. The TornasoleHook will store the tensors that match with the specified regex pattern. With this approach, users can store the tensors without explicitly creating a Collection object. The specified regex pattern will be associated with 'default' Collection and the SaveConfig object that is associated with the 'default' collection.
 
 #### Default Collections
-Currently, the tornasole\_mxnet hook creates Collection objects for 'weights', 'gradients', 'bias' and 'default'. These collections contain the regex pattern that match with tensors of type weights, gradient and bias. The regex pattern for the 'default' collection is set when user specifies _include\_regex_ with TornasoleHook or sets the _SaveAll=True_.  These collections use the SaveConfig parameter provided with the TornasoleHook initialization. The TornasoleHook will store the related tensors, if user does not specify any special collection with _include\_collections_ parameter. If user specifies a collection with _include\_collections_ the above default collections will not be in effect.
+Currently, the tornasole\_mxnet hook creates Collection objects for 'weights', 'gradients', 'biases' and 'default'. These collections contain the regex pattern that match with tensors of type weights, gradient and bias. The regex pattern for the 'default' collection is set when user specifies _include\_regex_ with TornasoleHook or sets the _SaveAll=True_.  These collections use the SaveConfig parameter provided with the TornasoleHook initialization. The TornasoleHook will store the related tensors, if user does not specify any special collection with _include\_collections_ parameter. If user specifies a collection with _include\_collections_ the above default collections will not be in effect.
 
 #### Custom Collections
 You can also create any other customized collection yourself.

sagemaker-docs/DeveloperGuide_PyTorch.md

@@ -290,7 +290,7 @@ def create_tornasole_hook(output_dir, module):
 
     # Create a hook that logs weights, biases, gradients and inputs outputs of model while training.
     hook = TornasoleHook(out_dir=output_dir, save_config=SaveConfig(save_steps=[i * 10 for i in range(5)]),
-			    include_collections=['weights', 'gradients', 'bias','l_mod'])
+			    include_collections=['weights', 'gradients', 'biases','l_mod'])
 ```
 
 Here is how to register the above hook.

sagemaker-docs/FirstPartyRules.md

@@ -195,6 +195,9 @@ for step 21 is compared with the loss for step 9. The next step where loss is ch
 since 10 steps after 21 is 31, and at 31 and 32 loss is not being saved.
 - `diff_percent`: float (default is 0.0)  (between 0.0 and 100.0)
 The minimum difference in percentage that loss should be lower by. By default, the rule just checks if loss is going down. If you want to specify a stricter check that loss is going down fast enough, you might want to pass diff_percent.
+- `mode`: string
+The name of tornasole mode to query tensor values for rule checking. 
+If this is not passed, the rule checks for eval mode, then training mode and then global mode in this order.
 
 ```python
 rules_specification = [

tests/analysis/config.yaml

@@ -89,9 +89,9 @@
   - tensorflow
   - *Disable
   - [*tf_mnist,
-     --lr 0.001 --tornasole_train_frequency 10 --random_seed True,
+     --lr 0.001 --tornasole_train_frequency 10 --random_seed True --num_steps 100,
      *invoker,
-     --rule_name lossnotdecreasing --flag True --end_step 1000
+     --rule_name lossnotdecreasing --flag True --end_step 100
     ]
 -
   - loss_not_decreasing/tf/false
@@ -100,7 +100,7 @@
   - [*simple,
      --lr 0.05 --scale 1 --steps 1009 --tornasole_frequency 13 --random_seed True,
      *invoker,
-     --rule_name lossnotdecreasing --flag False --num_steps 100 --min_difference 12
+     --rule_name lossnotdecreasing --flag False --end_step 50 --diff_percent 50
     ]
 
 # test cases for mxnet

tests/core/test_collections.py

@@ -3,9 +3,9 @@
   COLLECTIONS_FILE_NAME
 from tornasole.core.reduction_config import ReductionConfig
 from tornasole.core.save_config import SaveConfig, SaveConfigMode
-from tornasole.core.save_manager import SaveManager
 from tornasole.core.modes import ModeKeys
-
+from tornasole.pytorch.hook import TornasoleHook
+import datetime
 
 def test_export_load():
   # with none as save config
@@ -71,12 +71,15 @@ def test_collection_defaults_to_hook_config():
   cm.create_collection('foo')
   cm.get('foo').set_save_config({ModeKeys.EVAL: SaveConfigMode(save_interval=20)})
 
-  sm = SaveManager(
-    collection_manager=cm,
-    include_collections_names=['foo'],
-    default_reduction_config=ReductionConfig(),
-    default_save_config={ModeKeys.TRAIN: SaveConfigMode(save_interval=10)},
-  )
+
+  hook = TornasoleHook(
+          out_dir='/tmp/test_collections/' + str(datetime.datetime.now()),
+          save_config={ModeKeys.TRAIN: SaveConfigMode(save_interval=10)},
+          include_collections=['foo'],
+          reduction_config=ReductionConfig(save_raw_tensor=True))
+  hook.collection_manager = cm
   assert cm.get('foo').save_config.mode_save_configs[ModeKeys.TRAIN] is None
-  sm.prepare()
-  assert cm.get('foo').save_config.mode_save_configs[ModeKeys.TRAIN].save_interval == 10
+  assert cm.get('foo').reduction_config is None
+  hook._prepare_collections()
+  assert cm.get('foo').save_config.mode_save_configs[ModeKeys.TRAIN].save_interval == 10
+  assert cm.get('foo').reduction_config.save_raw_tensor is True

tests/mxnet/mnist_gluon_model.py

@@ -120,4 +120,4 @@ def run_mnist_gluon_model(hook=None, hybridize=False, set_modes=False, register_
                 valid_acc/len(valid_data), time.time()-tic))
 
     # for tests we have to call cleanup ourselves as destructor won't be called now
-    # hook.cleanup()
+     # hook._cleanup()

tests/mxnet/test_hook_all_zero.py

@@ -18,7 +18,7 @@ def test_hook_all_zero(hook=None, out_dir=None):
     run_id = 'trial_' + datetime.now().strftime('%Y%m%d-%H%M%S%f')
     out_dir = './newlogsRunTest/' + run_id
     print("Registering the hook with out_dir {0}".format(out_dir))
-    hook = t_hook(out_dir=out_dir, save_config=save_config, include_collections=['ReluActivation','weights', 'bias','gradients'])
+    hook = t_hook(out_dir=out_dir, save_config=save_config, include_collections=['ReluActivation','weights', 'biases','gradients'])
   run_mnist_gluon_model(hook=hook, num_steps_train=10, num_steps_eval=10, make_input_zero=True)