Add more logging to check_supervision in case it is failing

Summary:
For a OneWay message like `ControllerActor::check_supervision`, if it fails halfway through
it won't send a message to the client, and it won't reschedule itself to run again.
We need to have logs to make sure the message is sent.

I suspect that occasionally when a worker fails with a DeviceException it isn't getting propagated
back to the client

Reviewed By: shayne-fletcher

Differential Revision: D75232803

fbshipit-source-id: 25c6fa2e927a762305b8bbd73a181174a1129bda

Author: dulinriley

controller/src/lib.rs

@@ -519,10 +519,12 @@ impl ControllerMessageHandler for ControllerActor {
                     address: failed_state.proc_addr.to_string(),
                     backtrace: failure_reason,
                 });
+                tracing::error!("Sending failure to client: {exc:?}");
                 // Seq does not matter as the client will raise device error immediately before setting the results.
                 self.client()?
                     .result(this, Seq::default(), Some(Err(exc)))
                     .await?;
+                tracing::error!("Failure successfully sent to client");
 
                 // No need to set history failures as we are directly sending back failure results.
             }

python/monarch/common/client.py

@@ -302,6 +302,7 @@ def _handle_pending_result(self, output: MessageResult) -> None:
         self.last_processed_seq = max(self.last_processed_seq, seq)
 
         if error is not None:
+            logging.error("Received error for seq %s: %s", seq, error)
             # We should not have set result if we have an error.
             assert result is None
             if not isinstance(error, RemoteException):