Initial Integration of node-shell - iohk-monitoring. (#26)

* [#20] Added iohk-monitoring as dependence in stack.yaml.

* [#20] Ease nix build.

* [#20] Fix nix.

* [#20] Renamed LoggingLayer contents.

* [#20] Further renaming.

* [#20] more functions in LoggingLayer

Signed-off-by: Alexander Diemand <[email protected]>

* [#20] upstream updated

Signed-off-by: Alexander Diemand <[email protected]>

* Fix some review points.

* [#20] adapted to new code structure

Signed-off-by: Alexander Diemand <[email protected]>

* [#20] adhere to naming convention

Signed-off-by: Alexander Diemand <[email protected]>

* Fix BuildKite Nix builds.

* [#20] global configuration

Signed-off-by: Alexander Diemand <[email protected]>

Author: Andreas Triantafyllos

.buildkite/pipeline.yml

@@ -1,6 +1,8 @@
 steps:
   - label: 'nix build'
     command: 'nix-build release.nix'
+    agents:
+      system: x86_64-linux
   - label: 'stack rebuild'
     env:
       AWS_REGION: us-west-1

ARCHITECTURE.md

@@ -23,7 +23,7 @@ For example, let's suppose we want to create a very simple logging/monitoring _f
 We see that the _feature_ requires a specific *configuration and has some global exception handling*.
 The specific configuration can be something really specific to logging, for example. We may need to read the default log level, whether it's info, debug, warn or something else - we need to define the level of information we want to save in the logs. For that, we require a specific configuration. That configuration is not part of the default configuration we usually parse when we start the _cardano-shell_ - when we parse the initial key configuration, we parse what *every feature requires*, not
 something specific for each _feature_.
-Then we have the _feature_ ready for usage and we can use the functions it contains. We will restrict this a bit later, but for now, let's suppose we can use all the functions in the module export. 
+Then we have the _feature_ ready for usage and we can use the functions it contains. We will restrict this a bit later, but for now, let's suppose we can use all the functions in the module export.
 And so, all is well. We are done! But wait! What about the other _features_? Ok, so I lied, we are not done. We need to able to communicate with the other nodes. What do we need for that? Networking! So we need to construct the _networking feature_. And what does networking use? It uses loggging! How does that look like?
 
 ![networking_deps](https://user-images.githubusercontent.com/6264437/48932488-84ae8800-eefc-11e8-89ff-5c8d45dc0f46.jpg)
@@ -50,7 +50,7 @@ So let's draw some pictures here and make this a bit more clear. Let's simplify
 
 ![feature_layer_1](https://user-images.githubusercontent.com/6264437/48933434-89753b00-ef00-11e8-9edf-5f424caaba07.jpg)
 
-When we complete the construction of this logging/monitoring _feature_, we get the initialized _layer_, which is the interface toward the _feature_. It's essentially a record of functions. 
+When we complete the construction of this logging/monitoring _feature_, we get the initialized _layer_, which is the interface toward the _feature_. It's essentially a record of functions.
 Let's make this a bit more concrete. Let's say we have an extremely simple interface for _logging_ and all we want our users (developers) to use is this:
 ```
 logInfo     :: Text -> IO ()
@@ -87,7 +87,7 @@ I lied again. I simplified it a bit. If you take a look at the actual _layer_ ex
 ![feature_layer_abstract_effect](https://user-images.githubusercontent.com/6264437/48934055-db1ec500-ef02-11e8-910a-a11885661bb2.jpg)
 
 In the example, we suppose that the interface is abstract (the `m` type is abstract) and we later on define it to be `forall m. (MonadIO m, MonadLog m) => m`, so an _mtl style effect_ which has two constraints on it - it can log and do IO, which sounds good. But there is nothing perfect and there are trade-offs everywhere. Even in our nice little example. How?
-Let me show you. The abstract type is now constrained by two types - `MonadIO` and `MonadLog`. 
+Let me show you. The abstract type is now constrained by two types - `MonadIO` and `MonadLog`.
 
 ![feature_layer_effect_escape_1](https://user-images.githubusercontent.com/6264437/48934364-fdfda900-ef03-11e8-8631-bcdb2f283eab.jpg)
 
@@ -97,7 +97,7 @@ isn't that one of the reasons we use Haskell in the first place? Even worse, see
 ![feature_layer_effect_escape_2](https://user-images.githubusercontent.com/6264437/48934561-d65b1080-ef04-11e8-808a-b83bee9a188b.jpg)
 
 But that's not the worst part! The worst part is yet to come!
-What happens when we use _logging layer_ in our _networking feature_ function? 
+What happens when we use _logging layer_ in our _networking feature_ function?
 
 ![feature_layer_effect_escape_3](https://user-images.githubusercontent.com/6264437/48934599-fdb1dd80-ef04-11e8-984b-4d2f4cc2eef2.jpg)
 
@@ -114,9 +114,9 @@ Well, we can fix this. The price is hard-to-read type errors if you miss somethi
 So we can simply use `Rank2Types` extension and use a layer that has the constraints defined *per function*:
 ```
 data LoggingLayer = LoggingLayer
-    { iolLogDebug   :: forall m. (MonadIO m) => Text -> m ()
-    , iolLogInfo    :: forall m. (MonadIO m) => Text -> m ()
-    , iolNonIo      :: forall m. (MonadThrow m) => m ()
+    { llLogDebug   :: forall m. (MonadIO m) => Text -> m ()
+    , llLogInfo    :: forall m. (MonadIO m) => Text -> m ()
+    , llLockNonIO  :: forall m. (MonadThrow m) => m ()
     }
 ```
 

app/Cardano/Shell/Features/Logging.hs

@@ -4,50 +4,32 @@
 module Cardano.Shell.Features.Logging
     ( LoggingLayer (..)
     , createLoggingFeature
+    , Trace
     ) where
 
-import           Cardano.Prelude
+import           Cardano.Prelude hiding (trace)
 
-import           Control.Exception.Safe (MonadThrow)
+import           Cardano.BM.Configuration.Static (defaultConfigStdout)
+import           Cardano.BM.Setup (setupTrace)
+import           Cardano.BM.Trace (Trace)
+import qualified Cardano.BM.Trace as Trace
 
 import           Cardano.Shell.Types (CardanoConfiguration, CardanoEnvironment,
-                                      CardanoFeature (..),
-                                      CardanoFeatureInit (..),
-                                      NoDependency (..))
+                     CardanoFeature (..), CardanoFeatureInit (..),
+                     NoDependency (..), ccLogConfig, ccLogPath)
 
 --------------------------------------------------------------------------------
 -- Loggging feature
 --------------------------------------------------------------------------------
 
---------------------------------
--- Exceptions
---------------------------------
-
-data LoggingException
-    = MissingLogFileException
-    deriving (Eq, Show)
-
-instance Exception LoggingException
-
 --------------------------------
 -- Configuration
 --------------------------------
 
--- Ideas picked up from https://github.com/input-output-hk/cardano-sl/blob/develop/util/src/Pos/Util/Log/LoggerConfig.hs
-
-data RotationParameters = RotationParameters
-    { _rpLogLimitBytes :: !Word64  -- ^ max size of file in bytes
-    , _rpMaxAgeHours   :: !Word    -- ^ hours
-    , _rpKeepFilesNum  :: !Word    -- ^ number of files to keep
-    } deriving (Generic, Show, Eq)
-
-
-testRotationParameters :: RotationParameters
-testRotationParameters = RotationParameters
-    { _rpLogLimitBytes = 10
-    , _rpMaxAgeHours   = 3
-    , _rpKeepFilesNum  = 5
-    }
+data LoggingParameters = LoggingParameters
+    { configFp :: FilePath
+    , prefixFp :: FilePath
+    } deriving (Show, Eq)
 
 --------------------------------
 -- Layer
@@ -60,33 +42,36 @@ testRotationParameters = RotationParameters
 -- the functions effects and constraining the user (programmer) of those function to use specific effects in them.
 -- https://github.com/input-output-hk/cardano-sl/blob/develop/util/src/Pos/Util/Log/LogSafe.hs
 data LoggingLayer = LoggingLayer
-    { iolLogDebug :: forall m. (MonadIO m) => Text -> m ()
-    , iolLogInfo  :: forall m. (MonadIO m) => Text -> m ()
-    , iolNonIo    :: forall m. (MonadThrow m) => m ()
-    }
-
-testLoggingLayer :: LoggingLayer
-testLoggingLayer = LoggingLayer
-    { iolLogDebug               = liftIO . putTextLn
-    , iolLogInfo                = liftIO . putTextLn
-    , iolNonIo                  = pure ()
+    { llStartTrace  :: forall m. (MonadIO m) => Trace m
+    , llLogDebug    :: forall m. (MonadIO m) => Trace m -> Text -> m ()
+    , llLogInfo     :: forall m. (MonadIO m) => Trace m -> Text -> m ()
+    , llLogNotice   :: forall m. (MonadIO m) => Trace m -> Text -> m ()
+    , llLogWarning  :: forall m. (MonadIO m) => Trace m -> Text -> m ()
+    , llLogError    :: forall m. (MonadIO m) => Trace m -> Text -> m ()
+    , llAppendName  :: forall m. (MonadIO m) => Text -> Trace m -> m (Trace m)
     }
 
 --------------------------------
 -- Feature
 --------------------------------
 
-type LoggingCardanoFeature = CardanoFeatureInit NoDependency RotationParameters LoggingLayer
+type LoggingCardanoFeature = CardanoFeatureInit NoDependency LoggingParameters LoggingLayer
 
 createLoggingFeature :: CardanoEnvironment -> CardanoConfiguration -> IO (LoggingLayer, CardanoFeature)
 createLoggingFeature cardanoEnvironment cardanoConfiguration = do
     -- we parse any additional configuration if there is any
     -- We don't know where the user wants to fetch the additional configuration from, it could be from
     -- the filesystem, so we give him the most flexible/powerful context, @IO@.
-    loggingConfiguration    <-  pure testRotationParameters
+    loggingConfiguration    <-  pure $ LoggingParameters
+                                    (ccLogConfig cardanoConfiguration)
+                                    (ccLogPath cardanoConfiguration)
 
     -- we construct the layer
-    loggingLayer            <- (featureInit loggingCardanoFeatureInit) cardanoEnvironment NoDependency cardanoConfiguration loggingConfiguration
+    loggingLayer            <- (featureInit loggingCardanoFeatureInit)
+                               cardanoEnvironment
+                               NoDependency
+                               cardanoConfiguration
+                               loggingConfiguration
 
     -- we construct the cardano feature
     let cardanoFeature      = loggingCardanoFeature loggingCardanoFeatureInit loggingLayer
@@ -96,30 +81,32 @@ createLoggingFeature cardanoEnvironment cardanoConfiguration = do
 
 loggingCardanoFeatureInit :: LoggingCardanoFeature
 loggingCardanoFeatureInit = CardanoFeatureInit
-    { featureType                   = "LoggingMonitoringFeature"
-    , featureInit                   = featureInit'
-    , featureCleanup                = featureCleanup'
+    { featureType    = "LoggingMonitoringFeature"
+    , featureInit    = initLogging
+    , featureCleanup = cleanupLogging
     }
   where
-    featureInit' :: CardanoEnvironment -> NoDependency -> CardanoConfiguration -> RotationParameters -> IO LoggingLayer
-    featureInit' = actualLoggingFeature
-      where
-        actualLoggingFeature :: CardanoEnvironment -> NoDependency -> CardanoConfiguration -> RotationParameters -> IO LoggingLayer
-        actualLoggingFeature _ _ _ _ = do
-            --putTextLn "Starting up logging!"
-            pure testLoggingLayer
-
-    featureCleanup' :: LoggingLayer -> IO ()
-    featureCleanup' _ = pure () --putTextLn "Shutting down logging feature!" -- save a file, for example
+    initLogging :: CardanoEnvironment -> NoDependency -> CardanoConfiguration -> LoggingParameters -> IO LoggingLayer
+    initLogging _ _ _ _logparams = do
+        cfg <- defaultConfigStdout  -- TODO initialize from 'configFp logparams'
+        baseTrace <- setupTrace (Right cfg) "simple"
+        pure $ LoggingLayer
+                { llStartTrace  = Trace.natTrace liftIO baseTrace
+                , llLogDebug    = Trace.logDebug
+                , llLogInfo     = Trace.logInfo
+                , llLogNotice   = Trace.logNotice
+                , llLogWarning  = Trace.logWarning
+                , llLogError    = Trace.logError
+                , llAppendName  = Trace.appendName
+                }
+    cleanupLogging :: LoggingLayer -> IO ()
+    cleanupLogging _ = pure ()  -- TODO
 
 loggingCardanoFeature :: LoggingCardanoFeature -> LoggingLayer -> CardanoFeature
 loggingCardanoFeature loggingCardanoFeature' loggingLayer = CardanoFeature
     { featureName       = featureType loggingCardanoFeature'
     , featureStart      = liftIO $ do
-        --putTextLn "Starting up loggingCardanoFeature!"
-        void $ pure loggingLayer -- or whatever it means for YOU (a specific team)
+        void $ pure loggingLayer
     , featureShutdown   = liftIO $ do
-        --putTextLn "Shutting down loggingCardanoFeature!"
         (featureCleanup loggingCardanoFeature') loggingLayer
     }
-

app/Cardano/Shell/Features/Networking.hs

@@ -55,7 +55,10 @@ data NetworkLayer = NetworkLayer
 testNetworkLayer :: LoggingLayer -> NetworkLayer
 testNetworkLayer loggingLayer = NetworkLayer
     { sendToNodes       = \_ -> pure "SEND"
-    , readFromNodes     = \_ -> iolNonIo loggingLayer >> pure "READ"
+    , readFromNodes     = \_ -> do
+                                let m = "READ"
+                                llLogInfo loggingLayer (llStartTrace loggingLayer) m
+                                pure m
     }
 
 --------------------------------

cardano-shell.cabal

@@ -73,6 +73,8 @@ executable cardano-shell-exe
     , cardano-prelude
     -- util
     , safe-exceptions
+    -- features
+    , iohk-monitoring
   default-language:    Haskell2010
   default-extensions:  NoImplicitPrelude
                        OverloadedStrings

cardano-shell.nix

@@ -1,7 +1,7 @@
 { mkDerivation, base, cardano-prelude, concurrency, contravariant
 , dejafu, dhall, directory, ekg-core, formatting, hspec
-, hspec-contrib, hunit-dejafu, QuickCheck, safe-exceptions, stdenv
-, text, transformers
+, hspec-contrib, hunit-dejafu, iohk-monitoring, QuickCheck
+, safe-exceptions, stdenv, text, transformers
 }:
 mkDerivation {
   pname = "cardano-shell";
@@ -11,7 +11,8 @@ mkDerivation {
   isExecutable = true;
   libraryHaskellDepends = [
     base cardano-prelude concurrency contravariant dhall directory
-    ekg-core formatting QuickCheck safe-exceptions text transformers
+    ekg-core formatting iohk-monitoring QuickCheck safe-exceptions text
+    transformers
   ];
   executableHaskellDepends = [
     base cardano-prelude safe-exceptions

iohk-monitoring.nix

@@ -0,0 +1,31 @@
+{ mkDerivation, aeson, Cabal, array, async, auto-update, base, bytestring
+, clock, containers, contravariant, directory, ekg, ekg-core, fetchgit
+, filepath, katip, lens, mtl, process, QuickCheck, random
+, safe-exceptions, semigroups, stdenv, stm, tasty, tasty-hunit
+, tasty-quickcheck, template-haskell, text, time, time-units
+, transformers, unix, unordered-containers, void, yaml
+}:
+mkDerivation {
+  pname = "iohk-monitoring";
+  version = "0.1.0.0";
+  src = fetchgit {
+    url = "https://github.com/input-output-hk/iohk-monitoring-framework";
+    sha256 = "01bm1b5qfgilwlj6ygmri6l4fydd2p2awg2z9isj8jwkglv9shga";
+    rev = "9ab53237c579698c3bf892580a7b4df2d097e270";
+    fetchSubmodules = false;
+  };
+  libraryHaskellDepends = [
+    aeson Cabal array async auto-update base bytestring clock containers
+    contravariant directory ekg ekg-core filepath katip lens mtl
+    safe-exceptions stm template-haskell text time time-units
+    transformers unix unordered-containers yaml
+  ];
+  testHaskellDepends = [
+    Cabal array base bytestring clock containers mtl process QuickCheck
+    random semigroups stm tasty tasty-hunit tasty-quickcheck text time
+    time-units transformers void
+  ];
+  description = "logging, benchmarking and monitoring framework";
+  license = stdenv.lib.licenses.mit;
+  hydraPlatforms = stdenv.lib.platforms.none;
+}

release.nix

@@ -5,8 +5,10 @@ let
         overrides = haskellPackagesNew: haskellPackagesOld: rec {
           cardano-prelude   = haskellPackagesNew.callPackage ./cardano-prelude.nix {
             #canonical-json = pkgs.haskell.lib.dontCheck (haskellPackagesNew.callHackage "canonical-json" "0.5.0.1" {});
-            canonical-json = pkgs.haskell.lib.dontCheck (haskellPackagesNew.callPackage ./canonical-json.nix {});
+            canonical-json    = pkgs.haskell.lib.dontCheck (haskellPackagesNew.callPackage ./canonical-json.nix {});
           };
+
+          iohk-monitoring   = haskellPackagesNew.callPackage ./iohk-monitoring.nix { };
           cardano-shell     = haskellPackagesNew.callPackage ./cardano-shell.nix { };
         };
       };

src/Cardano/Shell/Types.hs

@@ -2,11 +2,10 @@
 
 module Cardano.Shell.Types where
 
-import Cardano.Prelude
+import           Cardano.Prelude
 
-import Control.Concurrent.Classy (MonadConc)
+import           Control.Concurrent.Classy (MonadConc)
 
---import qualified Katip as Katip
 import qualified System.Metrics as Ekg
 
 -- | The top level module we use to run the key functions.
@@ -20,13 +19,15 @@ data ApplicationEnvironment
 
 -- | A simple function to inform us.
 applicationProductionMode :: ApplicationEnvironment -> Bool
-applicationProductionMode Production    = True
-applicationProductionMode _             = False
+applicationProductionMode Production = True
+applicationProductionMode _          = False
 
 -- | Cardano configuration
 data CardanoConfiguration = CardanoConfiguration
     { ccLogPath                 :: !FilePath
     -- ^ The location of the log files on the filesystem.
+    , ccLogConfig               :: !FilePath
+    -- ^ The path to the log configuration.
     , ccDBPath                  :: !FilePath
     -- ^ The location of the DB on the filesystem.
     , ccApplicationLockFile     :: !FilePath
@@ -38,15 +39,14 @@ data CardanoConfiguration = CardanoConfiguration
 -- | The common runtime environment for all features in the server.
 -- All features have access to this environment.
 data CardanoEnvironment = CardanoEnvironment
-    { ceLogEnv      :: Text --Katip.LogEnv
-    , ceEkgStore    :: Ekg.Store
+    { ceLogEnv   :: Text
+    , ceEkgStore :: Ekg.Store
      -- ...
     }
 
 -- | Initalise 'ServerEnv'
 initializeCardanoEnvironment :: IO CardanoEnvironment
 initializeCardanoEnvironment = do
-  --  logenv   <- Katip.initLogEnv (...) (...)
     ekgStore <- Ekg.newStore
     return CardanoEnvironment
         { ceLogEnv      = "To implement"
@@ -55,7 +55,7 @@ initializeCardanoEnvironment = do
 
 loadCardanoConfiguration :: IO CardanoConfiguration
 loadCardanoConfiguration = do
-    pure $ CardanoConfiguration mempty mempty mempty
+    pure $ CardanoConfiguration mempty mempty mempty mempty
 
 -- | The option to not have any additional dependency for the @CardanoFeature@.
 data NoDependency = NoDependency
@@ -81,11 +81,11 @@ data CardanoFeatureInit dependency configuration layer = CardanoFeatureInit
 
 -- | The interface for the running feature, the high-level interface we use for running it.
 data CardanoFeature = CardanoFeature
-    { featureName       :: Text
+    { featureName     :: Text
     -- ^ The name of the feature.
-    , featureStart      :: forall m. (MonadIO m, MonadConc m) => m ()
+    , featureStart    :: forall m. (MonadIO m, MonadConc m) => m ()
     -- ^ What we call when we start the feature.
-    , featureShutdown   :: forall m. (MonadIO m, MonadConc m) => m ()
+    , featureShutdown :: forall m. (MonadIO m, MonadConc m) => m ()
     -- ^ What we call when we shut down the feature.
     }
 

stack-shell.nix

@@ -11,7 +11,7 @@ let
 
   pkgs = import nixpkgs { inherit config; };
 
-in 
+in
   with pkgs;
 
   haskell.lib.buildStackProject {