[Kaizen] improve logging configuration

[jvdp]

akka.loglevel should be set to DEBUG so that messages with this level can actually show up in the logs, if we configure them appropriately. The filters and levels defined in logback.xml are applied after and thus will still filter all debug messages by default.

This should not cause significant overhead: https://doc.akka.io/docs/akka/current/logging.html#slf4j

[dzajkowski]

please follow up on Petra's comment.

[jvdp]

please follow up on Petra's comment.

So I did some basic performance checks with the getBlock endpoint (which logs a couple pretty big messages on DEBUG) and there is no noticable performance degradation. (If anything it became faster?)

[jvdp]

I added a change to filter out the many messages like:

2021-05-10 15:41:28,710 [i.i.s.d.e.v.DiscoveryService$ServiceImpl] - Storing the ENR for Peer(id = 75fba1a45cbeba353361e8ec1ee2ce276fd5f88e78816024f3363b411e1a647bfb043eaec95fc101a8ae1c1a9d420cb2095c7665db3d029b526bce602ec65f86, address = /x.x.x.x:30303)

[dzajkowski]

ok so I think this is overthinking it. the consequence of passing TRACE to the log system is:

[ERROR] [05/13/2021 09:51:24.512] [run-main-0] [EventStream(akka://mantis_system)] unknown akka.loglevel TRACE
akka.event.Logging$LoggerException:
	at akka.event.LoggingBus.$anonfun$startDefaultLoggers$1(Logging.scala:113)
	at akka.event.LoggingBus.$anonfun$startDefaultLoggers$1$adapted(Logging.scala:110)
	at scala.Option.getOrElse(Option.scala:201)
	at akka.event.LoggingBus.startDefaultLoggers(Logging.scala:110)
	at akka.event.LoggingBus.startDefaultLoggers$(Logging.scala:108)
	at akka.event.EventStream.startDefaultLoggers(EventStream.scala:26)
	at akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:605)
	at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:1021)
	at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:1017)
	at akka.actor.ActorSystemImpl._start(ActorSystem.scala:1017)
	at akka.actor.ActorSystemImpl.start(ActorSystem.scala:1040)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:272)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:316)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:260)

I think this is fine and it's a better solution to use ${logging.logs-level} instead of trying to guess what a good level might be.
maybe a note on the logs-level setting that notes this interaction.
eg.

# Note: if this level is set to TRACE the akka.loglevel needs to be adjusted (akka does not know TRACE)

[jvdp]

Well, that won't work as expected when overall "logs-level" is info but you configure some specific parts for a level lower than that. (Which is what I'm doing a lot now.)

[leo-bogastry]

For me this feels like a bit of personal preference. How often do we use a log level of trace anyway? For that I think I prefer Dom's perspective

[jvdp]

Writing a top-level comment to clarify the reason for this PR: it's a bit hidden (and surprising, at least to me) that you have to change the akka.loglevel when you set the LOGSLEVEL env var to DEBUG or set a fine-grained log level for a package prefix to DEBUG in logback.xml. There's no indication that this is needed, other than debug messages mysteriously not appearing. This PR aims to address both those use cases.