CDRIVER-3775 mongoc_structured_log

[ghost]

This is a revival of an old pull request by @alcaeus to add structured logging to the C driver. (#684)

Structured logging is a new subsystem, independent from unstructured logging. The original PR introduced structured logging with a global scope, similar in design to unstructured logging. This version redesigns it to be per-client or per-pool. Internally, there is a structured logging "instance" owned by the mongoc_topology_t. A separate public mongoc_structured_log_opts_t object provides a configuration API, and the opts can then be passed on to the client or client pool in order to apply new log settings.

The internal API has been redesigned. The previous PR required each new type of log message to have functions and structures associated with that specific message format. In this redesign, I've split the single callback into a variable length list of callbacks, and added a set of macros to make it straightforward to build these function tables. This has some nice properties:

• Submitting a log entry still doesn't require any deep copies, json serialization, or any memory allocation, only a stack-allocated table of references.
• Now the mongoc_structured_log() call site explicitly names the keys that are included in the resulting document.
• Now it's easy to define ad-hoc log entries or add new values to existing log entries.
• The list-based approach also makes it easy to define reusable building blocks. This is immediately useful for command redaction and for server descriptions.

Here's a sample invocation:

   mongoc_structured_log (
      client->topology->structured_log,
      MONGOC_STRUCTURED_LOG_LEVEL_DEBUG,
      MONGOC_STRUCTURED_LOG_COMPONENT_COMMAND,
      "Command started",
      int32 ("requestId", client->cluster.request_id),
      server_description (server_stream->sd, SERVER_HOST, SERVER_PORT, SERVER_CONNECTION_ID, SERVICE_ID),
      utf8_n ("databaseName", cursor->ns, cursor->dblen),
      utf8 ("commandName", "getMore"),
      int64 ("operationId", cursor->operation_id),
      bson_as_json ("command", &doc));

The macros are explained by doc comments in mongoc-structured-log-private.h.

This PR includes updated unified tests from the command logging and monitoring spec, which now pass. This required several other changes to the unified test runner.

Contents:

• Implement and document a structured logging facility
• Structured logging items for command and reply redaction
• Unified test runner support for: createEntities, observeLogMessages, waitForEvent, $$matchAsDocument, $$matchAsRoot
• Add serverDescriptionChangedEvent to the unified test runner and unify its two event serialization systems
• Private serialization utilities, *_append_contents_to_bson
• Sync unified tests from the command-logging-and-monitoring spec
• Enough command logging to pass the CLAM tests
• bson-dsl support for oid() values
• Private utilities for dealing with zero'ed oids
• Minor drive-by cleanup

CDRIVER-3775 is the epic for structured logging in general. This PR currently covers:

• CDRIVER-4485 (Except for prose test)
• CDRIVER-4486
• CDRIVER-4535
• CDRIVER-4553

Follow-up changes will address the remaining parts of CDRIVER-3775. In particular, these major items are out of scope for this PR:

• Re-evaluation of JSON serialization for large documents, payloads, ellipsis, and character boundaries (includes CDRIVER-4814)
• Prose tests for command logging
• Cleanup of legacy OP_KILL_CURSORS code (now CDRIVER-5823)
• SDAM logging, and associated refactoring
• Server selection logging, and associated refactoring
• CDRIVER-4560
• CDRIVER-4566
• CDRIVER-4758
• CDRIVER-4812
• CDRIVER-5717

[kevinAlbs]

Great work. LGTM with minor comments addressed. I like the use of atomics to avoid repeated logs from invalid environment variable values.

[ghost]

Re the scan-build-macos-14-arm64-clang failure that remains on evergreen, it seems to me like some of the build hosts may have an inconsistent environment? The failing runs here are unable to locate a scan-build binary. On the master branch runs that succeed, the script locates a locally installed scan-build from homebrew.

[kevinAlbs]

build hosts may have an inconsistent environment?

Seems unexpected. Asked on DEVPROD-12817.

[jmikola]

Updating copyright dates on headers is probably the only necessary change (and maybe the bson_t formatting in docs if you care about that).

I'll leave a provisional LGTM to avoid holding this up further.