CDRIVER-4485 string append and truncation fixes for structured logging (#1826)

This PR is a follow-up for #1795, #1821, and #1816. The new internal string and json APIs are used to implement missing functionality and tests for the structured logging subsystem.

* Adds public APIs for the max_document_length within a mongoc_structured_log_opts_t.
* Addresses the remainder of CDRIVER-4485: JSON documents within structured log messages are truncated correctly according to the rules set out in the Logging specification.
* Addresses the remainder of CDRIVER-4486 by implementing prose tests.
* Addresses CDRIVER-4814. Command or payload data beyond the truncation limit no longer degrades logging performance. (New serializer for command-with-payload)
* Unified tests use the suggested max document length of 10000 from the spec
* Replace atomic counter with atomic flag for environment error guards
* Tests for environment defaults now skip if relevant variables are set externally

Author: mdbmes

src/libmongoc/CMakeLists.txt

@@ -1096,6 +1096,7 @@ set (test-libmongoc-sources
    ${PROJECT_SOURCE_DIR}/tests/test-mongoc-collection-find-with-opts.c
    ${PROJECT_SOURCE_DIR}/tests/test-mongoc-collection-find.c
    ${PROJECT_SOURCE_DIR}/tests/test-mongoc-collection.c
+   ${PROJECT_SOURCE_DIR}/tests/test-mongoc-command-logging-and-monitoring.c
    ${PROJECT_SOURCE_DIR}/tests/test-mongoc-command-monitoring.c
    ${PROJECT_SOURCE_DIR}/tests/test-mongoc-connection-uri.c
    ${PROJECT_SOURCE_DIR}/tests/test-mongoc-counters.c

src/libmongoc/doc/mongoc_structured_log_opts_get_max_document_length.rst

@@ -0,0 +1,26 @@
+:man_page: mongoc_structured_log_opts_get_max_document_length
+
+mongoc_structured_log_opts_get_max_document_length()
+====================================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+  size_t
+  mongoc_structured_log_opts_get_max_document_length (const mongoc_structured_log_opts_t *opts);
+
+Parameters
+----------
+
+* ``opts``: Structured log options, allocated with :symbol:`mongoc_structured_log_opts_new`.
+
+Returns
+-------
+
+Returns the current maximum document length set in ``opts``, as a ``size_t``.
+
+.. seealso::
+
+  | :doc:`structured_log`

src/libmongoc/doc/mongoc_structured_log_opts_new.rst

@@ -31,7 +31,7 @@ Environment Variables
 This is a full list of the captured environment variables.
 
 * ``MONGODB_LOG_MAX_DOCUMENT_LENGTH``: Maximum length for JSON-serialized documents that appear within a log message.
-  It may be a number, in bytes, or ``unlimited`` (case insensitive).
+  It may be a number, in bytes, or ``unlimited`` (case insensitive) to choose an implementation-specific value near the maximum representable length.
   By default, the limit is 1000 bytes.
   This limit affects interior documents like commands and replies, not the total length of a structured log message.
 

src/libmongoc/doc/mongoc_structured_log_opts_set_max_document_length.rst

@@ -0,0 +1,33 @@
+:man_page: mongoc_structured_log_opts_set_max_document_length
+
+mongoc_structured_log_opts_set_max_document_length()
+====================================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+   bool
+   mongoc_structured_log_opts_set_max_document_length (mongoc_structured_log_opts_t *opts,
+                                                       size_t max_document_length);
+
+Sets a maximum length for BSON documents that appear serialized in JSON form as part of a structured log message.
+
+Serialized JSON will be truncated at this limit, interpreted as a count of UTF-8 encoded bytes. Truncation will be indicated with a ``...`` suffix, the length of which is not included in the max document length. If truncation at the exact indicated length would split a valid UTF-8 sequence, we instead truncate the document earlier at the nearest boundary between code points.
+
+Parameters
+----------
+
+* ``opts``: Structured log options, allocated with :symbol:`mongoc_structured_log_opts_new`.
+* ``max_document_length``: Maximum length for each embedded JSON document, in bytes, not including an ellipsis (``...``) added to indicate truncation. Values near or above ``INT_MAX`` will be rejected.
+
+Returns
+-------
+
+Returns ``true`` on success, or ``false`` if the supplied maximum length is too large.
+
+.. seealso::
+
+  | :doc:`structured_log`
+  | :symbol:`mongoc_structured_log_opts_set_max_document_length_from_env`

src/libmongoc/doc/mongoc_structured_log_opts_set_max_document_length_from_env.rst

@@ -0,0 +1,34 @@
+:man_page: mongoc_structured_log_opts_set_max_document_length_from_env
+
+mongoc_structured_log_opts_set_max_document_length_from_env()
+=============================================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+   bool
+   mongoc_structured_log_opts_set_max_document_length_from_env (mongoc_structured_log_opts_t *opts);
+
+Sets a maximum document length from the ``MONGODB_LOG_MAX_DOCUMENT_LENGTH`` environment variable, if a valid setting is found.
+See :symbol:`mongoc_structured_log_opts_new` for a description of the supported environment variable formats.
+
+Parse errors may cause a warning message, delivered via unstructured logging.
+
+This happens automatically when :symbol:`mongoc_structured_log_opts_new` establishes defaults.
+Any subsequent programmatic modifications to the :symbol:`mongoc_structured_log_opts_t` will override the environment variable settings.
+For applications that desire the opposite behavior, where environment variables may override programmatic settings, they may call ``mongoc_structured_log_opts_set_max_document_length_from_env()`` after calling :symbol:`mongoc_structured_log_opts_set_max_document_length`.
+This will process the environment a second time, allowing it to override customized defaults.
+
+Returns
+-------
+
+Returns ``true`` on success: either a valid environment setting was found, or the value is unset and ``opts`` will not be modified.
+If warnings are encountered in the environment, returns ``false`` and may log additional information to the unstructured logging facility.
+Note that, by design, these errors are by default non-fatal.
+When :symbol:`mongoc_structured_log_opts_new` internally calls this function, it ignores the return value.
+
+.. seealso::
+
+  | :doc:`structured_log`

src/libmongoc/doc/mongoc_structured_log_opts_t.rst

@@ -29,6 +29,9 @@ Functions
   mongoc_structured_log_opts_set_max_level_for_all_components
   mongoc_structured_log_opts_set_max_levels_from_env
   mongoc_structured_log_opts_get_max_level_for_component
+  mongoc_structured_log_opts_set_max_document_length
+  mongoc_structured_log_opts_set_max_document_length_from_env
+  mongoc_structured_log_opts_get_max_document_length
 
 .. seealso::
 

src/libmongoc/src/mongoc/mongoc-structured-log-private.h

@@ -34,6 +34,18 @@ typedef struct mongoc_structured_log_instance_t mongoc_structured_log_instance_t
 #define MONGOC_STRUCTURED_LOG_DEFAULT_LEVEL MONGOC_STRUCTURED_LOG_LEVEL_WARNING
 #define MONGOC_STRUCTURED_LOG_DEFAULT_MAX_DOCUMENT_LENGTH 1000
 
+/**
+ * @brief maximum possible value of the 'maximum document length' setting, enforced when reading settings from the
+ * environment.
+ *
+ * Maximum document length applies to a single serialized JSON document within the log.
+ * The overall log document, when serialized as BSON, will be subject to BSON_MAX_SIZE.
+ * At a minimum we should leave room for BSON headers and for the JSON truncation marker ("...").
+ * Choose to leave a little more room, as it's more useful to truncate the huge document early
+ * rather than fail in bson_append_utf8().
+ */
+#define MONGOC_STRUCTURED_LOG_MAXIMUM_MAX_DOCUMENT_LENGTH ((uint32_t) BSON_MAX_SIZE - 4096u)
+
 /**
  * @brief Allocate a new instance of the structured logging system
  * @param opts Options, copied into the new instance.