CDRIVER-5641: BSON Binary Vector Subtype Support (#1868)

* Resolves CDRIVER-5641 by documenting and implementing new BSON Binary Vector APIs
* Synced bson_binary_vector and bson_corpus spec tests from specifications commit 585b797c110b6709f81def6200b946b94c8d9c55
* tests: Full support for the existing version of the spec test suite.
* tests: TODOs mark spec test improvements that depend on DRIVERS-3095 and DRIVERS-3097.
* tests: Additional API usage and fuzzing.
* New BSON Binary APIs: bson_append_binary_uninit, bson_iter_binary_equal, bson_iter_binary_subtype, bson_iter_overwrite_binary
* Extend existing big-endian byte swapping API for 32-bit float
* Compiler support for restricted pointer aliasing using BSON_RESTRICT.
* Added note about the differing public and private API of bson_iter_binary and friends
* ASSERT_MEMCMP improvements: eval inputs once inside parens, show a hex dump on mismatch, include error location.

Author: mdbmes

src/libbson/doc/api.rst

@@ -25,5 +25,6 @@ API Reference
   bson_writer_t
   bson_get_monotonic_time
   bson_memory
+  binary_vector
   version
   legacy_extended_json

src/libbson/doc/binary_vector.rst

@@ -0,0 +1,149 @@
+:man_page: libbson_binary_vector
+
+BSON Binary Vector subtype
+==========================
+
+In Libbson, we use the term *Vector* to refer to a data representation for compact storage of uniform elements, defined by the `BSON Binary Subtype 9 - Vector <https://github.com/mongodb/specifications/blob/master/source/bson-binary-vector/bson-binary-vector.md>`_ specification.
+
+Libbson includes API support for Vectors:
+
+* The *view* APIs provide an efficient way to access elements of Vector fields that reside within :symbol:`bson_t` storage.
+* Integration between *views* and other Libbson features: append, array builder, iter.
+* Vectors can be converted to and from a plain BSON Array, subject to the specification's type conversion rules.
+
+The specification currently defines three element types, which Libbson interprets as:
+
+* ``int8``: signed integer elements, equivalent to C ``int8_t``.
+* ``float32``: IEEE 754 floating point, 32 bits per element, least-significant byte first. After alignment and byte swapping, elements are equivalent to C ``float``.
+* ``packed_bit``: single-bit integer elements, packed most-significant bit first. Accessible in packed form as C ``uint8_t`` or as unpacked elements using C ``bool``.
+
+Vector Views
+------------
+
+.. toctree::
+  :titlesonly:
+  :maxdepth: 1
+
+  bson_vector_int8_view_t
+  bson_vector_int8_const_view_t
+  bson_vector_float32_view_t
+  bson_vector_float32_const_view_t
+  bson_vector_packed_bit_view_t
+  bson_vector_packed_bit_const_view_t
+
+Integration
+-----------
+
+* Allocating Vectors inside :symbol:`bson_t`:
+
+  .. toctree::
+    :titlesonly:
+    :maxdepth: 1
+
+    bson_append_vector_int8_uninit
+    bson_append_vector_float32_uninit
+    bson_append_vector_packed_bit_uninit
+
+* Accessing an existing Vector via :symbol:`bson_iter_t`:
+
+  .. code-block:: c
+
+    #define BSON_ITER_HOLDS_VECTOR(iter) /* ... */
+    #define BSON_ITER_HOLDS_VECTOR_INT8(iter) /* ... */
+    #define BSON_ITER_HOLDS_VECTOR_FLOAT32(iter) /* ... */
+    #define BSON_ITER_HOLDS_VECTOR_PACKED_BIT(iter) /* ... */
+
+  .. toctree::
+    :titlesonly:
+    :maxdepth: 1
+
+    bson_vector_int8_view_from_iter
+    bson_vector_int8_const_view_from_iter
+    bson_vector_float32_view_from_iter
+    bson_vector_float32_const_view_from_iter
+    bson_vector_packed_bit_view_from_iter
+    bson_vector_packed_bit_const_view_from_iter
+
+Array Conversion
+----------------
+
+* Polymorphic array-from-vector:
+
+  .. toctree::
+    :titlesonly:
+    :maxdepth: 1
+
+    bson_append_array_from_vector
+
+* Type specific array-from-vector:
+
+  .. toctree::
+    :titlesonly:
+    :maxdepth: 1
+
+    bson_append_array_from_vector_int8
+    bson_append_array_from_vector_float32
+    bson_append_array_from_vector_packed_bit
+
+* Using :symbol:`bson_array_builder_t` for array-from-vector:
+
+  .. toctree::
+    :titlesonly:
+    :maxdepth: 1
+
+    bson_array_builder_append_vector_int8_elements
+    bson_array_builder_append_vector_float32_elements
+    bson_array_builder_append_vector_packed_bit_elements
+    bson_array_builder_append_vector_elements
+
+* Type specific vector-from-array:
+
+  .. toctree::
+    :titlesonly:
+    :maxdepth: 1
+
+    bson_append_vector_int8_from_array
+    bson_append_vector_float32_from_array
+    bson_append_vector_packed_bit_from_array
+
+Additional Definitions
+----------------------
+
+* Binary subtype:
+
+  .. code-block:: c
+
+    typedef enum {
+      BSON_SUBTYPE_VECTOR = 0x09,
+      /* ... */
+    } bson_subtype_t;
+
+* Byte length of the Vector header:
+
+  .. code-block:: c
+
+    // Length of the required header for BSON_SUBTYPE_VECTOR, in bytes
+    #define BSON_VECTOR_HEADER_LEN 2
+
+* Byte length for a Vector with specific element type and count:
+
+  .. toctree::
+    :titlesonly:
+    :maxdepth: 1
+
+    bson_vector_int8_binary_data_length
+    bson_vector_float32_binary_data_length
+    bson_vector_packed_bit_binary_data_length
+
+* Errors:
+
+  .. code-block:: c
+
+    // Error "domain"
+    #define BSON_ERROR_VECTOR 4
+
+  .. toctree::
+    :titlesonly:
+    :maxdepth: 1
+
+    bson_vector_error_code_t

src/libbson/doc/bson_append_array_from_vector.rst

@@ -0,0 +1,40 @@
+:man_page: bson_append_array_from_vector
+
+bson_append_array_from_vector()
+===============================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+  #define BSON_APPEND_ARRAY_FROM_VECTOR(b, key, iter) \
+     bson_append_array_from_vector (b, key, (int) strlen (key), iter)
+
+  bool
+  bson_append_array_from_vector (bson_t *bson,
+                                 const char *key,
+                                 int key_length,
+                                 const bson_iter_t *iter);
+
+Parameters
+----------
+
+* ``bson``: A :symbol:`bson_t`.
+* ``key``: An ASCII C string containing the name of the field.
+* ``key_length``: The length of ``key`` in bytes, or -1 to determine the length with ``strlen()``.
+* ``iter``: A :symbol:`bson_iter_t` pointing to any supported :doc:`binary_vector` field.
+
+Description
+-----------
+
+Converts the Vector pointed to by ``iter`` into a plain BSON Array, written to ``bson`` under the name ``key``.
+
+Returns
+-------
+
+Returns ``true`` if the operation was applied successfully. The function fails if appending the array grows ``bson`` larger than INT32_MAX, or if ``iter`` doesn't point to a valid recognized Vector type.
+
+.. seealso::
+
+  | :symbol:`bson_array_builder_append_vector_elements`

src/libbson/doc/bson_append_array_from_vector_float32.rst

@@ -0,0 +1,40 @@
+:man_page: bson_append_array_from_vector_float32
+
+bson_append_array_from_vector_float32()
+=======================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+  #define BSON_APPEND_ARRAY_FROM_VECTOR_FLOAT32(b, key, view) \
+     bson_append_array_from_vector_float32 (b, key, (int) strlen (key), view)
+
+  bool
+  bson_append_array_from_vector_float32 (bson_t *bson,
+                                         const char *key,
+                                         int key_length,
+                                         bson_vector_float32_const_view_t view);
+
+Parameters
+----------
+
+* ``bson``: A :symbol:`bson_t`.
+* ``key``: An ASCII C string containing the name of the field.
+* ``key_length``: The length of ``key`` in bytes, or -1 to determine the length with ``strlen()``.
+* ``view``: A :symbol:`bson_vector_float32_const_view_t` pointing to validated ``float32`` :doc:`binary_vector` data.
+
+Description
+-----------
+
+Converts the Vector pointed to by ``view`` into a plain BSON Array, written to ``bson`` under the name ``key``.
+
+Returns
+-------
+
+Returns ``true`` if the operation was applied successfully. The function fails if appending the array grows ``bson`` larger than INT32_MAX.
+
+.. seealso::
+
+  | :symbol:`bson_array_builder_append_vector_float32_elements`

src/libbson/doc/bson_append_array_from_vector_int8.rst

@@ -0,0 +1,40 @@
+:man_page: bson_append_array_from_vector_int8
+
+bson_append_array_from_vector_int8()
+====================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+  #define BSON_APPEND_ARRAY_FROM_VECTOR_INT8(b, key, view) \
+     bson_append_array_from_vector_int8 (b, key, (int) strlen (key), view)
+
+  bool
+  bson_append_array_from_vector_int8 (bson_t *bson,
+                                      const char *key,
+                                      int key_length,
+                                      bson_vector_int8_const_view_t view);
+
+Parameters
+----------
+
+* ``bson``: A :symbol:`bson_t`.
+* ``key``: An ASCII C string containing the name of the field.
+* ``key_length``: The length of ``key`` in bytes, or -1 to determine the length with ``strlen()``.
+* ``view``: A :symbol:`bson_vector_int8_const_view_t` pointing to validated ``int8`` :doc:`binary_vector` data.
+
+Description
+-----------
+
+Converts the Vector pointed to by ``view`` into a plain BSON Array, written to ``bson`` under the name ``key``.
+
+Returns
+-------
+
+Returns ``true`` if the operation was applied successfully. The function fails if appending the array grows ``bson`` larger than INT32_MAX.
+
+.. seealso::
+
+  | :symbol:`bson_array_builder_append_vector_int8_elements`

src/libbson/doc/bson_append_array_from_vector_packed_bit.rst

@@ -0,0 +1,40 @@
+:man_page: bson_append_array_from_vector_packed_bit
+
+bson_append_array_from_vector_packed_bit()
+==========================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+  #define BSON_APPEND_ARRAY_FROM_VECTOR_PACKED_BIT(b, key, view) \
+     bson_append_array_from_vector_packed_bit (b, key, (int) strlen (key), view)
+
+  bool
+  bson_append_array_from_vector_packed_bit (bson_t *bson,
+                                            const char *key,
+                                            int key_length,
+                                            bson_vector_packed_bit_const_view_t view);
+
+Parameters
+----------
+
+* ``bson``: A :symbol:`bson_t`.
+* ``key``: An ASCII C string containing the name of the field.
+* ``key_length``: The length of ``key`` in bytes, or -1 to determine the length with ``strlen()``.
+* ``view``: A :symbol:`bson_vector_packed_bit_const_view_t` pointing to validated ``packed_bit`` :doc:`binary_vector` data.
+
+Description
+-----------
+
+Converts the Vector pointed to by ``view`` into a plain BSON Array, written to ``bson`` under the name ``key``.
+
+Returns
+-------
+
+Returns ``true`` if the operation was applied successfully. The function fails if appending the array grows ``bson`` larger than INT32_MAX.
+
+.. seealso::
+
+  | :symbol:`bson_array_builder_append_vector_packed_bit_elements`

src/libbson/doc/bson_append_binary.rst

@@ -38,3 +38,7 @@ Returns
 -------
 
 Returns ``true`` if the operation was applied successfully. The function will fail if appending ``binary`` grows ``bson`` larger than INT32_MAX.
+
+.. seealso::
+
+  | :symbol:`bson_append_binary_uninit`

src/libbson/doc/bson_append_binary_uninit.rst

@@ -0,0 +1,43 @@
+:man_page: bson_append_binary_uninit
+
+bson_append_binary_uninit()
+===========================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+  #define BSON_APPEND_BINARY_UNINIT(b, key, subtype, val, len) \
+     bson_append_binary_uninit (b, key, (int) strlen (key), subtype, val, len)
+
+  bool
+  bson_append_binary_uninit (bson_t *bson,
+                             const char *key,
+                             int key_length,
+                             bson_subtype_t subtype,
+                             uint8_t **binary,
+                             uint32_t length);
+
+Parameters
+----------
+
+* ``bson``: A :symbol:`bson_t`.
+* ``key``: The key name.
+* ``key_length``: The length of ``key`` in bytes or -1 to use strlen().
+* ``subtype``: A bson_subtype_t indicating the binary subtype.
+* ``binary``: Location for a pointer that will receive a writable pointer to the uninitialized binary data block.
+* ``length``: The length of ``buffer`` in bytes.
+
+Description
+-----------
+
+The :symbol:`bson_append_binary_uninit()` function is an alternative to :symbol:`bson_append_binary()` which allows applications to assemble the contents of the binary field within the :symbol:`bson_t`, without an additional temporary buffer.
+On success, the caller MUST write to every byte of the binary data block if the resulting :symbol:`bson_t` is to be used.
+The buffer that ``binary`` points to is only valid until the iterator's :symbol:`bson_t` is otherwise modified or freed.
+
+
+Returns
+-------
+
+Returns ``true`` if the uninitialized ``binary`` item was appended. The function will fail if appending ``binary`` grows ``bson`` larger than INT32_MAX.