CDRIVER-4317 Add support for rewrapManyDataKey and keyMaterial (#991)

* Move existing CSE tests into legacy subdirectory

* Add client_side_encryption/unified to unified test suite

* Add ClientEncryption.createKey()

* Add unified test support for ClientEncryption.createKey()

* Add const to parameters of result_from_*() functions

* Refactor and export result_from_bulk_write() helper in result.c

* Implement mongoc_client_encryption_rewrap_many_datakey()

* Add unified tests support for rewrapManyDataKey tests

* Add support for custom key material in dataKeyOpts

* Add test runner support for custom key material

* Add unified tests for createKey()

* Update skip-tests.txt with new paths to legacy subdirectory

* Add prose test for custom key material

* Number CSE prose tests

* Add documentation for mongoc_client_encryption_create_key

* Add documentation for mongoc_client_encryption_datakey_opts_set_keymaterial

* Add documentation for mongoc_client_encryption_rewrap_many_datakey

Author: eramongodb

.evergreen/skip-tests.txt

@@ -75,7 +75,7 @@
 
 /transactions/legacy/mongos-recovery-token/"commitTransaction retry fails on new mongos" # fails with server selection timeout (CDRIVER-4268)
 /crud/unified/aggregate-out-readConcern/"readConcern available with out stage" # server error on sharded: "PlanExecutor error" (CDRIVER-4161)
-/client_side_encryption/azureKMS/"Insert a document with auto encryption using Azure KMS provider" # failing on RHEL 7.0 / RHEL 7.1 (CDRIVER-3814)
+/client_side_encryption/legacy/azureKMS/"Insert a document with auto encryption using Azure KMS provider" # failing on RHEL 7.0 / RHEL 7.1 (CDRIVER-3814)
 /transactions/legacy/pin-mongos/"unpin after transient error within a transaction and commit" # (CDRIVER-4351) server selection timeout (on ASAN Tests Ubuntu 18.04 build variant)
 /Samples # (CDRIVER-4352) strange "heartbeat failed" error
 /server_discovery_and_monitoring/monitoring/heartbeat/pooled/dns # (CDRIVER-4353) this initially seemed like a zSeries w/ RHEL8 issue, but it also appeared on arm64 w/ Ubuntu 18.04
@@ -86,5 +86,5 @@
 /sessions/unified/snapshot-sessions/"countDocuments operation with snapshot" # (CDRIVER-4355) error: checking expectResult:  { "$numberInt" : "2" }
 /load_balancers/non-lb-connection-establishment/"operations against non-load balanced clusters fail if URI contains loadBalanced=true" # (CDRIVER-4356) error: expected error to contain "Driver attempted to initialize in load balancing mode, but the server does not support this mode", but got: "BSON field 'hello.loadBalanced' is an unknown field."
 
-/client_side_encryption/explain  # Failure with diverging behavior between csfle and mongocryptd
+/client_side_encryption/legacy/explain  # Failure with diverging behavior between csfle and mongocryptd
 /client_side_encryption/bypass_spawning_mongocryptd/mongocryptdBypassSpawn  # Fails if csfle is visible

src/libmongoc/doc/api.rst

@@ -15,6 +15,7 @@ API Reference
    mongoc_change_stream_t
    mongoc_client_encryption_t
    mongoc_client_encryption_datakey_opts_t
+   mongoc_client_encryption_rewrap_many_datakey_result_t
    mongoc_client_encryption_encrypt_opts_t
    mongoc_client_encryption_opts_t
    mongoc_client_pool_t

src/libmongoc/doc/mongoc_client_encryption_create_datakey.rst

@@ -16,28 +16,8 @@ Synopsis
       bson_value_t *keyid,
       bson_error_t *error);
 
-Creates a new key document in the key vault collection and sets ``keyid`` to the UUID of the
-newly created key if ``keyid`` is not NULL. The new key can be used to configure automatic encryption (see :symbol:`mongoc_client_enable_auto_encryption()` and :symbol:`mongoc_client_pool_enable_auto_encryption()`) or for explicit encryption (see :symbol:`mongoc_client_encryption_encrypt()`).
-
-The created key document is inserted into the key vault collection (identified via :symbol:`mongoc_client_encryption_opts_set_keyvault_namespace()`) with majority write concern.
-
-``keyid`` is always initialized (even on error). Caller must call :symbol:`bson_value_destroy()` on ``keyid`` to free.
-
-Parameters
-----------
-
-* ``client_encryption``: A :symbol:`mongoc_client_encryption_t`.
-* ``kms_provider``: A string identifying the Key Management Service (KMS) provider used to encrypt the datakey (e.g. "aws" or "local").
-* ``opts``: A :symbol:`mongoc_client_encryption_datakey_opts_t`
-* ``keyid``: The resulting UUID key ID of the newly created key.
-* ``error``: A :symbol:`bson_error_t`
-
-Returns
--------
-
-Returns ``true`` if successful. Returns ``false`` and sets ``error`` otherwise.
+Alias function equivalent to :symbol:`mongoc_client_encryption_create_key`.
 
 .. seealso::
 
   | :symbol:`mongoc_client_encryption_datakey_opts_t`
-

src/libmongoc/doc/mongoc_client_encryption_create_key.rst

@@ -0,0 +1,42 @@
+:man_page: mongoc_client_encryption_create_key
+
+mongoc_client_encryption_create_key()
+=====================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+   bool
+   mongoc_client_encryption_create_key (
+      mongoc_client_encryption_t *client_encryption,
+      const char *kms_provider,
+      mongoc_client_encryption_datakey_opts_t *opts,
+      bson_value_t *keyid,
+      bson_error_t *error);
+
+Creates a new key document in the key vault collection and sets ``keyid`` to the UUID of the
+newly created key if ``keyid`` is not NULL. The new key can be used to configure automatic encryption (see :symbol:`mongoc_client_enable_auto_encryption()` and :symbol:`mongoc_client_pool_enable_auto_encryption()`) or for explicit encryption (see :symbol:`mongoc_client_encryption_encrypt()`).
+
+The created key document is inserted into the key vault collection (identified via :symbol:`mongoc_client_encryption_opts_set_keyvault_namespace()`) with majority write concern.
+
+``keyid`` is always initialized (even on error). Caller must call :symbol:`bson_value_destroy()` on ``keyid`` to free.
+
+Parameters
+----------
+
+* ``client_encryption``: A :symbol:`mongoc_client_encryption_t`.
+* ``kms_provider``: A string identifying the Key Management Service (KMS) provider used to encrypt the datakey (e.g. "aws" or "local").
+* ``opts``: A :symbol:`mongoc_client_encryption_datakey_opts_t`
+* ``keyid``: The resulting UUID key ID of the newly created key.
+* ``error``: A :symbol:`bson_error_t`
+
+Returns
+-------
+
+Returns ``true`` if successful. Returns ``false`` and sets ``error`` otherwise.
+
+.. seealso::
+
+  | :symbol:`mongoc_client_encryption_datakey_opts_t`

src/libmongoc/doc/mongoc_client_encryption_datakey_opts_set_keymaterial.rst

@@ -0,0 +1,29 @@
+:man_page: mongoc_client_encryption_datakey_opts_set_keymaterial
+
+mongoc_client_encryption_datakey_opts_set_keymaterial()
+=======================================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+   void
+   mongoc_client_encryption_datakey_opts_set_keymaterial (
+      mongoc_client_encryption_datakey_opts_t *opts,
+      const uint8_t *data,
+      uint32_t len);
+
+Sets the custom key material to be used by the data key for encryption and decryption.
+
+Parameters
+----------
+
+* ``opts``: A :symbol:`mongoc_client_encryption_datakey_opts_t`
+* ``data``: A pointer to the bytes constituting the custom key material.
+* ``len``: The length of the bytes constituting the custom key material.
+
+Description
+-----------
+
+Key material is used to encrypt and decrypt data. If custom key material is not provided, the key material for the new data key is generated from a cryptographically secure random device.

src/libmongoc/doc/mongoc_client_encryption_datakey_opts_t.rst

@@ -25,8 +25,9 @@ Used to set options for :symbol:`mongoc_client_encryption_create_datakey()`.
     mongoc_client_encryption_datakey_opts_destroy
     mongoc_client_encryption_datakey_opts_set_masterkey
     mongoc_client_encryption_datakey_opts_set_keyaltnames
+    mongoc_client_encryption_datakey_opts_set_keymaterial
 
 .. seealso::
 
+  | :symbol:`mongoc_client_encryption_create_key()`
   | :symbol:`mongoc_client_encryption_create_datakey()`
-

src/libmongoc/doc/mongoc_client_encryption_rewrap_many_datakey.rst

@@ -0,0 +1,51 @@
+:man_page: mongoc_client_encryption_rewrap_many_datakey
+
+mongoc_client_encryption_rewrap_many_datakey()
+==============================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+   bool
+   mongoc_client_encryption_rewrap_many_datakey (
+      mongoc_client_encryption_t *client_encryption,
+      const bson_t *filter,
+      const char *provider,
+      const bson_t *master_key,
+      mongoc_client_encryption_rewrap_many_datakey_result_t *result,
+      bson_error_t *error);
+
+Rewraps zero or more data keys in the key vault collection that match the
+provided ``filter``.
+
+A ``NULL`` argument for ``filter`` is equivalent to being given an empty
+document (match all).
+
+If ``provider`` is ``NULL``, rewraps matching data keys with their current KMS
+provider.
+
+If ``provider`` is not ``NULL``, rewraps matching data keys with the new KMS
+provider as described by ``master_key``. The ``master_key`` document must
+conform to the `Client Side Encryption specification
+<https://github.com/mongodb/specifications/blob/master/source/client-side-encryption/client-side-encryption.rst#masterkey>`_.
+
+Parameters
+----------
+
+* ``client_encryption``: A :symbol:`mongoc_client_encryption_t`.
+* ``filter``: The filter to use when finding data keys to rewrap in the key vault collection.
+* ``provider``: The new KMS provider to use to encrypt the data keys, or ``NULL`` to use the current KMS provider(s).
+* ``master_key``: The master key fields corresponding to the new KMS provider when ``provider`` is not ``NULL``.
+* ``result``: An optional :symbol:`mongoc_client_encryption_rewrap_many_datakey_result_t`.
+* ``error``: A :symbol:`bson_error_t` set on failure.
+
+Returns
+-------
+
+Returns ``true`` if successful. Returns ``false`` and sets ``error`` otherwise.
+
+.. seealso::
+
+  | :symbol:`mongoc_client_encryption_rewrap_many_datakey_result_t`

src/libmongoc/doc/mongoc_client_encryption_rewrap_many_datakey_result_destroy.rst

@@ -0,0 +1,20 @@
+:man_page: mongoc_client_encryption_rewrap_many_datakey_result_destroy
+
+mongoc_client_encryption_rewrap_many_datakey_result_destroy()
+=============================================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+   void
+   mongoc_client_encryption_rewrap_many_datakey_result_destroy (
+      mongoc_client_encryption_rewrap_many_datakey_result_t *result);
+
+Frees resources of a :symbol:`mongoc_client_encryption_rewrap_many_datakey_result_t` created with :symbol:`mongoc_client_encryption_rewrap_many_datakey_result_new()`. Does nothing if ``NULL`` is passed.
+
+Parameters
+----------
+
+* ``result``: A :symbol:`mongoc_client_encryption_rewrap_many_datakey_result_t`.

src/libmongoc/doc/mongoc_client_encryption_rewrap_many_datakey_result_get_bulk_write_result.rst

@@ -0,0 +1,30 @@
+:man_page: mongoc_client_encryption_rewrap_many_datakey_result_get_bulk_write_result
+
+mongoc_client_encryption_rewrap_many_datakey_result_get_bulk_write_result()
+===========================================================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+   const bson_t *
+   mongoc_client_encryption_rewrap_many_datakey_result_get_bulk_write_result (
+      mongoc_client_encryption_rewrap_many_datakey_result_t *result)
+      BSON_GNUC_WARN_UNUSED_RESULT;
+
+Get the bulk write result set by a successful call to :symbol:`mongoc_client_encryption_rewrap_many_datakey()`.
+
+Parameters
+----------
+
+* ``result``: A :symbol:`mongoc_client_encryption_rewrap_many_datakey_result_t`.
+
+Returns
+-------
+
+A :symbol:`bson_t` that must not be modified or freed if ``result`` is not ``NULL``. Otherwise, returns ``NULL``.
+
+.. seealso::
+
+  | :symbol:`mongoc_client_encryption_rewrap_many_datakey()`

src/libmongoc/doc/mongoc_client_encryption_rewrap_many_datakey_result_new.rst

@@ -0,0 +1,18 @@
+:man_page: mongoc_client_encryption_rewrap_many_datakey_result_new 
+
+mongoc_client_encryption_rewrap_many_datakey_result_new()
+=========================================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+  mongoc_client_encryption_rewrap_many_datakey_result_t *
+  mongoc_client_encryption_rewrap_many_datakey_result_new (void)
+     BSON_GNUC_WARN_UNUSED_RESULT;
+
+Returns
+-------
+
+A new :symbol:`mongoc_client_encryption_rewrap_many_datakey_result_t` that must be freed with :symbol:`mongoc_client_encryption_rewrap_many_datakey_result_destroy()`.

src/libmongoc/doc/mongoc_client_encryption_rewrap_many_datakey_result_t.rst

@@ -0,0 +1,32 @@
+:man_page: mongoc_client_encryption_rewrap_many_datakey_result_t
+
+mongoc_client_encryption_rewrap_many_datakey_result_t
+=====================================================
+
+Synopsis
+--------
+
+.. code-block:: c
+
+   typedef struct _mongoc_client_encryption_rewrap_many_datakey_result_t
+      mongoc_client_encryption_rewrap_many_datakey_result_t;
+
+
+Used to access the result of :symbol:`mongoc_client_encryption_rewrap_many_datakey()`.
+
+.. only:: html
+
+  Functions
+  ---------
+
+  .. toctree::
+    :titlesonly:
+    :maxdepth: 1
+
+    mongoc_client_encryption_rewrap_many_datakey_result_new
+    mongoc_client_encryption_rewrap_many_datakey_result_destroy
+    mongoc_client_encryption_rewrap_many_datakey_result_get_bulk_write_result
+
+.. seealso::
+
+  | :symbol:`mongoc_client_encryption_rewrap_many_datakey()`

src/libmongoc/doc/mongoc_client_encryption_t.rst

@@ -34,7 +34,9 @@ The key vault client, configured via :symbol:`mongoc_client_encryption_opts_set_
 
     mongoc_client_encryption_new
     mongoc_client_encryption_destroy
+    mongoc_client_encryption_create_key
     mongoc_client_encryption_create_datakey
+    mongoc_client_encryption_rewrap_many_datakey
     mongoc_client_encryption_encrypt
     mongoc_client_encryption_decrypt
 
@@ -47,4 +49,3 @@ The key vault client, configured via :symbol:`mongoc_client_encryption_opts_set_
   | The guide for :doc:`Using Client-Side Field Level Encryption <using_client_side_encryption>` for libmongoc
 
   | The MongoDB Manual for `Client-Side Field Level Encryption <https://docs.mongodb.com/manual/core/security-client-side-encryption/>`_
-