DOCSP-30655 and DOCSP-30645 QE contention factor and cardinality (#3383)

nvillahermosa-mdb · jason-price-mongodb · Chris Cho · web-flow · commit 3ecc5ab9bb71 · 2023-08-03T11:35:17.000-04:00
* Docsp 28305 qe on disk format wire protocol (#3244) * Cleaned feature branch * Internal PR feedback * Fixed lingering merge text * External review: removed write amplification for delete operations * Revert "Remove insertmany from QE restricted operations (#3251)" (#3296) This reverts commit f1377c73483ceed9744d4d48647b56295706dcdc. * Docsp 29188 remove insertmany from restricted (#3300) * One line fix * Removed wording around future release functionality for index compaction. Left key creation language because there's a separate ticket for that content. * Light editorial cleanup, removed refs to technical preview * Attempted to clean up wording around unique index limitations * Attempted to clarify limitation around validation settings * PR feedback * Syntax fix * Docsp 28249 qe redaction (#3291) * Rebase to latest state of qe-equality-ga * Cleaned up old :doc: directives * Removed self reference links * Cleaned up old version references * Spellcheck * Build cleanup * Changed collstats redaction per SERVER-75266 * Changed collstats redaction per SERVER-75266--amend * Moved log redaction to the existing redaction heading * Moved log redaction to the existing redaction heading--amend * Moved log redaction to the existing redaction heading--amend * PR feedback * Cleanup from qe-equality-ga branch diversion * PR feedback * Copy edit, passive voice/future * Added shortdesc to limitations * Shortdesc and editorial cleanup * Rebase cleanup * Internal review feedback * External PR feedback * Fle sample app refactor (#3397) * add java tutorial source only * maven pom.xml for java build * update object property passing * add vm options * update variables per sync meeting * Java tutorial naming updates * c# updates * c# updates * fix indents * update testing instructions * python - naming updates * go tutorial - naming updates * start/end tags and readme * Go tutorial: added comment labels * python include tags and readme * add kms placeholder * add envrc_template * update README * rename project * start/end tags * remove extra method * clean up * Delete QueryableEncryption.csproj * Go tutorial: add readme, sample environment template small updates * remove whitespace * fix label * Java tutorial: add labels * refactored to add auto dek * c# key auto generation * refactored tutorial template * js feedback * python auto-key * python replace main script * python tutorial fix * java tutorial auto key creation * create/find first draft * first draft tutorial text * typo * Go tutorial updates for auto key creation * Python tutorial cleanup * remove encryptedFieldsMap * tutorial text feedback * Add CMK step, fix errors, add Azure tutorial * admonition for persisting keyId * keyId admonition edits * cc feedback * c# cleanup * fix compile error * move return statements * add project and fix README * updates to admonition * PRR fixes to admonition * cc feedback * PRR fixes for PyMongo tutorial * remove insert client from PyMongo tutorial * apply changes to azure page * envrc updates for PyMongo tutorial * apply changes to gcp page * PRR fix for PyMongo tutorial: check insert result * apply changes to kmip page * adds refactored mongosh sample app * fixes mongosh kmip issue * Java tutorial dotenv and README updates * Java README, add dotenv to deps * update variable names per code review * code review suggestions * Golang tutorial updates and various README updates * fix encrypted fields map * fix kms * start adding language tabs * PRR review fixes for Java tutorial * add comment in Python tutorial * PRR fixes for Golang tutorial * fix for relocated files * c# edits * go edits * java edits * python edits * add tabs for all languages * fix go merge conflict * fix go merge conflict * update node variable names per code review * remove insert client * update README files * provide more detail in the README * adds package.json to mongosh and updates README * removes package.json * bd c# feedback * fix merge error * README updates for Java and Python, requirements update for Python * bd c# feedback * Java and Golang README updates * envrc fixes * node readme fix * updates to READMEs * fix link to keys and key vaults * go tutorial fix placeholder * fix copypasta * fix driver tab ids * encryptionCollectionName -> encryptedCollectionName and encryptionDatabaseName -> encryptedDatabaseName * checks for existing master-key.txt before generating new file * checks value of acknowledged field on insert results * updates README * remove create insert client step * no need to specify shared lib in mongosh * clean up * tutorial fixes * code fixes for tutorial * Go fix comment structure * Python code: update placeholder comments * mongosh updates * Bailey feedback and requested changes * updated code comments to prevent confusion about placeholders * mongosh - updated code comments to prevent confusion about placeholders * fix driver tabs for nodejs and java-sync * small aws fixes * azure tutorial * fix language literalinclude references * azure tutorial * gcp tutorial * path updates * do not install mongosh via homebrew for QE * fix java paths aws * tabid fix for java-sync * tabid fix for nodejs * Update README.md * do not install mongosh via homebrew for QE * tabid and indentation fixes * direnv install * remove data * removes master-key * removes .envrc * change insert-patient-document -> insert-document * snippet fixes * literalinclude fixes * fix tabids and include paths * fix references * kmip tutorial + code changes * update go version * shell placeholder text * fix includes references * shell placeholder text azure * shell placeholder text gcp * shell placeholder text kmip * quick start draft * Java KMIP update * quick start fixes * quick start fixes * kmip include comment fix * Clarify Java KMIP certificates and TLS options * fix go code * update ref tags * more ref tags + Learn More sections * rename tutorials and quick start * fix go code * fix python comment * update text * update import * Java envrc_template fix * link to README in Quickstart * quick-start fixes + automatic encryption wording * reformat cmk from command line * automatic encryption wording * formatting * formatting * golang -> go * Go kmip comment name fix * refactor branch logic * add data models to aws tutorial * java tutorial - updates for quickstart * fix c# data models * python tutorial - fix comment boundaries * add C# data models + fix includes * python - show kms_provider_credentials * auto > automatic * add placeholder * update java dependencies to latest * update READMEs to include mention of release candidate * bd c# feedback * move c# data models * update kmsProviders variable * link to readmes in environment variables admonition * re-adding deleted method * java kmip add link * tutorial fixes * move start and end comments for kmsProviders * mongosh fixes * mongosh kmsProviderCredentials variable * mongosh updates * add go models to tutorials and quick start * go syntax highlight * spacing * add shell tab * bd c# feedback * kmip fixes * gcp fix * go - fix comment boundaries * remove mongosh * fix build error * staging build * remove duplicates --------- Co-authored-by: Jordan Smith <jordan.smith@mongodb.com> Co-authored-by: Mike Woofter <mike.woofter@mongodb.com> Co-authored-by: Mike Woofter <108414937+mongoKart@users.noreply.github.com> Co-authored-by: Joseph Dougherty <joseph.dougherty@mongodb.com> Co-authored-by: jmd-mongo <73852296+jmd-mongo@users.noreply.github.com> * Adding includes for contention factor * build * Added ref to contention from CSFLE doc * Added contention configuration * Merge fixes from rebasing to master * Removed misleading info on security guarantees * External review feedback * Clarified cases for increasing contention * Clarified contention factor use case * Typo * Whitespace fix --------- Co-authored-by: jason-price-mongodb <69260375+jason-price-mongodb@users.noreply.github.com> Co-authored-by: Chris Cho <chris.cho@mongodb.com> Co-authored-by: Jordan Smith <jordan.smith@mongodb.com> Co-authored-by: Mike Woofter <mike.woofter@mongodb.com> Co-authored-by: Mike Woofter <108414937+mongoKart@users.noreply.github.com> Co-authored-by: Joseph Dougherty <joseph.dougherty@mongodb.com> Co-authored-by: jmd-mongo <73852296+jmd-mongo@users.noreply.github.com>
diff --git a/source/core/csfle/fundamentals/manual-encryption.txt b/source/core/csfle/fundamentals/manual-encryption.txt
@@ -102,7 +102,7 @@ section of this guide.
 Automatic Decryption
 ~~~~~~~~~~~~~~~~~~~~
 
-To decrypt your fields automatically, you must configure your
+To decrypt your fields automatically, configure your
 ``MongoClient`` instance as follows:
 
 - Specify your {+key-vault-long+}
diff --git a/source/core/queryable-encryption/features.txt b/source/core/queryable-encryption/features.txt
@@ -30,8 +30,6 @@ encrypt data before transporting it over the network using fully
 randomized encryption, while maintaining queryability.
 Sensitive data is transparently encrypted and decrypted by the client
 and only communicated to and from the server in encrypted form.
-The security guarantees for sensitive fields containing both low
-cardinality (low-frequency) data and high cardinality data are identical
 
 Unlike :ref:`Client-Side Field Level Encryption <manual-csfle-feature>`
 that can use :ref:`Deterministic Encryption <csfle-deterministic-encryption>`,
diff --git a/source/core/queryable-encryption/fundamentals/encrypt-and-query.txt b/source/core/queryable-encryption/fundamentals/encrypt-and-query.txt
@@ -17,10 +17,27 @@ Overview
 
 Learn about the following {+qe+} topics:
 
+- Considerations when enabling queries on an encrypted field.
 - How to specify fields for encryption.
-- How to specify whether an encrypted field is queryable when you create a collection.
+- How to configure an encrypted field so that it is queryable.
 - Query types and which ones you can use on encrypted fields.
-- Considerations when enabling queries on an encrypted field.
+- How to optimize query performance on encrypted fields.
+
+Considerations when Enabling Querying
+-------------------------------------
+
+When you use {+qe+}, you can choose whether to make an encrypted field queryable.
+If you don't need to perform CRUD operations that require you
+to query an encrypted field, you may not need to enable querying on that field.
+You can still retrieve the entire document by querying other fields that are queryable or not encrypted.
+
+When you make encrypted fields queryable, {+qe+} creates an index for each encrypted field, which
+can make write operations on that field take longer. When a write operation updates
+an indexed field, MongoDB also updates the related index.
+
+When you create an encrypted collection, MongoDB creates 
+:ref:`two metadata collections <qe-metadata-collections>`, increasing 
+the storage space requirements.
 
 .. _qe-specify-fields-for-encryption:
 
@@ -147,8 +164,8 @@ to each entry that includes the key:
 
 .. _qe-enable-queries:
 
-Specify Fields for Querying
----------------------------
+Configure Fields for Querying
+-----------------------------
 
 Include the ``queries`` property on fields you want to make queryable in your JSON
 schema. This enables an authorized client to issue read and write
@@ -187,6 +204,21 @@ Add the ``queries`` property to the previous example schema to make the
          },
       ]
    }
+
+.. _qe-contention:
+
+Configure Contention Factor
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Include the ``contention`` property on queryable fields to prefer either
+find performance, or write and update performance.
+
+.. include:: /includes/fact-qe-csfle-contention.rst
+
+Example
++++++++
+
+.. include:: /includes/example-qe-csfle-contention.rst
 .. _qe-query-types:
 
 Query Types
@@ -229,23 +261,6 @@ following :term:`BSON` types:
 - ``array``
 - ``javascriptWithScope`` (*Deprecated in MongoDB 4.4*)
 
-
-Considerations when Enabling Querying
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-When you use {+qe+}, you can choose whether to make an encrypted field queryable.
-If you don't need to perform query operations, or write operations that require you
-to query an encrypted field, you may not need to enable querying on that field.
-You can still retrieve the entire document by querying other fields that are queryable or not encrypted.
-
-When you make encrypted fields queryable, {+qe+} creates an index for each encrypted field, which
-can make write operations on that field take longer. When a write operation updates
-an indexed field, MongoDB also updates the related index.
-
-When you create an encrypted collection, MongoDB creates 
-:ref:`two metadata collections <qe-metadata-collections>`, increasing 
-the storage space requirements.
-
 Client and Server Schemas
 -------------------------
 
diff --git a/source/core/queryable-encryption/fundamentals/manual-encryption.txt b/source/core/queryable-encryption/fundamentals/manual-encryption.txt
@@ -66,7 +66,7 @@ instance. Specify the following:
 - The value to be encrypted
 - The algorithm used, either ``Indexed`` or ``Unindexed``
 - The ID of the {+dek-long+}
-- The contention factor (if you are using the ``Indexed`` algorithm)
+- The :ref:`contention factor <qe-contention>` (if you are using the ``Indexed`` algorithm)
 - If performing a read operation, set the query type defined for your 
   field (if you are using the ``Indexed`` algorithm)
 
diff --git a/source/includes/example-qe-csfle-contention.rst b/source/includes/example-qe-csfle-contention.rst
@@ -0,0 +1,34 @@
+The example below sets ``contention`` to 0 for the low cardinality
+Social Security Number (SSN) and patient ID fields, since these are
+unique identifiers that shouldn't repeat in the data set:
+
+.. code-block:: javascript
+   :emphasize-lines: 7,13
+
+   const encryptedFieldsObject = {
+      fields: [
+         {
+            path: "patientId",
+            bsonType: "int",
+            queries: { queryType: "equality",
+                       contention: "0"}
+         },
+         {
+            path: "patientInfo.ssn",
+            bsonType: "string",
+            queries: { queryType: "equality",
+                       contention: "0"}
+         },
+         ...
+      ]
+   }
+
+.. Example context from Kenn White:
+.. - full name (unencrypted, ~750 possible values)
+.. - mobile (encrypted, high cardinality)
+.. - SSN (encrypted, high cardinality)
+.. - Address (unencrypted,high cardinality)
+.. - DOB between 1930-1990 (unencrypted, ~22K values)
+.. - gender (encrypted, Male/Female/Non-binary)
+.. - creditCard.type (encrypted, 4 types)
+.. - creditCard.expiry (encrypted, ~84 possible values)
diff --git a/source/includes/fact-qe-csfle-contention.rst b/source/includes/fact-qe-csfle-contention.rst
@@ -0,0 +1,33 @@
+Inserting the same field/value pair into multiple documents in close
+succession can cause conflicts that delay insert operations.
+
+MongoDB tracks the occurrences of each field/value pair in an
+encrypted collection using an internal counter. The contention factor
+partitions this counter, similar to an array. This minimizes issues with
+incrementing the counter when using ``insert``, ``update``, or ``findAndModify`` to add or modify an encrypted field
+with the same field/value pair in close succession. ``contention = 0``
+creates an array with one element
+at index 0. ``contention = 4`` creates an array with 5 elements at
+indexes 0-4. MongoDB increments a random array element during insert. If
+unset, ``contention`` defaults to 8.
+
+High contention improves the performance of insert and update operations on low cardinality fields, but decreases find performance.
+
+Consider increasing ``contention`` above the default value of 8 only if:
+
+- The field has low cardinality or low selectivity. A ``state`` field
+  may have 50 values, but if 99% of the data points use ``{state: NY}``,
+  that pair is likely to cause contention.
+
+- Write and update operations frequently modify the field. Since high
+  contention values sacrifice find performance in favor of write and
+  update operations, the benefit of a high contention factor for a
+  rarely updated field is unlikely to outweigh the drawback.
+
+Consider decreasing ``contention`` if:
+
+- The field is high cardinality and contains entirely unique values,
+  such as a credit card number.
+
+- The field is often queried, but never or rarely updated. In this
+  case, find performance is preferable to write and update performance.
