Skip to content

Commit aa28f78

Browse files
kevinAlbskevinAlbs
authored
DRIVERS-2524 do not create or drop eccCollection (#1396)
* do not create or drop the eccCollection * DRIVERS-2586 do not document `escCollection` and `ecocCollection` options * remove eccCollection from fle2v2-CreateCollection * remove `eccCollection` from `encryptedFields` data files * regenerate fle2v2 tests * remove eccCollection from fle2v2-EncryptedFields-vs-EncryptedFieldsMap * remove eccCollection from fle2v2-Range-WrongType * assert eccCollection is not created * remove incorrect comment * add wire version check and test * remove `escCollection` and `ecocCollection` from test data * regenerate tests * add $$exists to legacy format * add $$exists to tests Assert that state collections names are not sent to server * test encryptedFields is consulted for metadata collection names * change SHOULD to MUST * use YAML anchors for collection names * add comment that ecc collection is no longer created for QEv2 * remove `encryptedFieldsMap with cyclic entries does not loop` * use YAML anchors for encryptedFields * use `null`, not $$exists * Revert "add $$exists to legacy format" This reverts commit 72280f9. * swap order of `base64` and `subType` * remove unnecessary anchor * regenerate fle2v2-CreateCollection-OldServer
1 parent 364464f commit aa28f78

File tree

125 files changed

+481
-1724
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

125 files changed

+481
-1724
lines changed

‎source/client-side-encryption/client-side-encryption.rst

Lines changed: 27 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,6 @@ encryptedFields
115115
116116
{
117117
"escCollection": "enxcol_.CollectionName.esc",
118-
"eccCollection": "enxcol_.CollectionName.ecc",
119118
"ecocCollection": "enxcol_.CollectionName.ecoc",
120119
"fields": [
121120
{
@@ -134,7 +133,6 @@ encryptedFields
134133
135134
The acronyms within ``encryptedFields`` are defined as follows:
136135

137-
* ECC: Encrypted Cache Collection
138136
* ECOC: Encrypted Compaction Collection
139137
* ESC: Encrypted State Collection
140138

@@ -914,6 +912,17 @@ Drivers MUST support a BSON document option named ``encryptedFields`` for any
914912
will be interpreted by the helper method and MUST be passed to the `create`_
915913
command.
916914

915+
.. note::
916+
Users are not expected to set the ``escCollection`` and ``ecocCollection`` in
917+
``encryptedFields``. SERVER-74069 added server-side validation for those fields
918+
and no longer allows names to deviate from the following:
919+
920+
- ``enxcol_.<collectionName>.esc``
921+
- ``enxcol_.<collectionName>.ecoc`
922+
923+
Drivers MUST NOT document the ``escCollection`` and ``ecocCollection``
924+
options.
925+
917926
For a helper function, ``CreateCollection(collectionName, collectionOptions)``
918927
with the name of the database associated as `dbName`, look up the encrypted
919928
fields ``encryptedFields`` for the collection as
@@ -924,17 +933,16 @@ If a set of ``encryptedFields`` was found, then do the following operations. If
924933
any of the following operations error, the remaining operations are not
925934
attempted:
926935

936+
- Check the wire version of the writable server. If the wire version is less
937+
than 21 (for server 7.0.0), return an error containing the error message:
938+
"Driver support of Queryable Encryption is incompatible with server. Upgrade
939+
server to use Queryable Encryption."
927940
- Create the collection with name ``encryptedFields["escCollection"]`` as a
928941
clustered collection using the options
929942
``{clusteredIndex: {key: {_id: 1}, unique: true}}``. If
930943
``encryptedFields["escCollection"]`` is not set, use the collection name
931944
``enxcol_.<collectionName>.esc``. Creating this collection MUST NOT check if
932945
the collection namespace is in the ``AutoEncryptionOpts.encryptedFieldsMap``.
933-
- Create the collection with name ``encryptedFields["eccCollection"]`` as a
934-
clustered collection using the options
935-
``{clusteredIndex: {key: {_id: 1}, unique: true}}``. If
936-
``encryptedFields["eccCollection"]`` is not set, use the collection name
937-
``enxcol_.<collectionName>.ecc``. Creating this collection MUST NOT check if
938946
the collection namespace is in the ``AutoEncryptionOpts.encryptedFieldsMap``.
939947
- Create the collection with name ``encryptedFields["ecocCollection"]`` as a
940948
clustered collection using the options
@@ -1002,6 +1010,17 @@ Drivers MUST support a BSON document option named ``encryptedFields`` for any
10021010
``Collection.drop()``). This option will only be interpreted by the helper
10031011
method and MUST NOT be passed to the `drop`_ command.
10041012

1013+
.. note::
1014+
Users are not expected to set the ``escCollection`` and ``ecocCollection`` in
1015+
``encryptedFields``. SERVER-74069 added server-side validation for those fields
1016+
and no longer allows names to deviate from the following:
1017+
1018+
- ``enxcol_.<collectionName>.esc``
1019+
- ``enxcol_.<collectionName>.ecoc`
1020+
1021+
Drivers SHOULD NOT document the ``escCollection`` and ``ecocCollection``
1022+
options.
1023+
10051024
For a helper function ``DropCollection(dropOptions)`` with associated collection
10061025
named `collName` and database named `dbName`, look up the encrypted fields
10071026
``encryptedFields`` as `GetEncryptedFields(dropOptions, collName, dbname, true)`
@@ -1015,9 +1034,6 @@ are not attempted. A ``namespace not found`` error returned from the server
10151034
- Drop the collection with name ``encryptedFields["escCollection"]``. If
10161035
``encryptedFields["escCollection"]`` is not set, use the collection name
10171036
``enxcol_.<collectionName>.esc``.
1018-
- Drop the collection with name ``encryptedFields["eccCollection"]``. If
1019-
``encryptedFields["eccCollection"]`` is not set, use the collection name
1020-
``enxcol_.<collectionName>.ecc``.
10211037
- Drop the collection with name ``encryptedFields["ecocCollection"]``. If
10221038
``encryptedFields["ecocCollection"]`` is not set, use the collection name
10231039
``enxcol_.<collectionName>.ecoc``.
@@ -2714,6 +2730,7 @@ explicit session parameter as described in the
27142730
Changelog
27152731
=========
27162732
2733+
:2023-03-30: Remove ECC collection
27172734
:2023-02-01: Replace ``DataKeyOpts`` with ``masterKey`` in ``createEncryptedCollection``.
27182735
:2023-01-31: ``createEncryptedCollection`` does not check AutoEncryptionOptions for the encryptedFieldsMap.
27192736
:2023-01-30: Return ``encryptedFields`` on ``CreateCollection`` error.

‎source/client-side-encryption/etc/data/encryptedFields-Range-Date.json

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
11
{
2-
"escCollection": "enxcol_.default.esc",
3-
"eccCollection": "enxcol_.default.ecc",
4-
"ecocCollection": "enxcol_.default.ecoc",
52
"fields": [
63
{
74
"keyId": {
@@ -33,4 +30,4 @@
3330
}
3431
}
3532
]
36-
}
33+
}

‎source/client-side-encryption/etc/data/encryptedFields-Range-Decimal.json

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
11
{
2-
"escCollection": "enxcol_.default.esc",
3-
"eccCollection": "enxcol_.default.ecc",
4-
"ecocCollection": "enxcol_.default.ecoc",
52
"fields": [
63
{
74
"keyId": {
@@ -23,4 +20,4 @@
2320
}
2421
}
2522
]
26-
}
23+
}

‎source/client-side-encryption/etc/data/encryptedFields-Range-DecimalPrecision.json

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
11
{
2-
"escCollection": "enxcol_.default.esc",
3-
"eccCollection": "enxcol_.default.ecc",
4-
"ecocCollection": "enxcol_.default.ecoc",
52
"fields": [
63
{
74
"keyId": {
@@ -32,4 +29,4 @@
3229
}
3330
}
3431
]
35-
}
32+
}

‎source/client-side-encryption/etc/data/encryptedFields-Range-Double.json

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
11
{
2-
"escCollection": "enxcol_.default.esc",
3-
"eccCollection": "enxcol_.default.ecc",
4-
"ecocCollection": "enxcol_.default.ecoc",
52
"fields": [
63
{
74
"keyId": {
@@ -23,4 +20,4 @@
2320
}
2421
}
2522
]
26-
}
23+
}

‎source/client-side-encryption/etc/data/encryptedFields-Range-DoublePrecision.json

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
11
{
2-
"escCollection": "enxcol_.default.esc",
3-
"eccCollection": "enxcol_.default.ecc",
4-
"ecocCollection": "enxcol_.default.ecoc",
52
"fields": [
63
{
74
"keyId": {
@@ -32,4 +29,4 @@
3229
}
3330
}
3431
]
35-
}
32+
}

‎source/client-side-encryption/etc/data/encryptedFields-Range-Int.json

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
11
{
2-
"escCollection": "enxcol_.default.esc",
3-
"eccCollection": "enxcol_.default.ecc",
4-
"ecocCollection": "enxcol_.default.ecoc",
52
"fields": [
63
{
74
"keyId": {
@@ -29,4 +26,4 @@
2926
}
3027
}
3128
]
32-
}
29+
}

‎source/client-side-encryption/etc/data/encryptedFields-Range-Long.json

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
11
{
2-
"escCollection": "enxcol_.default.esc",
3-
"eccCollection": "enxcol_.default.ecc",
4-
"ecocCollection": "enxcol_.default.ecoc",
52
"fields": [
63
{
74
"keyId": {
@@ -29,4 +26,4 @@
2926
}
3027
}
3128
]
32-
}
29+
}

‎source/client-side-encryption/etc/data/encryptedFields.json

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
11
{
2-
"escCollection": "enxcol_.default.esc",
3-
"eccCollection": "enxcol_.default.ecc",
4-
"ecocCollection": "enxcol_.default.ecoc",
52
"fields": [
63
{
74
"keyId": {
@@ -30,4 +27,4 @@
3027
"bsonType": "string"
3128
}
3229
]
33-
}
30+
}

‎source/client-side-encryption/etc/test-templates/fle2v2-EncryptedFields-vs-EncryptedFieldsMap.yml.template

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,6 @@ tests:
1818
local: {{ local_provider() }}
1919
encryptedFieldsMap: {
2020
"default.default": {
21-
"escCollection": "enxcol_.default.esc",
22-
"eccCollection": "enxcol_.default.ecc",
23-
"ecocCollection": "enxcol_.default.ecoc",
2421
"fields": []
2522
}
2623
}

‎source/client-side-encryption/etc/test-templates/fle2v2-EncryptedFieldsMap-defaults.yml.template

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,9 +45,7 @@ tests:
4545
},
4646
"schema": {
4747
"default.default": {
48-
"fields": [],
49-
"escCollection": "enxcol_.default.esc",
50-
"ecocCollection": "enxcol_.default.ecoc"
48+
"fields": []
5149
}
5250
}
5351
}

‎source/client-side-encryption/tests/legacy/fle2v2-BypassQueryAnalysis.json

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,6 @@
1414
"collection_name": "default",
1515
"data": [],
1616
"encrypted_fields": {
17-
"escCollection": "enxcol_.default.esc",
18-
"eccCollection": "enxcol_.default.ecc",
19-
"ecocCollection": "enxcol_.default.ecoc",
2017
"fields": [
2118
{
2219
"keyId": {

‎source/client-side-encryption/tests/legacy/fle2v2-BypassQueryAnalysis.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ runOn:
88
database_name: &database_name "default"
99
collection_name: &collection_name "default"
1010
data: []
11-
encrypted_fields: &encrypted_fields {'escCollection': 'enxcol_.default.esc', 'eccCollection': 'enxcol_.default.ecc', 'ecocCollection': 'enxcol_.default.ecoc', 'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
11+
encrypted_fields: &encrypted_fields {'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
1212
key_vault_data: [{'_id': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'keyMaterial': {'$binary': {'base64': 'sHe0kz57YW7v8g9VP9sf/+K1ex4JqKc5rf/URX3n3p8XdZ6+15uXPaSayC6adWbNxkFskuMCOifDoTT+rkqMtFkDclOy884RuGGtUysq3X7zkAWYTKi8QAfKkajvVbZl2y23UqgVasdQu3OVBQCrH/xY00nNAs/52e958nVjBuzQkSb1T8pKJAyjZsHJ60+FtnfafDZSTAIBJYn7UWBCwQ==', 'subType': '00'}}, 'creationDate': {'$date': {'$numberLong': '1648914851981'}}, 'updateDate': {'$date': {'$numberLong': '1648914851981'}}, 'status': {'$numberInt': '0'}, 'masterKey': {'provider': 'local'}} ]
1313

1414
tests:

‎source/client-side-encryption/tests/legacy/fle2v2-Compact.json

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,6 @@
1414
"collection_name": "default",
1515
"data": [],
1616
"encrypted_fields": {
17-
"escCollection": "enxcol_.default.esc",
18-
"eccCollection": "enxcol_.default.ecc",
19-
"ecocCollection": "enxcol_.default.ecoc",
2017
"fields": [
2118
{
2219
"keyId": {

‎source/client-side-encryption/tests/legacy/fle2v2-Compact.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ runOn:
88
database_name: &database_name "default"
99
collection_name: &collection_name "default"
1010
data: []
11-
encrypted_fields: &encrypted_fields {'escCollection': 'enxcol_.default.esc', 'eccCollection': 'enxcol_.default.ecc', 'ecocCollection': 'enxcol_.default.ecoc', 'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
11+
encrypted_fields: &encrypted_fields {'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
1212
key_vault_data: [ {'_id': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'keyMaterial': {'$binary': {'base64': 'sHe0kz57YW7v8g9VP9sf/+K1ex4JqKc5rf/URX3n3p8XdZ6+15uXPaSayC6adWbNxkFskuMCOifDoTT+rkqMtFkDclOy884RuGGtUysq3X7zkAWYTKi8QAfKkajvVbZl2y23UqgVasdQu3OVBQCrH/xY00nNAs/52e958nVjBuzQkSb1T8pKJAyjZsHJ60+FtnfafDZSTAIBJYn7UWBCwQ==', 'subType': '00'}}, 'creationDate': {'$date': {'$numberLong': '1648914851981'}}, 'updateDate': {'$date': {'$numberLong': '1648914851981'}}, 'status': {'$numberInt': '0'}, 'masterKey': {'provider': 'local'}}, {'_id': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'keyMaterial': {'$binary': {'base64': 'HBk9BWihXExNDvTp1lUxOuxuZK2Pe2ZdVdlsxPEBkiO1bS4mG5NNDsQ7zVxJAH8BtdOYp72Ku4Y3nwc0BUpIKsvAKX4eYXtlhv5zUQxWdeNFhg9qK7qb8nqhnnLeT0f25jFSqzWJoT379hfwDeu0bebJHr35QrJ8myZdPMTEDYF08QYQ48ShRBli0S+QzBHHAQiM2iJNr4svg2WR8JSeWQ==', 'subType': '00'}}, 'creationDate': {'$date': {'$numberLong': '1648914851981'}}, 'updateDate': {'$date': {'$numberLong': '1648914851981'}}, 'status': {'$numberInt': '0'}, 'masterKey': {'provider': 'local'}} ]
1313
tests:
1414
- description: "Compact works"

‎source/client-side-encryption/tests/legacy/fle2v2-CreateCollection-OldServer.json

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
{
2+
"runOn": [
3+
{
4+
"minServerVersion": "6.0.0",
5+
"maxServerVersion": "6.3.99",
6+
"topology": [
7+
"replicaset",
8+
"sharded",
9+
"load-balanced"
10+
]
11+
}
12+
],
13+
"database_name": "default",
14+
"collection_name": "default",
15+
"tests": [
16+
{
17+
"description": "driver returns an error if creating a QEv2 collection on unsupported server",
18+
"clientOptions": {
19+
"autoEncryptOpts": {
20+
"kmsProviders": {
21+
"aws": {}
22+
},
23+
"encryptedFieldsMap": {
24+
"default.encryptedCollection": {
25+
"fields": [
26+
{
27+
"path": "firstName",
28+
"bsonType": "string",
29+
"keyId": {
30+
"$binary": {
31+
"base64": "AAAAAAAAAAAAAAAAAAAAAA==",
32+
"subType": "04"
33+
}
34+
}
35+
}
36+
]
37+
}
38+
}
39+
}
40+
},
41+
"operations": [
42+
{
43+
"name": "dropCollection",
44+
"object": "database",
45+
"arguments": {
46+
"collection": "encryptedCollection"
47+
}
48+
},
49+
{
50+
"name": "createCollection",
51+
"object": "database",
52+
"arguments": {
53+
"collection": "encryptedCollection"
54+
},
55+
"result": {
56+
"errorContains": "Driver support of Queryable Encryption is incompatible with server. Upgrade server to use Queryable Encryption."
57+
}
58+
}
59+
]
60+
}
61+
]
62+
}

‎source/client-side-encryption/tests/legacy/fle2v2-CreateCollection-OldServer.yml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Requires libmongocrypt 1.8.0.
2+
runOn:
3+
- minServerVersion: "6.0.0"
4+
maxServerVersion: "6.3.99"
5+
# FLE 2 Encrypted collections are not supported on standalone.
6+
topology: [ "replicaset", "sharded", "load-balanced" ]
7+
8+
database_name: &database_name "default"
9+
collection_name: &collection_name "default"
10+
11+
tests:
12+
- description: "driver returns an error if creating a QEv2 collection on unsupported server"
13+
clientOptions:
14+
autoEncryptOpts:
15+
kmsProviders:
16+
aws: {} # Credentials filled in from environment.
17+
encryptedFieldsMap:
18+
default.encryptedCollection: {
19+
"fields": [
20+
{
21+
"path": "firstName",
22+
"bsonType": "string",
23+
"keyId": { "$binary": { "base64": "AAAAAAAAAAAAAAAAAAAAAA==", "subType": "04" }}
24+
}
25+
]
26+
}
27+
28+
operations:
29+
# Do an initial drop to remove collections that may exist from previous test runs.
30+
- name: dropCollection
31+
object: database
32+
arguments:
33+
collection: "encryptedCollection"
34+
- name: createCollection
35+
object: database
36+
arguments:
37+
collection: "encryptedCollection"
38+
result:
39+
errorContains: "Driver support of Queryable Encryption is incompatible with server. Upgrade server to use Queryable Encryption."

0 commit comments

Comments
 (0)