Skip to content

test(NODE-4227): add explicit encryption prose tests #3297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jun 29, 2022
Merged

test(NODE-4227): add explicit encryption prose tests #3297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
52dec16
test(NODE-4227): add explicit encryption prose tests
durran Jun 23, 2022
306eace
test(NODE-4227): bypass query analysis in fle opts
durran Jun 23, 2022
d6e5e44
test(NODE-4227): reorder prose tests
durran Jun 23, 2022
c9d468c
test(NODE-4227): fix tests
durran Jun 23, 2022
5789257
test(NODE-4227): use csfle alpha 3
durran Jun 23, 2022
c526461
test(NODE-4227): update per suggestions
durran Jun 27, 2022
3cf6751
chore(NODE-4227): fix lint errors
durran Jun 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .evergreen/run-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ else
source "$DRIVERS_TOOLS"/.evergreen/csfle/set-temp-creds.sh
fi

npm install mongodb-client-encryption@">=2.2.0-alpha.2"
npm install mongodb-client-encryption@">=2.2.0-alpha.3"
npm install @mongodb-js/zstd
npm install snappy

Expand Down
293 changes: 293 additions & 0 deletions test/integration/client-side-encryption/client_side_encryption.prose.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,14 @@ const metadata = {
}
};

const eeMetadata = {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#12 prose test section for explicit encryption requires server 6.0 and higher and not standalone.

requires: {
clientSideEncryption: true,
mongodb: '>=6.0.0',
topology: ['replicaset', 'sharded']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about load balanced? is that considered more like a standalone?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't run the other fle tests against load balanced clusters so I was keeping in line with that. The spec in general pre-dates the load balancer spec and does not mention that topology at all, and enabling it got some strange setup errors that I felt were outside the scope of the ticket. I can create a ticket to look into running all the fle tests with load balancers as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should check with the spec owners regarding this, and maybe file a drivers ticket to add lb to the testing matrix if it makes sense to do so, then the node ticket will automatically come down to us

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for context, the reason for requiring these topologies is that QE requires transactions, which in turn requires a replset topology. I guess technically one could run a load balancer in front of a single standalone node, but I assume that won’t happen in practice or in tests?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as a follow up, other drivers are not running prose tests against a load balancer. However, the LB spec says it should run all tests in unified format so we'll need to make sure the new FLE unified tests are running against a load balanced topology.

}
};

// Tests for the ClientEncryption type are not included as part of the YAML tests.

// In the prose tests LOCAL_MASTERKEY refers to the following base64:
Expand Down Expand Up @@ -1426,6 +1434,291 @@ describe('Client Side Encryption Prose Tests', metadata, function () {
});
});

context('12. Explicit Encryption', eeMetadata, function () {
const data = path.join(__dirname, '..', '..', 'spec', 'client-side-encryption', 'etc', 'data');
let encryptedFields;
let key1Document;
let key1Id;
let setupClient;
let keyVaultClient;
let clientEncryption;
let encryptedClient;

beforeEach(async function () {
const mongodbClientEncryption = this.configuration.mongodbClientEncryption;
// Load the file encryptedFields.json as encryptedFields.
encryptedFields = EJSON.parse(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data for the encryptedFields option and key document are now provided in 2 new files in the etc/ directory in the csfle spec directory.

await fs.promises.readFile(path.join(data, 'encryptedFields.json')),
{ relaxed: false }
);
// Load the file key1-document.json as key1Document.
key1Document = EJSON.parse(
await fs.promises.readFile(path.join(data, 'keys', 'key1-document.json')),
{ relaxed: false }
);
// Read the "_id" field of key1Document as key1ID.
key1Id = key1Document._id;
setupClient = this.configuration.newClient();
// Drop and create the collection db.explicit_encryption using encryptedFields as an option.
const db = setupClient.db('db');
await dropCollection(db, 'explicit_encryption', { encryptedFields });
await db.createCollection('explicit_encryption', { encryptedFields });
// Drop and create the collection keyvault.datakeys.
const kdb = setupClient.db('keyvault');
await dropCollection(kdb, 'datakeys');
await kdb.createCollection('datakeys');
// Insert key1Document in keyvault.datakeys with majority write concern.
await kdb.collection('datakeys').insertOne(key1Document, { writeConcern: { w: 'majority' } });
// Create a MongoClient named keyVaultClient.
keyVaultClient = this.configuration.newClient();
// Create a ClientEncryption object named clientEncryption with these options:
// ClientEncryptionOpts {
// keyVaultClient: <keyVaultClient>;
// keyVaultNamespace: "keyvault.datakeys";
// kmsProviders: { "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } }
// }
clientEncryption = new mongodbClientEncryption.ClientEncryption(keyVaultClient, {
keyVaultNamespace: 'keyvault.datakeys',
kmsProviders: getKmsProviders(LOCAL_KEY),
bson: BSON
});
// Create a MongoClient named ``encryptedClient`` with these ``AutoEncryptionOpts``:
// AutoEncryptionOpts {
// keyVaultNamespace: "keyvault.datakeys";
// kmsProviders: { "local": { "key": <base64 decoding of LOCAL_MASTERKEY> } },
// bypassQueryAnalysis: true
// }
encryptedClient = this.configuration.newClient(
{},
{
autoEncryption: {
bypassQueryAnalysis: true,
keyVaultNamespace: 'keyvault.datakeys',
kmsProviders: getKmsProviders(LOCAL_KEY)
}
}
);
});

afterEach(async function () {
await setupClient.close();
await keyVaultClient.close();
await encryptedClient.close();
});

context('Case 1: can insert encrypted indexed and find', eeMetadata, function () {
let insertPayload;
let findPayload;

beforeEach(async function () {
// Use clientEncryption to encrypt the value "encrypted indexed value" with these EncryptOpts:
// class EncryptOpts {
// keyId : <key1ID>
// algorithm: "Indexed",
// }
// Store the result in insertPayload.
insertPayload = await clientEncryption.encrypt('encrypted indexed value', {
keyId: key1Id,
algorithm: 'Indexed'
});
// Use encryptedClient to insert the document { "encryptedIndexed": <insertPayload> }
// into db.explicit_encryption.
await encryptedClient.db('db').collection('explicit_encryption').insertOne({
encryptedIndexed: insertPayload
});
// Use clientEncryption to encrypt the value "encrypted indexed value" with these EncryptOpts:
// class EncryptOpts {
// keyId : <key1ID>
// algorithm: "Indexed",
// queryType: Equality
// }
// Store the result in findPayload.
findPayload = await clientEncryption.encrypt('encrypted indexed value', {
keyId: key1Id,
algorithm: 'Indexed',
queryType: 'equality'
});
});

it('returns the decrypted value', async function () {
// Use encryptedClient to run a "find" operation on the db.explicit_encryption
// collection with the filter { "encryptedIndexed": <findPayload> }.
// Assert one document is returned containing the field
// { "encryptedIndexed": "encrypted indexed value" }.
const collection = encryptedClient.db('db').collection('explicit_encryption');
const result = await collection.findOne({ encryptedIndexed: findPayload });
expect(result).to.have.property('encryptedIndexed', 'encrypted indexed value');
});
});

context(
'Case 2: can insert encrypted indexed and find with non-zero contention',
eeMetadata,
function () {
let findPayload;
let findPayload2;

beforeEach(async function () {
for (let i = 0; i < 10; i++) {
// Use clientEncryption to encrypt the value "encrypted indexed value" with these EncryptOpts:
// class EncryptOpts {
// keyId : <key1ID>
// algorithm: "Indexed",
// contentionFactor: 10
// }
// Store the result in insertPayload.
const insertPayload = await clientEncryption.encrypt('encrypted indexed value', {
keyId: key1Id,
algorithm: 'Indexed',
contentionFactor: 10
});
// Use encryptedClient to insert the document { "encryptedIndexed": <insertPayload> }
// into db.explicit_encryption.
await encryptedClient.db('db').collection('explicit_encryption').insertOne({
encryptedIndexed: insertPayload
});
// Repeat the above steps 10 times to insert 10 total documents.
// The insertPayload must be regenerated each iteration.
}
// Use clientEncryption to encrypt the value "encrypted indexed value" with these EncryptOpts:
// class EncryptOpts {
// keyId : <key1ID>
// algorithm: "Indexed",
// queryType: Equality
// }
// Store the result in findPayload.
findPayload = await clientEncryption.encrypt('encrypted indexed value', {
keyId: key1Id,
algorithm: 'Indexed',
queryType: 'equality'
});
// Use clientEncryption to encrypt the value "encrypted indexed value" with these EncryptOpts:
// class EncryptOpts {
// keyId : <key1ID>
// algorithm: "Indexed",
// queryType: Equality,
// contentionFactor: 10
// }
// Store the result in findPayload2.
findPayload2 = await clientEncryption.encrypt('encrypted indexed value', {
keyId: key1Id,
algorithm: 'Indexed',
queryType: 'equality',
contentionFactor: 10
});
});

it('returns less than the total documents with no contention', async function () {
// Use encryptedClient to run a "find" operation on the db.explicit_encryption
// collection with the filter { "encryptedIndexed": <findPayload> }.
// Assert less than 10 documents are returned. 0 documents may be returned.
// Assert each returned document contains the field
// { "encryptedIndexed": "encrypted indexed value" }.
const collection = encryptedClient.db('db').collection('explicit_encryption');
const result = await collection.find({ encryptedIndexed: findPayload }).toArray();
expect(result.length).to.be.below(10);
for (const doc of result) {
expect(doc).to.have.property('encryptedIndexed', 'encrypted indexed value');
}
});

it('returns all documents with contention', async function () {
// Use encryptedClient to run a "find" operation on the db.explicit_encryption
// collection with the filter { "encryptedIndexed": <findPayload2> }.
// Assert 10 documents are returned. Assert each returned document contains the
// field { "encryptedIndexed": "encrypted indexed value" }.
const collection = encryptedClient.db('db').collection('explicit_encryption');
const result = await collection.find({ encryptedIndexed: findPayload2 }).toArray();
Copy link
Member Author

@durran durran Jun 23, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pointing out I find this absolutely crazy. A value is directly encrypted on line 1603 with a contention factor equal to the amount of documents in the collection that would match the original value of the field. But each field's encrypted value would not match the encrypted value passed to the find in theory as all the bin data would be different for each document on insert. But it still matches all of them. Mind blown.

expect(result.length).to.equal(10);
for (const doc of result) {
expect(doc).to.have.property('encryptedIndexed', 'encrypted indexed value');
}
});
}
);

context('Case 3: can insert encrypted unindexed', eeMetadata, function () {
let insertPayload;

beforeEach(async function () {
// Use clientEncryption to encrypt the value "encrypted unindexed value" with these EncryptOpts:
// class EncryptOpts {
// keyId : <key1ID>
// algorithm: "Unindexed"
// }
// Store the result in insertPayload.
insertPayload = await clientEncryption.encrypt('encrypted unindexed value', {
keyId: key1Id,
algorithm: 'Unindexed'
});
// Use encryptedClient to insert the document { "_id": 1, "encryptedUnindexed": <insertPayload> }
// into db.explicit_encryption.
await encryptedClient.db('db').collection('explicit_encryption').insertOne({
_id: 1,
encryptedUnindexed: insertPayload
});
});

it('returns unindexed documents', async function () {
// Use encryptedClient to run a "find" operation on the db.explicit_encryption
// collection with the filter { "_id": 1 }.
// Assert one document is returned containing the field
// { "encryptedUnindexed": "encrypted unindexed value" }.
const collection = encryptedClient.db('db').collection('explicit_encryption');
const result = await collection.findOne({ _id: 1 });
expect(result).to.have.property('encryptedUnindexed', 'encrypted unindexed value');
});
});

context('Case 4: can roundtrip encrypted indexed', eeMetadata, function () {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These last 2 are just encrypt/decrypt roundtrips. So pretty straightforward.

let payload;

beforeEach(async function () {
// Use clientEncryption to encrypt the value "encrypted indexed value" with these EncryptOpts:
// class EncryptOpts {
// keyId : <key1ID>
// algorithm: "Indexed",
// }
// Store the result in payload.
payload = await clientEncryption.encrypt('encrypted indexed value', {
keyId: key1Id,
algorithm: 'Indexed'
});
});

it('decrypts the value', async function () {
// Use clientEncryption to decrypt payload. Assert the returned value
// equals "encrypted indexed value".
const result = await clientEncryption.decrypt(payload);
expect(result).equals('encrypted indexed value');
});
});

context('Case 5: can roundtrip encrypted unindexed', eeMetadata, function () {
let payload;

beforeEach(async function () {
// Use clientEncryption to encrypt the value "encrypted unindexed value" with these EncryptOpts:
// class EncryptOpts {
// keyId : <key1ID>
// algorithm: "Unindexed",
// }
// Store the result in payload.
payload = await clientEncryption.encrypt('encrypted unindexed value', {
keyId: key1Id,
algorithm: 'Unindexed'
});
});

it('decrypts the value', async function () {
// Use clientEncryption to decrypt payload. Assert the returned value
// equals "encrypted unindexed value".
const result = await clientEncryption.decrypt(payload);
expect(result).equals('encrypted unindexed value');
});
});
});

context('14. Decryption Events', metadata, function () {
let setupClient;
let clientEncryption;
Expand Down
4 changes: 2 additions & 2 deletions test/integration/shared.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ function delay(timeout) {
});
}

function dropCollection(dbObj, collectionName) {
return dbObj.dropCollection(collectionName).catch(ignoreNsNotFound);
function dropCollection(dbObj, collectionName, options = {}) {
return dbObj.dropCollection(collectionName, options).catch(ignoreNsNotFound);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just adding the ability to pass options to the dropCollection helper. If we pass encryptedFields here an extra 3 collections will get dropped. (The esc/ecc/ecoc collections)

}

function filterForCommands(commands, bag) {
Expand Down
33 changes: 33 additions & 0 deletions test/spec/client-side-encryption/etc/data/encryptedFields.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
"$binary": {
"base64": "EjRWeBI0mHYSNBI0VniQEg==",
"subType": "04"
}
},
"path": "encryptedIndexed",
"bsonType": "string",
"queries": {
"queryType": "equality",
"contention": {
"$numberLong": "0"
}
}
},
{
"keyId": {
"$binary": {
"base64": "q83vqxI0mHYSNBI0VniQEg==",
"subType": "04"
}
},
"path": "encryptedUnindexed",
"bsonType": "string"
}
]
}
30 changes: 30 additions & 0 deletions test/spec/client-side-encryption/etc/data/keys/key1-document.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"_id": {
"$binary": {
"base64": "EjRWeBI0mHYSNBI0VniQEg==",
"subType": "04"
}
},
"keyMaterial": {
"$binary": {
"base64": "sHe0kz57YW7v8g9VP9sf/+K1ex4JqKc5rf/URX3n3p8XdZ6+15uXPaSayC6adWbNxkFskuMCOifDoTT+rkqMtFkDclOy884RuGGtUysq3X7zkAWYTKi8QAfKkajvVbZl2y23UqgVasdQu3OVBQCrH/xY00nNAs/52e958nVjBuzQkSb1T8pKJAyjZsHJ60+FtnfafDZSTAIBJYn7UWBCwQ==",
"subType": "00"
}
},
"creationDate": {
"$date": {
"$numberLong": "1648914851981"
}
},
"updateDate": {
"$date": {
"$numberLong": "1648914851981"
}
},
"status": {
"$numberInt": "0"
},
"masterKey": {
"provider": "local"
}
}