Skip to content

PHPLIB-789: Snapshot query examples #900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 30, 2022
Merged

PHPLIB-789: Snapshot query examples #900

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
710392b
PHPLIB-789: Snapshot query examples
jmikola Mar 16, 2022
b6be7f1
Update spec links for StaleDbVersion workaround
jmikola Mar 18, 2022
0613f75
Update tests/DocumentationExamplesTest.php
jmikola Mar 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions tests/DocumentationExamplesTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,16 @@

use MongoDB\BSON\ObjectId;
use MongoDB\BSON\UTCDateTime;
use MongoDB\Collection;
use MongoDB\Database;
use MongoDB\Driver\Cursor;
use MongoDB\Driver\Exception\CommandException;
use MongoDB\Driver\Exception\Exception;
use MongoDB\Driver\ReadPreference;
use MongoDB\Driver\WriteConcern;

use function in_array;
use function microtime;
use function ob_end_clean;
use function ob_start;
use function var_dump;
Expand Down Expand Up @@ -1575,6 +1578,119 @@ public function testCausalConsistency(): void
ob_end_clean();
}

public function testSnapshotQueries(): void
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I observed SnapshotUnavailable(246) errors in CI builds. This PR will require work-arounds for replica sets and sharded clusters, which we can adapt from RUBY-2909.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added private waitForSnapshot() and preventStaleDbVersionError() methods for replica sets and sharded clusters, respectively.

{
if (version_compare($this->getServerVersion(), '5.0.0', '<')) {
$this->markTestSkipped('Snapshot queries outside of transactions are not supported');
Copy link
Member Author

@jmikola jmikola Mar 16, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See:

}

if (! ($this->isReplicaSet() || $this->isShardedClusterUsingReplicasets())) {
$this->markTestSkipped('Snapshot read concern is only supported with replicasets');
}

$client = static::createTestClient();

$catsCollection = $client->selectCollection('pets', 'cats');
$catsCollection->drop();
$catsCollection->insertMany([
['name' => 'Whiskers', 'color' => 'white', 'adoptable' => true],
['name' => 'Garfield', 'color' => 'orange', 'adoptable' => false],
]);

$dogsCollection = $client->selectCollection('pets', 'dogs');
$dogsCollection->drop();
$dogsCollection->insertMany([
['name' => 'Toto', 'color' => 'black', 'adoptable' => true],
['name' => 'Milo', 'color' => 'black', 'adoptable' => false],
['name' => 'Brian', 'color' => 'white', 'adoptable' => true],
]);

if ($this->isShardedCluster()) {
$this->preventStaleDbVersionError('pets', 'cats');
$this->preventStaleDbVersionError('pets', 'dogs');
} else {
$this->waitForSnapshot('pets', 'cats');
$this->waitForSnapshot('pets', 'dogs');
}

ob_start();

// Start Snapshot Query Example 1
$catsCollection = $client->selectCollection('pets', 'cats');
$dogsCollection = $client->selectCollection('pets', 'dogs');
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are purposely redefined in order to demonstrate selecting a collection from the client object within the example. For context, the docs team parses this file for code between various start/end comments and uses that to render language-specific examples in the MongoDB manual.


$session = $client->startSession(['snapshot' => true]);

$adoptablePetsCount = $catsCollection->aggregate(
[
['$match' => ['adoptable' => true]],
['$count' => 'adoptableCatsCount'],
],
['session' => $session]
)->toArray()[0]->adoptableCatsCount;

$adoptablePetsCount += $dogsCollection->aggregate(
[
['$match' => ['adoptable' => true]],
['$count' => 'adoptableDogsCount'],
],
['session' => $session]
)->toArray()[0]->adoptableDogsCount;

var_dump($adoptablePetsCount);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Various examples require printing output. Since we don't want that output to interfere with the test suite, we typically use output buffering to capture and discard it.

// End Snapshot Query Example 1

ob_end_clean();

$this->assertSame(3, $adoptablePetsCount);

$catsCollection->drop();
$dogsCollection->drop();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these examples use specific database and collection names, the cleanup logic in tearDown() does not apply and we manually clean up after each test.


$salesCollection = $client->selectCollection('retail', 'sales');
$salesCollection->drop();
$salesCollection->insertMany([
['shoeType' => 'boot', 'price' => 30, 'saleDate' => new UTCDateTime()],
]);

if ($this->isShardedCluster()) {
$this->preventStaleDbVersionError('retail', 'sales');
} else {
$this->waitForSnapshot('retail', 'sales');
}

// Start Snapshot Query Example 2
$salesCollection = $client->selectCollection('retail', 'sales');

$session = $client->startSession(['snapshot' => true]);

$totalDailySales = $salesCollection->aggregate(
[
[
'$match' => [
'$expr' => [
'$gt' => ['$saleDate', [
'$dateSubtract' => [
'startDate' => '$$NOW',
'unit' => 'day',
'amount' => 1,
],
],
],
],
],
],
['$count' => 'totalDailySales'],
],
['session' => $session]
)->toArray()[0]->totalDailySales;
// End Snapshot Query Example 2

$this->assertSame(1, $totalDailySales);

$salesCollection->drop();
}

/**
* @doesNotPerformAssertions
*/
Expand Down Expand Up @@ -1751,4 +1867,42 @@ private function assertInventoryCount($count): void
{
$this->assertCollectionCount($this->getDatabaseName() . '.' . $this->getCollectionName(), $count);
}

private function waitForSnapshot(string $databaseName, string $collectionName): void
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At a glance it seems to me, that the idea of custom function like waitForSnapshot can be useful for interacting with snapshots and in such case it can be organized in a library functionality?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this function was lifted from mongodb/mongo-ruby-driver@7c4117b#diff-7b02c2b4b34a58c688750366d0230014bb453e0381af394d736aa9d0a25538a5R374 in the Ruby driver.

The function itself is not generally useful, since we're targeting a specific collection. The server-side error message is:

Unable to read from a snapshot due to pending collection catalog changes; please retry the operation.

If a user were to encounter this, they would probably want to retry their original query. For our purposes here, ensuring any snapshot query (even one with a different query filter) succeeds seems to do the trick to avoid an error in the documentation example that calls this method.

{
$collection = new Collection($this->manager, $databaseName, $collectionName);
$session = $this->manager->startSession(['snapshot' => true]);

/* Retry until a snapshot query succeeds or ten seconds elapse,
* whichwever comes first.
*
* TODO: use hrtime() once the library requires PHP 7.3+ */
$retryUntil = microtime(true) + 10;

do {
try {
$collection->aggregate(
[['$match' => ['_id' => ['$exists' => true]]]],
['session' => $session]
);

break;
} catch (CommandException $e) {
if ($e->getCode() === 246 /* SnapshotUnavailable */) {
continue;
}

throw $e;
}
} while (microtime(true) < $retryUntil);
}

/**
* @see https://jira.mongodb.org/browse/SERVER-39704
*/
private function preventStaleDbVersionError(string $databaseName, string $collectionName): void
{
$collection = new Collection($this->manager, $databaseName, $collectionName);
$collection->distinct('foo');
}
}
2 changes: 1 addition & 1 deletion tests/SpecTests/TransactionsSpecTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,7 @@ private static function killAllSessions(): void
* Work around potential error executing distinct on sharded clusters.
*
* @param array $operations
* @see https://github.com/mongodb/specifications/tree/master/source/transactions/tests#why-do-tests-that-run-distinct-sometimes-fail-with-staledbversionts.
* @see https://github.com/mongodb/specifications/tree/master/source/transactions/tests#why-do-tests-that-run-distinct-sometimes-fail-with-staledbversion
*/
private function preventStaleDbVersionError(array $operations): void
{
Expand Down
2 changes: 1 addition & 1 deletion tests/UnifiedSpecTests/UnifiedTestRunner.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -442,7 +442,7 @@ private function prepareInitialData(array $initialData): void
/**
* Work around potential error executing distinct on sharded clusters.
*
* @see https://github.com/mongodb/specifications/blob/master/source/transactions/tests/README.rst#why-do-tests-that-run-distinct-sometimes-fail-with-staledbversion
* @see https://github.com/mongodb/specifications/blob/master/source/unified-test-format/unified-test-format.rst#staledbversion-errors-on-sharded-clusters
*/
private function preventStaleDbVersionError(array $operations, Context $context): void
{
Expand Down