Skip to content

Don't crash if Target cannot be persisted #2944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 1, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/firestore/.idea/runConfigurations/Unit_Tests.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions packages/firestore/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# Unreleased
- [fixed] Firestore now rejects `onSnapshot()` listeners if they cannot be
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and the next release note could probably be combined. Something like:

Firestore now rejects onSnapshot listeners are writes if they cannot be persisted in IndexedDB. ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prior change is going out this week. Unfortunately, this Changelog is not all that clean, but this is mostly due to the fact the version numbers are impossible to figure out. Maybe we can discuss this with the team and clean it up once and for all.

registered in IndexedDB. Previously, these errors crashed the client.
- [fixed] Firestore now rejects write operations if they cannot be persisted
in IndexedDB. Previously, these errors crashed the client.
- [fixed] Fixed a source of IndexedDB-related crashes for tabs that receive
Expand Down
40 changes: 25 additions & 15 deletions packages/firestore/src/core/event_manager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,19 @@ import { EventHandler } from '../util/misc';
import { ObjectMap } from '../util/obj_map';
import { Query } from './query';
import { SyncEngine, SyncEngineListener } from './sync_engine';
import { OnlineState, TargetId } from './types';
import { DocumentViewChange, ChangeType, ViewSnapshot } from './view_snapshot';
import { OnlineState } from './types';
import { ChangeType, DocumentViewChange, ViewSnapshot } from './view_snapshot';
import { logError } from '../util/log';
import { Code, FirestoreError } from '../util/error';

const LOG_TAG = 'EventManager';

/**
* Holds the listeners and the last received ViewSnapshot for a query being
* tracked by EventManager.
*/
class QueryListenersInfo {
viewSnap: ViewSnapshot | null = null;
targetId: TargetId = 0;
viewSnap: ViewSnapshot | undefined = undefined;
listeners: QueryListener[] = [];
}

Expand Down Expand Up @@ -59,16 +62,32 @@ export class EventManager implements SyncEngineListener {
this.syncEngine.subscribe(this);
}

listen(listener: QueryListener): Promise<TargetId> {
async listen(listener: QueryListener): Promise<void> {
Copy link
Contributor Author

@schmidt-sebastian schmidt-sebastian Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the return value since a failed target doesn't have an ID. The only consumer of the targetId result were the spec tests, which already verify the target ID in every step as part of expectedActiveTargets.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why doesn't failing to listen reject the promise here? If it did then you could return the TargetId in the successful case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I reject the Promise here, then FirestoreClient has to handle the rejected Promise. This is actually pretty nice and keeps the code in EventManager much cleaner. The calling code is also still clean, since it was simple to begin with.

The problem I see is that this makes this code non-testable. The spec tests don't use FirestoreClient, and the integration cannot inject failures. I have made the change regardless and added some special handling in SyncEngine. I am not sure if this is an improvement though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean; the spec test has to emulate this for itself. That's not completely awful but you're probably right that the other way was better because it was testable.

Would it help to put the error handling in SyncEngine?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think SyncEngine is not the right place for this since it can only convey two states: Success or failure. We really need three states: Success, failure (but don't crash AsyncQueue) and failure (crash AsyncQueue).
Unless we change SyncEngine.listen() return a tristate, we have to wrap any error thrown by SyncEngine in the same if (error.name === '...') check when we invoke the QueryListener.

Given this and the test ability, I moved the code back to EventListener. It's not as clean looking, but it will probably be easier to keep regression free.

const query = listener.query;
let firstListen = false;

let queryInfo = this.queries.get(query);
if (!queryInfo) {
firstListen = true;
queryInfo = new QueryListenersInfo();
this.queries.set(query, queryInfo);
}

if (firstListen) {
try {
queryInfo.viewSnap = await this.syncEngine.listen(query);
} catch (e) {
const msg = `Initialization of query '${query}' failed: ${e}`;
logError(LOG_TAG, msg);
if (e.name === 'IndexedDbTransactionError') {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may not be possible, but it seems like could benefit from handling these errors consistently in our API.

For example, we try to make FirestoreClient the place where we centrally handle dispatching onto the AsyncQueue. Should we try to similarly make handling retries (or error handling like this) a part of the SyncEngine?

This doesn't seem to work in this case because you're returning a successful promise in case of a known error, but that seems to be catering to the caller's use of this function in a way that makes this more complicated than it needs to be. Maybe the caller should handle this in the block submitted to the async queue?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did end up updating this, but as stated this pushes the error handling to both FirestoreClient and SpecTestRunner.

listener.onError(new FirestoreError(Code.UNAVAILABLE, msg));
} else {
throw e;
}
return;
}
}

this.queries.set(query, queryInfo);
queryInfo.listeners.push(listener);

// Run global snapshot listeners if a consistent snapshot has been emitted.
Expand All @@ -84,15 +103,6 @@ export class EventManager implements SyncEngineListener {
this.raiseSnapshotsInSyncEvent();
}
}

if (firstListen) {
return this.syncEngine.listen(query).then(targetId => {
queryInfo!.targetId = targetId;
return targetId;
});
} else {
return Promise.resolve(queryInfo.targetId);
}
}

async unlisten(listener: QueryListener): Promise<void> {
Expand Down
4 changes: 1 addition & 3 deletions packages/firestore/src/core/firestore_client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -393,9 +393,7 @@ export class FirestoreClient {
): QueryListener {
this.verifyNotTerminated();
const listener = new QueryListener(query, observer, options);
this.asyncQueue.enqueueAndForget(() => {
return this.eventMgr.listen(listener);
});
this.asyncQueue.enqueueAndForget(() => this.eventMgr.listen(listener));
return listener;
}

Expand Down
7 changes: 3 additions & 4 deletions packages/firestore/src/core/sync_engine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -213,9 +213,9 @@ export class SyncEngine implements RemoteSyncer {
/**
* Initiates the new listen, resolves promise when listen enqueued to the
* server. All the subsequent view snapshots or errors are sent to the
* subscribed handlers. Returns the targetId of the query.
* subscribed handlers. Returns the initial snapshot.
*/
async listen(query: Query): Promise<TargetId> {
async listen(query: Query): Promise<ViewSnapshot> {
this.assertSubscribed('listen()');

let targetId;
Expand Down Expand Up @@ -249,8 +249,7 @@ export class SyncEngine implements RemoteSyncer {
}
}

this.syncEngineListener!.onWatchChange([viewSnapshot]);
Copy link
Contributor Author

@schmidt-sebastian schmidt-sebastian Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to remove this indirection. onWatchChange doesn't fire if EventManager doesn't execute this.queries.set(query, queryInfo) first.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first I was concerned about this because it seemed like this would prevent calling onViewSnapshot on the listener, but actually it looks like code to do that was already explicitly in the EventManager (in if (queryInfo.viewSnap), and I'm guessing that the explicit code was just being treated as a no-op because previously queryInfo.viewSnap would be undefined before?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I fully understand this, but I hope that my previous comment reliefs your concerns. Let me know if it doesn't.

return targetId;
return viewSnapshot;
}

/**
Expand Down
34 changes: 34 additions & 0 deletions packages/firestore/test/unit/specs/recovery_spec.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import { describeSpec, specTest } from './describe_spec';
import { client, spec } from './spec_builder';
import { TimerId } from '../../../src/util/async_queue';
import { Query } from '../../../src/core/query';
import { Code } from '../../../src/util/error';
import { doc, path } from '../../util/helpers';

describeSpec('Persistence Recovery', ['no-ios', 'no-android'], () => {
Expand Down Expand Up @@ -162,4 +163,37 @@ describeSpec('Persistence Recovery', ['no-ios', 'no-android'], () => {
.watchAcksFull(query, 2, doc1, doc3)
.expectEvents(query, { metadata: [doc1, doc3] });
});

specTest('Fails targets that cannot be allocated', [], () => {
const query1 = Query.atPath(path('collection1'));
const query2 = Query.atPath(path('collection2'));
const query3 = Query.atPath(path('collection3'));
return spec()
.userListens(query1)
.watchAcksFull(query1, 1)
.expectEvents(query1, {})
.failDatabase()
.userListens(query2)
.expectEvents(query2, { errorCode: Code.UNAVAILABLE })
.recoverDatabase()
.userListens(query3)
.watchAcksFull(query3, 1)
.expectEvents(query3, {});
});

specTest('Can re-add failed target', [], () => {
const query1 = Query.atPath(path('collection1'));
const query2 = Query.atPath(path('collection2'));
return spec()
.userListens(query1)
.watchAcksFull(query1, 1)
.expectEvents(query1, {})
.failDatabase()
.userListens(query2)
.expectEvents(query2, { errorCode: Code.UNAVAILABLE })
.recoverDatabase()
.userListens(query2)
.watchAcksFull(query2, 1)
.expectEvents(query2, {});
});
});
39 changes: 29 additions & 10 deletions packages/firestore/test/unit/specs/spec_builder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ export class ClientMemoryState {
limboMapping: LimboMap = {};

limboIdGenerator: TargetIdGenerator = TargetIdGenerator.forSyncEngine();
injectFailures = false;

constructor() {
this.reset();
Expand Down Expand Up @@ -191,6 +192,14 @@ export class SpecBuilder {
return this.clientState.activeTargets;
}

private get injectFailures(): boolean {
return this.clientState.injectFailures;
}

private set injectFailures(injectFailures: boolean) {
this.clientState.injectFailures = injectFailures;
}

/**
* Exports the spec steps as a JSON object that be used in the spec runner.
*/
Expand Down Expand Up @@ -232,18 +241,26 @@ export class SpecBuilder {

const target = query.toTarget();
let targetId: TargetId = 0;
if (this.queryMapping.has(target)) {
targetId = this.queryMapping.get(target)!;

if (this.injectFailures) {
// Return a `userListens()` step but don't advance the target IDs.
this.currentStep = {
userListen: [targetId, SpecBuilder.queryToSpec(query)]
};
} else {
targetId = this.queryIdGenerator.next(target);
}
if (this.queryMapping.has(target)) {
targetId = this.queryMapping.get(target)!;
} else {
targetId = this.queryIdGenerator.next(target);
}

this.queryMapping.set(target, targetId);
this.addQueryToActiveTargets(targetId, query, resumeToken);
this.currentStep = {
userListen: [targetId, SpecBuilder.queryToSpec(query)],
expectedState: { activeTargets: { ...this.activeTargets } }
};
this.queryMapping.set(target, targetId);
this.addQueryToActiveTargets(targetId, query, resumeToken);
this.currentStep = {
userListen: [targetId, SpecBuilder.queryToSpec(query)],
expectedState: { activeTargets: { ...this.activeTargets } }
};
}
return this;
}

Expand Down Expand Up @@ -421,6 +438,7 @@ export class SpecBuilder {
/** Fails all database operations until `recoverDatabase()` is called. */
failDatabase(): this {
this.nextStep();
this.injectFailures = true;
this.currentStep = {
failDatabase: true
};
Expand All @@ -430,6 +448,7 @@ export class SpecBuilder {
/** Stops failing database operations. */
recoverDatabase(): this {
this.nextStep();
this.injectFailures = false;
this.currentStep = {
failDatabase: false
};
Expand Down
4 changes: 2 additions & 2 deletions packages/firestore/test/unit/specs/spec_test_components.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ import { Token } from '../../../src/api/credentials';
import { Observer } from '../../../src/core/event_manager';
import { ViewSnapshot } from '../../../src/core/view_snapshot';
import { Query } from '../../../src/core/query';
import { expectFirestoreError } from '../../util/helpers';
import { Mutation } from '../../../src/model/mutation';
import { expect } from 'chai';

/**
* A test-only MemoryPersistence implementation that is able to inject
Expand Down Expand Up @@ -367,7 +367,7 @@ export class EventAggregator implements Observer<ViewSnapshot> {
}

error(error: Error): void {
expectFirestoreError(error);
expect(error.name).to.equal('FirebaseError');
this.pushEvent({ query: this.query, error: error as FirestoreError });
}
}
Expand Down
55 changes: 32 additions & 23 deletions packages/firestore/test/unit/specs/spec_test_runner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ import {
deletedDoc,
deleteMutation,
doc,
expectFirestoreError,
validateFirestoreError,
filter,
key,
orderBy,
Expand Down Expand Up @@ -350,10 +350,16 @@ abstract class TestRunner {
}

private async doListen(listenSpec: SpecUserListen): Promise<void> {
const expectedTargetId = listenSpec[0];
let targetFailed = false;

const querySpec = listenSpec[1];
const query = parseQuery(querySpec);
const aggregator = new EventAggregator(query, this.pushEvent.bind(this));
const aggregator = new EventAggregator(query, e => {
if (e.error) {
targetFailed = true;
}
this.pushEvent(e);
});
// TODO(dimond): Allow customizing listen options in spec tests
const options = {
includeMetadataChanges: true,
Expand All @@ -362,27 +368,27 @@ abstract class TestRunner {
const queryListener = new QueryListener(query, aggregator, options);
this.queryListeners.set(query, queryListener);

await this.queue.enqueue(async () => {
const targetId = await this.eventManager.listen(queryListener);
expect(targetId).to.equal(
expectedTargetId,
'targetId assigned to listen'
);
});
await this.queue.enqueue(() => this.eventManager.listen(queryListener));

// Skip the backoff that may have been triggered by a previous call to
// `watchStreamCloses()`.
if (
this.queue.containsDelayedOperation(TimerId.ListenStreamConnectionBackoff)
) {
await this.queue.runDelayedOperationsEarly(
TimerId.ListenStreamConnectionBackoff
);
}
if (targetFailed) {
expect(this.persistence.injectFailures).to.be.true;
} else {
// Skip the backoff that may have been triggered by a previous call to
// `watchStreamCloses()`.
if (
this.queue.containsDelayedOperation(
TimerId.ListenStreamConnectionBackoff
)
) {
await this.queue.runDelayedOperationsEarly(
TimerId.ListenStreamConnectionBackoff
);
}

if (this.isPrimaryClient && this.networkEnabled) {
// Open should always have happened after a listen
await this.connection.waitForWatchOpen();
if (this.isPrimaryClient && this.networkEnabled) {
// Open should always have happened after a listen
await this.connection.waitForWatchOpen();
}
}
}

Expand Down Expand Up @@ -955,7 +961,10 @@ abstract class TestRunner {
const expectedQuery = parseQuery(expected.query);
expect(actual.query).to.deep.equal(expectedQuery);
if (expected.errorCode) {
expectFirestoreError(actual.error!);
validateFirestoreError(
mapCodeFromRpcCode(expected.errorCode),
actual.error!
);
} else {
const expectedChanges: DocumentViewChange[] = [];
if (expected.removed) {
Expand Down
9 changes: 7 additions & 2 deletions packages/firestore/test/util/helpers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ import { ByteString } from '../../src/util/byte_string';
import { PlatformSupport } from '../../src/platform/platform';
import { JsonProtoSerializer } from '../../src/remote/serializer';
import { Timestamp } from '../../src/api/timestamp';
import { Code, FirestoreError } from '../../src/util/error';

/* eslint-disable no-restricted-globals */

Expand Down Expand Up @@ -815,8 +816,12 @@ export function expectEqualitySets<T>(
}
}

export function expectFirestoreError(err: Error): void {
expect(err.name).to.equal('FirebaseError');
export function validateFirestoreError(
expectedCode: Code,
actualError: Error
): void {
expect(actualError.name).to.equal('FirebaseError');
expect((actualError as FirestoreError).code).to.equal(expectedCode);
}

export function forEachNumber<V>(
Expand Down