Skip to content

Mutable Documents #2383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Feb 25, 2021
Merged

Mutable Documents #2383

merged 16 commits into from
Feb 25, 2021

Conversation

schmidt-sebastian
Copy link
Contributor

@schmidt-sebastian schmidt-sebastian commented Jan 29, 2021

This is yet another big rewrite of the document/mutation logic that aims to provide stronger ownership guarantees for the C++ client.

With this PR, Documents and ObjectValues are now mutable. All mutations directly modify the Document and documents can transition from MaybeDocument to Document, NoDocument and UnknownDocument (and back). I think this is safe because every document lookup starts in the RemoteDocumentCache: In the SQLRemoteDocumentCache we always create new document anyways, and in the MemoryDocumentCache I now clone the documents.

I also merged all existing classes into Document. A Document that is not found in the RemoteDocumentCache is now INVALID, which allows mutations to transition the document to a valid state at a later point.

@googlebot googlebot added the cla: yes Override cla label Jan 29, 2021
@firebase firebase deleted a comment from google-oss-bot Jan 29, 2021
@google-oss-bot
Copy link
Contributor

google-oss-bot commented Jan 29, 2021

Coverage Report

Affected SDKs

  • firebase-firestore

    SDK overall coverage changed from 48.67% (52b1dca) to 48.73% (62be3327) by +0.05%.

    Click to show coverage changes in 21 files.
    Filename Base (52b1dca) Head (62be3327) Diff
    BundleSerializer.java 89.08% 89.12% +0.05%
    DefaultQueryEngine.java 95.74% 95.65% -0.09%
    DeleteMutation.java 83.33% 89.47% +6.14%
    DocumentReference.java 13.14% 13.24% +0.10%
    DocumentSet.java 83.56% 83.78% +0.22%
    DocumentSnapshot.java 36.84% 37.50% +0.66%
    DocumentViewChange.java 90.48% 90.91% +0.43%
    GrpcCallProvider.java 57.95% 68.18% +10.23%
    LocalSerializer.java 97.67% 97.58% -0.09%
    LocalStore.java 98.93% 99.28% +0.35%
    MutableDocument.java ? 98.31% ?
    Mutation.java 98.41% 98.15% -0.26%
    MutationBatch.java 89.55% 88.89% -0.66%
    ObjectValue.java 99.02% 99.04% +0.02%
    SQLiteRemoteDocumentCache.java 95.24% 95.29% +0.06%
    SetMutation.java 83.87% 86.49% +2.62%
    SnapshotVersion.java 93.75% 87.50% -6.25%
    SyncEngine.java 93.55% 93.59% +0.04%
    Timestamp.java 90.20% 80.39% -9.80%
    View.java 96.99% 96.82% -0.17%
    WatchChangeAggregator.java 98.17% 98.19% +0.02%

Test Logs

Notes

HTML coverage reports can be produced locally with ./gradlew <product>:checkCoverage.
Report files are located at <product-build-dir>/reports/jacoco/.

Head commit (62be3327) is created by Prow via merging commits: 52b1dca 6a81979.

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Jan 29, 2021

Binary Size Report

Affected SDKs

  • firebase-firestore

    Type Base (52b1dca) Head (62be3327) Diff
    aar 1.10 MB 1.10 MB -2.74 kB (-0.2%)
    apk (release) 3.19 MB 3.19 MB -692 B (-0.0%)
  • firebase-firestore-ktx

    Type Base (52b1dca) Head (62be3327) Diff
    apk (release) 3.67 MB 3.67 MB -996 B (-0.0%)

Test Logs

Notes

Head commit (62be3327) is created by Prow via merging commits: 52b1dca 6a81979.

Copy link
Contributor

@wilhuff wilhuff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In many ways, this is a replay of firebase/firebase-js-sdk#1886. mikelehen argued there that removing the type distinction between Document and MaybeDocument was harmful and I came to agree. Please review discussion in that PR and let's discuss.

The idea I was proposing w/r/t making mutations operate on a mutable document builder was narrow: within the mutation types, operate on the builder, but then once done applying mutations, convert to the appropriate existing immutable document type afterward. This would significantly limit the blast radius of the change.

@wilhuff wilhuff assigned schmidt-sebastian and unassigned wilhuff Feb 1, 2021
@schmidt-sebastian
Copy link
Contributor Author

I have tried to re-introduce the MaybeDocument chain by changing the current document to be a sort of builder for the old types. I am however having a bit of trouble finding a good place to draw the line between where the MutableDocument and the MaybeDocument should be used. The obvious place the LocalDocumentsView. It's input sources would be MutableDocuments (e.g. the RemoteDocumentStore and all mutation logic) and it would return MaybeDocuments. This however falls apart in two places:

  • When the LocalStore first writes a new mutation, it uses the LocalDocumentsView to generate the documents that contain the base state. It then passes these MaybeDocuments to the MutationBatch to quickly apply just the new mutation - but the MutationBatch APIs now takes MutableDocuments. This can be solved by going back to the LocalDocumentsView to compute the new state, but that means that we perform some work twice.

  • When the LocalStore applies RemoteEvents, it should also ideally take the input as MutableDocuments. If the documents inside of a RemoteEvent are not MutableDocuments, then we would need to convert them before applying mutations. If, however, these documents are MutableDocuments, then all logic in LocalStore applyRemoteEvent would have to deal with all this new state and mostly stay as is in this PR. Since that logic really is the one part of our code base that does any of "tricky" stuff with the internal document state, I am not sure if that defeats the purpose of re-introducing MaybeDocuments.

The last argument also applies to drawing the boundary at LocalStore - if RemoteStore, LocalStore, LocalDocumentsView and RemoteDocumentCache all use MutableDocuments, then the remaining part of the code that uses Documents would be fairly minimal - and plus, we would have to run a conversion between MutableDocumentMap and MaybeDocumentMap for every read from LocalStore.

I am probably missing something here, but these are my thoughts from today.

@schmidt-sebastian
Copy link
Contributor Author

Ping :)

if (doc instanceof Document) {
result = result.insert(doc.getKey(), (Document) doc);
Document doc = getDocument(DocumentKey.fromPath(path));
if (doc.exists()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though Document's new implementation handles this correctly, this doesn't read as if it works: unknown documents exist, but we just don't necessarily know their contents. Michael and I were discussing calling this definitelyExists to logically exclude the unknown case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I renamed all these status getters to reflect the type of the document. This is now isFoundDocument(). There is still some room for improvement, so suggestions welcome.

FWIW, in my opinion unknown documents also definitely exist.

}
mutation.applyToLocalView(document, batch.getLocalWriteTime());
if (document.isValid()) {
results = results.insert(key, document);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now mishandling the case where the base doc was in the results and is deleted by the mutation. If the document is invalid you need to remove its key from the results map.

Query snapshots should never include non-existent documents, but this would allow them to be there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rewrote this to match the old pattern. These documents already get removed by computeDocChanges (which is why the tests didn't break). I added a new test that would break though (doesNotIncludeDocumentsDeletedByMutation).

documentKey,
maybeDoc.getKey());
}
public void applyToLocalView(DocumentKey documentKey, Document document) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that documents are always non-null and also have their keys, what's the value in having a separate documentKey parameter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove it.

Document document = documentMap.get(key);
applyToLocalView(key, document);
if (!document.isValid()) {
document.asMissingDocument(SnapshotVersion.NONE);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm having a hard time reasoning about this change here because we're mixing a number of different terms. If this is going to go anywhere, we should standardize the nomenclature in here to make this easier to reason about.

A missing document has type NO_DOCUMENT and a !valid document has type INVALID. What's the point of coercing INVALID documents to NO_DOCUMENTs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know the answer to your question - but without this strange coercion the following assert fails:

value.set(getPatch());
value.set(transformResults);
document
.asFoundDocument(mutationResult.getVersion(), document.getData())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This API chain reads as if the we're generating a new document from the first. For example, with methods tend to be used in place of setters specifically in immutable APIs. Chaining isn't intrinsically a problem, it's just that the names don't suggest they're mutating in place. Since we're mutating this, it seems like you could just do setCommittedMutations.

as is trickier. What do you think of changing as to setTo, resetAs, or changeTo? I'm not 100% sold on any of those being any good, but as on its own doesn't work well because it's frequently used in methods that generate a view of a backing entity without changing it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly picked the API names because they looked nice, but it is true that they may suggest different behavior. I updated all calls to use set as the prefix.

public void applyToLocalView(Document document, Timestamp localWriteTime) {
verifyKeyMatches(document);

if (getPrecondition().isValidFor(document)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a style preference to apply here, but since I've heard Costa argue for it before, I'll repeat it: there's value in handling guard conditions with early returns. This reduces nesting and complexity (go/tott/455).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

if (this.updateTime != null) {
return (maybeDoc instanceof Document) && maybeDoc.getVersion().equals(this.updateTime);
return maybeDoc != null && maybeDoc.getVersion().equals(this.updateTime);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing to prevent a non-existing document from entering this method, so it seems like this should be maybeDoc != null && maybeDoc.exists() && .... This seems doubly-so since you're handling the exists case like that below. (NO_DOCUMENT and MISSING_DOCUMENT have versions but don't exist, and couldn't have the precondition apply at the server.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to include the isFoundDocument (previously exists()) call. I also made the argument non-nullable.

List<Value> transformResults =
serverTransformResults(maybeDoc, mutationResult.getTransformResults());
newData = transformObject(newData, transformResults);
Map<FieldPath, Value> transformResults =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elsewhere in this PR you changed getTransformResults to be non-nullable. That actually seems unrelated to the core of this change, and introduces the risk that it wouldn't be caught in review. Consider decoupling that into a separate change.

Otherwise, the line above needs to be a !isEmpty() test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separated this out as a stand alone PR: #2428

localValue.set(transformResults);
document.asFoundDocument(mutationResult.getVersion(), localValue).withCommittedMutations();
} else {
document.asFoundDocument(mutationResult.getVersion(), value.clone()).withCommittedMutations();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could avoid duplicating most of the arguments to asFoundDocument by following the old form of this method:

ObjectValue newData = value.clone();
if (mutationResult.getTransformResults() != null) {
  transformResults = ...;
  newData.set(transformResults);
}

return document.asFoundDcument(mutationResult.getVersion(), newData).withCommittedMutations();

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done (but some of this is no longer applicable)

schmidt-sebastian added a commit that referenced this pull request Feb 9, 2021
Gil asked that this should not be part of #2383, but I think it still makes sense as an indepedent simplification
Copy link
Contributor Author

@schmidt-sebastian schmidt-sebastian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feedback addressed (hopefully)

documentKey,
maybeDoc.getKey());
}
public void applyToLocalView(DocumentKey documentKey, Document document) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove it.

public void applyToLocalView(Document document, Timestamp localWriteTime) {
verifyKeyMatches(document);

if (getPrecondition().isValidFor(document)) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

if (doc instanceof Document) {
result = result.insert(doc.getKey(), (Document) doc);
Document doc = getDocument(DocumentKey.fromPath(path));
if (doc.exists()) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I renamed all these status getters to reflect the type of the document. This is now isFoundDocument(). There is still some room for improvement, so suggestions welcome.

FWIW, in my opinion unknown documents also definitely exist.

}
mutation.applyToLocalView(document, batch.getLocalWriteTime());
if (document.isValid()) {
results = results.insert(key, document);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rewrote this to match the old pattern. These documents already get removed by computeDocChanges (which is why the tests didn't break). I added a new test that would break though (doesNotIncludeDocumentsDeletedByMutation).

Document document = documentMap.get(key);
applyToLocalView(key, document);
if (!document.isValid()) {
document.asMissingDocument(SnapshotVersion.NONE);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know the answer to your question - but without this strange coercion the following assert fails:

value.set(getPatch());
value.set(transformResults);
document
.asFoundDocument(mutationResult.getVersion(), document.getData())
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly picked the API names because they looked nice, but it is true that they may suggest different behavior. I updated all calls to use set as the prefix.

if (this.updateTime != null) {
return (maybeDoc instanceof Document) && maybeDoc.getVersion().equals(this.updateTime);
return maybeDoc != null && maybeDoc.getVersion().equals(this.updateTime);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to include the isFoundDocument (previously exists()) call. I also made the argument non-nullable.

localValue.set(transformResults);
document.asFoundDocument(mutationResult.getVersion(), localValue).withCommittedMutations();
} else {
document.asFoundDocument(mutationResult.getVersion(), value.clone()).withCommittedMutations();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done (but some of this is no longer applicable)

@schmidt-sebastian schmidt-sebastian changed the title RFC: Mutable Documents Mutable Documents Feb 10, 2021
@schmidt-sebastian schmidt-sebastian requested review from dconeybe and wilhuff and removed request for wilhuff February 16, 2021 22:16
@schmidt-sebastian
Copy link
Contributor Author

I pushed a couple of renamings. This PR now uses setX for all methods that modify the document state. I also renamed Document to MutableDocument. I did experiment with adding a DocumentBuilder or keeping Document separate from MutableDocunent, but it leads to some strange APIs in the RemoteDocumentCache as it would store Documents but return Builders. Let me know what you think.

Copy link
Contributor

@dconeybe dconeybe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My review is very much incomplete, but I thought I'd post the comments I've left so far on 7 of the 92 files in this review :) Most of my comments concern naming and vestigial documentation that needs updating.


@Override
public int hashCode() {
return key.hashCode();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am concerned about using MutableDocument instances as keys in a table or values in a set. I'm sure everything is fine in this PR; however, future changes may accidentally change a MutableDocument that is stored in a set or used as a key and cause undefined behavior of the collection. It's kind of like storing StringBuilder objects in a Set instead of String objects. If this bug ever did happen then it would be challenging to find the root cause because the bug would be quite removed from the problematic mutation of the MutableDocument. What are your thoughts on this risk?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uff. The risk is absolutely there, especially as we further modify this code in the coming years. I spend the last couple of days trying to see how we could mitigate this risk without creating additional document copies (that would defeat the purpose of the PR a bit). In the end, I added a Document interface that provides getters but no setters. These Documents are now used in Views. While this does not 100% remove the risk as the documents can still be mutated in other parts of the code, the vast majority of the code base is now shielded from regressions. Let me know what you think.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a pretty good compromise. I like it.

boolean hasCommittedMutations;

public MutableDocument(DocumentKey key) {
this.key = key;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: Consider calling the private constructor to do this work. IMO it's more readable and explicit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

currentLevel.put(currentSegment, nextLevel);
currentLevel = nextLevel;
}
public void set(Map<FieldPath, Value> data) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you choose a different name for the set() method? Its name implies that it clobbers all existing data but in fact it acts more like a "set each" or "set all" method. So... maybe setAll() is a better name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setAll() sounds good to me. It matches the Java Collections add/addAll methods.

* Applies any overlays from `currentOverlays` that exist at `currentPath` and returns the merged
* data at `currentPath` (or null if there were no changes).
*
* @param currentPath The path at the current nesting level. Can be set toFieldValue.EMPTY_PATH to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: toFieldValue.EMPTY_PATH

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

* an `UnknownDocument` can be returned.
* Applies this mutation to the given Document for the purposes of computing a new remote document
* If the input document doesn't match the expected state (e.g. it is null or outdated), an
* `UnknownDocument` may be created
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"may be created" is somewhat misleading, since the MutableDocument is modified in place. Please update.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

* Applies this mutation to the given MaybeDocument for the purposes of computing the new local
* view of a document. Both the input and returned documents can be null.
* Applies this mutation to the given Document for the purposes of computing the new local view of
* a document. Both the input and returned documents can be null.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Both the input and returned documents can be null" is now somewhat incorrect since the method now returns null. Please update.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@Nullable MaybeDocument maybeDoc, List<Value> serverTransformResults) {
ArrayList<Value> transformResults = new ArrayList<>(fieldTransforms.size());
protected Map<FieldPath, Value> serverTransformResults(
MutableDocument maybeDoc, List<Value> serverTransformResults) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rename maybeDoc. Also on line 193.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor

@dconeybe dconeybe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is another round of feedback. My brain has turned to mush so I'll resume, and hopefully finish, my first pass of the review tomorrow.

return new Document(
key(key), version(version), wrapObject(data), Document.DocumentState.SYNCED);
public static MutableDocument doc(String key, long version, Map<String, Object> data) {
return new MutableDocument(key(key)).setFoundDocument(version(version), wrapObject(data));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For readability and maintainability, could these overloaded doc() methods simply delegate to a "canonical" method? The code duplication between these methods suggest that they all do something a little different but they appear to do the same thing but some overloaded versions "massage" the input arguments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The canonical overload does not yet exist but I created one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you actually implement the canonical overload of doc()? I don't see it (but I could very easily be using GitHub's reviews incorrectly).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

firestore, (Document) doc, /*fromCache=*/ false, /*hasPendingWrites=*/ false);
} else if (doc instanceof NoDocument) {
firestore,
(MutableDocument) doc,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cast to MutableDocument appear to be unnecessary. Please remove it if it is not required.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

boolean hasLocalMutations;
boolean hasCommittedMutations;

public MutableDocument(DocumentKey key) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about making the MutableDocument constructor private and instead providing "factory functions" to create new instances? This idea came to mind after reviewing the rest of this PR and seeing a lot of new MutableDocument(key).setFoundDocument(...) calls, which begs the following question: "in what state would the newly-created MutableDocument have been had setFoundDocument() not been called?".

If these call sites were instead MutableDocument.newFoundDocument(...) then it would be immediately obvious that a new object is being created and will initially be in the "found" state.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was part of my initial attempts, but I abandoned it because it requires me to have both the factory methods as well as the methods to transition documents (as those are used in the mutations).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is true; however, IMO having both "setters" and "factory methods" is a reasonable tradeoff in the name of readability at the call sites.

if (maybeDoc instanceof Document) {
return (Document) maybeDoc;
} else if (maybeDoc instanceof NoDocument) {
MutableDocument document = result.getResult();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the object returned from result.getResult() potentially nullable (as it was annotated to be in the original code)? If so, you will need to perform explicit null checks before invoking methods like isFoundDocument() and isNoDocument() on it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not nullable anymore. What used to be null Document is now an invalid document.

Document newDoc = null;
MaybeDocument maybeDoc = entry.getValue();
MutableDocument oldDoc = oldDocumentSet.getDocument(key);
MutableDocument newDoc = null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like how you've changed the logic here to improve readability. The old code initialized newDoc to null, set it to non-null if the entry was a "found" document, then set it back to null if it didn't match the query. Now, it's initialized to null then set to non-null only if the entry is a "found" document that matches the query.

I think this could be improved from a readability perspective even further by not initializing newDoc and assigning it exactly once. Namely, I'm thinking something like this:

MutableDocument newDoc;

if (! entry.getValue().isFoundDocument()) {
  newDoc = null;
} else {
  hardAssert(...);
  if (query.matches(entry.getValue())) {
    newDoc = entry.getValue();
  } else {
    newDoc = null;
  }
}

Another idea would be to move the hardAssert to after initializing newDoc, which IMO makes the initialization of newDoc even easier to understand:

MutableDocument newDoc =
    entry.getValue().isFoundDocument() && query.matches(entry.getValue())
        ? entry.getValue()
        : null;

if (newDoc != null) {
  hardAssert(...);
}

Anyways, this is an optional suggestion to consider.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated this a bit in my commit that introduces the Document interface. It is simpler now, but it does bring back the old pattern of assigning newDoc back to null. Let me know what you think.

I also removed the assert since it just asserts that key is entry.getKey(), which is the assignment in the first line of this block.

} else {
throw fail("Unknown document type %s", document.getClass().getCanonicalName());
throw fail("Unknown Document %s", document);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider rewording this error message since the concept of an "unknown document" is already taken, and would have caused the "else if" branch above to be entered had the document been "unknown".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

if (readTime.compareTo(sinceReadTime) <= 0) {
continue;
}
MutableDocument doc = entry.getValue().first;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally, I prefer the "guard" style from the original code. The new code mixes an "if" statement and a guard, which is arguably more confusing. For readability, could you switch to exclusively use the "guard" style?

i.e.

while (iterator.hasNext()) {
  Map.Entry<DocumentKey, Pair<MutableDocument, SnapshotVersion>> entry = iterator.next();

  DocumentKey key = entry.getKey();
  if (!queryPath.isPrefixOf(key.getPath())) {
    break;
  }

  MutableDocument doc = entry.getValue().first;
  if (! doc.isFoundDocument()) {
    continue;
  }
  
  SnapshotVersion readTime = entry.getValue().second;
  if (readTime.compareTo(sinceReadTime) <= 0) {
    continue;
  }

  if (!query.matches(doc)) {
    continue;
  }
  
  result = result.insert(doc.getKey(), doc.clone());
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

@@ -37,10 +35,10 @@
* <p>The cache key is extracted from {@code maybeDocument.key}. If there is already a cache entry
* for the key, it will be replaced.
*
* @param maybeDocument A Document or NoDocument to put in the cache.
* @param document A Document or NoDocument to put in the cache.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: The mention of NoDocument here is obsolete. Also, Document should not begin with an uppercase letter because it is no longer the name of a class. Also on lines 50 and 58.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated this entire file.

* @return The cached Document or NoDocument entries indexed by key. If an entry is not cached,
* the corresponding key will be mapped to a null value.
* @return The cached Document or NoDocument entries indexed by key. If an entry is not cached, an
* invalid document is returned.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This statement that an "invalid document is returned" is incorrect because the method returns a Map, not a document. The original comment can probably be tweaked, something like "If an entry is not cached, the corresponding key will be mapped to an invalid document."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. Done.

DocumentKey documentKey, MutableDocument document, MutationBatchResult batchResult) {
hardAssert(
document.getKey().equals(documentKey),
"applyToRemoteDocument: key %s doesn't match maybeDoc key %s",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: fix this outdated mention of "maybeDoc" in this assertion message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Copy link
Contributor

@dconeybe dconeybe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this LGTM. My comments about naming are "optional" since coming up with the perfect names is proving to be challenging and probably not worth a bunch of back and forth. Moreover, we can always change then names later.

boolean hasLocalMutations;
boolean hasCommittedMutations;

public MutableDocument(DocumentKey key) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is true; however, IMO having both "setters" and "factory methods" is a reasonable tradeoff in the name of readability at the call sites.

}

/** Changes the document type to FOUND_DOCUMENT and sets the given version and data. */
public MutableDocument setFoundDocument(SnapshotVersion version, ObjectValue value) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about reinitializeAsFoundDocument(), reinitializeAsNoDocument(), etc? Also, IMO "reset" is a good companion to the other "set" methods and conveys that the scope of the changes made are the entire object instead of a particular field in the object.

return this;
}

public MutableDocument setCommittedMutations() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO setCommittedMutations() sounds like you are setting the list/collection of committed mutations as opposed to a boolean flag indicating that there are committed mutations. If you rename the enum value to HAS_COMMITTED_MUTATIONS then it will be more internally consistent :) You pick what you prefer and we'll move on. Naming is hard.

private enum DocumentType {
/**
* Represents the initial state of a MutableDocument when only the document key is known.
* Invalid documents transition to other states as mutations are applied. If a document remain
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: "remain invalids" -> "remains invalid".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed


@Override
public int hashCode() {
return key.hashCode();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a pretty good compromise. I like it.

return new Document(
key(key), version(version), wrapObject(data), Document.DocumentState.SYNCED);
public static MutableDocument doc(String key, long version, Map<String, Object> data) {
return new MutableDocument(key(key)).setFoundDocument(version(version), wrapObject(data));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you actually implement the canonical overload of doc()? I don't see it (but I could very easily be using GitHub's reviews incorrectly).

@@ -74,38 +71,31 @@ IndexManager getIndexManager() {
/**
* Returns the the local view of the document identified by {@code key}.
*
* @return Local view of the document or null if we don't have any cached state for it.
* @return Local view of the document or a null if we don't have any cached state for it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that this method returns null, which makes this javadoc @return tag incorrect: "or a null if we don't have any cached state for it". If it does indeed potentially return null, add back the @nullable annotation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not, it returns an invalid document instead of null. Fixed.

}

return document;
}

// Returns the view of the given {@code docs} as they would appear after applying all mutations in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This javadoc comment for applyLocalMutationsToDocuments() is incorrect since the method returns void: "Returns the view of the given". Please correct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Thanks for catching this as well.

@@ -162,13 +154,13 @@ static SnapshotVersion getPostMutationVersion(@Nullable MaybeDocument maybeDoc)
* result of applying a transform) for use after a mutation containing transforms has been
* acknowledged by the server.
*
* @param maybeDoc The current state of the document after applying all previous mutations.
* @param mutableDocument The current state of the document after applying all previous mutations.
* @param serverTransformResults The transform results received by the server.
* @return The transform results list.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ultranit: "list" is no longer accurate, since the return type has changed to a map. Please update. Also on line 192.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed (here and below)

Copy link
Contributor Author

@schmidt-sebastian schmidt-sebastian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review! Next one will be at least 10x as long :) (okay, maybe not)

private enum DocumentType {
/**
* Represents the initial state of a MutableDocument when only the document key is known.
* Invalid documents transition to other states as mutations are applied. If a document remain
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

return new Document(
key(key), version(version), wrapObject(data), Document.DocumentState.SYNCED);
public static MutableDocument doc(String key, long version, Map<String, Object> data) {
return new MutableDocument(key(key)).setFoundDocument(version(version), wrapObject(data));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -74,38 +71,31 @@ IndexManager getIndexManager() {
/**
* Returns the the local view of the document identified by {@code key}.
*
* @return Local view of the document or null if we don't have any cached state for it.
* @return Local view of the document or a null if we don't have any cached state for it.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not, it returns an invalid document instead of null. Fixed.

}

return document;
}

// Returns the view of the given {@code docs} as they would appear after applying all mutations in
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Thanks for catching this as well.

@@ -162,13 +154,13 @@ static SnapshotVersion getPostMutationVersion(@Nullable MaybeDocument maybeDoc)
* result of applying a transform) for use after a mutation containing transforms has been
* acknowledged by the server.
*
* @param maybeDoc The current state of the document after applying all previous mutations.
* @param mutableDocument The current state of the document after applying all previous mutations.
* @param serverTransformResults The transform results received by the server.
* @return The transform results list.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed (here and below)

@google-oss-bot
Copy link
Contributor

@schmidt-sebastian: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
smoke-tests 6a81979 link /test smoke-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@schmidt-sebastian schmidt-sebastian merged commit 979000c into master Feb 25, 2021
@schmidt-sebastian schmidt-sebastian deleted the mrschmidt/mutabledocuments branch February 25, 2021 21:38
@firebase firebase locked and limited conversation to collaborators Mar 28, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants