-
Notifications
You must be signed in to change notification settings - Fork 624
Protobuf-backed FieldValues #1184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Protobuf-backed FieldValues #1184
Conversation
b85fba2
to
433da87
Compare
cf6369a
to
282380f
Compare
282380f
to
1602062
Compare
Codecov Report
Continue to review full report at Codecov.
|
@@ -222,7 +222,7 @@ public void testWriteTheSameServerTimestampAcrossWrites() { | |||
assertTrue(localSnap.getMetadata().hasPendingWrites()); | |||
assertEquals(asList(map("when", null), map("when", null)), querySnapshotToValues(localSnap)); | |||
|
|||
QuerySnapshot serverSnap = accumulator.await(); | |||
QuerySnapshot serverSnap = accumulator.awaitRemoteEvent(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR somehow changed the timing of events so much that the fromCache:false
event is now delivered before hasPendingWrites:false
. I am not 100% convinced that this is not due to a bug, but if I look at the test logs everything looks sane.
onlineCallbackDone.setResult(null); | ||
} | ||
}); | ||
ListenerRegistration listenerRegistration = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the change from #1188
@@ -32,16 +32,24 @@ | |||
import com.google.firebase.Timestamp; | |||
import com.google.firebase.firestore.GeoPoint; | |||
import com.google.firebase.firestore.model.mutation.FieldMask; | |||
import com.google.firebase.firestore.model.protovalue.ObjectValue; | |||
import com.google.firebase.firestore.model.protovalue.PrimitiveValue; | |||
import com.google.firebase.firestore.model.value.BlobValue; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI: A lot of the diff in this file brings the test more in line with master
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've done a partial review but want to give you the feedback I have so far.
codecov.yml
Outdated
@@ -9,12 +9,14 @@ coverage: | |||
default: off | |||
FirebaseCommon: | |||
flags: FirebaseCommon | |||
informational: true | |||
informational: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these changes intended?
If so, since these are changes to FirebaseCommon's configuration, maybe pull them out into a PR sent to someone that owns that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I merged master
into rewritefieldvalue
and rewritefieldvalue
into this branch, but forgot to push rewritefieldvalue
. This should be fixed.
@@ -533,7 +533,7 @@ private Object getInternal( | |||
if (val != null) { | |||
UserDataWriter userDataWriter = | |||
new UserDataWriter(firestore, timestampsInSnapshots, serverTimestampBehavior); | |||
return userDataWriter.convertValue(val); | |||
return userDataWriter.convertValue(val.getProto()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like having overloads (or just a public interface) that accepts FieldValue/ObjectValue would make this code less cumbersome to deal with.
Then again, if this is the only caller, maybe this is better.
Just a thought; no action required.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This pattern is used in two places in the code and in a bunch of test cases in UserWriterTest. I added a helper to UserDataWriterTest, but kept the usages in the code the same.
@@ -398,35 +389,37 @@ private void parseSentinelFieldValue( | |||
* @return The parsed value, or {@code null} if the value was a FieldValue sentinel that should | |||
* not be included in the resulting parsed data. | |||
*/ | |||
private FieldValue parseScalarValue(Object input, ParseContext context) { | |||
private Value parseScalarValue(Object input, ParseContext context) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This code essentially duplicates Values.valueOf
. Is there no way to unite these?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a TODO for now. I would like to remove the individual field value wrappers first.
@@ -32,8 +32,8 @@ public boolean matches(Document doc) { | |||
if (!(other instanceof ArrayValue)) { | |||
return false; | |||
} | |||
for (FieldValue val : ((ArrayValue) other).getInternalValue()) { | |||
if (arrayValue.getInternalValue().contains(val)) { | |||
for (FieldValue val : ((ArrayValue) other).getValues()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like this would be cheaper if we iterated over the proto Value
elements and compared for equality using ProtoValues
directly.
Feel free to defer into another PR that address concerns like these.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. More cleanup and the removal of ArrayValue will come in a follow-up.
*/ | ||
public boolean contains(FieldValue value) { | ||
for (Value element : internalValue.getArrayValue().getValuesList()) { | ||
if (value.equals(FieldValue.valueOf(element))) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you pull the proto out of the value outside the loop you can use ProtoValues.equals here instead which avoids the reflective type checking and allocation inherent in FieldValue.valueOf
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I updated the callers of this method to do exactly that.
import java.util.Set; | ||
|
||
/** A structured object value stored in Firestore. */ | ||
public class ObjectValue extends FieldValue { | ||
private static final ObjectValue EMPTY_INSTANCE = | ||
new ObjectValue(ImmutableSortedMap.Builder.emptyMap(Util.<String>comparator())); | ||
new ObjectValue( | ||
com.google.firestore.v1.Value.newBuilder() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shorten this to use the imported Value
and MapValue
? Here and below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
@@ -165,7 +166,8 @@ public void setLocalDocumentsView(LocalDocumentsView localDocuments) { | |||
private static double estimateFilterSelectivity(Filter filter) { | |||
hardAssert(filter instanceof FieldFilter, "Filter type expected to be FieldFilter"); | |||
FieldFilter fieldFilter = (FieldFilter) filter; | |||
if (fieldFilter.getValue().equals(null) || fieldFilter.getValue().equals(DoubleValue.NaN)) { | |||
FieldValue filterValue = fieldFilter.getValue(); | |||
if (NullValue.nullValue().equals(filterValue) || DoubleValue.NaN.equals(filterValue)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It strikes me here that we're inconsistent about how we handle these singletons. We could have NullValue.NULL
instead, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
} | ||
baseObject = baseObject.set(transform.getFieldPath(), coercedValue.getProto()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that baseObject
is a Builder, this assignment here seems duplicative. The builder only does return this
for chaining, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is not needed anymore. We also don't use chaining anywhere - I wonder if we should remove it from the Builder API.
} else { | ||
return new PrimitiveValue(value); | ||
public static @Nullable FieldValue valueOf(@Nullable Value value) { | ||
if (value != null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use an early return to reduce nesting?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
this.localWriteTime = localWriteTime; | ||
this.previousValue = previousValue; | ||
public ServerTimestampValue(Value value) { | ||
super(value); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hardAssert that isServerTimestamp(value)
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Also made the constructor package-private.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few more thoughts.
@@ -167,8 +179,8 @@ public void testConvertsTimestampValue() { | |||
for (Timestamp d : testCases) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like the other tests are using a single letter variable that's short for the type. Make this Timestamp t
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
field("foo.bar"), | ||
new ServerTimestampValue(timestamp, StringValue.valueOf("bar-value"))) | ||
.build(); | ||
com.google.firebase.firestore.model.value.FieldValue fieldValue = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Import FieldValue and shorten?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test already imports com.google.firebase.firestore.FieldValue
.
@@ -74,7 +74,7 @@ public void setsNestedField() { | |||
builder.set(field("a.b"), fooValue); | |||
builder.set(field("c.d.e"), fooValue); | |||
ObjectValue object = builder.build(); | |||
assertEquals(wrapObject("a", map("b", fooValue), "c", map("d", map("e", fooValue))), object); | |||
assertEquals(wrapObject("a", map("b", fooString), "c", map("d", map("e", fooString))), object); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fooString
isn't adding a whole lot of value here, and it's inconsistent with the other string literals here. Consider reverting that.
Also, why not pass the values as arguments to map
? That seemed like a great way to explicitly compose expected values.
@@ -118,8 +127,13 @@ public void testAddsNewFields() { | |||
@Test | |||
public void testAddsMultipleNewFields() { | |||
ObjectValue object = ObjectValue.emptyObject(); | |||
object = object.toBuilder().set(field("a"), valueOf("a")).build(); | |||
object = object.toBuilder().set(field("b"), valueOf("b")).set(field("c"), valueOf("c")).build(); | |||
object = object.toBuilder().set(field("a"), wrap("a").getProto()).build(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW, valueOf("a")
seems cleaner than wrap("a").getProto()
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Replaced this single occurrence with a call to the helper setField
.
@@ -137,7 +151,7 @@ public void testImplicitlyCreatesObjects() { | |||
@Test | |||
public void testCanOverwritePrimitivesWithObjects() { | |||
ObjectValue old = wrapObject("a", map("b", "old")); | |||
ObjectValue mod = setField(old, "a", map("b", "mod")); | |||
ObjectValue mod = setField(old, "a", wrapObject("b", "mod")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should add an overload for setField
that allows map
to work inline so that we don't have think about this distinction.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
com.google.firestore.v1.ArrayValue.newBuilder(); | ||
for (FieldValue subValue : elements) { | ||
arrayBuilder.addValues(encodeValue(subValue)); | ||
private ArrayValue encodeArrayTransformElements(List<FieldValue> elementsProto) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
List<FieldValue>
still seems like "elements" more than "elementsProto": we're asking each element to get its underlying proto.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
for (int i = 0; i < count; i++) { | ||
result.add(decodeValue(elementsProto.getValues(i))); | ||
private List<FieldValue> decodeArrayTransformElements(List<Value> elementsProto) { | ||
List<FieldValue> result = new ArrayList<>(elementsProto.size()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to make the array transform elements just a model.ArrayValue?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added TODO
@@ -648,7 +482,7 @@ public MutationResult decodeMutationResult( | |||
if (transformResultsCount > 0) { | |||
transformResults = new ArrayList<>(transformResultsCount); | |||
for (int i = 0; i < transformResultsCount; i++) { | |||
transformResults.add(decodeValue(proto.getTransformResults(i))); | |||
transformResults.add(FieldValue.valueOf(proto.getTransformResults(i))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similarly: can we make transform results a model.ArrayValue so we don't have to convert?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added TODO
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feedback addressed. I will work on a couple of follow-up PRs to address the newly added TODOs.
@@ -533,7 +533,7 @@ private Object getInternal( | |||
if (val != null) { | |||
UserDataWriter userDataWriter = | |||
new UserDataWriter(firestore, timestampsInSnapshots, serverTimestampBehavior); | |||
return userDataWriter.convertValue(val); | |||
return userDataWriter.convertValue(val.getProto()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This pattern is used in two places in the code and in a bunch of test cases in UserWriterTest. I added a helper to UserDataWriterTest, but kept the usages in the code the same.
@@ -398,35 +389,37 @@ private void parseSentinelFieldValue( | |||
* @return The parsed value, or {@code null} if the value was a FieldValue sentinel that should | |||
* not be included in the resulting parsed data. | |||
*/ | |||
private FieldValue parseScalarValue(Object input, ParseContext context) { | |||
private Value parseScalarValue(Object input, ParseContext context) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a TODO for now. I would like to remove the individual field value wrappers first.
@@ -32,8 +32,8 @@ public boolean matches(Document doc) { | |||
if (!(other instanceof ArrayValue)) { | |||
return false; | |||
} | |||
for (FieldValue val : ((ArrayValue) other).getInternalValue()) { | |||
if (arrayValue.getInternalValue().contains(val)) { | |||
for (FieldValue val : ((ArrayValue) other).getValues()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. More cleanup and the removal of ArrayValue will come in a follow-up.
@@ -165,7 +166,8 @@ public void setLocalDocumentsView(LocalDocumentsView localDocuments) { | |||
private static double estimateFilterSelectivity(Filter filter) { | |||
hardAssert(filter instanceof FieldFilter, "Filter type expected to be FieldFilter"); | |||
FieldFilter fieldFilter = (FieldFilter) filter; | |||
if (fieldFilter.getValue().equals(null) || fieldFilter.getValue().equals(DoubleValue.NaN)) { | |||
FieldValue filterValue = fieldFilter.getValue(); | |||
if (NullValue.nullValue().equals(filterValue) || DoubleValue.NaN.equals(filterValue)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
} | ||
baseObject = baseObject.set(transform.getFieldPath(), coercedValue.getProto()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is not needed anymore. We also don't use chaining anywhere - I wonder if we should remove it from the Builder API.
for (int i = 0; i < count; i++) { | ||
result.add(decodeValue(elementsProto.getValues(i))); | ||
private List<FieldValue> decodeArrayTransformElements(List<Value> elementsProto) { | ||
List<FieldValue> result = new ArrayList<>(elementsProto.size()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added TODO
@@ -648,7 +482,7 @@ public MutationResult decodeMutationResult( | |||
if (transformResultsCount > 0) { | |||
transformResults = new ArrayList<>(transformResultsCount); | |||
for (int i = 0; i < transformResultsCount; i++) { | |||
transformResults.add(decodeValue(proto.getTransformResults(i))); | |||
transformResults.add(FieldValue.valueOf(proto.getTransformResults(i))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added TODO
@@ -118,8 +127,13 @@ public void testAddsNewFields() { | |||
@Test | |||
public void testAddsMultipleNewFields() { | |||
ObjectValue object = ObjectValue.emptyObject(); | |||
object = object.toBuilder().set(field("a"), valueOf("a")).build(); | |||
object = object.toBuilder().set(field("b"), valueOf("b")).set(field("c"), valueOf("c")).build(); | |||
object = object.toBuilder().set(field("a"), wrap("a").getProto()).build(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Replaced this single occurrence with a call to the helper setField
.
@@ -137,7 +151,7 @@ public void testImplicitlyCreatesObjects() { | |||
@Test | |||
public void testCanOverwritePrimitivesWithObjects() { | |||
ObjectValue old = wrapObject("a", map("b", "old")); | |||
ObjectValue mod = setField(old, "a", map("b", "mod")); | |||
ObjectValue mod = setField(old, "a", wrapObject("b", "mod")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
field("foo.bar"), | ||
new ServerTimestampValue(timestamp, StringValue.valueOf("bar-value"))) | ||
.build(); | ||
com.google.firebase.firestore.model.value.FieldValue fieldValue = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test already imports com.google.firebase.firestore.FieldValue
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This finishes up the Protobuf FieldValue rewrite:
protovalue
classes tovalue
packageThe individual FieldValue types (IntegerValue, DoubleValue, etc) still exists, which makes the diff considerably smaller and keeps their current factory methods.
Performance with master for returning X docs out a collection of size Y:
20 out of 1000: 3692 505 382 310 524 228 409 356
1000 out of 1000: 8813 1512 1066 1748 997 1114 1145
With this branch:
20 out of 1000: 714 238 279 311 277 339 245 308
1000 out of 1000: 1344 1186 677 767 726 657 673 636
I will get more numbers (once I have fixed the Performance spec tests, but I will do that tomorrow).