Skip to content

Add ability to delete indexes #3118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Nov 12, 2021
Merged

Add ability to delete indexes #3118

merged 12 commits into from
Nov 12, 2021

Conversation

schmidt-sebastian
Copy link
Contributor

This PR diffs the index configuration that the user provides with the currently active definition and removes indexes that are no longer used

@schmidt-sebastian
Add ability to delete indexes
eae822b 
This PR diffs the index configuration that the user provides with the currently active definition and removes indexes that are no longer used
@google-cla google-cla bot added the cla: yes Override cla label Nov 9, 2021
@schmidt-sebastian
Copy link
Contributor Author

cc @thebrianchen

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Nov 9, 2021 •
edited
Loading

Coverage Report

Affected SDKs

  • firebase-firestore

    SDK overall coverage changed from 44.86% (fe64756) to 45.04% (0474c6bf) by +0.18%.

    Filename Base (fe64756) Head (0474c6bf) Diff
    AutoValue_FieldIndex_Segment.java 34.78% 56.52% +21.74%
    DeleteMutation.java 90.00% 95.00% +5.00%
    FieldIndex.java 76.92% 88.06% +11.14%
    LocalStore.java 97.48% 99.38% +1.89%
    MemoryIndexManager.java 72.00% 66.67% -5.33%
    SQLiteIndexManager.java 95.82% 95.97% +0.15%
    Util.java 53.57% 62.28% +8.71%

Test Logs

Notes

HTML coverage reports can be produced locally with ./gradlew <product>:checkCoverage.
Report files are located at <product-build-dir>/reports/jacoco/.

Head commit (0474c6bf) is created by Prow via merging commits: fe64756 1dec73a.

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Nov 9, 2021 •
edited
Loading

Binary Size Report

Affected SDKs

  • firebase-firestore

    Type Base (fe64756) Head (0474c6bf) Diff
    aar 1.22 MB 1.22 MB +2.17 kB (+0.2%)
    apk (release) 3.32 MB 3.32 MB +212 B (+0.0%)

Test Logs

Notes

Head commit (0474c6bf) is created by Prow via merging commits: fe64756 1dec73a.

wu-hui
wu-hui reviewed Nov 11, 2021
View reviewed changes
Copy link
Contributor

@wu-hui wu-hui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on what is in the PR.

I only left one comment, let me know if you want to simply add a TODO, I'll approve in that case.

firebase-firestore/src/main/java/com/google/firebase/firestore/local/LocalStore.java Outdated
Collection<FieldIndex> existingIndexes = indexManager.getFieldIndexes();

// Delete indices that are no longer used.
for (FieldIndex existingIndex : existingIndexes) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit worried about performance here. With a 30 indexes setup, this could easily lead up to 1800 (or 1000 realistically) operations.

I want to suggest to explore something like a bloom filter to help. But I realize this might also be premature optimization. So how about adding a TODO to followup?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, can the two loop be merged, such that we only call matchesConstraints half times?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rewrote the entire thing to run in O(n log n). Can you take another look?

wu-hui
wu-hui reviewed Nov 12, 2021
View reviewed changes
firebase-firestore/src/main/java/com/google/firebase/firestore/local/LocalStore.java Outdated

if (existingValue != null && updatedValue != null) {
int cmp = FieldIndex.SEMANTIC_COMPARATOR.compare(existingValue, updatedValue);
if (cmp < 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not obvious immediately why comparison result leads to deleted or updated. Can we add some comment? Ideally a high level comment for the method, and some implementation comment to explain details.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added comments to the more generic implementation, as well as tests.

firebase-firestore/src/main/java/com/google/firebase/firestore/local/LocalStore.java Outdated
@Nullable FieldIndex updatedValue = advanceIterator(updatedIt);

while (existingValue != null || updatedValue != null) {
boolean deleted = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think they can be named deleteExistingIndex and writeNewIndex?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually realized yesterday that I need to re-use this functionality in a follow up PR, so I changed this method to be more generic and moved it to Util.

@wu-hui wu-hui assigned schmidt-sebastian and unassigned wu-hui Nov 12, 2021
@google-oss-bot
Copy link
Contributor

google-oss-bot commented Nov 12, 2021 •
edited
Loading

@schmidt-sebastian: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
smoke-tests 1dec73a link /test smoke-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@schmidt-sebastian
Copy link
Contributor Author

/test device-check-changed

wu-hui
wu-hui approved these changes Nov 12, 2021
View reviewed changes
Copy link
Contributor

@wu-hui wu-hui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@wu-hui wu-hui removed their assignment Nov 12, 2021
@schmidt-sebastian schmidt-sebastian merged commit d481861 into master Nov 12, 2021
@schmidt-sebastian schmidt-sebastian deleted the mrschmidt/deleteindices branch November 12, 2021 19:00
@schmidt-sebastian schmidt-sebastian mentioned this pull request Nov 22, 2021
Merged
@firebase firebase locked and limited conversation to collaborators Dec 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@wu-hui wu-hui wu-hui approved these changes

Assignees

@schmidt-sebastian schmidt-sebastian

Labels
api: firestore cla: yes Override cla size/L
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@schmidt-sebastian @google-oss-bot @wu-hui