-
Notifications
You must be signed in to change notification settings - Fork 948
Add README.md about spec tests #4424
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| # Firestore Spec Tests | ||
|
|
||
| The Firestore Spec Tests are a suite of unit tests that fake out external | ||
| sources of events so that they can be completely controlled from the tests | ||
| themselves and verify the correct handling of these events. | ||
|
|
||
| There are three sources of external events in the SDK: user writes, input from | ||
| the persistence layer, and input from the watch stream. The spec tests fake all | ||
| of these. Since these external events are asynchronous in nature, in the real | ||
| world there is some non-determinism in the ordering of events. The spec tests | ||
| allow these orderings to be specified explicitly, enabling orderings that are | ||
| difficult or impossible to force when connected to a real backend. In addition, | ||
| when running under Node, the IndexedDb persistence layer is faked. | ||
|
|
||
| The Firestore Spec Tests are portable and must be kept in sync between the Web | ||
| SDK (this SDK), Android SDK, and iOS SDK. The Web SDK contains the authoritative | ||
| test definitions and they must be manually "exported" to the Android SDK and iOS | ||
| SDK when they change. | ||
|
|
||
| ## Running the Tests | ||
|
|
||
| The spec tests are part of the unit test suite of each SDK. Therefore, to run | ||
| the spec tests simply run the entire unit test suite and the spec tests will be | ||
| executed as a part of it. | ||
|
|
||
| ## Test Tags | ||
|
|
||
| Each spec test can specify zero or many "tags" that influence the test's | ||
| execution. Some of the commonly-used tags are documented here; see | ||
| [describe_spec.ts](describe_spec.ts) for the full set of supported tags. | ||
|
|
||
| - `exclusive` - Only run this spec test and skip all others. It can be useful | ||
| to apply this tag to a test while it is under active development or debugging. | ||
| Tests should never be checked into source control with this tag. | ||
|
|
||
| - `no-android` and `no-ios` - Do not run this spec test on Android and/or iOS, | ||
| respectively. It may be useful to specify these tags if the functionality has | ||
| not been or never will be ported to these platforms. | ||
|
|
||
| ## Exporting the Tests | ||
|
|
||
| To export the tests use the [generate_spec_json.sh](../generate_spec_json.sh) | ||
| script, specifying a single argument whose value is the destination directory | ||
| into which to place the generated files. | ||
|
|
||
| #### Exporting the Tests to the Android SDK | ||
|
|
||
| To export the spec tests to the Android SDK, run the following command: | ||
|
|
||
| ``` | ||
| cd ~/firebase-js-sdk/packages/firestore/test/unit && | ||
| ./generate_spec_json.sh ~/firebase-android-sdk/firebase-firestore/src/test/resources/json | ||
| ``` | ||
|
|
||
| This command assumes that this Git repository is cloned into `~/firebase-js-sdk` | ||
| and the Android SDK is cloned into `~/firebase-android-sdk`. | ||
|
|
||
| #### Exporting the Tests to the iOS SDK | ||
|
|
||
| To export the spec tests to the iOS SDK, run the following command: | ||
|
|
||
| ``` | ||
| cd ~/firebase-js-sdk/packages/firestore/test/unit && | ||
| ./generate_spec_json.sh ~/firebase-ios-sdk/Firestore/Example/Tests/SpecTests/json | ||
| ``` | ||
|
|
||
| This command assumes that this Git repository is cloned into `~/firebase-js-sdk` | ||
| and the iOS SDK is cloned into `~/firebase-ios-sdk`. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, this is a good starting point for a spec test introduction. I think the complications arise when someone wants to go past these setup steps, but I am honestly not quite sure how we could teach someone how to write a spec test over a README. The actual tests require an insane amount of technical knowledge, and I would be wary to put all this in writing.
Even if we want to focus on the high level nature of these tests, we may still want to expand on this and write about the tags. We should mention "exclusive", "no-ios" and "no-android" (and maybe "no-web", but I don't know when that would make sense).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good point about the tags. I've added information about the tags. I agree that trying to document the spec tests in their entirety is not feasible. I am mostly writing this so that the next time I need to work on the spec tests I have something to remind me of the basics, and this will hopefully be helpful to future authors as well.