-
Notifications
You must be signed in to change notification settings - Fork 80
[docs] Add documentation about writing tests for IndexStoreDB #35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,124 @@ | ||
# Development | ||
|
||
This document contains notes about development and testing of IndexStoreDB. | ||
|
||
## Table of Contents | ||
|
||
* [Writing Tests](#writing-tests) | ||
* [Tibs, the "Test Index Build System"](Tibs.md) | ||
|
||
## Writing Tests | ||
|
||
As much as is practical, all code should be covered by tests. New tests can be added under the `Tests` directory and should use `XCTest`. The rest of this section will describe the additional tools available in the `ISDBTestSupport` module to make it easier to write good and efficient tests. | ||
|
||
Most indexer tests follow a pattern: | ||
|
||
1. Build and index a test project | ||
2. Perform index queries | ||
3. Compare the results against known locations in the test code | ||
4. (Optional) modify the code and repeat. | ||
|
||
### Test Projects (Fixtures) | ||
|
||
Index test projects should be put in the `Tests/INPUTS` directory, and use the [Tibs](Tibs.md) build system to define their sources and targets. An example test project might look like: | ||
|
||
``` | ||
Tests/ | ||
INPUTS/ | ||
MyTestProj/ | ||
a.swift | ||
b.swift | ||
c.cpp | ||
``` | ||
|
||
Where `project.json` describes the project's targets, for example | ||
|
||
``` | ||
{ "sources": ["a.swift", "b.swift", "c.cpp"] } | ||
``` | ||
|
||
Tibs supports more advanced project configurations, such as multiple swift modules with dependencies, etc. For much more information about Tibs, including what features it supports and how it works, see [Tibs.md](Tibs.md). | ||
|
||
### TibsTestWorkspace | ||
|
||
The `TibsTestWorkspace` pulls together the various pieces needed for working with tests, including building the project, creating an IndexStoreDB instance, and keeping it up to date after modifying sources. | ||
|
||
To create a `TibsTestWorkspace`, use the `staticTibsTestWorkspace` and `mutableTibsTestWorkspace` methods. Tests that do not mutate sources should use `staticTibsTestWorkspace`. | ||
|
||
```swift | ||
func testFoo() { | ||
|
||
// Loads an immutable workspace for the test project "MyTestProj" and resolves | ||
// all of its targets, ready to be indexed. | ||
guard let ws = try staticTibsTestWorkspace(name: "MyTestProj") else { return } | ||
|
||
// Perform the build and read the produced index data. | ||
try ws.buildAndIndex() | ||
|
||
// See the results | ||
ws.index.occurrences(ofUSR: "<some usr>", ...) | ||
|
||
... | ||
} | ||
``` | ||
|
||
#### Source Locations | ||
|
||
It is common to want to refer to specific locations in the source code of a test project. This is supported using inline comment syntax. | ||
|
||
```swift | ||
Test.swift: | ||
func /*myFuncDef*/myFunc() { | ||
/*myFuncCall*/myFunc() | ||
} | ||
``` | ||
|
||
In a test, these locations can be referenced by name. The named location is immediately after the comment. | ||
|
||
```swift | ||
|
||
let loc = ws.testLoc("myFuncDef") | ||
// TestLocation(url: ..., line: 1, column: 19) | ||
``` | ||
|
||
A common use of a `TestLocation` is to form a `SymbolOccurrence` at that location. | ||
|
||
```swift | ||
let occurrence = Symbol(...).at(ws.testLoc("myFuncDef"), roles: .definition) | ||
``` | ||
|
||
#### Comparing SymbolOccurrences | ||
|
||
Rather than attempting to compare `SymbolOccurrences` for equality, prefer using the custom XCTest assertion method `checkOccurrences`. This API avoids accidentally comparing details that are undefined (e.g. the order of occurrences), or likely to change frequently (e.g. `SymbolRole`s are checked to be a superset instead of an exact match, as we often add new roles). | ||
|
||
``` | ||
checkOccurrences(ws.index.occurrences(ofUSR: sym.usr, roles: .all), expected: [ | ||
sym.at(ws.testLoc("c"), roles: .definition), | ||
sym.at(ws.testLoc("c_call"), roles: .call), | ||
]) | ||
``` | ||
|
||
#### Mutating Test Sources | ||
|
||
In order to test changes to the index, we create a mutable test workspace. The mutable project takes care of copying the sources to a new location. Afterwards, we can mutate the sources using the `TibsTestWorkspace.edit(rebuild: Bool, block: ...)` method. | ||
|
||
``` | ||
func testMutating() { | ||
guard let ws = try mutableTibsTestWorkspace(name: "MyTestProj") else { return } | ||
try ws.buildAndIndex() | ||
|
||
// Check initial state... | ||
|
||
// Perform modifications to "url"! | ||
ws.edit(rebuild: true) { editor, files in | ||
|
||
// Read the original contents and add a new line. | ||
let new = try files.get(url).appending("func /*moreCodeDef*/moreCode() {}") | ||
|
||
// Provide the new contents for "url". | ||
editor.write(new, to: url) | ||
} | ||
} | ||
``` | ||
|
||
After we return from `edit()`, the sources are modified and any changes to stored source locations are reflected. We can now `buildAndIndex()` to update the index, or as a convenience we can pass `rebuild: true` to `edit`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,128 @@ | ||
# Tibs | ||
|
||
Tibs ("Test Index Build System") is a simple and flexible build system designed for test projects of IndexStoreDB and SourceKit-LSP. Tibs can incrementally build or rebuild the index data and generated module files for a Swift and/or C-family language test project. Tibs can also dump the compiler arguments to a clang-compatible JSON compilation database. It is *not designed to compile to executable code*. | ||
|
||
Tibs is implemented using [Ninja](https://ninja-build.org), which introduces a new dependency in IndexStoreDB when running tests. | ||
|
||
Tibs projects are described by a `project.json` file containing one or more targets. Typically, a test case will use a project fixture located in the `Tests/INPUTS` directory. | ||
|
||
## Project | ||
|
||
A Tibs project is described by `project.json` manifest file. The top-level entity is a target, which may depend on other targets. | ||
|
||
Example: | ||
|
||
``` | ||
{ | ||
"targets": [ | ||
{ | ||
"name": "mytarget", | ||
"swift_flags": ["-warnings-as-errors"], | ||
"sources": ["a.swift", "b.swift"], | ||
"dependencies": ["dep1"], | ||
}, | ||
{ | ||
"name": "dep1", | ||
"sources": ["dep1.swift"], | ||
}, | ||
] | ||
} | ||
``` | ||
|
||
As a convenience, if the project consists of a single target, it can be written at the top level and the name can be omitted, e.g. | ||
|
||
``` | ||
{ "sources": ["main.swift"] } | ||
``` | ||
|
||
### Targets | ||
|
||
Targets have the following fields: | ||
|
||
* "name": The name of the target. If omitted, it will be "main". | ||
* "sources": Array of source file paths. | ||
* "bridging_header": Optional path to a Swift bridging header. | ||
* "dependencies": Optional array of target names that this target depends on. This is useful for Swift module dependencies. | ||
* "swift_flags": Optional array of extra compiler arguments for Swift. | ||
* "clang_flags": Optional array of extra compiler arguments for Clang. | ||
|
||
The directory containing the `project.json` manifest file is considered the project's root directory and any relative paths are relative to that. | ||
|
||
## Building Tibs Projects | ||
|
||
Most tests using Tibs will use the APIs provided by the `TibsTestWorkspace` class in the `ISDBTestSupport` module, which provides a high-level API for working with a complete Tibs project. The following information is for anyone needing to work using the lower-level APIs from `ISDBTibs` or the `tibs` command-line tool. | ||
|
||
At a high level, both the library and command-line interfaces for Tibs work similarly: given a project and toolchain (containing executables clang, swiftc, ninja, etc.), produce a Ninja build description file and a compilation database. When built using `ninja`, all of the generated Swift modules are created and the raw index data is produced. | ||
|
||
### ISDBTibs Library | ||
|
||
The `TibsBuilder` class provides a library interface for building Tibs projects. It is responsible for fully resolving all the dependencies, build outputs, and compiler arguments. It also has APIs for executing the build using the `ninja` command-line tool and outputting a compilation database (`compile_commands.json`). | ||
|
||
#### Example | ||
|
||
The inputs to the build process are a toolchain, project, and build directory. | ||
|
||
``` | ||
import ISDBTibs | ||
|
||
let toolchain = TibsToolchain(...) | ||
let projectDir: URL = ... | ||
let buildDir: URL = ... | ||
``` | ||
|
||
We load the `project.json` manifest from the project directory. | ||
|
||
``` | ||
let manifest = try TibsManifest.load(projectRoot: projectDir) | ||
``` | ||
|
||
From a manifest, we can create a `TibsBuilder` and examine the fully resolved targets. | ||
|
||
``` | ||
let builder = try TibsBuilder( | ||
manifest: manifest, | ||
sourceRoot: projectDir, | ||
buildRoot: buildDir, | ||
toolchain: toolchain) | ||
|
||
for target in builder.targets { | ||
... | ||
} | ||
``` | ||
|
||
Finally, we can write the build files and build, or incrementally rebuild the project's index data and modules. | ||
|
||
``` | ||
try builder.writeBuildFiles() | ||
try builder.build() | ||
// edit sources | ||
try builder.build() | ||
``` | ||
|
||
### tibs Command-line Tool | ||
|
||
As a convenience, we also provide a command-line interface `tibs`. It uses the `ISDBTibs` library to write out build files, which can then be built using `ninja`. | ||
|
||
#### Example | ||
|
||
Running `tibs` will write out the necessary Ninja build file and compilation database. | ||
|
||
``` | ||
mkdir build | ||
cd build | ||
tibs <path to project directory, containing project.json> | ||
``` | ||
|
||
To execute the build, use the `ninja` command | ||
|
||
``` | ||
ninja | ||
``` | ||
|
||
The build directory will be populated with the generated swift modules and index data. | ||
|
||
## FAQ | ||
|
||
### Why not use the Swift Package Manager? | ||
|
||
The primary reason not to use the Swift Package Manager (SwiftPM) for our test projects is that SwiftPM's model is stricter than other build systems we need to support, and stricter than we want for our testing support. For example, we want to be able to test mixed language targets (using bridging headers), and to perform only the module-generation and indexing parts of the build without emitting object code. We need to be able to add features to our test support that would break the clean model that SwiftPM provides. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this
INPUTS
orInputs
? There are both in the document.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's
INPUTS
, since I wanted it to be visually obvious it's a special directory. Fixed.