Skip to content

[docs] Add documentation about writing tests for IndexStoreDB #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 6, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 124 additions & 0 deletions Documentation/Development.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# Development

This document contains notes about development and testing of IndexStoreDB.

## Table of Contents

* [Writing Tests](#writing-tests)
* [Tibs, the "Test Index Build System"](Tibs.md)

## Writing Tests

As much as is practical, all code should be covered by tests. New tests can be added under the `Tests` directory and should use `XCTest`. The rest of this section will describe the additional tools available in the `ISDBTestSupport` module to make it easier to write good and efficient tests.

Most indexer tests follow a pattern:

1. Build and index a test project
2. Perform index queries
3. Compare the results against known locations in the test code
4. (Optional) modify the code and repeat.

### Test Projects (Fixtures)

Index test projects should be put in the `Tests/INPUTS` directory, and use the [Tibs](Tibs.md) build system to define their sources and targets. An example test project might look like:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this INPUTS or Inputs? There are both in the document.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's INPUTS, since I wanted it to be visually obvious it's a special directory. Fixed.


```
Tests/
INPUTS/
MyTestProj/
a.swift
b.swift
c.cpp
```

Where `project.json` describes the project's targets, for example

```
{ "sources": ["a.swift", "b.swift", "c.cpp"] }
```

Tibs supports more advanced project configurations, such as multiple swift modules with dependencies, etc. For much more information about Tibs, including what features it supports and how it works, see [Tibs.md](Tibs.md).

### TibsTestWorkspace

The `TibsTestWorkspace` pulls together the various pieces needed for working with tests, including building the project, creating an IndexStoreDB instance, and keeping it up to date after modifying sources.

To create a `TibsTestWorkspace`, use the `staticTibsTestWorkspace` and `mutableTibsTestWorkspace` methods. Tests that do not mutate sources should use `staticTibsTestWorkspace`.

```swift
func testFoo() {

// Loads an immutable workspace for the test project "MyTestProj" and resolves
// all of its targets, ready to be indexed.
guard let ws = try staticTibsTestWorkspace(name: "MyTestProj") else { return }

// Perform the build and read the produced index data.
try ws.buildAndIndex()

// See the results
ws.index.occurrences(ofUSR: "<some usr>", ...)

...
}
```

#### Source Locations

It is common to want to refer to specific locations in the source code of a test project. This is supported using inline comment syntax.

```swift
Test.swift:
func /*myFuncDef*/myFunc() {
/*myFuncCall*/myFunc()
}
```

In a test, these locations can be referenced by name. The named location is immediately after the comment.

```swift

let loc = ws.testLoc("myFuncDef")
// TestLocation(url: ..., line: 1, column: 19)
```

A common use of a `TestLocation` is to form a `SymbolOccurrence` at that location.

```swift
let occurrence = Symbol(...).at(ws.testLoc("myFuncDef"), roles: .definition)
```

#### Comparing SymbolOccurrences

Rather than attempting to compare `SymbolOccurrences` for equality, prefer using the custom XCTest assertion method `checkOccurrences`. This API avoids accidentally comparing details that are undefined (e.g. the order of occurrences), or likely to change frequently (e.g. `SymbolRole`s are checked to be a superset instead of an exact match, as we often add new roles).

```
checkOccurrences(ws.index.occurrences(ofUSR: sym.usr, roles: .all), expected: [
sym.at(ws.testLoc("c"), roles: .definition),
sym.at(ws.testLoc("c_call"), roles: .call),
])
```

#### Mutating Test Sources

In order to test changes to the index, we create a mutable test workspace. The mutable project takes care of copying the sources to a new location. Afterwards, we can mutate the sources using the `TibsTestWorkspace.edit(rebuild: Bool, block: ...)` method.

```
func testMutating() {
guard let ws = try mutableTibsTestWorkspace(name: "MyTestProj") else { return }
try ws.buildAndIndex()

// Check initial state...

// Perform modifications to "url"!
ws.edit(rebuild: true) { editor, files in

// Read the original contents and add a new line.
let new = try files.get(url).appending("func /*moreCodeDef*/moreCode() {}")

// Provide the new contents for "url".
editor.write(new, to: url)
}
}
```

After we return from `edit()`, the sources are modified and any changes to stored source locations are reflected. We can now `buildAndIndex()` to update the index, or as a convenience we can pass `rebuild: true` to `edit`.
128 changes: 128 additions & 0 deletions Documentation/Tibs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# Tibs

Tibs ("Test Index Build System") is a simple and flexible build system designed for test projects of IndexStoreDB and SourceKit-LSP. Tibs can incrementally build or rebuild the index data and generated module files for a Swift and/or C-family language test project. Tibs can also dump the compiler arguments to a clang-compatible JSON compilation database. It is *not designed to compile to executable code*.

Tibs is implemented using [Ninja](https://ninja-build.org), which introduces a new dependency in IndexStoreDB when running tests.

Tibs projects are described by a `project.json` file containing one or more targets. Typically, a test case will use a project fixture located in the `Tests/INPUTS` directory.

## Project

A Tibs project is described by `project.json` manifest file. The top-level entity is a target, which may depend on other targets.

Example:

```
{
"targets": [
{
"name": "mytarget",
"swift_flags": ["-warnings-as-errors"],
"sources": ["a.swift", "b.swift"],
"dependencies": ["dep1"],
},
{
"name": "dep1",
"sources": ["dep1.swift"],
},
]
}
```

As a convenience, if the project consists of a single target, it can be written at the top level and the name can be omitted, e.g.

```
{ "sources": ["main.swift"] }
```

### Targets

Targets have the following fields:

* "name": The name of the target. If omitted, it will be "main".
* "sources": Array of source file paths.
* "bridging_header": Optional path to a Swift bridging header.
* "dependencies": Optional array of target names that this target depends on. This is useful for Swift module dependencies.
* "swift_flags": Optional array of extra compiler arguments for Swift.
* "clang_flags": Optional array of extra compiler arguments for Clang.

The directory containing the `project.json` manifest file is considered the project's root directory and any relative paths are relative to that.

## Building Tibs Projects

Most tests using Tibs will use the APIs provided by the `TibsTestWorkspace` class in the `ISDBTestSupport` module, which provides a high-level API for working with a complete Tibs project. The following information is for anyone needing to work using the lower-level APIs from `ISDBTibs` or the `tibs` command-line tool.

At a high level, both the library and command-line interfaces for Tibs work similarly: given a project and toolchain (containing executables clang, swiftc, ninja, etc.), produce a Ninja build description file and a compilation database. When built using `ninja`, all of the generated Swift modules are created and the raw index data is produced.

### ISDBTibs Library

The `TibsBuilder` class provides a library interface for building Tibs projects. It is responsible for fully resolving all the dependencies, build outputs, and compiler arguments. It also has APIs for executing the build using the `ninja` command-line tool and outputting a compilation database (`compile_commands.json`).

#### Example

The inputs to the build process are a toolchain, project, and build directory.

```
import ISDBTibs

let toolchain = TibsToolchain(...)
let projectDir: URL = ...
let buildDir: URL = ...
```

We load the `project.json` manifest from the project directory.

```
let manifest = try TibsManifest.load(projectRoot: projectDir)
```

From a manifest, we can create a `TibsBuilder` and examine the fully resolved targets.

```
let builder = try TibsBuilder(
manifest: manifest,
sourceRoot: projectDir,
buildRoot: buildDir,
toolchain: toolchain)

for target in builder.targets {
...
}
```

Finally, we can write the build files and build, or incrementally rebuild the project's index data and modules.

```
try builder.writeBuildFiles()
try builder.build()
// edit sources
try builder.build()
```

### tibs Command-line Tool

As a convenience, we also provide a command-line interface `tibs`. It uses the `ISDBTibs` library to write out build files, which can then be built using `ninja`.

#### Example

Running `tibs` will write out the necessary Ninja build file and compilation database.

```
mkdir build
cd build
tibs <path to project directory, containing project.json>
```

To execute the build, use the `ninja` command

```
ninja
```

The build directory will be populated with the generated swift modules and index data.

## FAQ

### Why not use the Swift Package Manager?

The primary reason not to use the Swift Package Manager (SwiftPM) for our test projects is that SwiftPM's model is stricter than other build systems we need to support, and stricter than we want for our testing support. For example, we want to be able to test mixed language targets (using bridging headers), and to perform only the module-generation and indexing parts of the build without emitting object code. We need to be able to add features to our test support that would break the clean model that SwiftPM provides.
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,7 @@ The C++ code in the index requires `libdispatch`, but unlike Swift code, it cann
```sh
$ swift build -Xcxx -I<path_to_swift_toolchain>/usr/lib/swift -Xcxx -I<path_to_swift_toolchain>/usr/lib/swift/Block
```

## Development

For more information about developing IndexStoreDB, see [Development](Documentation/Development.md).