Skip to content

Ensure that integration tests execute the same code as the extension #1131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 11, 2024

Conversation

matthewbastien
Copy link
Member

Build the test code and source into ./dist rather than ./out. This fixes two issues:

  1. The test code used to import different source modules from VS Code which caused issues during integration tests.
  2. The code coverage for integration tests was inconsistent as it was coming from code executed from both the dist and out folders.

The downside of this approach is that building in watch mode can overwrite files when you try to run a test at the same time. Thankfully, this can be easily solved by performing a separate build every time the extension host is launched. There is no noticeable performance impact with this as esbuild is quite fast.

@award999
Copy link
Contributor

award999 commented Oct 9, 2024

Is there no change in coverage, i.e. same numbers? Does breakpoints work in test source, as well as extension code?

@matthewbastien
Copy link
Member Author

matthewbastien commented Oct 9, 2024

Yes, I verified that breakpoints are working still for our launch configurations. As for the code coverage changing: it definitely did, but I'm not sure there's a perfect solution for this.

Before my change:

> FAST_TEST_RUN=1 npm run integration-test -- --coverage

---------------------------------|---------|----------|---------|---------|
File                             | % Stmts | % Branch | % Funcs | % Lines |
---------------------------------|---------|----------|---------|---------|
All files                        |   59.29 |    72.73 |   62.59 |   59.29 |

After my change:

> FAST_TEST_RUN=1 npm run integration-test -- --coverage

---------------------------------|---------|----------|---------|---------|
File                             | % Stmts | % Branch | % Funcs | % Lines |
---------------------------------|---------|----------|---------|---------|
All files                        |   74.33 |    78.36 |   74.68 |   74.33 |

Neither of these numbers are going to be particularly accurate as the extension host is executing code that we aren't actually performing any assertions on. It is, however, a more accurate representation of what code is actually being executed during the test run as its all coming from the same source now. Before, unexecuted code from out/src/** and dist/** were being combined to decrease the overall coverage score.

It's impossible to get accurate information here, unfortunately. We cannot differentiate between the two even with the old method as sometimes we execute code by retrieving the extension API which would execute code from ./dist/** and other times we import modules directly from ./out/src/**. I think this unification of our code coverage output just makes it easier to consume.

@matthewbastien matthewbastien merged commit d106e4c into swiftlang:main Oct 11, 2024
8 checks passed
@matthewbastien matthewbastien deleted the fix-code-coverage branch October 11, 2024 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants