Skip to content

Split functional tests into per file so they don't run out of memory during failures #13534

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Aug 20, 2020

Conversation

rchiodo
Copy link

@rchiodo rchiodo commented Aug 19, 2020

This gets the windows tests to run again. However there's some differences (see last run I did)
https://dev.azure.com/ms/vscode-python/_build/results?buildId=103115&view=results

  1. Failures are split up by more than just OS now. They'll be split up by file too.
  2. It takes longer. About 40 minutes longer on Windows.

I believe I also fixed the root cause of the memory problem, opening files without a workspace means we can't use relative paths in the contents manager (and crashes in it will cause an infinite loop of starting a server).

@rchiodo rchiodo self-assigned this Aug 19, 2020
@rchiodo rchiodo added the no-changelog No news entry required label Aug 19, 2020
@@ -2,25 +2,36 @@
{
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THis file didn't change as much as it looks. I just formatted it.

@@ -3449,6 +3449,7 @@
"test:functional:perf": "node --inspect-brk ./node_modules/mocha/bin/_mocha --require source-map-support/register --config ./build/.mocha.functional.perf.json",
"test:functional:memleak": "node --inspect-brk ./node_modules/mocha/bin/_mocha --require source-map-support/register --config ./build/.mocha.functional.json",
"test:functional:cover": "npm run test:functional",
"test:functional:split": "node ./build/ci/scripts/runFunctionalTests.js",
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THis was necessary to get node to have the correct environment when starting. The test-phases.yaml will run this command using npm run test:functional:split

@codecov-commenter
Copy link

codecov-commenter commented Aug 20, 2020

Codecov Report

Merging #13534 into master will decrease coverage by 0.10%.
The diff coverage is 45.45%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #13534      +/-   ##
==========================================
- Coverage   59.82%   59.71%   -0.11%     
==========================================
  Files         670      670              
  Lines       37335    37425      +90     
  Branches     5344     5371      +27     
==========================================
+ Hits        22334    22348      +14     
- Misses      13847    13911      +64     
- Partials     1154     1166      +12     
Impacted Files Coverage Δ
src/client/datascience/jupyter/jupyterServer.ts 9.90% <0.00%> (-0.19%) ⬇️
...datascience/jupyter/liveshare/hostJupyterServer.ts 11.81% <0.00%> (-0.19%) ⬇️
src/client/datascience/jupyter/jupyterSession.ts 72.34% <58.82%> (-4.77%) ⬇️
src/client/common/utils/platform.ts 64.70% <0.00%> (-11.77%) ⬇️
src/client/datascience/crossProcessLock.ts 79.41% <0.00%> (-11.77%) ⬇️
src/client/datascience/jupyter/kernels/helpers.ts 35.51% <0.00%> (-10.27%) ⬇️
...nt/datascience/notebookStorage/vscNotebookModel.ts 61.11% <0.00%> (-4.80%) ⬇️
src/client/common/utils/misc.ts 73.33% <0.00%> (-2.23%) ⬇️
src/client/linters/pydocstyle.ts 86.66% <0.00%> (-2.23%) ⬇️
src/client/datascience/debugLocationTracker.ts 76.56% <0.00%> (-1.57%) ⬇️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49e3ac7...a6189ac. Read the comment docs.

@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@@ -263,6 +263,10 @@ export class JupyterServerBase implements INotebookServer {
throw new Error('You forgot to override createNotebookInstance');
}

protected get isDisposed(): boolean {
throw new Error('You forgot to override isDisposed');

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not return isDisposed?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class can't be abstract because the mixin stuff I use further up won't allow it. So this function is supposed to be overridden.

@rchiodo rchiodo merged commit b07f9a5 into master Aug 20, 2020
@rchiodo rchiodo deleted the rchiodo/split_functional branch August 20, 2020 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no-changelog No news entry required
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants