Add smoke tests to GH Action (#9964)

* Update branch name (for testing)

* TypeScript tests

* python unit tests

* ipython unit tests

* single, multi, debugger tests

* functional tests

* Comment out VSIX build+upload when testing the action

* Cache out/ directory

* rename step id to more specific pythonFiles

* single workspace tests

* multi workspace tests

* debugger tests

* Add todo for when the PR is ready for review

* Fix out/ cache key

* Fix pythonFiles cache key

* Retry single workspace tests

* Use same cache key everywhere

* Try out multi workspace tests

* Try out debugger tests

* Uncomment compile + upload step

* run venv tests

* Update branch name to trigger workflow

* Adding comments + enable full test suite

* Finalize versions of python we want to use

* Try 3.7 instead of 3.8

* Run the workflow only on upstream master

* Clean up comments

* Revert to 3.8

* Update .github/workflows/ci.yml

Co-Authored-By: Brett Cannon <[email protected]>

* Apply suggestions from code review

Co-Authored-By: Brett Cannon <[email protected]>

* Rename test suite matrix key

* Try bang operator

* Make python cache keys more specific

* Move repo condition up

* Run a subset of tests to see if stuff still works

* Update the branch to test the workflow

* Try leaving no space between ! and the expression

* Undo ! testing

* Update pythonFiles cache key

* Smoke tests

* Don't install vsce globally

* Where my env var value at

* well this one didn't work

* Use correct env var syntax

* Remove curly brackets

* Use xvfb for smoke tests

* Take prep work outside of xvfb step

* Uncomment things

* Comment things out for testing

* Merge master, fix things

* forgot to comment condition

* Uncomment testing checks

* Newlines

* Whitespaces #2

* Run tests on windows

* Remember to package the Python files

* Update build number earlier

* Use existing VSIX artifact instead of repackaging

* Skip repo condition for a test run

* List directory content (debug)

* Update ci.yml

* Update ci.yml

* Undo download artifact changes to compare

* Re-add brackets

* Force update build nb on bash shell

* Do not call gulp:cleanExceptTests 😒

* Download artifact in place

* Only run npx tsc -p ./

* Comment windows for now, re-run all prep (comparison)

* Correct path for artifact download

* Extract vsix from download location

* Remove path when downloading

* Cleanup now that the smoke tests run

* Remove repo conditions (ready to be merged)

* Newline

* Don't need the python files in smoke tests

* Ready for re-review

* npm ci --prefer-offline

Co-authored-by: Brett Cannon <[email protected]>

Author: kimadeline

.github/workflows/ci.yml

@@ -161,6 +161,12 @@ jobs:
       - name: Clean directory
         run: npm run clean
 
+      # Use the GITHUB_RUN_ID environment variable to update the build number.
+      # GITHUB_RUN_ID is a unique number for each run within a repository.
+      # This number does not change if you re-run the workflow run.
+      - name: Update build number
+        run: npm run updateBuildNumber -- --buildNumber $GITHUB_RUN_ID
+
       - name: Package the VSIX
         run: npm run package
 
@@ -180,7 +186,7 @@ jobs:
       matrix:
         # We're not running CI on macOS for now because it's one less matrix entry to lower the number of runners used,
         # macOS runners are expensive, and we assume that Ubuntu is enough to cover the UNIX case.
-        os: [ubuntu-latest]
+        os: [ubuntu-latest, windows-latest]
         # Run the tests on the oldest and most recent versions of Python.
         python: [2.7, 3.8]
         test-suite: [ts-unit, python-unit, venv, single-workspace, multi-workspace, debugger, functional]
@@ -346,3 +352,49 @@ jobs:
       - name: Run functional tests
         run: npm run test:functional
         if: matrix.test-suite == 'functional'
+
+  smoke-tests:
+    name: Smoke tests
+    # The value of runs-on is the OS of the current job (specified in the strategy matrix below) instead of being hardcoded.
+    runs-on: ${{ matrix.os }}
+    needs: [build-vsix]
+    if: github.repository == 'microsoft/vscode-python'
+    strategy:
+      matrix:
+        # We're not running CI on macOS for now because it's one less matrix entry to lower the number of runners used,
+        # macOS runners are expensive, and we assume that Ubuntu is enough to cover the UNIX case.
+        os: [ubuntu-latest, windows-latest]
+        python: [3.8]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Retrieve cached npm files
+        uses: actions/cache@v1
+        with:
+          path: ~/.npm
+          key: ${{runner.os}}-${{env.CACHE_NPM_DEPS}}-${{hashFiles('package-lock.json')}}
+
+      - name: Install dependencies (npm ci)
+        run: npm ci --prefer-offline
+
+      - name: Download VSIX
+        uses: actions/download-artifact@v1
+        with:
+          name: ${{env.ARTIFACT_NAME_VSIX}}
+
+      # Extract the artifact from its download folder (./${{env.ARTIFACT_NAME_VSIX}}) to the repo root,
+      # then delete the download folder and compile the source code.
+      - name: Prepare for smoke tests
+        run: |
+           mv ${{env.ARTIFACT_NAME_VSIX}}/* .
+           rm -r ${{env.ARTIFACT_NAME_VSIX}}
+           npx tsc -p ./
+        shell: bash
+
+      - name: Run smoke tests
+        env:
+          DISPLAY: 10
+        uses: GabrielBB/[email protected]
+        with:
+          run: node ./out/test/smokeTest.js