v3 CLI compiling E2E test suite

.changeset/beige-pears-explode.md

@@ -0,0 +1,5 @@
+---
+"trigger.dev": patch
+---
+
+Add an e2e suite to test compiling with v3 CLI.

.github/workflows/e2e.yml

@@ -1,9 +1,53 @@
-name: "🧪 E2E Tests"
+name: "E2E"
 on:
   workflow_call:
+    inputs:
+      package:
+        description: The identifier of the job to run
+        default: webapp
+        required: false
+        type: string
 jobs:
-  e2e:
-    name: "🧪 E2E Tests"
+  cli-v3:
+    name: "🧪 CLI v3 tests"
+    if: inputs.package == 'cli-v3' || inputs.package == ''
+    runs-on: buildjet-8vcpu-ubuntu-2204
+    strategy:
+      fail-fast: false
+      matrix:
+        package-manager: ["npm", "pnpm", "yarn"]
+    steps:
+      - name: ⬇️ Checkout repo
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+
+      - name: ⎔ Setup pnpm
+        uses: pnpm/[email protected]
+        with:
+          version: 8.15.5
+
+      - name: ⎔ Setup node
+        uses: buildjet/setup-node@v3
+        with:
+          node-version: 20.11.1
+          cache: "pnpm"
+
+      - name: 📥 Download deps
+        run: pnpm install --frozen-lockfile --filter trigger.dev...
+
+      - name: 🔧 Build v3 cli monorepo dependencies
+        run: pnpm run build --filter trigger.dev^...
+
+      - name: 🔧 Build worker template files
+        run: pnpm --filter trigger.dev run build:workers
+
+      - name: Run E2E Tests
+        run: |
+          PM=${{ matrix.package-manager }} pnpm --filter trigger.dev run test:e2e
+  webapp:
+    name: "🧪 Webapp tests"
+    if: inputs.package == 'webapp' || inputs.package == ''
     runs-on: buildjet-16vcpu-ubuntu-2204
     steps:
       - name: 🐳 Login to Docker Hub

.github/workflows/pr_checks.yml

@@ -29,4 +29,6 @@ jobs:
 
   # e2e:
   #   uses: ./.github/workflows/e2e.yml
+  #   with:
+  #     package: webapp
   #   secrets: inherit

.github/workflows/publish.yml

@@ -49,9 +49,11 @@ jobs:
     uses: ./.github/workflows/unit-tests.yml
     secrets: inherit
 
-  # e2e:
-  #   uses: ./.github/workflows/e2e.yml
-  #   secrets: inherit
+  e2e:
+    uses: ./.github/workflows/e2e.yml
+    with:
+      package: cli-v3
+    secrets: inherit
 
   publish:
     needs: [typecheck, units]

packages/cli-v3/e2e/README.md

@@ -0,0 +1,177 @@
+# Trigger.dev CLI E2E suite
+
+E2E test suite for the Trigger.dev v3 CLI.
+
+Note: this only works with Trigger.dev v3 projects and later. There is no E2E test suite for the [@trigger.dev/cli](https://www.npmjs.com/package/@trigger.dev/cli) package yet.
+
+Trigger.dev is an open source platform that makes it easy to create event-driven background tasks directly in your existing project.
+
+## Description
+
+This suite aims to test the outputs fo the `triggerdev deploy` command.
+To do so, it runs the deploy code against fixture projects that are located under `packages/cli-v3/e2e/fixtures/`.
+Those fixtures reproduce minimal project structure and contents, in order to reproduce known bugs and run fast.
+
+**Notes**
+- The suite uses vitest
+- Everything happens locally
+- There is no login required
+- There is not real project reference needed
+- No docker image is created or built, instead, the bundled worker file is started with node directly inside the vitest process
+
+## Usage
+
+If you have not done it yet, build the CLI:
+
+```sh
+pnpm run build --filter trigger.dev
+```
+
+Then, run the v3 CLI E2E test suite:
+
+```sh
+pnpm --filter trigger.dev run test:e2e
+```
+
+| Option                 | Description                                                                  |
+| ---------------------- | ---------------------------------------------------------------------------- |
+| `MOD=<fixture-name>`   | The name of any folder directly nested under `packages/cli-v3/e2e/fixtures/` |
+| `PM=<package-manager>` | The package manager to use. One of `npm`, `pnpm`, `yarn`. Defaults to `npm`  |
+
+Example:
+
+```sh
+MOD=server-only PM=yarn pnpm --filter trigger.dev run test:e2e
+```
+
+This will run the test suite for the `server-only` fixture using `yarn` to install and resolve dependencies.
+
+## Debugging
+
+When debugging an issue with the `triggerdev deploy` or `triggerdev dev` command, it is recommended to reproduce it with a minimal project fixture in the e2e suite.
+Check [Adding a fixture](#adding-a-fixture) for more information.
+
+Then run:
+
+```sh
+MOD=<fixture-name> pnpm run test:e2e
+```
+
+This will test your fixture project, and generate outputs in the `packages/cli-v3/e2e/fixtures/<fixture-name>/.trigger` folder, so you can easily debug.
+
+## Adding a fixture
+
+1. Create a new `packages/cli-v3/e2e/fixtures/<fixture-name>` folder.
+
+    It will hold the project to test.
+
+2. Add a `package.json` file in your `packages/cli-v3/e2e/fixtures/<fixture-name>` folder.
+
+    Use the following template:
+
+    ```json package.json
+    {
+      "name": "<fixture-name>",
+      "private": true,
+      "engines": {
+        "pnpm": "8.15.5",
+        "yarn": "4.2.2"
+      },
+      "packageManager": "[email protected]"
+    }
+    ```
+
+    > The `engines` field is used to store the versions of pnpm and yarn to use when running the suite.
+
+3. Add an empty `pnpm-workspace.yaml` in your `packages/cli-v3/e2e/fixtures/<fixture-name>` folder.
+
+    This is necessary to prevent the Trigger.dev monorepo from handling this project.
+    Please check https://github.com/pnpm/pnpm/issues/2412 for more inforation.
+
+4. Add an empty `yarn.lock` in your fixture folder.
+
+    This is necessary to allow to use `yarn` without having a warning on the current project being a `pnpm` project.
+
+5. Install the fixture dependencies and generate lockfiles.
+
+    Like you would in any project.
+    E.g. if your fixture contains a trigger task that uses the `jsdom` library:
+
+    ```sh
+    cd packages/cli-v3/e2e/fixtures/<fixture-name>
+    corepack use [email protected]
+    pnpm install jsdom
+    ```
+
+    > This will update the `package.json` and generate the `pnpm-lock.yaml` file.
+
+6. To run the test suite against multiple package manager, we need to generate the other lockfiles.
+
+    ```sh
+    cd packages/cli-v3/e2e/fixtures/<fixture-name>
+    rm -rf node_modules
+    npm install
+    rm -rf node_modules
+    corepack use yarn # will update the yarn lockfile
+    ```
+
+    > Do it in this order, otherwise `npm install` will update the existing `yarn.lock` file with legacy version 1.
+
+7. Create a new `packages/cli-v3/e2e/fixtures/trigger` folder, and create a trigger task in it.
+
+    Here is an example:
+
+    ```javascript
+    import { task } from "@trigger.dev/sdk/v3";
+
+    export const helloWorldTask = task({
+      id: "hello-world",
+      run: async (payload) => {
+        console.log("Hello, World!", payload);
+      },
+    });
+    ```
+
+8. Add a trigger configuration file.
+
+    The configuration file is mandatory here, the E2E suite does not execute `trigger.dev` commands.
+
+    ```javascript
+    export const config = {
+      project: "<fixture-name>",
+      triggerDirectories: ["./trigger"],
+    };
+    ```
+
+    > The project reference can be anything here, as the suite runs locally without connecting to the platform.
+
+9.  Commit your changes.
+
+10. Add your fixture test configuration in `testCases.json`.
+
+    ```json testCases.json
+    [
+      ...
+      {
+        "name": "<fixture-name>",
+      },
+      ...
+    ]
+    ```
+
+    You can configure your test case by adding other properties to the JSON object. Here is the `TestCase` type for reference:
+
+    ```typescript
+    type TestCase = {
+      name: string;
+      skipTypecheck?: boolean;
+      wantConfigNotFoundError?: boolean;
+      wantBadConfigError?: boolean;
+      wantCompilationError?: boolean;
+      wantWorkerError?: boolean;
+      wantDependenciesError?: boolean;
+      wantInstallationError?: boolean;
+    };
+    ```
+
+    > You might expect a specific error at a specific test, so use those configuration option at your discretion.