Run Blazor E2E tests on SauceLabs #18456

ajaybhargavb · 2020-01-20T07:10:56Z

Fixes #17013

This PR sets up a remote web driver that connects to SauceLabs using a proxy.

Added a script that will be launched as a separate process that launches a persistent proxy connection with SauceLabs
Added necessary options to E2ETestOptions to accomodate SauceLabs configs
Updated server test fixtures to use a custom hostname as the server url instead of directly pointing to 127.0.0.1

ajaybhargavb

I've verified this works manually.

Things to do:

Provide a config to customize which browser/platform to run on
Add a way to report test pass/fail status to SauceLabs (right now it shows unknown)
Setup an azure pipeline and add a .yml file for the blazor e2e tests

In the meantime, @javiercn @SteveSandersonMS can you take a look at this PR?

src/Components/test/E2ETest/e2eTestSettings.json

src/Shared/E2ETesting/BrowserFixture.cs

ajaybhargavb · 2020-01-20T07:13:05Z

src/Shared/E2ETesting/SauceConnectServer.cs

+
+namespace Microsoft.AspNetCore.E2ETesting
+{
+    public class SauceConnectServer : IDisposable


Most of this file is copy-paste from SeleniumStandaloneServer

Is there room to consolidate the two types? It might be difficult to remember to update both types if we change the code in the other.

javiercn · 2020-01-21T14:16:18Z

src/Components/test/E2ETest/scripts/sauce.ts

@@ -0,0 +1,82 @@
+import { EOL } from "os";


Why is this a typescript file? There's nothing typescript specific here and we shouldn't need typescript for something as simple as this.

Can we change this to be a plain JavaScript file? Any recent version of node will handle modern javascript.

I took the easy path as we do similar things in SignalR which is in typescript. IMO there is no downside to using typescript for this. It is nicer :)

javiercn

Looks like a promising start!

A couple of things to take into account.

We likely want to run the tests on saucelabs in addition to running them locally on the CI. Does this change make that possible? It seems to me that it replaces one server with the other.

If we wanted to run them in both places we would have to have some extra step on the CI to do that, which I'm fine with, just want you to think about it.

Other than that, this is definitely going on the right direction. Great job so far!

ajaybhargavb · 2020-01-21T18:51:56Z

We likely want to run the tests on saucelabs in addition to running them locally on the CI. Does this change make that possible? It seems to me that it replaces one server with the other.

Yes, that is the goal. I know this is currently hardcoded to true but this will actually come from msbuild.

If we wanted to run them in both places we would have to have some extra step on the CI to do that, which I'm fine with, just want you to think about it.

Yes, I am adding a .yml file similar to https://github.com/dotnet/aspnetcore/blob/master/.azure/pipelines/signalr-daily-tests.yml. This will set all the necessary env vars and run build with testing on SauceLabs enabled.

ajaybhargavb · 2020-01-22T09:01:31Z

Updated.

.azure/pipelines/blazor-daily-tests.yml

pranavkm · 2020-01-23T02:01:15Z

.azure/pipelines/blazor-daily-tests.yml

+    jobDisplayName: "Blazor Daily Tests"
+    afterBuild:
+    # macOS/Chrome
+    - script: 'dotnet test --filter "StandaloneAppTest"'


I'm happy to start with this, but is there a limitation that requires this?

.azure/pipelines/blazor-daily-tests.yml

pranavkm · 2020-01-23T02:11:54Z

src/Components/Blazor/Server/src/MonoDebugProxy/ws-proxy/MonoProxy.cs

@@ -461,7 +461,7 @@ async Task GetDetails(int msg_id, int object_id, CancellationToken token, string
 				{
 					result = var_list
 				});
-			} catch (Exception e) {
+			} catch (Exception) {


no beuno. We should fix this in Mono and flow it through. But given how small these changes are, I think we can live with this for now.

src/Shared/E2ETesting/BrowserFixture.cs

src/Shared/E2ETesting/E2ETestOptions.cs

pranavkm · 2020-01-23T02:18:17Z

src/Shared/E2ETesting/SauceConnectServer.cs

+
+namespace Microsoft.AspNetCore.E2ETesting
+{
+    public class SauceConnectServer : IDisposable


Is there room to consolidate the two types? It might be difficult to remember to update both types if we change the code in the other.

ajaybhargavb · 2020-01-28T01:54:02Z

Merging this. Real device testing on iPhones is currently blocked on a saucelabs subscription issue. I'll send a separate PR to enable that.

ajaybhargavb commented Jan 20, 2020

View reviewed changes

ajaybhargavb requested review from javiercn and SteveSandersonMS January 20, 2020 07:16

javiercn added the area-blazor Includes: Blazor, Razor Components label Jan 21, 2020

javiercn reviewed Jan 21, 2020

View reviewed changes

Run Blazor E2E tests on SauceLabs

b42a399

ajaybhargavb force-pushed the ajbaaska/multi-device-testing branch 2 times, most recently from 3313670 to dc8dd35 Compare January 22, 2020 03:30

ajaybhargavb marked this pull request as ready for review January 22, 2020 03:32

Added azure pipeline

745d69e

ajaybhargavb force-pushed the ajbaaska/multi-device-testing branch 2 times, most recently from e0cce92 to ef5129b Compare January 22, 2020 08:59

ajaybhargavb commented Jan 22, 2020

View reviewed changes

.azure/pipelines/blazor-daily-tests.yml Outdated Show resolved Hide resolved

ajaybhargavb requested a review from pranavkm January 22, 2020 09:08

ajaybhargavb force-pushed the ajbaaska/multi-device-testing branch 2 times, most recently from c92f4e5 to e3bbb7e Compare January 23, 2020 00:15

pranavkm approved these changes Jan 23, 2020

View reviewed changes

update yml

02c3ddb

ajaybhargavb force-pushed the ajbaaska/multi-device-testing branch 3 times, most recently from 6b9d5ff to bbdccab Compare January 23, 2020 21:35

Update meta

df74243

ajaybhargavb force-pushed the ajbaaska/multi-device-testing branch from bbdccab to fae2886 Compare January 27, 2020 23:50

More changes

9dbbddc

ajaybhargavb force-pushed the ajbaaska/multi-device-testing branch from fae2886 to 9dbbddc Compare January 28, 2020 01:18

ajaybhargavb merged commit 24be299 into blazor-wasm Jan 28, 2020

ajaybhargavb deleted the ajbaaska/multi-device-testing branch January 28, 2020 01:54

Run Blazor E2E tests on SauceLabs #18456

Run Blazor E2E tests on SauceLabs #18456

Uh oh!

Conversation

ajaybhargavb commented Jan 20, 2020

Uh oh!

ajaybhargavb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ajaybhargavb Jan 20, 2020

Choose a reason for hiding this comment

Uh oh!

pranavkm Jan 23, 2020

Choose a reason for hiding this comment

Uh oh!

javiercn Jan 21, 2020

Choose a reason for hiding this comment

Uh oh!

ajaybhargavb Jan 21, 2020

Choose a reason for hiding this comment

Uh oh!

javiercn left a comment

Choose a reason for hiding this comment

Uh oh!

ajaybhargavb commented Jan 21, 2020

Uh oh!

ajaybhargavb commented Jan 22, 2020

Uh oh!

Uh oh!

Uh oh!

pranavkm Jan 23, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pranavkm Jan 23, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pranavkm Jan 23, 2020

Choose a reason for hiding this comment

Uh oh!

ajaybhargavb commented Jan 28, 2020

Uh oh!

Uh oh!

ajaybhargavb left a comment •

edited

Loading