Skip to content

test: Make some replay tests less flaky #10418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 31, 2024
Merged

test: Make some replay tests less flaky #10418

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
067b1e1
test: Make some replay tests less flaky
mydea Jan 30, 2024
b6757a4
use large runner for flaky test detection
mydea Jan 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,7 @@ jobs:
name: Browser Unit Tests
needs: [job_get_metadata, job_build]
timeout-minutes: 10
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- name: Check out current commit (${{ needs.job_get_metadata.outputs.commit_label }})
uses: actions/checkout@v4
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/flaky-test-detector.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ concurrency:

jobs:
flaky-detector:
runs-on: ubuntu-20.04
runs-on: ubuntu-20.04-large-js
timeout-minutes: 60
name: 'Check tests for flakiness'
# Also skip if PR is from master -> develop
Expand Down
49 changes: 27 additions & 22 deletions ...integration-tests/suites/replay/extendNetworkBreadcrumbs/fetch/captureRequestBody/test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ import { expect } from '@playwright/test';
import { sentryTest } from '../../../../../utils/fixtures';
import { envelopeRequestParser, waitForErrorRequest } from '../../../../../utils/helpers';
import {
getCustomRecordingEvents,
collectReplayRequests,
getReplayPerformanceSpans,
shouldSkipReplayTest,
waitForReplayRequest,
} from '../../../../../utils/replayHelpers';

sentryTest('captures text request body', async ({ getLocalTestPath, page, browserName }) => {
Expand All @@ -30,7 +30,9 @@ sentryTest('captures text request body', async ({ getLocalTestPath, page, browse
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -65,9 +67,8 @@ sentryTest('captures text request body', async ({ getLocalTestPath, page, browse
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down Expand Up @@ -109,7 +110,9 @@ sentryTest('captures JSON request body', async ({ getLocalTestPath, page, browse
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -144,9 +147,8 @@ sentryTest('captures JSON request body', async ({ getLocalTestPath, page, browse
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down Expand Up @@ -188,7 +190,9 @@ sentryTest('captures non-text request body', async ({ getLocalTestPath, page, br
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -227,9 +231,8 @@ sentryTest('captures non-text request body', async ({ getLocalTestPath, page, br
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down Expand Up @@ -271,7 +274,9 @@ sentryTest('captures text request body when matching relative URL', async ({ get
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestUrl({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -306,9 +311,8 @@ sentryTest('captures text request body when matching relative URL', async ({ get
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down Expand Up @@ -348,7 +352,9 @@ sentryTest('does not capture request body when URL does not match', async ({ get
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -383,9 +389,8 @@ sentryTest('does not capture request body when URL does not match', async ({ get
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down
49 changes: 27 additions & 22 deletions ...egration-tests/suites/replay/extendNetworkBreadcrumbs/fetch/captureRequestHeaders/test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ import { expect } from '@playwright/test';
import { sentryTest } from '../../../../../utils/fixtures';
import { envelopeRequestParser, waitForErrorRequest } from '../../../../../utils/helpers';
import {
getCustomRecordingEvents,
collectReplayRequests,
getReplayPerformanceSpans,
shouldSkipReplayTest,
waitForReplayRequest,
} from '../../../../../utils/replayHelpers';

sentryTest('handles empty/missing request headers', async ({ getLocalTestPath, page, browserName }) => {
Expand All @@ -28,7 +28,9 @@ sentryTest('handles empty/missing request headers', async ({ getLocalTestPath, p
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -61,9 +63,8 @@ sentryTest('handles empty/missing request headers', async ({ getLocalTestPath, p
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down Expand Up @@ -100,7 +101,9 @@ sentryTest('captures request headers as POJO', async ({ getLocalTestPath, page,
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -140,9 +143,8 @@ sentryTest('captures request headers as POJO', async ({ getLocalTestPath, page,
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down Expand Up @@ -184,7 +186,9 @@ sentryTest('captures request headers on Request', async ({ getLocalTestPath, pag
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -224,9 +228,8 @@ sentryTest('captures request headers on Request', async ({ getLocalTestPath, pag
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down Expand Up @@ -267,7 +270,9 @@ sentryTest('captures request headers as Headers instance', async ({ getLocalTest
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });

Expand Down Expand Up @@ -308,9 +313,8 @@ sentryTest('captures request headers as Headers instance', async ({ getLocalTest
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down Expand Up @@ -351,7 +355,9 @@ sentryTest('does not captures request headers if URL does not match', async ({ g
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -391,9 +397,8 @@ sentryTest('does not captures request headers if URL does not match', async ({ g
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down
22 changes: 12 additions & 10 deletions ...integration-tests/suites/replay/extendNetworkBreadcrumbs/fetch/captureRequestSize/test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ import { expect } from '@playwright/test';
import { sentryTest } from '../../../../../utils/fixtures';
import { envelopeRequestParser, waitForErrorRequest } from '../../../../../utils/helpers';
import {
getCustomRecordingEvents,
collectReplayRequests,
getReplayPerformanceSpans,
shouldSkipReplayTest,
waitForReplayRequest,
} from '../../../../../utils/replayHelpers';

sentryTest('captures request body size when body is sent', async ({ getLocalTestPath, page }) => {
Expand All @@ -28,7 +28,9 @@ sentryTest('captures request body size when body is sent', async ({ getLocalTest
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -63,9 +65,8 @@ sentryTest('captures request body size when body is sent', async ({ getLocalTest
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down Expand Up @@ -112,7 +113,9 @@ sentryTest('captures request size from non-text request body', async ({ getLocal
});

const requestPromise = waitForErrorRequest(page);
const replayRequestPromise1 = waitForReplayRequest(page, 0);
const replayRequestPromise = collectReplayRequests(page, recordingEvents => {
return getReplayPerformanceSpans(recordingEvents).some(span => span.op === 'resource.fetch');
});

const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);
Expand Down Expand Up @@ -149,9 +152,8 @@ sentryTest('captures request size from non-text request body', async ({ getLocal
},
});

const replayReq1 = await replayRequestPromise1;
const { performanceSpans: performanceSpans1 } = getCustomRecordingEvents(replayReq1);
expect(performanceSpans1.filter(span => span.op === 'resource.fetch')).toEqual([
const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
method: 'POST',
Expand Down
Loading