Crashlytics smoke test stubs #2737

mrichards · 2021-06-16T21:37:21Z

No description provided.

google-oss-bot · 2021-06-16T21:52:15Z

Coverage Report

Affected SDKs

firebase-database

SDK overall coverage changed from 50.02% (1b0fdcf) to 50.07% (83fe61c0) by +0.05%.

Filename Base (1b0fdcf) Head (83fe61c0) Diff

ChildChangeAccumulator.java 83.33% 96.67% +13.33%

QueryParams.java 89.76% 90.36% +0.60%
firebase-messaging

SDK overall coverage changed from 83.56% (1b0fdcf) to 83.43% (83fe61c0) by -0.14%.

Filename Base (1b0fdcf) Head (83fe61c0) Diff

TopicsSubscriber.java 91.38% 89.66% -1.72%

Test Logs

Notes

HTML coverage reports can be produced locally with ./gradlew <product>:checkCoverage.
Report files are located at <product-build-dir>/reports/jacoco/.

Head commit (83fe61c0) is created by Prow via merging commits: 1b0fdcf 2f1569a.

google-oss-bot · 2021-06-16T21:55:30Z

Binary Size Report

Affected SDKs

firebase-crashlytics

Type Base (1b0fdcf) Head (83fe61c0) Diff

aar 330 kB 330 kB +26 B (+0.0%)

apk (release) 825 kB 825 kB -64 B (-0.0%)
firebase-crashlytics-ktx

Type Base (1b0fdcf) Head (83fe61c0) Diff

apk (release) 1.31 MB 1.31 MB -68 B (-0.0%)

Test Logs

Notes

Head commit (83fe61c0) is created by Prow via merging commits: 1b0fdcf 2f1569a.

google-oss-bot · 2021-06-16T22:53:22Z

Macrobenchmark Report

Affected SDKs

Measurements are for head commit (a53c473). Diffing against base commit (c7d8a80) is working in progress.

baseline

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	122.0 ms	154.0 ms	134.5 ms	153.0 ms	153.8 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	41.0 ms	65.0 ms	51.5 ms	61.4 ms	65.0 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	58.0 ms	92.0 ms	75.0 ms	83.7 ms	91.6 ms

firebase-common

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	127.0 ms	211.0 ms	143.0 ms	162.7 ms	208.1 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	48.0 ms	67.0 ms	52.5 ms	56.3 ms	65.5 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	57.0 ms	97.0 ms	73.0 ms	85.1 ms	94.9 ms

firebase-config

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	117.0 ms	174.0 ms	141.0 ms	159.3 ms	171.7 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	43.0 ms	89.0 ms	53.5 ms	62.5 ms	84.8 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	56.0 ms	89.0 ms	70.5 ms	83.2 ms	88.2 ms

firebase-crashlytics

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	149.0 ms	181.0 ms	164.5 ms	178.0 ms	180.4 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	38.0 ms	62.0 ms	50.0 ms	60.0 ms	61.6 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	49.0 ms	105.0 ms	72.0 ms	91.5 ms	103.3 ms

firebase-database

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	120.0 ms	171.0 ms	140.5 ms	147.1 ms	168.3 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	43.0 ms	72.0 ms	50.5 ms	58.4 ms	70.1 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	57.0 ms	88.0 ms	73.0 ms	81.1 ms	86.9 ms

firebase-dynamic-links

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	138.0 ms	167.0 ms	152.5 ms	162.1 ms	166.2 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	38.0 ms	75.0 ms	59.5 ms	73.0 ms	74.6 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	56.0 ms	88.0 ms	77.5 ms	84.4 ms	88.0 ms

firebase-firestore

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	127.0 ms	162.0 ms	142.0 ms	148.7 ms	160.7 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	35.0 ms	68.0 ms	52.5 ms	64.0 ms	67.2 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	52.0 ms	90.0 ms	74.5 ms	85.3 ms	89.6 ms

firebase-functions

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	124.0 ms	170.0 ms	147.0 ms	160.1 ms	168.3 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	43.0 ms	73.0 ms	52.5 ms	60.1 ms	70.7 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	53.0 ms	95.0 ms	75.5 ms	92.2 ms	94.8 ms

firebase-inappmessaging-display

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	155.0 ms	204.0 ms	181.0 ms	189.4 ms	201.9 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	38.0 ms	66.0 ms	54.0 ms	62.3 ms	65.8 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	64.0 ms	108.0 ms	81.0 ms	94.0 ms	105.3 ms

firebase-messaging

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	130.0 ms	178.0 ms	147.5 ms	161.8 ms	176.3 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	34.0 ms	67.0 ms	56.5 ms	62.4 ms	66.8 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	54.0 ms	94.0 ms	70.5 ms	80.1 ms	91.5 ms

firebase-perf

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	151.0 ms	178.0 ms	159.0 ms	173.0 ms	177.1 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	41.0 ms	73.0 ms	53.0 ms	68.3 ms	72.6 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	58.0 ms	101.0 ms	72.5 ms	87.4 ms	99.1 ms

firebase-storage

Name	Device	Min	Max	P50	P90	P99
BenchmarkTest.startup[mode=COLD]	flame-30-en-portrait	131.0 ms	164.0 ms	148.5 ms	158.3 ms	163.4 ms
BenchmarkTest.startup[mode=HOT]	flame-30-en-portrait	38.0 ms	70.0 ms	52.0 ms	65.2 ms	69.4 ms
BenchmarkTest.startup[mode=WARM]	flame-30-en-portrait	63.0 ms	86.0 ms	76.0 ms	80.5 ms	85.8 ms

Test Logs

Head (a53c473): https://android-ci.firebaseopensource.com/view/gcs/android-ci/pr-logs/pull/firebase_firebase-android-sdk/2737/macrobenchmark/1407006117829021700

One test validates the Analytics integration; the rest are simple stubbs for now.

…ests

yifanyang

Thanks for adding this test! LGTM.

yifanyang · 2021-06-30T17:53:21Z

smoke-tests/src/main/java/com/google/firebase/testing/CrashlyticsTest.java

+      Object core = coreField.get(crashlytics);
+      Field breadcrumbSourceField = core.getClass().getDeclaredField("breadcrumbSource");
+      breadcrumbSourceField.setAccessible(true);
+      BreadcrumbSource breadcrumbSource = (BreadcrumbSource)breadcrumbSourceField.get(core);


Suggested change

BreadcrumbSource breadcrumbSource = (BreadcrumbSource)breadcrumbSourceField.get(core);

BreadcrumbSource breadcrumbSource = (BreadcrumbSource) breadcrumbSourceField.get(core);

yifanyang · 2021-06-30T18:02:43Z

/retest

yifanyang · 2021-06-30T18:07:25Z

@vkryachko Adding you as a reviewer about the reflections and additional proguard rules.

vkryachko · 2021-06-30T18:11:43Z

smoke-tests/src/main/java/com/google/firebase/testing/CrashlyticsTest.java

+    try {
+      // need to use reflection to get the breadcrumb handler.
+      FirebaseCrashlytics crashlytics = FirebaseCrashlytics.getInstance();
+      Field coreField = crashlytics.getClass().getDeclaredField("core");


I feel like this use of reflection could cause maintenance problems down the road.

There seems to be an easy workaround, given the following observation: core is not a private field and is visible within the com.google.firebase.crashlytics package. @mrichards Can we move this test into src/main/java/com/google/firebase/crashlytics and access the field directly?

As for breadcrumbSource, can we make it public and annotation with @VisibleForTesting? (could actually be used with core above as well)

@vkryachko Please take a look at the updated implementation. I moved CrashlyticsTest to the crashlytics package, because making core public in FirebaseCrashlytics makes it visible in the IDE's autocomplete for customers, even when @VisibleForTesting was used.

LMK if this is consistent with your suggestion.

vkryachko

LGTM with minor nits

vkryachko · 2021-07-09T18:08:22Z

firebase-crashlytics/src/main/java/com/google/firebase/crashlytics/FirebaseCrashlytics.java

@@ -156,7 +157,8 @@ public Void call() throws Exception {
    return new FirebaseCrashlytics(core);
  }

-  private final CrashlyticsCore core;
+  // package-private for smoke tests
+  final CrashlyticsCore core;


Could you still add @VisibleForTesting?

vkryachko · 2021-07-09T18:09:04Z

...ashlytics/src/main/java/com/google/firebase/crashlytics/internal/common/CrashlyticsCore.java

  private final IdManager idManager;
-  private final BreadcrumbSource breadcrumbSource;
+
+  @VisibleForTesting(otherwise = VisibleForTesting.PRIVATE)


I think otherwise = Private is the default, so no need to explicitly specify

google-oss-bot · 2021-07-09T19:02:39Z

@mrichards: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
device-check-changed	`2f1569a`	link	`/test device-check-changed`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

google-oss-bot added the do-not-merge/work-in-progress label Jun 16, 2021

googlebot added the cla: yes Override cla label Jun 16, 2021

google-oss-bot added the size/M label Jun 16, 2021

mrichards force-pushed the crashlytics-smoke-tests branch 5 times, most recently from 5727629 to acfe867 Compare June 25, 2021 19:26

mrichards force-pushed the crashlytics-smoke-tests branch 2 times, most recently from 1431376 to 03afcd7 Compare June 29, 2021 18:24

Crashlytics smoke tests

5ab83f5

One test validates the Analytics integration; the rest are simple stubbs for now.

mrichards force-pushed the crashlytics-smoke-tests branch from 03afcd7 to 5ab83f5 Compare June 29, 2021 18:55

Merge remote-tracking branch 'origin/master' into crashlytics-smoke-t…

6675140

…ests

mrichards marked this pull request as ready for review June 29, 2021 20:07

google-oss-bot removed the do-not-merge/work-in-progress label Jun 29, 2021

mrichards requested a review from yifanyang June 29, 2021 20:07

mrichards changed the title ~~Crashlytics smoke test stubbs~~ Crashlytics smoke test stubs Jun 29, 2021

yifanyang approved these changes Jun 30, 2021

View reviewed changes

yifanyang requested a review from vkryachko June 30, 2021 18:06

vkryachko reviewed Jun 30, 2021

View reviewed changes

mrichards added 2 commits July 9, 2021 13:43

Removed reflection from Crashlytics smoke tests.

cf7130c

Merge branch 'master' into crashlytics-smoke-tests

db24d0b

google-oss-bot added size/L and removed size/M labels Jul 9, 2021

vkryachko approved these changes Jul 9, 2021

View reviewed changes

mrichards added 2 commits July 9, 2021 14:14

Updated @VisibleForTesting annotations

67b4c63

Applied gJF

2f1569a

mrichards merged commit a78f6bd into master Jul 9, 2021

mrichards deleted the crashlytics-smoke-tests branch July 9, 2021 18:53

This was referenced Jul 13, 2021

Rollback #2785, which made bom generation and crashlytics smoke test fail. #2825

Merged

Temporarily disbale a smoke test case for Crashlytics. #2831

Merged

firebase locked and limited conversation to collaborators Aug 9, 2021

	BreadcrumbSource breadcrumbSource = (BreadcrumbSource)breadcrumbSourceField.get(core);
	BreadcrumbSource breadcrumbSource = (BreadcrumbSource) breadcrumbSourceField.get(core);

Crashlytics smoke test stubs #2737

Crashlytics smoke test stubs #2737

Uh oh!

Conversation

mrichards commented Jun 16, 2021

Uh oh!

google-oss-bot commented Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Affected SDKs

firebase-database

firebase-messaging

Test Logs

Notes

Uh oh!

google-oss-bot commented Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Report

Affected SDKs

firebase-crashlytics

firebase-crashlytics-ktx

Test Logs

Notes

Uh oh!

google-oss-bot commented Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Macrobenchmark Report

Affected SDKs

baseline

firebase-common

firebase-config

firebase-crashlytics

firebase-database

firebase-dynamic-links

firebase-firestore

firebase-functions

firebase-inappmessaging-display

firebase-messaging

firebase-perf

firebase-storage

Test Logs

Uh oh!

yifanyang left a comment

Choose a reason for hiding this comment

Uh oh!

yifanyang Jun 30, 2021

Choose a reason for hiding this comment

Uh oh!

yifanyang commented Jun 30, 2021

Uh oh!

yifanyang commented Jun 30, 2021

Uh oh!

vkryachko Jun 30, 2021

Choose a reason for hiding this comment

Uh oh!

mrichards Jul 9, 2021

Choose a reason for hiding this comment

Uh oh!

vkryachko left a comment

Choose a reason for hiding this comment

Uh oh!

vkryachko Jul 9, 2021

Choose a reason for hiding this comment

Uh oh!

vkryachko Jul 9, 2021

Choose a reason for hiding this comment

Uh oh!

google-oss-bot commented Jul 9, 2021

Uh oh!

Uh oh!

google-oss-bot commented Jun 16, 2021 •

edited

Loading

`firebase-database`

`firebase-messaging`

google-oss-bot commented Jun 16, 2021 •

edited

Loading

`firebase-crashlytics`

`firebase-crashlytics-ktx`

google-oss-bot commented Jun 16, 2021 •

edited

Loading