Add new smoke tests

smoke-tests/README.md

@@ -0,0 +1,82 @@
+# Firebase Smoke Test Suite
+
+This directory contains smoke tests for Firebase on Android. These tests are
+intended to verify the integrations between different components and versions of
+Firebase. The tests should not include additional overhead, such as user
+interfaces. However, the tests should strive to remain similar to real use
+cases. As such, these tests run on devices or emulators (no Robolectric). This
+is a work in progress, and the following list shows what is complete:
+
+- [x] Create first set of tests to replace old test apps.
+- [ ] Reliably run smoke tests on CI.
+- [ ] Support version matrices.
+- [ ] Extend to collect system health metrics.
+
+# Test Synchronization
+
+Tests on devices usually involve at least two threads: the main thread and the
+testing thread. Either thread should be sufficient for testing, but most users
+will likely invoke Firebase methods from the main thread. Integration tests
+already verify the use of the testing thread, so it is important to gain
+coverage from the main thread as well. Therefore, this framework provides a
+simple solution to easily share state between threads. Most tests will consist
+of two methods. Both may be present in the same file for simplicity and ease of
+understanding. One method will be executed by the test runner on the testing
+thread, while the other, is invoked via a lambda on the main thread:
+
+```java
+@Test
+public void foo() throws Exception {
+  TaskChannel<Foo> channel = new TaskChannel<>();
+  MainThread.run(() -> test_foo(channel));
+
+  Foo foo = channel.waitForSuccess();
+  assertThat(foo)...
+}
+
+private void test_foo(TaskChannel<Foo> channel) {
+  ...
+}
+```
+
+Channels should be used to send results back to the testing thread. The
+`AbstractChannel` class provides the core implementation to send a failure or
+success value back to the test method. Failures are always exceptions, and
+multiple exceptions may be sent per test case. Only one success value may be
+sent. Failures dominate over successful values.
+
+Firebase methods are often difficult to test due to their asynchronous nature.
+This framework provides a few subclasses of `AbstractChannel` to eliminate some
+boilerplate, concentrate the synchronization logic in one place, and shorten the
+tests to focus on the actual Firebase methods under test. The most common
+subclass is `TaskChannel`, and it simplifies chaining and adding listeners to
+Google Play Services tasks. Below is an example (Firebase methods truncated for
+brevity):
+
+```java
+private void test_readAfterSignIn(TaskChannel<Document> channel) {
+  FirebaseAuth auth = FirebaseAuth.getInstance();
+  FirebaseDatabase db = FirebaseDatabase.getInstance();
+
+  channel.trapFailure(auth.signIn()).andThen(a -> {
+    channel.sendOutcome(db.get("path/to/document"));
+  });
+}
+```
+
+Methods like `trapFailure` and `sendOutcome` automatically direct failures from
+tasks to the testing channel. There are also specialized channels for Firestore
+and Database, because they rely on custom listeners. See the source code for
+more details.
+
+# Building the Tests
+
+This Gradle project is split into flavors for each Firebase product. The Android
+plugin adds additional, compound flavors, such as `androidTestDatabase`. All
+test code lives in one of these testing variants. Common infrastructure belongs
+directly in `androidTest`. Do not add any source code to `main` or a non-testing
+variant, such as `firestore`.
+
+As an exception, there is a single activity in the `main` variant. This is used
+to build the APK under test. However, all test logic belongs in the testing
+variants.

smoke-tests/build.gradle

@@ -0,0 +1,104 @@
+// Copyright 2018 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+buildscript {
+  repositories {
+    google()
+    jcenter()
+  }
+
+  dependencies {
+    classpath "com.android.tools.build:gradle:3.3.2"
+    classpath "com.google.gms:google-services:4.1.0"
+  }
+}
+
+apply plugin: "com.android.application"
+
+android {
+  compileSdkVersion 24
+
+  compileOptions {
+    sourceCompatibility JavaVersion.VERSION_1_8
+    targetCompatibility JavaVersion.VERSION_1_8
+  }
+
+  defaultConfig {
+    minSdkVersion 16
+    multiDexEnabled true
+    testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
+  }
+
+  flavorDimensions "systemUnderTest"
+
+  productFlavors {
+    database {
+      dimension "systemUnderTest"
+    }
+
+    firestore {
+      dimension "systemUnderTest"
+    }
+
+    storage {
+      dimension "systemUnderTest"
+    }
+  }
+}
+
+repositories {
+  google()
+  jcenter()
+}
+
+dependencies {
+  // Generally, depencies need to be placed under the test variants, such as androidTestDatabase.
+  // However, to ensure we don't have build conflicts and test errors, we also need to put the same
+  // dependencies on the "app" variants.
+
+  // Common
+  implementation "com.android.support.test:runner:1.0.2"
+  implementation "com.google.android.gms:play-services-tasks:16.0.1"
+  implementation "com.google.firebase:firebase-core:16.0.7"
+  implementation "com.google.truth:truth:0.43"
+
+  androidTestImplementation "com.android.support.test:runner:1.0.2"
+  androidTestImplementation "com.google.android.gms:play-services-tasks:16.0.1"
+  androidTestImplementation "com.google.firebase:firebase-core:16.0.7"
+  androidTestImplementation "com.google.truth:truth:0.43"
+  androidTestImplementation "junit:junit:4.12"
+
+  // Database
+  databaseImplementation "com.google.firebase:firebase-auth:16.1.0"
+  databaseImplementation "com.google.firebase:firebase-database:16.1.0"
+
+  androidTestDatabaseImplementation "com.google.firebase:firebase-auth:16.1.0"
+  androidTestDatabaseImplementation "com.google.firebase:firebase-database:16.1.0"
+
+  // Firestore
+  firestoreImplementation "com.google.firebase:firebase-auth:16.1.0"
+  firestoreImplementation "com.google.firebase:firebase-firestore:18.1.0"
+
+  androidTestFirestoreImplementation "com.google.firebase:firebase-auth:16.1.0"
+  androidTestFirestoreImplementation "com.google.firebase:firebase-firestore:18.1.0"
+
+  // Storage
+  storageImplementation "com.google.firebase:firebase-auth:16.1.0"
+  storageImplementation "com.google.firebase:firebase-storage:16.1.0"
+
+  androidTestStorageImplementation "com.google.firebase:firebase-auth:16.1.0"
+  androidTestStorageImplementation "com.google.firebase:firebase-storage:16.1.0"
+}
+
+apply plugin: "com.google.gms.google-services"

smoke-tests/src/androidTest/java/com/google/firebase/testing/common/AbstractChannel.java

@@ -0,0 +1,97 @@
+// Copyright 2018 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.firebase.testing.common;
+
+import com.google.android.gms.tasks.TaskCompletionSource;
+import com.google.android.gms.tasks.Tasks;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+/**
+ * An abstract channel for sending test results across threads.
+ *
+ * <p>This enables test code to run on the main thread and signal the test thread when to stop
+ * blocking. Tests may send multiple errors that will then be thrown on the test thread. However, a
+ * test may only send success once. After this is done, nothing else can be sent.
+ */
+public abstract class AbstractChannel<T> {
+
+  private final TaskCompletionSource<T> implementation = new TaskCompletionSource<>();
+
+  /** Runs the test target on the main thread, trapping any exception into the channel. */
+  protected static <U extends AbstractChannel<R>, R> U runTarget(Target<U> target, U channel) {
+    MainThread.run(
+        () -> {
+          try {
+            target.run(channel);
+          } catch (Exception ex) {
+            channel.fail(ex);
+          }
+        });
+
+    return channel;
+  }
+
+  /**
+   * Sends a failure back to the testing thread.
+   *
+   * <p>This method will always send an exception to the testing thread. If an exception has already
+   * been sent, this method will chain the new exception to the previous. If a successful value has
+   * already been sent, this method will override it with the failure. Note, it is recommended to
+   * send only one value through the channel. This method is safe to invoke from any thread.
+   */
+  protected void fail(Exception err) {
+    boolean isSet = implementation.trySetException(err);
+
+    if (!isSet) {
+      implementation.getTask().getException().addSuppressed(err);
+    }
+  }
+
+  /**
+   * Sends a successful value back to the testing thread.
+   *
+   * <p>This method will only send the value if no value has been sent. It is an error to invoke
+   * this method multiple times or after invoking {@link #fail}. This method is safe to invoke from
+   * any thread.
+   */
+  protected void succeed(T val) {
+    implementation.setResult(val);
+  }
+
+  /** Waits 30 seconds to receive the successful value. */
+  public T waitForSuccess() throws InterruptedException {
+    return waitForSuccess(30, TimeUnit.SECONDS);
+  }
+
+  /**
+   * Waits for up to the request time for the sending thread to send a successful value.
+   *
+   * <p>If the sender does not send success within the specified time, this method throws an {@link
+   * AssertionError} and chains any received errors to it. This method is safe to invoke from any
+   * thread.
+   */
+  public T waitForSuccess(long duration, TimeUnit unit) throws InterruptedException {
+    try {
+      return Tasks.await(implementation.getTask(), duration, unit);
+    } catch (ExecutionException ex) {
+      throw new AssertionError("Test completed with errors", ex.getCause());
+    } catch (TimeoutException ex) {
+      String message = String.format("Test did not complete within %s %s", duration, unit);
+      throw new AssertionError(message);
+    }
+  }
+}

smoke-tests/src/androidTest/java/com/google/firebase/testing/common/MainThread.java

@@ -0,0 +1,39 @@
+// Copyright 2018 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.firebase.testing.common;
+
+import android.os.Handler;
+import android.os.Looper;
+
+/**
+ * Convenience class for interacting with Android.
+ *
+ * <p>For now, this only consists of the {@link #run} method.
+ */
+public final class MainThread {
+
+  private static Handler handler = null;
+
+  private MainThread() {}
+
+  /** Runs the {@link Runnable} on the main thread. */
+  public static void run(Runnable r) {
+    if (handler == null) {
+      handler = new Handler(Looper.getMainLooper());
+    }
+
+    handler.post(r);
+  }
+}

smoke-tests/src/androidTest/java/com/google/firebase/testing/common/Target.java

@@ -0,0 +1,26 @@
+// Copyright 2018 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.firebase.testing.common;
+
+/**
+ * A test target.
+ *
+ * <p>This interface is similar to {@link Runnable}, but this interface's {@link #run} method takes
+ * an instance of {@link T} as input. This is intended to be a channel.
+ */
+public interface Target<T> {
+
+  void run(T channel);
+}