Update Docs With Incluster Testing Support (#526)

* doc/test-framework: update test framework walkthrough

* doc/sdk-cli-reference.md: update for incluster testing support

Author: AlexNPavel

commands/operator-sdk/cmd/test/cluster.go

@@ -56,7 +56,7 @@ func NewTestClusterCmd() *cobra.Command {
 	testCmd.Flags().StringVar(&tcConfig.kubeconfig, "kubeconfig", defaultKubeConfig, "Kubeconfig path")
 	testCmd.Flags().StringVar(&tcConfig.imagePullPolicy, "image-pull-policy", "Always", "Set test pod image pull policy. Allowed values: Always, Never")
 	testCmd.Flags().StringVar(&tcConfig.serviceAccount, "service-account", "default", "Service account to run tests on")
-	testCmd.Flags().IntVar(&tcConfig.pendingTimeout, "pending-timeout", 60, "Timeout for testing pod in pending state")
+	testCmd.Flags().IntVar(&tcConfig.pendingTimeout, "pending-timeout", 60, "Timeout in seconds for testing pod to stay in pending state (default 60s)")
 
 	return testCmd
 }

doc/sdk-cli-reference.md

@@ -9,16 +9,21 @@ Usage:
 
 ### Args
 
-* image - is the container image to be built, e.g. "quay.io/example/operator:v0.0.1". This image will be automatically set in the deployment manifests.
+* image - is the container image to be built, e.g. "quay.io/example/operator:v0.0.1".
 
 ### Flags
-
+* `--enable-tests` bool - enable in-cluster testing by adding test binary to the image
+* `--namespaced-manifest` string - path of namespaced resources manifest for tests (default "deploy/operator.yaml")
+* `--test-location` string - location of tests (default "./test/e2e")
 * `-h, --help` - help for build
 
+
 ### Use
 
-The operator-sdk build command compiles the code, builds the executables,
-and generates Kubernetes manifests. After build completes, the image would be built locally in docker. Then it needs to be pushed to remote registry.
+The operator-sdk build command compiles the code and builds the executables. After build completes, the image is built locally in docker. Then it needs to be pushed to a remote registry.
+
+If `--enable-tests` is set, the build command will also build the testing binary, add it to the docker image, and generate
+a `deploy/test-pod.yaml` file that allows a user to run the tests as a pod on a cluster.
 
 ### Example:
 
@@ -158,30 +163,61 @@ Create app-operator/.gitignore
 
 ## test
 
-### Flags
+### Available Commands
 
-* `-t, --test-location` **(required)** string - location of e2e test files
-* `-k, --kubeconfig` string - location of kubeconfig for kubernetes cluster
-* `-g, --global-init` string - location of global resource manifest yaml file
-* `-n, --namespaced-init` string - location of namespaced resource manifest yaml file
-* `-f, --go-test-flags` string - extra arguments to pass to `go test` (e.g. -f "-v -parallel=2")
-* `-h, --help` - help for test
+#### local
+Runs the tests locally
 
-### Use
+##### Args
+* <test-location> string - location of e2e test files (e.g. "./test/e2e/")
 
-The operator-sdk test command runs go tests built using the Operator SDK's test framework.
+##### Flags
+* `--kubeconfig` string - location of kubeconfig for kubernetes cluster (default "~/.kube/config")
+* `--global-manifest` string - path to manifest for global resources (default "deploy/crd.yaml)
+* `--namespaced-manifest` string - path to manifest for per-test, namespaced resources (default: combines deploy/sa.yaml, deploy/rbac.yaml, and deploy/operator.yaml)
+* `--go-test-flags` string - extra arguments to pass to `go test` (e.g. -f "-v -parallel=2")
+* `-h, --help` - help for local
 
-### Example:
+##### Use
+
+The operator-sdk test command runs go tests built using the Operator SDK's test framework.
 
-#### Test
+##### Example:
 
 ```bash
-operator-sdk test --test-location ./test/e2e/
+$ operator-sdk test local ./test/e2e/
 
 # Output:
 ok  	github.com/operator-framework/operator-sdk-samples/memcached-operator/test/e2e	20.410s
 ```
 
+#### cluster
+Runs the e2e tests packaged in an operator image as a pod in the cluster
+
+##### Args
+* <image-name> string - the operator image that is used to run the tests in a pod (e.g. "quay.io/example/memcached-operator:v0.0.1")
+
+##### Flags
+* `--kubeconfig` string - location of kubeconfig for kubernetes cluster (default "~/.kube/config")
+* `--image-pull-policy` string - set test pod image pull policy. Allowed values: Always, Never (default "Always")
+* `--namespace` string - namespace to run tests in (default "default")
+* `--pending-timeout` int - timeout in seconds for testing pod to stay in pending state (default 60s)
+* `--service-account` string - service account to run tests on (default "default")
+* `--help` - help for cluster
+
+##### Use
+
+The operator-sdk test command runs go tests embedded in an operator image built using the Operator SDK.
+
+##### Example:
+
+```bash
+$ operator-sdk test cluster quay.io/example/memcached-operator:v0.0.1
+
+# Output:
+Test Successfully Completed
+```
+
 ## up
 
 ### Available Commands

doc/test-framework/writing-e2e-tests.md

@@ -194,31 +194,73 @@ functions will automatically be run since they were deferred when the TestCtx wa
 
 ## Running the Tests
 
-To make running the tests simpler, the `operator-sdk` CLI tool has a `test` subcommand that configures some
-default test settings, such as locations of the manifest files for your global resource manifest file (by default `deploy/crd.yaml`) and your namespaced manifest file (by defualt `deploy/rbac.yaml` concatenated with `deploy/operator.yaml`), and allows the user to configure these runtime options. To use it, run the
-`operator-sdk test` command in your project root and pass the location of the tests using the
-`--test-location` flag. You can use `--help` to view the other configuration options and use
-`--go-test-flags` to pass in arguments to `go test`. Here is an example command:
+To make running the tests simpler, the `operator-sdk` CLI tool has a `test` subcommand that can configure
+default test settings, such as locations of your global resource manifest file (by default
+`deploy/crd.yaml`) and your namespaced resource manifest file (by default `deploy/sa.yaml` concatenated with
+`deploy/rbac.yaml` and `deploy/operator.yaml`), and allows the user to configure runtime options. There are 2 ways to use the
+subcommand: local and cluster.
+### Local
+To run the tests locally, run the `operator-sdk test local` command in your project root and pass the location of the tests
+as an argument. You can use `--help` to view the other configuration options and use `--go-test-flags` to pass in arguments to `go test`. Here is an example command:
 
 ```shell
-$ operator-sdk test --test-location ./test/e2e --go-test-flags "-v -parallel=2"
+$ operator-sdk test local ./test/e2e --go-test-flags "-v -parallel=2"
 ```
 
-For more documentation on the `operator-sdk test` command, see the [SDK CLI Reference][sdk-cli-ref] doc.
+For more documentation on the `operator-sdk test local` command, see the [SDK CLI Reference][sdk-cli-ref] doc.
 
 For advanced use cases, it is possible to run the tests via `go test` directly. As long as all flags defined
 in [MainEntry][main-entry-link] are declared, the tests will run correctly. Running the tests directly with missing flags
-will result in undefined behavior. This is an example `go test` equivalent to the `operator-sdk test` example above:
+will result in undefined behavior. This is an example `go test` equivalent to the `operator-sdk test local` example above:
 
 ```shell
-# Combine rbac and operator manifest into namespaced manifest
-$ cp deploy/rbac.yaml deploy/namespace-init.yaml
+# Combine sa, rbac, operator manifest into namespaced manifest
+$ cp deploy/sa.yaml deploy/namespace-init.yaml
+$ echo -e "\n---\n" >> deploy/namespace-init.yaml
+$ cat deploy/rbac.yaml >> deploy/namespace-init.yaml
 $ echo -e "\n---\n" >> deploy/namespace-init.yaml
 $ cat deploy/operator.yaml >> deploy/namespace-init.yaml
 # Run tests
 $ go test ./test/e2e/... -root=$(pwd) -kubeconfig=$HOME/.kube/config -globalMan deploy/crd.yaml -namespacedMan deploy/namespace-init.yaml -v -parallel=2
 ```
 
+### Cluster
+
+Another way to run the tests is from within a kubernetes cluster. To do this, you first need to build an image with
+the testing binary embedded by using the `operator-sdk build` command and using the `--enable-tests` flag to enable tests:
+
+```shell
+$ operator-sdk build quay.io/example/memcached-operator:v0.0.1 --enable-tests
+```
+
+Note that the namespaced yaml must be up to date before running this command. The `build` subcommand will warn you
+if it finds a deployment in the namespaced manifest with an image that doesn't match the argument you provided. The
+`operator-sdk build` command has other flags for configuring the tests that can be viewed with the `--help` flag
+or at the [SDK CLI Reference][sdk-cli-ref].
+
+Once the image is ready, the tests are ready to be run. To run the tests, make sure you have all global resources
+and a namespace with proper rbac configured:
+
+```shell
+$ kubectl create -f deploy/crd.yaml
+$ kubectl create namespace memcached-test
+$ kubectl create -f deploy/sa.yaml -n memcached-test
+$ kubectl create -f deploy/rbac.yaml -n memcached-test
+```
+
+Once you have your environment properly configured, you can start the tests using the `operator-sdk test cluster` command:
+
+```shell
+$ operator-sdk test cluster quay.io/example/memcached-operator:v0.0.1 --namespace memcached-test
+
+Example Output:
+Test Successfully Completed
+```
+
+The `test cluster` command will deploy a test pod in the given namespace that will run the e2e tests packaged in the image.
+The command will wait until the tests succeed (pod phase=`Succeeded`) or fail (pod phase=`Failed`).
+If the tests fail, the command will output the test pod logs which should be the standard go test error logs.
+
 ## Manual Cleanup
 
 While the test framework provides utilities that allow the test to automatically be cleaned up when done,