tox integrations with invoke and docker (#1632)

Author: chayim

.github/workflows/integration.yaml

@@ -2,12 +2,62 @@ name: CI
 
 on:
   push:
+    paths-ignore:
+      - 'docs/**'
+      - '**/*.rst'
+      - '**/*.md'
   pull_request:
+    paths-ignore:
+      - 'docs/**'
+      - '**/*.rst'
+      - '**/*.md'
 
 jobs:
-  integration:
+
+  lint:
+    name: Code linters
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+      - name: run code linters
+        run: |
+          pip install -r dev_requirements.txt
+          invoke linters
+
+  run-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 6
+      matrix:
+        python-version: ['3.6', '3.7', '3.8', '3.9', '3.10', 'pypy-3.7']
+    env:
+      ACTIONS_ALLOW_UNSECURE_COMMANDS: true
+    name: Python ${{ matrix.python-version }} tests
+    steps:
+      - uses: actions/checkout@v2
+      - name: install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: run tests
+        run: |
+          pip install -r dev_requirements.txt
+          invoke tests
+
+  build_package:
+    name: Validate building and installing the package
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
-      - name: test
-        run: make test
+      - name: install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+      - name: build and install
+        run: |
+          pip install invoke
+          invoke package

.gitignore

@@ -14,3 +14,4 @@ vagrant/.vagrant
 env
 venv
 coverage.xml
+.venv

CONTRIBUTING.rst

@@ -29,25 +29,36 @@ Here's how to get started with your code contribution:
 
 1. Create your own fork of redis-py
 2. Do the changes in your fork
-3. If you need a development environment, run ``make dev``
-4. While developing, make sure the tests pass by running ``make test``
+3. Create a virtualenv and install the development dependencies from the dev_requirements.txt file:
+    a. python -m venv .venv
+    b. source .venv/bin/activate
+    c. pip install -r dev_requirements.txt
+3. If you need a development environment, run ``invoke devenv``
+4. While developing, make sure the tests pass by running ``invoke tests``
 5. If you like the change and think the project could use it, send a pull request
 
+To see what else is part of the automation, run ``invoke -l``
+
 The Development Environment
 ---------------------------
 
-Running ``make dev`` will create a Docker-based development environment that starts the following containers:
+Running ``invoke devenv`` installs the development dependencies specified in the dev_requirements.txt. It starts all of the dockers used by this project, and leaves them running. These can be easily cleaned up with ``invoke clean``. NOTE: it is assumed that the user running these tests, can execute docker and its various commands.
 
 * A master Redis node
-* A slave Redis node
+* A Redis replica node
 * Three sentinel Redis nodes
-* A test container
+* A multi-python docker, with your source code mounted in /data
 
-The slave is a replica of the master node, using the `leader-follower replication <https://redis.io/topics/replication>`_ feature.
+The replica node, is a replica of the master node, using the `leader-follower replication <https://redis.io/topics/replication>`_ feature.
 
 The sentinels monitor the master node in a `sentinel high-availability configuration <https://redis.io/topics/sentinel>`_.
 
-Meanwhile, the `test` container hosts the code from your checkout of ``redis-py`` and allows running tests against many Python versions.
+Testing
+-------
+
+Each run of tox starts and stops the various dockers required.  Sometimes things get stuck, an ``invoke clean`` can help.
+
+Continuous Integration uses these same wrappers to run all of these tests against multiple versions of python. Feel free to test your changes against all the python versions supported, as declared by the tox.ini file (eg: tox -e py39). If you have the various python versions on your desktop, you can run *tox* by itself, to test all supported versions. Alternatively, as your source code is mounted in the **lots-of-pythons** docker, you can start exploring from there, with all supported python versions!
 
 Docker Tips
 ^^^^^^^^^^^
@@ -56,17 +67,17 @@ Following are a few tips that can help you work with the Docker-based developmen
 
 To get a bash shell inside of a container:
 
-``$ docker-compose run <service> /bin/bash``
- 
-**Note**: The term "service" refers to the "services" defined in the ``docker-compose.yml`` file: "master", "slave", "sentinel_1", "sentinel_2", "sentinel_3", "test".
+``$ docker run -it <service> /bin/bash``
+
+**Note**: The term "service" refers to the "services" defined in the ``tox.ini`` file at the top of the repo: "master", "replicaof", "sentinel_1", "sentinel_2", "sentinel_3".
 
 Containers run a minimal Debian image that probably lacks tools you want to use. To install packages, first get a bash session (see previous tip) and then run:
 
 ``$ apt update && apt install <package>``
 
-You can see the combined logging output of all containers like this:
+You can see the logging output of a containers like this:
 
-``$ docker-compose logs``
+``$ docker logs -f <service>``
 
 The command `make test` runs all tests in all tested Python environments. To run the tests in a single environment, like Python 3.6, use a command like this:
 
@@ -81,13 +92,11 @@ Our test suite uses ``pytest``. You can run a specific test suite against a spec
 Troubleshooting
 ^^^^^^^^^^^^^^^
 If you get any errors when running ``make dev`` or ``make test``, make sure that you
-are using supported versions of Docker and docker-compose.
+are using supported versions of Docker.
 
-The included Dockerfiles and docker-compose.yml file work with the following
-versions of Docker and docker-compose:
+Please try at least versions of Docker.
 
 * Docker 19.03.12
-* docker-compose 1.26.2
 
 How to Report a Bug
 -------------------

Dockerfile

@@ -0,0 +1,5 @@
+flake8>=3.9.2
+pytest==6.2.5
+tox==3.24.4
+tox-docker==3.1.0
+invoke==1.6.0

Makefile

@@ -0,0 +1,4 @@
+bind replica 127.0.0.1
+port 6380
+save ""
+replicaof master 6379

dev_requirements.txt

@@ -0,0 +1,60 @@
+import os
+import shutil
+from invoke import task, run
+
+with open('tox.ini') as fp:
+    lines = fp.read().split("\n")
+    dockers = [line.split("=")[1].strip() for line in lines
+               if line.find("name") != -1]
+
+
+@task
+def devenv(c):
+    """Builds a development environment: downloads, and starts all dockers
+    specified in the tox.ini file.
+    """
+    clean(c)
+    cmd = 'tox -e devenv'
+    for d in dockers:
+        cmd += " --docker-dont-stop={}".format(d)
+    print("Running: {}".format(cmd))
+    run(cmd)
+
+
+@task
+def linters(c):
+    """Run code linters"""
+    run("flake8")
+
+
+@task
+def all_tests(c):
+    """Run all linters, and tests in redis-py. This assumes you have all
+    the python versions specified in the tox.ini file.
+    """
+    linters(c)
+    tests(c)
+
+
+@task
+def tests(c):
+    """Run the redis-py test suite against the current python,
+    with and without hiredis.
+    """
+    run("tox -e plain -e hiredis")
+
+
+@task
+def clean(c):
+    """Stop all dockers, and clean up the built binaries, if generated."""
+    if os.path.isdir("build"):
+        shutil.rmtree("build")
+    if os.path.isdir("dist"):
+        shutil.rmtree("dist")
+    run("docker rm -f {}".format(' '.join(dockers)))
+
+
+@task
+def package(c):
+    """Create the python packages"""
+    run("python setup.py build install")