Sandbox: Reduce footprint of Buildout

- The sandbox setup is now based on vanilla Python and virtualenv.
- A convenient bootstrap script is provided, which can be used like
  `source bootstrap.sh`.
- The test runner has not been changed, it is still `zope.testrunner`.
- The command line invocation interface `./bin/test` has not been
  changed.
- Buildout is now only used for downloading the CrateDB tarball using
  `hexagonit.recipe.download`.

References:
- #428
- crate-workbench/test-buildout-python310#1

Author: amotl

.github/workflows/nightly.yml

@@ -28,11 +28,13 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
 
-      - name: Install dependencies
+      - name: Propagate matrix information
         run: |
+          source bootstrap.sh
           ./devtools/setup_ci.sh --cratedb-version=${{ matrix.cratedb-version }} --sqlalchemy-version=${{ matrix.sqla-version }}
 
       - name: Invoke tests
         run: |
-          bin/flake8
-          bin/coverage run bin/test -vv1
+          source bootstrap.sh
+          flake8 src bin
+          coverage run bin/test -vv1

.github/workflows/tests.yml

@@ -35,15 +35,17 @@ jobs:
           brew install gnu-getopt
           echo "/usr/local/opt/gnu-getopt/bin" >> $GITHUB_PATH
 
-      - name: Install dependencies
+      - name: Propagate matrix information
         run: |
+          source bootstrap.sh
           ./devtools/setup_ci.sh --cratedb-version=${{ matrix.cratedb-version }} --sqlalchemy-version=${{ matrix.sqla-version }}
 
       - name: Invoke tests
         run: |
-          bin/flake8
-          bin/coverage run bin/test -vv1
-          bin/coverage xml
+          source bootstrap.sh
+          flake8 src bin
+          coverage run bin/test -vv1
+          coverage xml
 
       # https://github.com/codecov/codecov-action
       - name: Upload coverage results to Codecov

.gitignore

@@ -6,7 +6,9 @@ coverage.xml
 .tox/
 *.DS_Store
 *.pyc
-bin/
+bin/*
+!bin/test
+!bin/sphinx
 build/
 crate-python.iml
 crate.egg-info

DEVELOP.rst

@@ -5,27 +5,22 @@ Developer guide
 Setup
 =====
 
-This project uses buildout_ to set up the development environment.
+To start things off, bootstrap the sandbox environment::
 
-To start things off, create a Python virtualenv and install buildout::
+    source bootstrap.sh
 
-    python3 -m venv .venv
-    source .venv/bin/activate
+This command should automatically install all prerequisites for the development
+sandbox and drop you into the virtualenv, ready for invoking further commands.
 
-    # Workaround for Python 3.5
-    python -m pip install --upgrade "setuptools>=31,<51"
-
-    pip install zc.buildout==2.13.4
-
-Then, run::
-
-    buildout -N
 
 Running tests
 =============
 
 All tests will be invoked using the Python interpreter that was used when
-creating the Python virtualenv. The test runner is zope.testrunner_.
+creating the Python virtualenv. The test runner is `zope.testrunner`_.
+
+Some examples are outlined below. In order to learn about more details,
+see, for example, `useful command-line options for zope-testrunner`_.
 
 Run all tests::
 
@@ -41,18 +36,18 @@ Ignore specific test directories::
 
     ./bin/test -vvvv --ignore_dir=testing
 
-You can run the tests against multiple Python interpreters with tox_::
+You can run the tests against multiple Python interpreters with `tox`_::
 
-    ./bin/tox
+    tox
 
 To do this, you will need the respective Python interpreter versions available
 on your ``$PATH``.
 
 To run against a single interpreter, you can also invoke::
 
-    ./bin/tox -e py37
+    tox -e py37
 
-*Note*: before running the tests, make sure to stop all CrateDB instances which
+*Note*: Before running the tests, make sure to stop all CrateDB instances which
 are listening on the default CrateDB transport port to avoid side effects with
 the test layer.
 
@@ -121,5 +116,6 @@ nothing special you need to do to get the live docs to update.
 .. _Sphinx: http://sphinx-doc.org/
 .. _tox: http://testrun.org/tox/latest/
 .. _twine: https://pypi.python.org/pypi/twine
-.. _zope.testrunner: https://pypi.python.org/pypi/zope.testrunner/4.4.1
+.. _useful command-line options for zope-testrunner: https://pypi.org/project/zope.testrunner/#some-useful-command-line-options-to-get-you-started
 .. _versions hosted on ReadTheDocs: https://readthedocs.org/projects/crate-python/versions/
+.. _zope.testrunner: https://pypi.org/project/zope.testrunner/

base.cfg

@@ -0,0 +1,15 @@
+#!/usr/bin/env python
+import os
+import sys
+
+join = os.path.join
+base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
+base = os.path.dirname(base)
+
+
+sys.argv.extend(['-b', 'html', '-E', 'docs', './out/html'])
+import sphinx.cmd.build  # noqa:E402
+
+
+if __name__ == '__main__':
+    sys.exit(sphinx.cmd.build.main())

bin/sphinx

@@ -0,0 +1,16 @@
+#!/usr/bin/env python
+import os
+import sys
+import zope.testrunner
+
+join = os.path.join
+base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
+base = os.path.dirname(base)
+
+
+sys.argv[0] = os.path.abspath(sys.argv[0])
+
+if __name__ == '__main__':
+    sys.exit(zope.testrunner.run((['--auto-color', '--verbose']) + [
+        '--test-path', join(base, 'src'),
+        ]))

bin/test

@@ -0,0 +1,90 @@
+#!/bin/bash
+#
+# Bootstrap sandbox environment for crate-python
+#
+# - Create a Python virtualenv
+# - Install all dependency packages and modules
+# - Install package in editable mode
+# - Drop user into an activated virtualenv
+#
+# Synopsis::
+#
+#     source bootstrap.sh
+#
+
+
+# Trace all invocations.
+# set -x
+
+
+function print_header() {
+    printf '=%.0s' {1..42}; echo
+    echo "$1"
+    printf '=%.0s' {1..42}; echo
+}
+
+function ensure_virtualenv() {
+    # Create a Python virtualenv with current version of Python 3.
+    # TODO: Maybe take `pyenv` into account.
+    if [[ ! -d .venv ]]; then
+        python3 -m venv .venv
+    fi
+}
+
+function activate_virtualenv() {
+    # Activate Python virtualenv.
+    source .venv/bin/activate
+}
+
+function before_setup() {
+
+    # When `wheel` is installed, Python will build `wheel` packages from all
+    # acquired `sdist` packages and will store them into `~/.cache/pip`, where
+    # they will be picked up by the caching machinery and will be reused on
+    # subsequent invocations when run on CI. This makes a *significant*
+    # difference on total runtime on CI, it is about 2x faster.
+    #
+    # Otherwise, there will be admonitions like:
+    #   Using legacy 'setup.py install' for foobar, since package 'wheel' is
+    #   not installed.
+    #
+    pip install wheel
+
+    BUILDOUT_VERSION=${BUILDOUT_VERSION:-2.13.7}
+    pip install "zc.buildout==${BUILDOUT_VERSION}"
+
+}
+
+function setup_package() {
+
+    # Install package in editable mode.
+    pip install --editable=.[sqlalchemy,test,doc]
+
+}
+
+function run_buildout() {
+    buildout -N
+}
+
+function finalize() {
+
+    # Some steps before dropping into the activated virtualenv.
+    echo
+    python -c 'import sqlalchemy; print(f"SQLAlchemy version: {sqlalchemy.__version__}")'
+
+}
+
+function main() {
+    ensure_virtualenv
+    activate_virtualenv
+    before_setup
+    setup_package
+    run_buildout
+    finalize
+}
+
+function lint() {
+    flake8
+}
+
+main

bootstrap.sh

@@ -1,24 +1,20 @@
 [buildout]
-extends = base.cfg
-parts += tox
-         sphinx
-         scripts
+extends = versions.cfg
+versions = versions
+show-picked-versions = true
+parts = crate
 
-[scripts]
-recipe = zc.recipe.egg:script
-eggs = wheel
-       docutils
-       twine
+[crate:linux]
+recipe = hexagonit.recipe.download
+url = https://cdn.crate.io/downloads/releases/crate-${versions:crate_server}.tar.gz
+strip-top-level-dir = true
 
-[tox]
-recipe = gp.recipe.tox
+[crate:macosx]
+recipe = hexagonit.recipe.download
+url = https://cdn.crate.io/downloads/releases/cratedb/x64_mac/crate-${versions:crate_server}.tar.gz
+strip-top-level-dir = true
 
-[sphinx]
-recipe = zc.recipe.egg:script
-eggs = sphinx
-       crate-docs-theme
-relative-paths=true
-scripts = sphinx-build=sphinx
-initialization =
-    sys.argv.extend(['-N', '-q', '-b', 'html',
-                     '-E', 'docs', '${buildout:directory}/out/html'])
+[crate:windows]
+recipe = hexagonit.recipe.download
+url = https://cdn.crate.io/downloads/releases/cratedb/x64_windows/crate-${versions:crate_server}.zip
+strip-top-level-dir = true

buildout.cfg

@@ -48,7 +48,7 @@ fi
 
 # check if tag to create has already been created
 WORKING_DIR=`dirname $0`
-VERSION=`$WORKING_DIR/../bin/py setup.py --version`
+VERSION=`python setup.py --version`
 EXISTS=`git tag | grep $VERSION`
 
 if [ "$VERSION" == "$EXISTS" ]

devtools/create_tag.sh

@@ -46,23 +46,17 @@ function main() {
   # Let's go.
   echo "Invoking tests with CrateDB ${cratedb_version} and SQLAlchemy ${sqlalchemy_version}"
 
-  python -m pip install --upgrade pip
-
-  pip install zc.buildout==2.13.4
-
-  # Replace SQLAlchemy version.
-  sed -ir "s/SQLAlchemy.*/SQLAlchemy = ${sqlalchemy_version}/g" versions.cfg
+  # Install designated SQLAlchemy version.
+  pip install "sqlalchemy==${sqlalchemy_version}"
 
   # Replace CrateDB version.
   if [ ${cratedb_version} = "nightly" ]; then
-    sed -ir "s/releases/releases\/nightly/g" base.cfg
+    sed -ir "s/releases/releases\/nightly/g" buildout.cfg
     sed -ir "s/crate_server.*/crate_server = latest/g" versions.cfg
   else
     sed -ir "s/crate_server.*/crate_server = ${cratedb_version}/g" versions.cfg
   fi
 
-  buildout -n -c base.cfg
-
 }
 
 main "$@"

devtools/setup_ci.sh

@@ -2,4 +2,4 @@
 universal = 1
 
 [flake8]
-ignore = E501,C901, W504
+ignore = E501, C901, W504

setup.cfg

@@ -64,10 +64,17 @@ def read(path):
         ]
     },
     extras_require=dict(
-        test=['zope.testing>=4,<5',
+        sqlalchemy=['sqlalchemy>=1.0,<1.5',
+                    'geojson>=2.5.0'],
+        test=['tox>=3,<4',
+              'zope.testing>=4,<5',
+              'zope.testrunner>=5,<6',
               'zc.customdoctests>=1.0.1,<2',
-              'stopit>=1.1.2,<2'],
-        sqlalchemy=['sqlalchemy>=1.0,<1.5', 'geojson>=2.5.0']
+              'createcoverage>=1,<2',
+              'stopit>=1.1.2,<2',
+              'flake8>=4,<5'],
+        doc=['sphinx>=3,<4',
+             'crate-docs-theme'],
     ),
     python_requires='>=3.4',
     install_requires=requirements,