keithks · keithks · Nov 2, 2020 · Feb 17, 2019 · Mar 3, 2019 · Mar 7, 2019
diff --git a/.travis.yml b/.travis.yml
@@ -1,38 +1,45 @@
 language: python
 
+dist: xenial
+
 python:
     - 2.7
     - 3.4
-    - 3.5
-    - 3.6
-    - pypy
+    - 3.7
+    - 3.8
+    - pypy2.7-6.0
 
 env:
     - KAFKA_VERSION=0.8.2.2
     - KAFKA_VERSION=0.9.0.1
-    - KAFKA_VERSION=0.10.2.1
-    - KAFKA_VERSION=0.11.0.2
+    - KAFKA_VERSION=0.10.2.2
+    - KAFKA_VERSION=0.11.0.3
     - KAFKA_VERSION=1.1.1
+    - KAFKA_VERSION=2.4.0
+    - KAFKA_VERSION=2.5.0
 
 addons:
   apt:
     packages:
       - libsnappy-dev
+      - libzstd-dev
+      - openjdk-8-jdk
 
 cache:
   directories:
     - $HOME/.cache/pip
-    - servers/
+    - servers/dist
 
 before_install:
+    - source travis_java_install.sh
     - ./build_integration.sh
 
 install:
     - pip install tox coveralls
     - pip install .
 
 script:
-  - tox -e `if [ "$TRAVIS_PYTHON_VERSION" == "pypy" ]; then echo pypy; else echo py${TRAVIS_PYTHON_VERSION/./}; fi`
+  - tox -e `if [ "$TRAVIS_PYTHON_VERSION" == "pypy2.7-6.0" ]; then echo pypy; else echo py${TRAVIS_PYTHON_VERSION/./}; fi`
 
 after_success:
   - coveralls
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,3 +1,244 @@
+# 2.0.2 (Sep 29, 2020)
+
+Consumer
+* KIP-54: Implement sticky partition assignment strategy (aynroot / PR #2057)
+* Fix consumer deadlock when heartbeat thread request timeout (huangcuiyang / PR #2064)
+
+Compatibility
+* Python 3.8 support (Photonios / PR #2088)
+
+Cleanups
+* Bump dev requirements (jeffwidman / PR #2129)
+* Fix crc32c deprecation warning (crc32c==2.1) (jeffwidman / PR #2128)
+* Lint cleanup (jeffwidman / PR #2126)
+* Fix initialization order in KafkaClient (pecalleja / PR #2119)
+* Allow installing crc32c via extras (mishas / PR #2069)
+* Remove unused imports (jameslamb / PR #2046)
+
+Admin Client
+* Merge _find_coordinator_id methods (jeffwidman / PR #2127)
+* Feature: delete consumergroups (swenzel / PR #2040)
+* Allow configurable timeouts in admin client check version (sunnyakaxd / PR #2107)
+* Enhancement for Kafka Admin Client's "Describe Consumer Group" (Apurva007 / PR #2035)
+
+Protocol
+* Add support for zstd compression (gabriel-tincu / PR #2021)
+* Add protocol support for brokers 1.1.0 - 2.5.0 (gabriel-tincu / PR #2038)
+* Add ProduceRequest/ProduceResponse v6/v7/v8 (gabriel-tincu / PR #2020)
+* Fix parsing NULL header values (kvfi / PR #2024)
+
+Tests
+* Add 2.5.0 to automated CI tests (gabriel-tincu / PR #2038)
+* Add 2.1.1 to build_integration (gabriel-tincu / PR #2019)
+
+Documentation / Logging / Errors
+* Disable logging during producer object gc (gioele / PR #2043)
+* Update example.py; use threading instead of multiprocessing (Mostafa-Elmenbawy / PR #2081)
+* Fix typo in exception message (haracejacob / PR #2096)
+* Add kafka.structs docstrings (Mostafa-Elmenbawy / PR #2080)
+* Fix broken compatibility page link (anuragrana / PR #2045)
+* Rename README to README.md (qhzxc0015 / PR #2055)
+* Fix docs by adding SASL mention (jeffwidman / #1990)
+
+# 2.0.1 (Feb 19, 2020)
+
+Admin Client
+* KAFKA-8962: Use least_loaded_node() for AdminClient.describe_topics() (jeffwidman / PR #2000)
+* Fix AdminClient topic error parsing in MetadataResponse (jtribble / PR #1997)
+
+# 2.0.0 (Feb 10, 2020)
+
+This release includes breaking changes for any application code that has not
+migrated from older Simple-style classes to newer Kafka-style classes.
+
+Deprecation
+* Remove deprecated SimpleClient, Producer, Consumer, Unittest (jeffwidman / PR #1196)
+
+Admin Client
+* Use the controller for topic metadata requests (TylerLubeck / PR #1995)
+* Implement list_topics, describe_topics, and describe_cluster (TylerLubeck / PR #1993)
+* Implement __eq__ and __hash__ for ACL objects (TylerLubeck / PR #1955)
+* Fixes KafkaAdminClient returning `IncompatibleBrokerVersion` when passing an `api_version` (ian28223 / PR #1953)
+* Admin protocol updates (TylerLubeck / PR #1948)
+* Fix describe config for multi-broker clusters (jlandersen  / PR #1869)
+
+Miscellaneous Bugfixes / Improvements
+* Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl (swenzel / PR #1918)
+* Fix slots usage and use more slots (carsonip / PR #1987)
+* Optionally return OffsetAndMetadata from consumer.committed(tp) (dpkp / PR #1979)
+* Reset conn configs on exception in conn.check_version() (dpkp / PR #1977)
+* Do not block on sender thread join after timeout in producer.close() (dpkp / PR #1974)
+* Implement methods to convert a Struct object to a pythonic object (TylerLubeck / PR #1951)
+
+Test Infrastructure / Documentation / Maintenance
+* Update 2.4.0 resource files for sasl integration (dpkp)
+* Add kafka 2.4.0 to CI testing (vvuibert / PR #1972)
+* convert test_admin_integration to pytest (ulrikjohansson / PR #1923)
+* xfail test_describe_configs_topic_resource_returns_configs (dpkp / Issue #1929)
+* Add crc32c to README and docs (dpkp)
+* Improve docs for reconnect_backoff_max_ms (dpkp / PR #1976)
+* Fix simple typo: managementment -> management (timgates42 / PR #1966)
+* Fix typos (carsonip / PR #1938)
+* Fix doc import paths (jeffwidman / PR #1933)
+* Update docstring to match conn.py's (dabcoder / PR #1921)
+* Do not log topic-specific errors in full metadata fetch (dpkp / PR #1980)
+* Raise AssertionError if consumer closed in poll() (dpkp / PR #1978)
+* Log retriable coordinator NodeNotReady, TooManyInFlightRequests as debug not error (dpkp / PR #1975)
+* Remove unused import (jeffwidman)
+* Remove some dead code (jeffwidman)
+* Fix a benchmark to Use print() function in both Python 2 and Python 3 (cclauss / PR #1983)
+* Fix a test to use ==/!= to compare str, bytes, and int literals (cclauss / PR #1984)
+* Fix benchmarks to use pyperf (carsonip / PR #1986)
+* Remove unused/empty .gitsubmodules file (jeffwidman / PR #1928)
+* Remove deprecated `ConnectionError` (jeffwidman / PR #1816)
+
+
+# 1.4.7 (Sep 30, 2019)
+
+This is a minor release focused on KafkaConsumer performance, Admin Client
+improvements, and Client concurrency. The KafkaConsumer iterator implementation
+has been greatly simplified so that it just wraps consumer.poll(). The prior
+implementation will remain available for a few more releases using the optional
+KafkaConsumer config: `legacy_iterator=True` . This is expected to improve
+consumer throughput substantially and help reduce heartbeat failures / group
+rebalancing.
+
+Client
+* Send socket data via non-blocking IO with send buffer (dpkp / PR #1912)
+* Rely on socket selector to detect completed connection attempts (dpkp / PR #1909)
+* Improve connection lock handling; always use context manager (melor,dpkp / PR #1895)
+* Reduce client poll timeout when there are no in-flight requests (dpkp / PR #1823)
+
+KafkaConsumer
+* Do not use wakeup when sending fetch requests from consumer (dpkp / PR #1911)
+* Wrap `consumer.poll()` for KafkaConsumer iteration (dpkp / PR #1902)
+* Allow the coordinator to auto-commit on old brokers (justecorruptio / PR #1832)
+* Reduce internal client poll timeout for (legacy) consumer iterator interface (dpkp / PR #1824)
+* Use dedicated connection for group coordinator (dpkp / PR #1822)
+* Change coordinator lock acquisition order (dpkp / PR #1821)
+* Make `partitions_for_topic` a read-through cache (Baisang / PR #1781,#1809)
+* Fix consumer hanging indefinitely on topic deletion while rebalancing (commanderdishwasher / PR #1782)
+
+Miscellaneous Bugfixes / Improvements
+* Fix crc32c avilability on non-intel architectures (ossdev07 / PR #1904)
+* Load system default SSL CAs if `ssl_cafile` is not provided (iAnomaly / PR #1883)
+* Catch py3 TimeoutError in BrokerConnection send/recv (dpkp / PR #1820)
+* Added a function to determine if bootstrap is successfully connected (Wayde2014 / PR #1876)
+
+Admin Client
+* Add ACL api support to KafkaAdminClient (ulrikjohansson / PR #1833)
+* Add `sasl_kerberos_domain_name` config to KafkaAdminClient (jeffwidman / PR #1852)
+* Update `security_protocol` config documentation for KafkaAdminClient (cardy31 / PR #1849)
+* Break FindCoordinator into request/response methods in KafkaAdminClient (jeffwidman / PR #1871)
+* Break consumer operations into request / response methods in KafkaAdminClient (jeffwidman / PR #1845)
+* Parallelize calls to `_send_request_to_node()` in KafkaAdminClient (davidheitman / PR #1807)
+
+Test Infrastructure / Documentation / Maintenance
+* Add Kafka 2.3.0 to test matrix and compatibility docs (dpkp / PR #1915)
+* Convert remaining `KafkaConsumer` tests to `pytest` (jeffwidman / PR #1886)
+* Bump integration tests to 0.10.2.2 and 0.11.0.3 (jeffwidman / #1890)
+* Cleanup handling of `KAFKA_VERSION` env var in tests (jeffwidman / PR #1887)
+* Minor test cleanup (jeffwidman / PR #1885)
+* Use `socket.SOCK_STREAM` in test assertions (iv-m / PR #1879)
+* Sanity test for `consumer.topics()` and `consumer.partitions_for_topic()` (Baisang / PR #1829)
+* Cleanup seconds conversion in client poll timeout calculation (jeffwidman / PR #1825)
+* Remove unused imports (jeffwidman / PR #1808)
+* Cleanup python nits in RangePartitionAssignor (jeffwidman / PR #1805)
+* Update links to kafka consumer config docs (jeffwidman)
+* Fix minor documentation typos (carsonip / PR #1865)
+* Remove unused/weird comment line (jeffwidman / PR #1813)
+* Update docs for `api_version_auto_timeout_ms` (jeffwidman / PR #1812)
+
+
+# 1.4.6 (Apr 2, 2019)
+
+This is a patch release primarily focused on bugs related to concurrency,
+SSL connections and testing, and SASL authentication:
+
+Client Concurrency Issues (Race Conditions / Deadlocks)
+* Fix race condition in `protocol.send_bytes` (isamaru / PR #1752)
+* Do not call `state_change_callback` with lock (dpkp / PR #1775)
+* Additional BrokerConnection locks to synchronize protocol/IFR state (dpkp / PR #1768)
+* Send pending requests before waiting for responses (dpkp / PR #1762)
+* Avoid race condition on `client._conns` in send() (dpkp / PR #1772)
+* Hold lock during `client.check_version` (dpkp / PR #1771)
+
+Producer Wakeup / TimeoutError
+* Dont wakeup during `maybe_refresh_metadata` -- it is only called by poll() (dpkp / PR #1769)
+* Dont do client wakeup when sending from sender thread (dpkp / PR #1761)
+
+SSL - Python3.7 Support / Bootstrap Hostname Verification / Testing
+* Wrap SSL sockets after connecting for python3.7 compatibility (dpkp / PR #1754)
+* Allow configuration of SSL Ciphers (dpkp / PR #1755)
+* Maintain shadow cluster metadata for bootstrapping (dpkp / PR #1753)
+* Generate SSL certificates for local testing (dpkp / PR #1756)
+* Rename ssl.keystore.location and ssl.truststore.location config files (dpkp)
+* Reset reconnect backoff on SSL connection (dpkp / PR #1777)
+
+SASL - OAuthBearer support / api version bugfix
+* Fix 0.8.2 protocol quick detection / fix SASL version check (dpkp / PR #1763)
+* Update sasl configuration docstrings to include supported mechanisms (dpkp)
+* Support SASL OAuthBearer Authentication (pt2pham / PR #1750)
+
+Miscellaneous Bugfixes
+* Dont force metadata refresh when closing unneeded bootstrap connections (dpkp / PR #1773)
+* Fix possible AttributeError during conn._close_socket (dpkp / PR #1776)
+* Return connection state explicitly after close in connect() (dpkp / PR #1778)
+* Fix flaky conn tests that use time.time (dpkp / PR #1758)
+* Add py to requirements-dev (dpkp)
+* Fixups to benchmark scripts for py3 / new KafkaFixture interface (dpkp)
+
+
+# 1.4.5 (Mar 14, 2019)
+
+This release is primarily focused on addressing lock contention
+and other coordination issues between the KafkaConsumer and the
+background heartbeat thread that was introduced in the 1.4 release.
+
+Consumer
+* connections_max_idle_ms must be larger than request_timeout_ms (jeffwidman / PR #1688)
+* Avoid race condition during close() / join heartbeat thread (dpkp / PR #1735)
+* Use last offset from fetch v4 if available to avoid getting stuck in compacted topic (keithks / PR #1724)
+* Synchronize puts to KafkaConsumer protocol buffer during async sends (dpkp / PR #1733)
+* Improve KafkaConsumer join group / only enable Heartbeat Thread during stable group (dpkp / PR #1695)
+* Remove unused `skip_double_compressed_messages` (jeffwidman / PR #1677)
+* Fix commit_offsets_async() callback (Faqa / PR #1712)
+
+Client
+* Retry bootstrapping after backoff when necessary (dpkp / PR #1736)
+* Recheck connecting nodes sooner when refreshing metadata (dpkp / PR #1737)
+* Avoid probing broker versions twice on newer brokers (dpkp / PR #1738)
+* Move all network connections and writes to KafkaClient.poll() (dpkp / PR #1729)
+* Do not require client lock for read-only operations (dpkp / PR #1730)
+* Timeout all unconnected conns (incl SSL) after request_timeout_ms (dpkp / PR #1696)
+
+Admin Client
+* Fix AttributeError in response topic error codes checking (jeffwidman)
+* Fix response error checking in KafkaAdminClient send_to_controller (jeffwidman)
+* Fix NotControllerError check (jeffwidman)
+
+Core/Protocol
+* Fix default protocol parser version / 0.8.2 version probe (dpkp / PR #1740)
+* Make NotEnoughReplicasError/NotEnoughReplicasAfterAppendError retriable (le-linh / PR #1722)
+
+Bugfixes
+* Use copy() in metrics() to avoid thread safety issues (emeric254 / PR #1682)
+
+Test Infrastructure
+* Mock dns lookups in test_conn (dpkp / PR #1739)
+* Use test.fixtures.version not test.conftest.version to avoid warnings (dpkp / PR #1731)
+* Fix test_legacy_correct_metadata_response on x86 arch (stanislavlevin / PR #1718)
+* Travis CI: 'sudo' tag is now deprecated in Travis (cclauss / PR #1698)
+* Use Popen.communicate() instead of Popen.wait() (Baisang / PR #1689)
+
+Compatibility
+* Catch thrown OSError by python 3.7 when creating a connection (danjo133 / PR #1694)
+* Update travis test coverage: 2.7, 3.4, 3.7, pypy2.7 (jeffwidman, dpkp / PR #1614)
+* Drop dependency on sphinxcontrib-napoleon (stanislavlevin / PR #1715)
+* Remove unused import from kafka/producer/record_accumulator.py (jeffwidman / PR #1705)
+* Fix SSL connection testing in Python 3.7 (seanthegeek, silentben / PR #1669)
+
+
 # 1.4.4 (Nov 20, 2018)
 
 Bugfixes

diff --git a/Makefile b/Makefile
@@ -14,8 +14,8 @@ servers/$(KAFKA_VERSION)/kafka-bin:
 build-integration: servers/$(KAFKA_VERSION)/kafka-bin
 
 # Test and produce coverage using tox. This is the same as is run on Travis
-test36: build-integration
-	KAFKA_VERSION=$(KAFKA_VERSION) SCALA_VERSION=$(SCALA_VERSION) tox -e py36 -- $(FLAGS)
+test37: build-integration
+	KAFKA_VERSION=$(KAFKA_VERSION) SCALA_VERSION=$(SCALA_VERSION) tox -e py37 -- $(FLAGS)
 
 test27: build-integration
 	KAFKA_VERSION=$(KAFKA_VERSION) SCALA_VERSION=$(SCALA_VERSION) tox -e py27 -- $(FLAGS)
@@ -56,4 +56,4 @@ doc:
 	make -C docs html
 	@echo "open file://`pwd`/docs/_build/html/index.html"
 
-.PHONY: all test36 test27 test-local cov-local clean doc
+.PHONY: all test37 test27 test-local cov-local clean doc
diff --git a/README.rst b/README.rst
@@ -1,8 +1,8 @@
 Kafka Python client
 ------------------------
 
-.. image:: https://img.shields.io/badge/kafka-1.1%2C%201.0%2C%200.11%2C%200.10%2C%200.9%2C%200.8-brightgreen.svg
-    :target: https://kafka-python.readthedocs.io/compatibility.html
+.. image:: https://img.shields.io/badge/kafka-2.5%2C%202.4%2C%202.3%2C%202.2%2C%202.1%2C%202.0%2C%201.1%2C%201.0%2C%200.11%2C%200.10%2C%200.9%2C%200.8-brightgreen.svg
+    :target: https://kafka-python.readthedocs.io/en/master/compatibility.html
 .. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg
     :target: https://pypi.python.org/pypi/kafka-python
 .. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github
@@ -142,6 +142,14 @@ To enable snappy compression/decompression install python-snappy (also requires
 See <https://kafka-python.readthedocs.io/en/master/install.html#optional-snappy-install>
 for more information.
 
+Optimized CRC32 Validation
+**************************
+
+Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure
+python implementation for compatibility. To improve performance for high-throughput
+applications, kafka-python will use `crc32c` for optimized native code if installed.
+See https://pypi.org/project/crc32c/
+
 Protocol
 ********
 
@@ -150,11 +158,4 @@ for interacting with kafka brokers via the python repl. This is useful for
 testing, probing, and general experimentation. The protocol support is
 leveraged to enable a KafkaClient.check_version() method that
 probes a kafka broker and attempts to identify which version it is running
-(0.8.0 to 1.1+).
-
-Low-level
-*********
-
-Legacy support is maintained for low-level consumer and producer classes,
-SimpleConsumer and SimpleProducer. See
-<https://kafka-python.readthedocs.io/en/master/simple.html?highlight=SimpleProducer> for API details.
+(0.8.0 to 2.4+).
diff --git a/benchmarks/README b/benchmarks/README
diff --git a/benchmarks/README.md b/benchmarks/README.md
@@ -0,0 +1,4 @@
+The `record_batch_*` benchmarks in this section are written using
+``pyperf`` library, created by Victor Stinner. For more information on
+how to get reliable results of test runs please consult
+https://pyperf.readthedocs.io/en/latest/run_benchmark.html.
diff --git a/benchmarks/consumer_performance.py b/benchmarks/consumer_performance.py
@@ -29,7 +29,7 @@ def start_brokers(n):
     replicas = min(n, 3)
     print('-> {0} Brokers [{1} partitions / {2} replicas]'.format(n, partitions, replicas))
     brokers = [
-        KafkaFixture.instance(i, zk.host, zk.port, zk_chroot='',
+        KafkaFixture.instance(i, zk, zk_chroot='',
                               partitions=partitions, replicas=replicas)
         for i in range(n)
     ]
@@ -148,7 +148,7 @@ def get_args_parser():
         help='Topic for consumer test',
         default='kafka-python-benchmark-test')
     parser.add_argument(
-        '--num-records', type=long,
+        '--num-records', type=int,
         help='number of messages to consume',
         default=1000000)
     parser.add_argument(
@@ -157,7 +157,7 @@ def get_args_parser():
         default=100)
     parser.add_argument(
         '--consumer-config', type=str, nargs='+', default=(),
-        help='kafka consumer related configuaration properties like '
+        help='kafka consumer related configuration properties like '
              'bootstrap_servers,client_id etc..')
     parser.add_argument(
         '--fixture-compression', type=str,