Skip to content

Commit 720edb1

Browse files
BaisangBaisang
authored
Merge pull request #67 from Yelp/u/baisang/KAFKA-22203
Merge upstream 1.4.6
2 parents 22a4a8a + 65c02b7 commit 720edb1

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+1025
-1222
lines changed

‎.travis.yml

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,38 @@
11
language: python
22

3+
dist: xenial
4+
35
python:
46
- 2.7
57
- 3.4
68
- 3.5
79
- 3.6
8-
- pypy
10+
- pypy2.7-6.0
911

1012
env:
1113
- KAFKA_VERSION=1.1.0
1214

13-
sudo: false
14-
1515
addons:
1616
apt:
1717
packages:
1818
- libsnappy-dev
19+
- openjdk-8-jdk
1920

2021
cache:
2122
directories:
2223
- $HOME/.cache/pip
2324
- servers/
2425

2526
before_install:
27+
- source travis_java_install.sh
2628
- ./build_integration.sh
2729

2830
install:
2931
- pip install tox coveralls
3032
- pip install .
3133

3234
script:
33-
- tox -i https://pypi.python.org/simple -e `if [ "$TRAVIS_PYTHON_VERSION" == "pypy" ]; then echo pypy; else echo py${TRAVIS_PYTHON_VERSION/./}; fi`
35+
- tox -i https://pypi.python.org/simple -e `if [ "$TRAVIS_PYTHON_VERSION" == "pypy2.7-6.0" ]; then echo pypy; else echo py${TRAVIS_PYTHON_VERSION/./}; fi`
3436

3537
after_success:
3638
- coveralls

‎CHANGES.md

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,102 @@
1+
# 1.4.6.post1 (Jun 4, 2019)
2+
This release merges in changes from 1.4.5 and 1.4.6 upstream.
3+
The only key differences are we focus on py35 still instead of py36/py37, and
4+
we only build for versions 0.10.2.2 and 1.1.0 and 1.1.1
5+
6+
# 1.4.6 (Apr 2, 2019)
7+
8+
This is a patch release primarily focused on bugs related to concurrency,
9+
SSL connections and testing, and SASL authentication:
10+
11+
12+
Client Concurrency Issues (Race Conditions / Deadlocks)
13+
14+
* Fix race condition in `protocol.send_bytes` (isamaru / PR #1752)
15+
* Do not call `state_change_callback` with lock (dpkp / PR #1775)
16+
* Additional BrokerConnection locks to synchronize protocol/IFR state (dpkp / PR #1768)
17+
* Send pending requests before waiting for responses (dpkp / PR #1762)
18+
* Avoid race condition on `client._conns` in send() (dpkp / PR #1772)
19+
* Hold lock during `client.check_version` (dpkp / PR #1771)
20+
21+
Producer Wakeup / TimeoutError
22+
23+
* Dont wakeup during `maybe_refresh_metadata` -- it is only called by poll() (dpkp / PR #1769)
24+
* Dont do client wakeup when sending from sender thread (dpkp / PR #1761)
25+
26+
SSL - Python3.7 Support / Bootstrap Hostname Verification / Testing
27+
28+
* Wrap SSL sockets after connecting for python3.7 compatibility (dpkp / PR #1754)
29+
* Allow configuration of SSL Ciphers (dpkp / PR #1755)
30+
* Maintain shadow cluster metadata for bootstrapping (dpkp / PR #1753)
31+
* Generate SSL certificates for local testing (dpkp / PR #1756)
32+
* Rename ssl.keystore.location and ssl.truststore.location config files (dpkp)
33+
* Reset reconnect backoff on SSL connection (dpkp / PR #1777)
34+
35+
SASL - OAuthBearer support / api version bugfix
36+
37+
* Fix 0.8.2 protocol quick detection / fix SASL version check (dpkp / PR #1763)
38+
* Update sasl configuration docstrings to include supported mechanisms (dpkp)
39+
* Support SASL OAuthBearer Authentication (pt2pham / PR #1750)
40+
41+
Miscellaneous Bugfixes
42+
43+
* Dont force metadata refresh when closing unneeded bootstrap connections (dpkp / PR #1773)
44+
* Fix possible AttributeError during conn._close_socket (dpkp / PR #1776)
45+
* Return connection state explicitly after close in connect() (dpkp / PR #1778)
46+
* Fix flaky conn tests that use time.time (dpkp / PR #1758)
47+
* Add py to requirements-dev (dpkp)
48+
* Fixups to benchmark scripts for py3 / new KafkaFixture interface (dpkp)
49+
50+
51+
# 1.4.5 (Mar 14, 2019)
52+
53+
This release is primarily focused on addressing lock contention
54+
and other coordination issues between the KafkaConsumer and the
55+
background heartbeat thread that was introduced in the 1.4 release.
56+
57+
Consumer
58+
* connections_max_idle_ms must be larger than request_timeout_ms (jeffwidman / PR #1688)
59+
* Avoid race condition during close() / join heartbeat thread (dpkp / PR #1735)
60+
* Use last offset from fetch v4 if available to avoid getting stuck in compacted topic (keithks / PR #1724)
61+
* Synchronize puts to KafkaConsumer protocol buffer during async sends (dpkp / PR #1733)
62+
* Improve KafkaConsumer join group / only enable Heartbeat Thread during stable group (dpkp / PR #1695)
63+
* Remove unused `skip_double_compressed_messages` (jeffwidman / PR #1677)
64+
* Fix commit_offsets_async() callback (Faqa / PR #1712)
65+
66+
Client
67+
* Retry bootstrapping after backoff when necessary (dpkp / PR #1736)
68+
* Recheck connecting nodes sooner when refreshing metadata (dpkp / PR #1737)
69+
* Avoid probing broker versions twice on newer brokers (dpkp / PR #1738)
70+
* Move all network connections and writes to KafkaClient.poll() (dpkp / PR #1729)
71+
* Do not require client lock for read-only operations (dpkp / PR #1730)
72+
* Timeout all unconnected conns (incl SSL) after request_timeout_ms (dpkp / PR #1696)
73+
74+
Admin Client
75+
* Fix AttributeError in response topic error codes checking (jeffwidman)
76+
* Fix response error checking in KafkaAdminClient send_to_controller (jeffwidman)
77+
* Fix NotControllerError check (jeffwidman)
78+
79+
Core/Protocol
80+
* Fix default protocol parser version / 0.8.2 version probe (dpkp / PR #1740)
81+
* Make NotEnoughReplicasError/NotEnoughReplicasAfterAppendError retriable (le-linh / PR #1722)
82+
83+
Bugfixes
84+
* Use copy() in metrics() to avoid thread safety issues (emeric254 / PR #1682)
85+
86+
Test Infrastructure
87+
* Mock dns lookups in test_conn (dpkp / PR #1739)
88+
* Use test.fixtures.version not test.conftest.version to avoid warnings (dpkp / PR #1731)
89+
* Fix test_legacy_correct_metadata_response on x86 arch (stanislavlevin / PR #1718)
90+
* Travis CI: 'sudo' tag is now deprecated in Travis (cclauss / PR #1698)
91+
* Use Popen.communicate() instead of Popen.wait() (Baisang / PR #1689)
92+
93+
Compatibility
94+
* Catch thrown OSError by python 3.7 when creating a connection (danjo133 / PR #1694)
95+
* Update travis test coverage: 2.7, 3.4, 3.7, pypy2.7 (jeffwidman, dpkp / PR #1614)
96+
* Drop dependency on sphinxcontrib-napoleon (stanislavlevin / PR #1715)
97+
* Remove unused import from kafka/producer/record_accumulator.py (jeffwidman / PR #1705)
98+
* Fix SSL connection testing in Python 3.7 (seanthegeek, silentben / PR #1669)
99+
1100
# 1.4.4.post1 (Jan 10, 2019)
2101

3102
* Added proc.communicate() patch that got merged upstream to ensure tests don't deadlock

‎Dockerfile

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,16 @@
11
FROM ubuntu:xenial
22
ENV DEBIAN_FRONTEND=noninteractive
33

4-
RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" >> /etc/apt/sources.list
54
RUN echo "deb http://ppa.launchpad.net/fkrull/deadsnakes/ubuntu precise main" >> /etc/apt/sources.list
6-
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5BB92C09DB82666C C2518248EEA14886
7-
RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections
5+
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5BB92C09DB82666C
86
RUN apt-get update && apt-get install -y python2.7-dev \
97
python3.5-dev \
108
python-pkg-resources \
119
python-setuptools \
1210
python-virtualenv \
1311
libsnappy-dev \
14-
oracle-java8-installer \
12+
locales \
13+
openjdk-8-jdk \
1514
wget\
1615
g++ \
1716
ca-certificates \
@@ -25,7 +24,7 @@ RUN ln -s $PWD/pypy2-v5.8.0-linux64/bin/pypy /usr/local/bin/pypy
2524

2625
RUN /usr/sbin/locale-gen en_US.UTF-8
2726
ENV LANG en_US.UTF-8
28-
ENV JAVA_HOME="/usr/lib/jvm/java-8-oracle"
27+
ENV JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-amd64"
2928
ENV PATH="$PATH:$JAVA_HOME/bin"
3029

3130
COPY servers /work/servers

‎Makefile

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ all: test itest
1111
test:
1212
tox -e py27
1313
tox -e py35
14+
tox -e py36
1415

1516
unit_test_docker:
1617
docker build -t kafka_python_test .
@@ -40,8 +41,8 @@ servers/$(KAFKA_VERSION)/kafka-bin:
4041
build-integration: servers/$(KAFKA_VERSION)/kafka-bin
4142

4243
# Test and produce coverage using tox. This is the same as is run on Travis
43-
test36: build-integration
44-
KAFKA_VERSION=$(KAFKA_VERSION) SCALA_VERSION=$(SCALA_VERSION) tox -e py36 -- $(FLAGS)
44+
test35: build-integration
45+
KAFKA_VERSION=$(KAFKA_VERSION) SCALA_VERSION=$(SCALA_VERSION) tox -e py35 -- $(FLAGS)
4546

4647
test27: build-integration
4748
KAFKA_VERSION=$(KAFKA_VERSION) SCALA_VERSION=$(SCALA_VERSION) tox -e py27 -- $(FLAGS)
@@ -82,4 +83,8 @@ clean:
8283
find . -name '__pycache__' -delete
8384
docker rmi -f kafka_python_test
8485

85-
.PHONY: all test36 test27 test-local cov-local clean doc docs test
86+
doc:
87+
make -C docs html
88+
@echo "open file://`pwd`/docs/_build/html/index.html"
89+
90+
.PHONY: all test test35 test27 test-local cov-local clean doc

‎benchmarks/consumer_performance.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ def start_brokers(n):
2929
replicas = min(n, 3)
3030
print('-> {0} Brokers [{1} partitions / {2} replicas]'.format(n, partitions, replicas))
3131
brokers = [
32-
KafkaFixture.instance(i, zk.host, zk.port, zk_chroot='',
32+
KafkaFixture.instance(i, zk, zk_chroot='',
3333
partitions=partitions, replicas=replicas)
3434
for i in range(n)
3535
]
@@ -148,7 +148,7 @@ def get_args_parser():
148148
help='Topic for consumer test',
149149
default='kafka-python-benchmark-test')
150150
parser.add_argument(
151-
'--num-records', type=long,
151+
'--num-records', type=int,
152152
help='number of messages to consume',
153153
default=1000000)
154154
parser.add_argument(

‎benchmarks/producer_performance.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ def start_brokers(n):
2626
replicas = min(n, 3)
2727
print('-> {0} Brokers [{1} partitions / {2} replicas]'.format(n, partitions, replicas))
2828
brokers = [
29-
KafkaFixture.instance(i, zk.host, zk.port, zk_chroot='',
29+
KafkaFixture.instance(i, zk, zk_chroot='',
3030
partitions=partitions, replicas=replicas)
3131
for i in range(n)
3232
]
@@ -130,7 +130,7 @@ def get_args_parser():
130130
help='Topic name for test',
131131
default='kafka-python-benchmark-test')
132132
parser.add_argument(
133-
'--num-records', type=long,
133+
'--num-records', type=int,
134134
help='number of messages to produce',
135135
default=1000000)
136136
parser.add_argument(

‎docs/changelog.rst

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,109 @@ Logging / Error Messages
105105
* Log Heartbeat thread start / close for debugging (dpkp)
106106

107107

108+
1.4.6 (Apr 2, 2019)
109+
###################
110+
111+
This is a patch release primarily focused on bugs related to concurrency,
112+
SSL connections and testing, and SASL authentication:
113+
114+
Client Concurrency Issues (Race Conditions / Deadlocks)
115+
-------------------------------------------------------
116+
* Fix race condition in `protocol.send_bytes` (isamaru / PR #1752)
117+
* Do not call `state_change_callback` with lock (dpkp / PR #1775)
118+
* Additional BrokerConnection locks to synchronize protocol/IFR state (dpkp / PR #1768)
119+
* Send pending requests before waiting for responses (dpkp / PR #1762)
120+
* Avoid race condition on `client._conns` in send() (dpkp / PR #1772)
121+
* Hold lock during `client.check_version` (dpkp / PR #1771)
122+
123+
Producer Wakeup / TimeoutError
124+
------------------------------
125+
* Dont wakeup during `maybe_refresh_metadata` -- it is only called by poll() (dpkp / PR #1769)
126+
* Dont do client wakeup when sending from sender thread (dpkp / PR #1761)
127+
128+
SSL - Python3.7 Support / Bootstrap Hostname Verification / Testing
129+
-------------------------------------------------------------------
130+
* Wrap SSL sockets after connecting for python3.7 compatibility (dpkp / PR #1754)
131+
* Allow configuration of SSL Ciphers (dpkp / PR #1755)
132+
* Maintain shadow cluster metadata for bootstrapping (dpkp / PR #1753)
133+
* Generate SSL certificates for local testing (dpkp / PR #1756)
134+
* Rename ssl.keystore.location and ssl.truststore.location config files (dpkp)
135+
* Reset reconnect backoff on SSL connection (dpkp / PR #1777)
136+
137+
SASL - OAuthBearer support / api version bugfix
138+
-----------------------------------------------
139+
* Fix 0.8.2 protocol quick detection / fix SASL version check (dpkp / PR #1763)
140+
* Update sasl configuration docstrings to include supported mechanisms (dpkp)
141+
* Support SASL OAuthBearer Authentication (pt2pham / PR #1750)
142+
143+
Miscellaneous Bugfixes
144+
----------------------
145+
* Dont force metadata refresh when closing unneeded bootstrap connections (dpkp / PR #1773)
146+
* Fix possible AttributeError during conn._close_socket (dpkp / PR #1776)
147+
* Return connection state explicitly after close in connect() (dpkp / PR #1778)
148+
* Fix flaky conn tests that use time.time (dpkp / PR #1758)
149+
* Add py to requirements-dev (dpkp)
150+
* Fixups to benchmark scripts for py3 / new KafkaFixture interface (dpkp)
151+
152+
153+
1.4.5 (Mar 14, 2019)
154+
####################
155+
156+
This release is primarily focused on addressing lock contention
157+
and other coordination issues between the KafkaConsumer and the
158+
background heartbeat thread that was introduced in the 1.4 release.
159+
160+
Consumer
161+
--------
162+
* connections_max_idle_ms must be larger than request_timeout_ms (jeffwidman / PR #1688)
163+
* Avoid race condition during close() / join heartbeat thread (dpkp / PR #1735)
164+
* Use last offset from fetch v4 if available to avoid getting stuck in compacted topic (keithks / PR #1724)
165+
* Synchronize puts to KafkaConsumer protocol buffer during async sends (dpkp / PR #1733)
166+
* Improve KafkaConsumer join group / only enable Heartbeat Thread during stable group (dpkp / PR #1695)
167+
* Remove unused `skip_double_compressed_messages` (jeffwidman / PR #1677)
168+
* Fix commit_offsets_async() callback (Faqa / PR #1712)
169+
170+
Client
171+
------
172+
* Retry bootstrapping after backoff when necessary (dpkp / PR #1736)
173+
* Recheck connecting nodes sooner when refreshing metadata (dpkp / PR #1737)
174+
* Avoid probing broker versions twice on newer brokers (dpkp / PR #1738)
175+
* Move all network connections and writes to KafkaClient.poll() (dpkp / PR #1729)
176+
* Do not require client lock for read-only operations (dpkp / PR #1730)
177+
* Timeout all unconnected conns (incl SSL) after request_timeout_ms (dpkp / PR #1696)
178+
179+
Admin Client
180+
------------
181+
* Fix AttributeError in response topic error codes checking (jeffwidman)
182+
* Fix response error checking in KafkaAdminClient send_to_controller (jeffwidman)
183+
* Fix NotControllerError check (jeffwidman)
184+
185+
Core/Protocol
186+
-------------
187+
* Fix default protocol parser version / 0.8.2 version probe (dpkp / PR #1740)
188+
* Make NotEnoughReplicasError/NotEnoughReplicasAfterAppendError retriable (le-linh / PR #1722)
189+
190+
Bugfixes
191+
--------
192+
* Use copy() in metrics() to avoid thread safety issues (emeric254 / PR #1682)
193+
194+
Test Infrastructure
195+
-------------------
196+
* Mock dns lookups in test_conn (dpkp / PR #1739)
197+
* Use test.fixtures.version not test.conftest.version to avoid warnings (dpkp / PR #1731)
198+
* Fix test_legacy_correct_metadata_response on x86 arch (stanislavlevin / PR #1718)
199+
* Travis CI: 'sudo' tag is now deprecated in Travis (cclauss / PR #1698)
200+
* Use Popen.communicate() instead of Popen.wait() (Baisang / PR #1689)
201+
202+
Compatibility
203+
-------------
204+
* Catch thrown OSError by python 3.7 when creating a connection (danjo133 / PR #1694)
205+
* Update travis test coverage: 2.7, 3.4, 3.7, pypy2.7 (jeffwidman, dpkp / PR #1614)
206+
* Drop dependency on sphinxcontrib-napoleon (stanislavlevin / PR #1715)
207+
* Remove unused import from kafka/producer/record_accumulator.py (jeffwidman / PR #1705)
208+
* Fix SSL connection testing in Python 3.7 (seanthegeek, silentben / PR #1669)
209+
210+
108211
1.4.4 (Nov 20, 2018)
109212
##########
110213

‎docs/compatibility.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,6 @@ through 0.8.0 . kafka-python is not compatible with the 0.8.2-beta release.
1212
Because the kafka server protocol is backwards compatible, kafka-python is
1313
expected to work with newer broker releases as well (2.0+).
1414

15-
kafka-python is tested on python 2.7, 3.4, 3.5, 3.6 and pypy.
15+
kafka-python is tested on python 2.7, 3.4, 3.7, and pypy2.7.
1616

1717
Builds and tests via Travis-CI. See https://travis-ci.org/dpkp/kafka-python

‎docs/requirements.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
sphinx
2-
sphinxcontrib-napoleon
32
sphinx_rtd_theme
43

54
# Install kafka-python in editable mode

0 commit comments

Comments
 (0)