Skip to content

Cleanup install instructions for optional libs #2139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 15, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ documentation, please see readthedocs and/or python's inline help.

>>> pip install kafka-python


KafkaConsumer
*************

Expand Down Expand Up @@ -78,6 +79,7 @@ that expose basic message attributes: topic, partition, offset, key, and value:
>>> # Get consumer metrics
>>> metrics = consumer.metrics()


KafkaProducer
*************

Expand Down Expand Up @@ -124,6 +126,7 @@ for more details.
>>> # Get producer performance metrics
>>> metrics = producer.metrics()


Thread safety
*************

Expand All @@ -133,22 +136,30 @@ KafkaConsumer which cannot.
While it is possible to use the KafkaConsumer in a thread-local manner,
multiprocessing is recommended.


Compression
***********

kafka-python supports gzip compression/decompression natively. To produce or consume lz4
compressed messages, you should install python-lz4 (pip install lz4).
To enable snappy compression/decompression install python-snappy (also requires snappy library).
See <https://kafka-python.readthedocs.io/en/master/install.html#optional-snappy-install>
for more information.
kafka-python supports the following compression formats:

- gzip
- LZ4
- Snappy
- Zstandard (zstd)

gzip is supported natively, the others require installing additional libraries.
See <https://kafka-python.readthedocs.io/en/master/install.html> for more information.


Optimized CRC32 Validation
**************************

Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure
python implementation for compatibility. To improve performance for high-throughput
applications, kafka-python will use `crc32c` for optimized native code if installed.
See https://pypi.org/project/crc32c/
See <https://kafka-python.readthedocs.io/en/master/install.html> for installation instructions.
See https://pypi.org/project/crc32c/ for details on the underlying crc32c lib.


Protocol
********
Expand Down
25 changes: 20 additions & 5 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ documentation, please see readthedocs and/or python's inline help.

>>> pip install kafka-python


KafkaConsumer
*************

Expand Down Expand Up @@ -122,12 +123,26 @@ multiprocessing is recommended.
Compression
***********

kafka-python supports multiple compression types:
kafka-python supports the following compression formats:

- gzip
- LZ4
- Snappy
- Zstandard (zstd)

gzip is supported natively, the others require installing additional libraries.
See `Install <install.html>`_ for more information.


Optimized CRC32 Validation
**************************

Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure
python implementation for compatibility. To improve performance for high-throughput
applications, kafka-python will use `crc32c` for optimized native code if installed.
See `Install <install.html>`_ for installation instructions and
https://pypi.org/project/crc32c/ for details on the underlying crc32c lib.

- gzip : supported natively
- lz4 : requires `python-lz4 <https://pypi.org/project/lz4/>`_ installed
- snappy : requires the `python-snappy <https://pypi.org/project/python-snappy/>`_ package (which requires the snappy C library)
- zstd : requires the `python-zstandard <https://github.com/indygreg/python-zstandard>`_ package installed

Protocol
********
Expand Down
40 changes: 20 additions & 20 deletions docs/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,33 @@ Bleeding-Edge
pip install ./kafka-python


Optional LZ4 install
Optional crc32c install
***********************
Highly recommended if you are using Kafka 11+ brokers. For those `kafka-python`
uses a new message protocol version, that requires calculation of `crc32c`,
which differs from the `zlib.crc32` hash implementation. By default `kafka-python`
calculates it in pure python, which is quite slow. To speed it up we optionally
support https://pypi.python.org/pypi/crc32c package if it's installed.

.. code:: bash

pip install 'kafka-python[crc32c]'


Optional ZSTD install
********************

To enable LZ4 compression/decompression, install python-lz4:
To enable ZSTD compression/decompression, install python-zstandard:

>>> pip install lz4
>>> pip install 'kafka-python[zstd]'


Optional crc32c install
Optional LZ4 install
********************

To enable optimized CRC32 checksum validation, install crc32c:
To enable LZ4 compression/decompression, install python-lz4:

>>> pip install crc32c
>>> pip install 'kafka-python[lz4]'


Optional Snappy install
Expand Down Expand Up @@ -77,17 +90,4 @@ Install the `python-snappy` module

.. code:: bash

pip install python-snappy


Optional crc32c install
***********************
Highly recommended if you are using Kafka 11+ brokers. For those `kafka-python`
uses a new message protocol version, that requires calculation of `crc32c`,
which differs from `zlib.crc32` hash implementation. By default `kafka-python`
calculates it in pure python, which is quite slow. To speed it up we optionally
support https://pypi.python.org/pypi/crc32c package if it's installed.

.. code:: bash

pip install crc32c
pip install 'kafka-python[snappy]'