rabbitmq streams compression

[wrobell]

Is it possible to compress stream data stored by RabbitMQ Streams on a disk?

[acogoluegnes]

Yes, it's possible, you need to use "sub-batching", that is squeezing several messages in a regular message entry of the publish frame.

Structure of a simple message entry (note the very first bit is 0)

Structure of a sub-batch entry (note the very first bit is 1)

You can use "plain" sub-batch entries, you basically squeeze plain messages in a single "slot". There are also different supported compression algorithms.

This is a pure client-side feature, the broker does not compress and decompress, the client libraries do. This implies that the client library must be able to read sub-batch entries when consuming from a stream, a possibly decompress them (an entry in a chunk can be a sub-batch entry).

The Java client has a setting for sub-batching. Note message de-duplication is guaranteed when using sub-batching.