Skip to content

Follow AMQP spec for durable field #13918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 20, 2025
Merged

Follow AMQP spec for durable field #13918

merged 2 commits into from
May 20, 2025

Conversation

ansd
Copy link
Member

@ansd ansd commented May 19, 2025

The AMQP spec defines:

<field name="durable" type="boolean" default="false"/>

RabbitMQ 4.0 and 4.1 interpret the durable field as true if not set.
The idea was to favour safety over performance.
This complies with the AMQP spec because the spec allows other target or
node specific defaults for the durable field:

If the header section is omitted the receiver MUST assume the appropriate
default values (or the meaning implied by no value being set) for the fields
within the header unless other target or node specific defaults have otherwise
been set.

However, some client libraries completely omit the header section if the
app expliclity sets durable=false. This complies with the spec, but it
means that RabbitMQ cannot diffentiate between "client app forgot to set
the durable field" vs "client lib opted in for an optimisation omitting
the header section".

This is problematic with JMS message selectors where JMS apps can filter
on JMSDeliveryMode. To be able to correctly filter on JMSDeliveryMode,
RabbitMQ needs to know whether the JMS app sent the message as
PERSISTENT or NON_PERSISTENT.

Rather than relying on client libs to always send the header section
including the durable field, this commit makes RabbitMQ comply with the
default value for durable in the AMQP spec.
Some client lib maintainers accepted to send the header section, while
other maintainers refused to do so:
Azure/go-amqp#330
https://issues.apache.org/jira/browse/QPIDJMS-608

Likely the AMQP spec was designed to omit the header section when
performance is important, as is the case with durable=false. Omitting
the header section means saving a few bytes per message on the wire and
some marshalling and unmarshalling overhead on both client and server.

Therefore, it's better to push the "safe by default" behaviour from the broker
back to the client libs. Client libs should send messages as durable by
default unless the client app expliclity opts in to send messages as
non-durable. This is also what JMS does: By default JMS apps send
messages as PERSISTENT:

The message producer's default delivery mode is PERSISTENT.

Therefore, this commit also makes the AMQP Erlang client send messages as
durable, by default.

This commit will apply to RabbitMQ 4.2.
It's arguably not a breaking change because in RabbitMQ, message durability
is actually more determined by the queue type the message is sent to rather than the
durable field of the message:

  • Quroum queues and streams store messages durably (fsync or replicate)
    no matter what the durable field is
  • MQTT QoS 0 queues hold messages in memory no matter what the
    durable field is
  • Classic queues do not fsync even if the durable field is set to true

In addition, the RabbitMQ AMQP Java library introduced in RabbitMQ 4.0 sends messages with
durable=true:
https://github.com/rabbitmq/rabbitmq-amqp-java-client/blob/53e3dd6abbcbce8ca4f2257da56b314786b037cc/src/main/java/com/rabbitmq/client/amqp/impl/AmqpPublisher.java#L91

Nevertheless I added a breaking change section to the 4.2 release notes to be extra clear and transparent.

The tests for selecting messages by JMSDeliveryMode relying on the
behaviour in this commit can be found on the jms branch. They got green after rebasing onto this branch without setting the priority field.

@ansd ansd force-pushed the amqp-durable branch 3 times, most recently from 85d0c5b to be064ff Compare May 19, 2025 16:57
@ansd ansd added this to the 4.2.0 milestone May 19, 2025
@ansd ansd marked this pull request as ready for review May 20, 2025 05:51
ansd added a commit that referenced this pull request May 20, 2025
because this commit is rebased onto branch amqp-durable
containing #13918
ansd added 2 commits May 20, 2025 08:32
The AMQP spec defines:
```
<field name="durable" type="boolean" default="false"/>
```

RabbitMQ 4.0 and 4.1 interpret the durable field as true if not set.
The idea was to favour safety over performance.
This complies with the AMQP spec because the spec allows other target or
node specific defaults for the durable field:
> If the header section is omitted the receiver MUST assume the appropriate
> default values (or the meaning implied by no value being set) for the fields
> within the header unless other target or node specific defaults have otherwise
> been set.

However, some client libraries completely omit the header section if the
app expliclity sets durable=false. This complies with the spec, but it
means that RabbitMQ cannot diffentiate between "client app forgot to set
the durable field" vs "client lib opted in for an optimisation omitting
the header section".

This is problematic with JMS message selectors where JMS apps can filter
on JMSDeliveryMode. To be able to correctly filter on JMSDeliveryMode,
RabbitMQ needs to know whether the JMS app sent the message as
PERSISTENT or NON_PERSISTENT.

Rather than relying on client libs to always send the header section
including the durable field, this commit makes RabbitMQ comply with the
default value for durable in the AMQP spec.
Some client lib maintainers accepted to send the header section, while
other maintainers refused to do so:
Azure/go-amqp#330
https://issues.apache.org/jira/browse/QPIDJMS-608

Likely the AMQP spec was designed to omit the header section when
performance is important, as is the case with durable=false. Omitting
the header section means saving a few bytes per message on the wire and
some marshalling and unmarshalling overhead on both client and server.

Therefore, it's better to push the "safe by default" behaviour from the broker
back to the client libs. Client libs should send messages as durable by
default unless the client app expliclity opts in to send messages as
non-durable. This is also what JMS does: By default JMS apps send
messages as PERSISTENT:
> The message producer's default delivery mode is PERSISTENT.

Therefore, this commit also makes the AMQP Erlang client send messages as
durable, by default.

This commit will apply to RabbitMQ 4.2.
It's arguably not a breaking change because in RabbitMQ, message durability
is actually more determined by the queue type the message is sent to rather than the
durable field of the message:
* Quroum queues and streams store messages durably (fsync or replicate)
  no matter what the durable field is
* MQTT QoS 0 queues hold messages in memory no matter what the
  durable field is
* Classic queues do not fsync even if the durable field is set to true

In addition, the RabbitMQ AMQP Java library introduced in RabbitMQ 4.0 sends messages with
durable=true:
https://github.com/rabbitmq/rabbitmq-amqp-java-client/blob/53e3dd6abbcbce8ca4f2257da56b314786b037cc/src/main/java/com/rabbitmq/client/amqp/impl/AmqpPublisher.java#L91

The tests for selecting messages by JMSDeliveryMode relying on the
behaviour in this commit can be found on the `jms` branch.
@ansd ansd added the breaking-change Includes changes that are potentially breaking compared to prior versions label May 20, 2025
@ansd ansd merged commit 8546bcf into main May 20, 2025
273 of 274 checks passed
@ansd ansd deleted the amqp-durable branch May 20, 2025 11:20
ansd added a commit that referenced this pull request May 20, 2025
because this commit is rebased onto branch amqp-durable
containing #13918
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-go-client that referenced this pull request May 20, 2025
releaded to rabbitmq/rabbitmq-server#13918

Signed-off-by: Gabriele Santomaggio <[email protected]>
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-go-client that referenced this pull request May 20, 2025
releaded to rabbitmq/rabbitmq-server#13918

Signed-off-by: Gabriele Santomaggio <[email protected]>
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-go-client that referenced this pull request May 20, 2025
* set message durable as default
* related to rabbitmq/rabbitmq-server#13918
---------

Signed-off-by: Gabriele Santomaggio <[email protected]>
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-dotnet-client that referenced this pull request May 20, 2025
releaded to rabbitmq/rabbitmq-server#13918

Signed-off-by: Gabriele Santomaggio <[email protected]>
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-dotnet-client that referenced this pull request May 20, 2025
releaded to rabbitmq/rabbitmq-server#13918

Signed-off-by: Gabriele Santomaggio <[email protected]>
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-dotnet-client that referenced this pull request May 20, 2025
releaded to rabbitmq/rabbitmq-server#13918

Signed-off-by: Gabriele Santomaggio <[email protected]>
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-dotnet-client that referenced this pull request May 20, 2025
* set message durable as default
* related to rabbitmq/rabbitmq-server#13918
* The user can always decide to set durable=false btw
---------
Signed-off-by: Gabriele Santomaggio <[email protected]>
ansd added a commit that referenced this pull request May 21, 2025
because this commit is rebased onto branch amqp-durable
containing #13918
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-python-client that referenced this pull request May 21, 2025
releaded to rabbitmq/rabbitmq-server#13918

Signed-off-by: Gabriele Santomaggio <[email protected]>
ansd added a commit that referenced this pull request May 21, 2025
because this commit is rebased onto branch amqp-durable
containing #13918
ansd added a commit that referenced this pull request May 22, 2025
because this commit is rebased onto branch amqp-durable
containing #13918
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-python-client that referenced this pull request May 26, 2025
releaded to rabbitmq/rabbitmq-server#13918

Signed-off-by: Gabriele Santomaggio <[email protected]>
Gsantomaggio added a commit to rabbitmq/rabbitmq-amqp-python-client that referenced this pull request May 26, 2025
releaded to rabbitmq/rabbitmq-server#13918

Signed-off-by: Gabriele Santomaggio <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change Includes changes that are potentially breaking compared to prior versions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants