[Questions] Technique for introducing alternate exchanges to existing topologies

[HakShak]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

3.13.7

Erlang version used

26.2.x

Operating system (distribution) used

docker image

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

Steps to deploy RabbitMQ cluster

tested on single docker container

Steps to reproduce the behavior in question

We are looking to introduce alternate exchanges to our existing topologies. Since we can't modify existing exchanges, then we need to create new exchanges with the alternate exchange configured. This makes sense, so we did the following:

---
title: exchange topology
---
flowchart LR
   a[A]
   b[B]
   bAlt[AltB]

   a-- Exchange to exchange binding with # ---b
   b-- alternate exchange ---bAlt

Loading

This works great because the old publishers (A) and the new publishers (B) will both flow to exchange B and Rabbit dedupes any consumers with multiple bindings to both exchanges.

However, we realized that new publishers wouldn't get messages back to old consumers that haven't be updated, so we added another exchange to exchange binding:

flowchart LR
   a[A]
   b[B]
   bAlt[AltB]

   a-- Exchange to exchange binding with # ---b
   b-- Exchange to exchange binding with # ---a
   b-- alternate exchange ---bAlt

Loading

Surprisingly this works for routable message delivery (feels dirty). Old and new publishers' messages can be seen by old consumers.

However, this seems to break the alternate exchange configuration as any unroutable messages are just that, unroutable and don't make it to AltB.

Are we thinking about this wrong? (bidirectional e2e binding for migration, or is there a better way to phase out the old exchange?)
Or is this a bug with alternate exchanges that gets confused when the exchanges turn into memes?

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

Application code

# relevant messaging-related parts of your code go here

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed

[michaelklishin]

@HakShak there is nothing wrong with the strategy you are exploring. You can temporarily have two bindings between entities A (an exchange) and its routing destination (a queue or another exchange), and if the destination is another exchange, it can in turn be bound to one or more existing queues.

In fact, this ability to add more "routes" between A and B (or C via B, or whatever) has no downsides
other than the complexity, at least with the built-in exchange types.

rabbitmq_consistent_hashing exchange requires one binding between an exchange and a queue (or another exchange) because bindings are "weights" (buckets in the hashing ring) but that's an exchange type-specific restriction.

The only thing I have to add is that RabbitMQ does not deduplicate your messages in any way. It simply aggregates N routes to compute the final list of destinations (queues or other exchanges). If you have N bindings between A and B, that won't create duplicates but not because RabbitMQ tries to deduplicate messages. What it "deduplicates" (treats as a set with unique values) is the list of destinations.

[HakShak]

Thanks for the clarification!

Is it expected that alternate exchanges don't work if B had an alternate exchange and A doesn't, but both A and B are bound to each other?

(And no queues for that routing key bound to either exchange)

[michaelklishin]

Alternate exchanges are used only when the initial routing step produces an empty set of destinations. If that's not the case, then RabbitMQ quite logically assumes that alternatives are not necessary because a regular route exists.

[HakShak]

So if exchanges A and B are bound to each other then they always have a destination?

I ask specifically because after binding them to each other and publishing I get a "message not routable" popup from the admin UI.

[michaelklishin]

Simply binding A to B is not enough, the exchange type and routing key must produce a non-empty set of destinations. But for, say, a fanout exchange, having a binding will be enough (as I am sure you understand).

[michaelklishin]

Our own docs slightly correct my statements but match your observations.

It's a philosophical question whether the server should return the message as unroutable if it could not be routed using its original input and only the AE. I think both answers can be justified and as you can see, RabbitMQ does currently basic.return the message to the publisher.

In any case, for your migration goal, having duplicate bindings or an exchange "routing layer" via E2Es (exchange-to-exchange bindings) is fine and the right approach. Then over time you will delete some bindings you no longer need.

[HakShak]

Thanks again Michael! The double bindings eventually atrophy and get cleaned up when all publishers and consumers move over, that makes sense.

Maybe this is another thread, but I feel like failed explaining the weirder bit here. I want to rephrase this bit here:

It's a philosophical question whether the server should return the message as unroutable if it could not be routed using its original input and only the AE.

Say we have these exchanges:

This topology works really well from the sense that if old publishers are using v1 and new publishers are using v2, both old and new consumers get the messages. However, when this "cyclic bind" is in place and I publish a message to v1 or v2 exchanges I get this:

There are no deliveries in any queues. So the response is accurate, which is good, but it removes a nice tool of this "cyclic binding" between the exchanges which prevents us from needing to be more careful with consumers and publishers.

Maybe that's what you mean is the philosophical part?

If I remove the "cyclic binding":

• publishing an unmatched routing key to either v1 or v2 the message is delivered to v2-alt-queue, which makes sense. 👍
• publishing a matched routing key to v1 that isn't matched in v2 means the message is delivered to both v1-queue and v2-alt-queue. Counter-intuitive but makes sense?

The latter case is the reason to try out the "cyclic binding", but again, the alternate exchange doesn't seem to function with "cyclic bindings".

But maybe that part of the migration? Moving each binding over instead of pushing a wildcard from v1 to v2?

[michaelklishin]

I did not mean anything beyond "should an AE be a silent retry or a retry after an explicit error."

Cycle detection (which routing does not actually perform(, dead-lettering (which does perform cycle detection), and so on are not particularly relevant to the key question of migration.

[michaelklishin]

As an alternative, you can use two concurrent topologies: the original (OX) and new (NX) exchanges, where NX will be identical to OX except for the AE setting.

OX and NX then will presumably route different messages to the same sets of queues and your publishers need to publish to one or the other, and can migrate over time.

Whether this would be a more practical option than two sets of bindings plus E2Es, I am not in a position to tell.