RabbitMQ on Amazon ECS/Containerised with Persistent Storage

[PeteAUK]

I'm making good progress on getting RabbitMQ up and running on AWS. The plan is that we're going to have a second RabbitMQ installation on premises and the two will talk to each other using a Federation set up. What I do need though is some advice when it comes to the way RabbitMQ handles multiple containers and persistency.

Brief outline of the set up. I've created a ECS Service with a Load Balancer. This means that any messages to port 5672 get routed to an available Task (i.e. Container). The Service part ensures that I've X containers running and if I drop below that threshold it'll spin up new ones and add them to the load balancer. OK that's paraphrasing a little bit!

There are a number of things that may result in a container being dropped and a new one spun up - an unresponsive machine, me modifying something, zone taken offline due to a meteor etc. This runs the risk that messages in queues yet to be sent may get lost if they reside on local storage. In my head, the solution is a shared volume that can get connected to all the containers running. Within AWS this is an EFS volume - there is a bit of additional latency involved, but it's a volume you can attach to as many containers as you want to.

I've created a EFS volume that I've mounted to /var/lib/rabbitmq and this is where the issues start to occur.

Firstly the container builds the necessary files within the mnesia folder - this seems to take an age and creates a huge number of files - you're looking at 4/500mb within the quorum folder even though I have no quorum queues.

Secondly each container creates it's own set of these files using a "[email protected]" folder name. I misguidedly thought that everything would be put into a single location. I'm guessing this means that a brand new replacement container isn't going to pick up it's predecessors messages? It also means I'm going to end up with a massive amount of redundant data without any way of it being tidied up.

Lastly, I'm not even 100% sure that this is the "right" solution? The Load Balancer, auto spin up and Producer set up provides all of the necessary resilience should a RabbitMQ broker not be available when a message is produced, however I have no idea how to handle persistency of messages in queues.

Thanks in advance! Appreciate I may have missed something in the documentation - that'll teach me for trying to learn too many things all at once!

[mkuratczyk]

Autoscaling stateful services is not a simple problem in general, and certainly not with RabbitMQ right now. You can have a look at some of the problems involved in this thread: rabbitmq/cluster-operator#800 (comment)

[PeteAUK]

Thank you so much! I'd found I was just going round in circles trying to fathom this out and in short what I was trying to do isn't possible (or at least isn't without a huge amount of work). Going to have a rethink on the best set up for RabbitMQ on AWS - running it on containers within ECS (Fargate) appears to cause more issues than it solves!