[Questions] How to do Django -> RabbitMQ > Multiple Celery Workers, each inside a docker container?

[djaniel]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.2

Erlang version used

26.2.x

Operating system (distribution) used

Centos

How is RabbitMQ deployed?

Debian package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

Follow the documentation available online.

Steps to reproduce the behavior in question

Launch all services on the docker compose.

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

Hello fellow developers,

I have a question concerning the deployment of RabbitMQ. Currently, I have set two different computers, the first is running my django application, and the second is running Rabbit MQ and one worker written in Python(Celery). The second computer has more computational resources, it does the heavy-lifting. Now I am working on a containerized version of my web application and I would like to ask for your advise. My idea is to launch the following containers (image names in bold):

1. MySQL: database
2. Python: django installed, my app is copied into the image
3. RabbitMQ: the broker correctly configured to talk to my django app (container 2)
4. Python: celery installed, my worker code copied into the image. It connects to the broker to start receiving tasks.

Currently, containers 1 and 2 are set up and running.

The idea is that container 4 can be instantiated as many times as possible, depending on the workload of my django app. The broker would have one or many workers to work with.

What's your opinion? I would really appreciate your feedback.

[michaelklishin]

As long as your apps can connect to RabbitMQ nodes, RabbitMQ does not care where they run, whether they use containers or not.

See most relevant Docker documentation and what ports RabbitMQ uses.

If your question is how to increase the number of application instances, the answer is "you must decide on what metrics to monitor them, say, once a minute, and scale your instances accordingly.

The most common mistakes around this by far are

1. Using GET /api/queues to get one metric of one queue. That endpoint will return 60+ metrics for all queues in the cluster, that's extremely wasteful. Use GET /api/queues/{vhost}/{name} (the endpoint that returns the metrics of one specific queue)
2. Monitoring too frequently, trying to scale up or down too frequently. But this is not really something RabbitMQ-specific

[djaniel]

Hi,
Thank you for you answer. I tried and succeeded to set up the third and fourth container (as described on my first post).

Do you know how could I scale up and down programmatically?

I learned about the existence of the docker SDK that could allow me to launch more instances of my 'worker' container on demand. It is my understanding that kubernetes would be too much.

Thanks again.

[lukebakken]

Do you know how could I scale up and down programmatically?

I suggest asking elsewhere. Team RabbitMQ does not have time to answer basic questions about other software projects.

[michaelklishin]

@djaniel I believe my earlier response includes an answer. RabbitMQ does not care how or why or when you increase or reduce the number of consumers.

Usually this is done in response to a certain metric, such as a message backlog (in Ready state only, of course), and I have proactively provided some suggestions about what NOT to do.