Cannot get inter node TLS to work

[bsiara]

Describe the bug

I'm trying to configure inter node tls on 3 nodes cluster and still can not achive working that. Tryed rabbitmq docker image: rabbitmq:3.12.13-management-alpine
with docker-compose.yml:

version: '3'
services:
  rabbitmq:
    image: rabbitmq:3.12.13-management-alpine
    container_name: rabbitmq
    restart: always
    environment:
      TZ: Europe/Warsaw
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: adminpass
      #RABBITMQ_NODENAME: rabbit@rabbitmq1
      RABBITMQ_ERLANG_COOKIE: Cookie
    volumes:
      - /opt/rmq/local/config:/etc/rabbitmq
      - /opt/rmq/local/data:/var/lib/rabbitmq
      - /opt/rmq/local/log:/var/log/rabbitmq
    network_mode: "host"
    extra_hosts:
      - "rabbitmq1:192.168.125.133"
      - "rabbitmq2:192.168.125.55"
      - "rabbitmq3:192.168.125.246"
    logging:
      driver: "json-file"
      options:
        max-size: "5m"
        max-file: "2"
    ulimits:
      nproc: 65535
      nofile:
        soft: 100000
        hard: 200000

Configuration and log you can find in attachment

Certificates was generated using tls-gen/basic
When I try to conect using openssl all works well:

openssl s_client -connect 192.168.125.133:25672   -cert config/certs/client.crt -key config/certs/client.key -CAfile config/certs/ca.crt   -verify 8 -verify_ip 192.168.125.133
verify depth is 8
CONNECTED(00000003)
40E707B7887F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:308:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 300 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

rabbitmq-diagnosting can not connect to node:

rabbitmq-diagnostics cluster_status
Error: unable to perform an operation on node 'rabbit@rabbitmq2'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@rabbitmq2
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: [rabbit@rabbitmq2]

rabbit@rabbitmq2:
  * connected to epmd (port 4369) on rabbitmq2
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
  * TCP connection succeeded but Erlang distribution failed 
  * suggestion: check if the Erlang cookie is identical for all server nodes and CLI tools
  * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
  * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
   * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more


Current node details:
 * node name: 'rabbitmqcli-692-rabbit@rabbitmq2'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: Cookie

When I run the cluster nodes they did not see each other.
advanced.config.txt
enabled_pluginx.txt
inter_node_tls.config.txt
rabbitmq.conf.txt
rabbitmq.log.txt
rabbitmq-env.conf.txt

Reproduction steps

Run 3 nodes cluster with attached configuration

Expected behavior

Nodes join to cluster

Additional context

No response

[michaelklishin]

@bsiara we have ample evidence that inter-node TLS does work in RabbitMQ, or, well, the runtime.

TCP connection succeeded but Erlang distribution failed

is the log line you are looking for. When inter-node TLS is enabled, as the docs state,
CLI tools must be configured to use TLS as well since nodes will no longer accept non-TLS connections:

CLI tools also have to be configured to use a certificate/key pair as TLS-enabled
nodes won't accept unencrypted connections from CLI tools and peers

Our small team does not have the resources to troubleshoot inter-node TLS on Kubernetes for non-paying users, so please take it from here.

[michaelklishin]

As mentioned in the docs, RABBITMQ_CTL_ERL_ARGS is the env variable that should be used to configure -proto_dist inet_tls and the certificates for CLI tools.

I don't remember if CLI tools include rabbitmq-env.conf contents that the server uses, though. This may be the root cause, if they don't/didn't.

[lukebakken]

@bsiara I have a complete example here:

https://github.com/lukebakken/docker-rabbitmq-cluster/tree/tls

[Przemyslaw-Witczak]

Hello All,
I have similar issue after upgrade Erlang from 25.2.1 to 26.2
Did any one solved that problem?
"* TCP connection succeeded but Erlang distribution failed"

Hash of erlang.cookie returned from cli commands is same as logged by RabbitMq due to startup

[lukebakken]

Please use this project as a reference: #10823 (comment)

Team RabbitMQ will not diagnose TLS issues for you.

Since you're having issues after upgrading to 26.2, you probably have invalid settings in your inter-node configuration file. For instance, ensure that fail_if_no_peer_cert is ONLY in the server section:

https://github.com/lukebakken/docker-rabbitmq-cluster/blob/tls/rmq0/inter_node_tls.config#L7

[Przemyslaw-Witczak]

Yes, that's I already found. But thanks pon., 19 maj 2025, 16:57 użytkownik Luke Bakken ***@***.***> napisał:

…

Closed #10823 <#10823> as resolved. — Reply to this email directly, view it on GitHub <#10823>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKZ2Y445KE3LGZOWUSRXMAD27HWODAVCNFSM6AAAAAB5NRGWGOVHI2DSMVQWIX3LMV45UABFIRUXGY3VONZWS33OIV3GK3TUHI5E433UNFTGSY3BORUW63R3GE4TKMBYG4ZA> . You are receiving this because you commented.Message ID: <rabbitmq/rabbitmq-server/repo-discussions/10823/discussion_event/1950872@ github.com>