Log clearer message if TLS client connects to AMQP port

[ansd]

What?

If a TLS client app is misconfigured trying to connect to AMQP port 5672 instead to the AMQPS port 5671, this commit makes RabbitMQ log a more descriptive error message.

openssl s_client -connect localhost:5672 -tls1_3
openssl s_client -connect localhost:5672 -tls1_2

RabbitMQ logs prior to this commit:

[info] <0.1073.0> accepting AMQP connection [::1]:53535 -> [::1]:5672
[error] <0.1073.0> closing AMQP connection <0.1073.0> ([::1]:53535 -> [::1]:5672, duration: '0ms'):
[error] <0.1073.0> {bad_header,<<22,3,1,0,192,1,0,0>>}

[info] <0.1080.0> accepting AMQP connection [::1]:53577 -> [::1]:5672
[error] <0.1080.0> closing AMQP connection <0.1080.0> ([::1]:53577 -> [::1]:5672, duration: '1ms'):
[error] <0.1080.0> {bad_header,<<22,3,1,0,224,1,0,0>>}

RabbitMQ logs after this commit:

[info] <0.969.0> accepting AMQP connection [::1]:53632 -> [::1]:5672
[error] <0.969.0> closing AMQP connection <0.969.0> ([::1]:53632 -> [::1]:5672, duration: '0ms'):
[error] <0.969.0> {detected_unexpected_tls_header,<<22,3,1,0,192,1,0,0>>

[info] <0.975.0> accepting AMQP connection [::1]:53638 -> [::1]:5672
[error] <0.975.0> closing AMQP connection <0.975.0> ([::1]:53638 -> [::1]:5672, duration: '1ms'):
[error] <0.975.0> {detected_unexpected_tls_header,<<22,3,1,0,224,1,0,0>>}

Why?

I've seen numerous occurrences in the past few years where misconfigured TLS apps connected to the wrong port. Therefore, RabbitMQ trying to detect a TLS client and providing a more descriptive log message seems appropriate to me.

How?

The first few bytes of any TLS connection are:

Record Type (1 byte):
Always 0x16 (22 in decimal) for a Handshake message.

Version (2 bytes):
This represents the highest version of TLS that the client supports. Common values: 0x0301 → TLS 1.0 (or SSL 3.1)
0x0302 → TLS 1.1
0x0303 → TLS 1.2
0x0304 → TLS 1.3

Record Length (2 bytes):
Specifies the length of the following handshake message.

Handshake Type (1 byte, usually the 6th byte overall): Always 0x01 for ClientHello.

[michaelklishin]

Works as expected:

rabbitmq-diagnostics log_tail_stream
# => Streaming logs from node rabbit@{hostname} ...
# => 2025-03-17 18:23:36.919383-04:00 [info] <0.544.0> accepting # => AMQP connection [::1]:57812 -> [::1]:5672
# => 2025-03-17 18:23:36.919483-04:00 [error] <0.544.0> closing # => AMQP connection <0.544.0> ([::1]:57812 -> [::1]:5672, duration: '0ms'):
# => 2025-03-17 18:23:36.919483-04:00 [error] <0.544.0> {detected_unexpected_tls_header,<<22,3,1,0,224,1,0,0>>}