SERVER ALERT: Fatal - Unknown CA

[kotyara85]

rmq version: 3.9.8-alpine
we do sign certs with our own CA and they were added to the image's chain (build our of off of official image)
I followed the steps on the tls debug steps which all passed.
I can also wget to other resources using the same tls cert with no issues which means tls does work correctly

logs:

43
2021-11-06 17:52:47.545802+00:00 [noti] <0.2615.0> TLS client: In state connection received SERVER ALERT: Fatal - Unknown CA
42
2021-11-06 17:52:47.545802+00:00 [noti] <0.2615.0>
41
2021-11-06 17:52:47.546500+00:00 [warn] <0.2594.0> Federation exchange 'federated' in vhost '/' did not connect to exchange 'federated' in vhost '/' on amqps://rmq-cluster-1:30671. Reason: {error,
40
2021-11-06 17:52:47.546500+00:00 [warn] <0.2594.0>                                                                                                                                                                               {{socket_error,
39
2021-11-06 17:52:47.546500+00:00 [warn] <0.2594.0>                                                                                                                                                                                 {tls_alert,
38
2021-11-06 17:52:47.546500+00:00 [warn] <0.2594.0>                                                                                                                                                                                  {unknown_ca,
37
2021-11-06 17:52:47.546500+00:00 [warn] <0.2594.0>                                                                                                                                                                                   "TLS client: In state connection received SERVER ALERT: Fatal - Unknown CA\n"}}},
36
2021-11-06 17:52:47.546500+00:00 [warn] <0.2594.0>                                                                                                                                                                                {expecting,
35
2021-11-06 17:52:47.546500+00:00 [warn] <0.2594.0>                                                                                                                                                                                 'connection.start'}}}
34
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>     supervisor: {<0.2267.0>,rabbit_federation_link_sup}
33
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>     errorContext: child_terminated
32
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>     reason: {shutdown,restart}
31
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>     offender: [{pid,<0.2594.0>},
30
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                {id,{upstream,
29
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                        [{encrypted,
28
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                             <<"ABC">>}],
27
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                        <<"federated">>,<<"federated">>,
26
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                        <<"federation-link-stage">>,1000,1,5,none,none,false,
25
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                        'on-confirm',none,<<"stage">>,false,default,multiple}},
24
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                {mfargs,
23
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                    {rabbit_federation_exchange_link,start_link,
22
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                        [{{upstream,
21
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                              [{encrypted,
20
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                                   <<"ABC">>}],
19
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                              <<"federated">>,<<"federated">>,
18
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                              <<"federation-link-stage">>,1000,1,5,none,none,
17
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                              false,'on-confirm',none,<<"stage">>,false,
16
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                              default,multiple},
15
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                          {resource,<<"/">>,exchange,<<"federated">>}}]}},
14
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                {restart_type,{permanent,5}},
13
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                {shutdown,300000},
12
2021-11-06 17:52:47.547015+00:00 [erro] <0.2267.0>                {child_type,worker}]
11
2021-11-06 17:52:47.547551+00:00 [dbug] <0.2622.0> Closing all channels from connection '<[email protected]>' because it has been closed
10
2021-11-06 17:52:48.049804+00:00 [info] <0.2080.0> Supervisor {<0.2080.0>,rabbit_connection_sup}: child helper_sup started (<0.2081.0>): {rabbit_connection_helper_sup,start_link,[]}
9
2021-11-06 17:52:48.050013+00:00 [info] <0.2080.0> Supervisor {<0.2080.0>,rabbit_connection_sup}: child reader started (<0.2082.0>): {rabbit_reader,start_link,[<0.2081.0>,{acceptor,{0,0,0,0,0,0,0,0},5672}]}
8

Any suggestions?

Thanks

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

Please do not use issues for questions. GitHub Discussions have been around for many months.

[michaelklishin]

TLS client: In state connection received SERVER ALERT: Fatal - Unknown CA

means that it's a federation link (a TLS client) that is unaware of the server CA certificate. Without any details on what you did with wget I can only guess but you may or may not be comparing apples to oranges. wget, RabbitMQ server and Erlang client all have different ways of managing trusted certificate sets.

Relevant doc guides:

• Troubleshooting TLS
• Using TLS with Federation
• Erlang client TLS setup (as it is used by Federation internally)

[kotyara85]

Both curl and wget are using hosts chain to trusts certs so that does work.
I do append certs in the federation link using http style params when creating a federation link.
The same certs attached to all the nodes and pass verification using openssl.

What else can be debugged?

I deployed the cluster using rabbitmq operator. Certs attached to containers using volumes and present.
thanks

[michaelklishin]

We need a specific set of steps to reproduce using tls-gen produced certificate chains, ideally outside of Kubernetes. Certificate paths can be incorrect, we don't know what's in the files used exactly (e.g. how long the chain may be).

[kotyara85]

This fixed the problem on alpine ssl_options.cacertfile=/etc/ssl/certs/ca-certificates.crt