Feature flags: must have failed to write (but has been written..)

[Patrick-Remy]

I encounter a strange behavior of rabbitmq (tested with 3.8.9 and 3.9.8). The server always complies about not existing feature_flags file, but is indeed there

/ # cat /var/lib/rabbitmq/mnesia/rabbit@worker1-rabbitmq-feature_flags
[drop_unroutable_metric,empty_basic_get_metric,implicit_default_bindings,
 maintenance_mode_status,quorum_queue,stream_queue,user_limits,
 virtual_host_metadata].
/ # rabbitmqctl stop_app
Stopping rabbit application on node rabbit@worker1-rabbitmq ...
/ # cat /var/lib/rabbitmq/mnesia/rabbit@worker1-rabbitmq-feature_flags
[drop_unroutable_metric,empty_basic_get_metric,implicit_default_bindings,
 maintenance_mode_status,quorum_queue,stream_queue,user_limits,
 virtual_host_metadata].
/ # rabbitmqctl reset
Resetting node rabbit@worker1-rabbitmq ...
/ # cat /var/lib/rabbitmq/mnesia/rabbit@worker1-rabbitmq-feature_flags
[drop_unroutable_metric,empty_basic_get_metric,implicit_default_bindings,
 maintenance_mode_status,quorum_queue,stream_queue,user_limits,
 virtual_host_metadata].
/ # rabbitmqctl join_cluster rabbit@core-rabbitmq
Clustering node rabbit@worker1-rabbitmq with rabbit@core-rabbitmq

09:40:59.939 [warn]  Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@worker1-rabbitmq-feature_flags`:
09:40:59.939 [warn]  Feature flags:   - list of previously disabled feature flags now marked as such: [:maintenance_mode_status]

09:41:00.923 [warn]  Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@worker1-rabbitmq-feature_flags`:
09:41:00.924 [warn]  Feature flags:   - list of previously enabled feature flags now marked as such: [:maintenance_mode_status]

/ # ls -la /var/lib/rabbitmq/mnesia/rabbit@worker1-rabbitmq-feature_flags
-rw-r--r--    1 rabbitmq rabbitmq       163 Nov 12 09:41 /var/lib/rabbitmq/mnesia/rabbit@worker1-rabbitmq-feature_flags

/ # cat /var/lib/rabbitmq/mnesia/rabbit@worker1-rabbitmq-feature_flags
[drop_unroutable_metric,empty_basic_get_metric,implicit_default_bindings,
 maintenance_mode_status,quorum_queue,stream_queue,user_limits,
 virtual_host_metadata].

In my understanding the warnings make no sense:

1. the required file is always there (and has the correct permissions)
2. the content of the file did not change between any of the tasks
3. it warns about maintenance_mode_status flag (twice) and that it would disable it, but instead it is still present

Do you have any idea why this happens?

[johanrhodin]

I think this is the same issue as has been reported here rabbitmq/discussions#183 and here https://groups.google.com/g/rabbitmq-users/c/ICy8b8UzOr0/m/ylI6EaVVCQAJ. I also think @lukebakken is looking into it.

[lukebakken]

I remember reading the message but no, I'm not currently looking into it.