[Questions] Required plugin feature flags not written to file

[gomoripeti]

(the "Question" template is a bit weird that there is no text field to put my actual question in)

I noticed on a newly created 4.0.2 cluster and reproduced as bellow that the required feature flags of the later enabled MQTT plugin are not written to disk. My question is if this can cause any problem in practice? Required feature flags are enabled anyway, so I'm not sure in what edge case this might confuse RabbitMQ, but it is at least somewhat inconsistent as required feature flags of the rabbit app are present in the enabled feature flags file.

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.2

Erlang version used

26.2.x

Operating system (distribution) used

MacOS

How is RabbitMQ deployed?

Other

rabbitmq-diagnostics status output

Logs from node 1 (with sensitive values edited out)

2024-10-04 16:31:18.711561+02:00 [debug] <0.862.0> Plugins discovery: ignoring getopt, not a RabbitMQ plugin
2024-10-04 16:31:18.711621+02:00 [debug] <0.862.0> Plugins discovery: ignoring quantile_estimator, not a RabbitMQ plugin
2024-10-04 16:31:18.736266+02:00 [debug] <0.862.0> Plugins discovery: ignoring getopt, not a RabbitMQ plugin
2024-10-04 16:31:18.736315+02:00 [debug] <0.862.0> Plugins discovery: ignoring quantile_estimator, not a RabbitMQ plugin
2024-10-04 16:31:18.740796+02:00 [debug] <0.862.0> Feature flags: REFRESHING after applications load...
2024-10-04 16:31:18.740836+02:00 [debug] <0.340.0> Feature flags: registering controller globally before proceeding with task: refresh_after_app_load
2024-10-04 16:31:18.740866+02:00 [debug] <0.340.0> Feature flags: [global sync] @ rabbit@Peters-MBP
2024-10-04 16:31:18.740910+02:00 [debug] <0.340.0> Feature flags: [global register] @ rabbit@Peters-MBP
2024-10-04 16:31:18.740967+02:00 [debug] <0.340.0> Feature flags: controller globally registered; can proceed with task
2024-10-04 16:31:18.741529+02:00 [debug] <0.340.0> Feature flags: acquired lock before initializing registry (<0.340.0>)
2024-10-04 16:31:18.741549+02:00 [debug] <0.340.0> Feature flags: query feature flags in loaded applications
2024-10-04 16:31:18.758086+02:00 [debug] <0.340.0> Feature flags: time to find supported feature flags and deprecated features: 16522 us
2024-10-04 16:31:18.758124+02:00 [debug] <0.340.0> Feature flags: application `rabbitmq_mqtt` has 3 feature flags (including deprecated features)
2024-10-04 16:31:18.758149+02:00 [debug] <0.340.0> Feature flags: application `rabbitmq_management_agent` has 2 feature flags (including deprecated features)
2024-10-04 16:31:18.758162+02:00 [debug] <0.340.0> Feature flags: application `rabbitmq_management` has 1 feature flags (including deprecated features)
2024-10-04 16:31:18.758186+02:00 [debug] <0.340.0> Feature flags: application `rabbit` has 1 feature flags (including deprecated features)
2024-10-04 16:31:18.758197+02:00 [debug] <0.340.0> Feature flags: application `rabbit` has 22 feature flags (including deprecated features)
2024-10-04 16:31:18.758298+02:00 [debug] <0.340.0> Feature flags: application `rabbitmq_management_agent` has 1 feature flags (including deprecated features)
2024-10-04 16:31:18.758390+02:00 [debug] <0.340.0> Feature flags: application `rabbit` has 1 feature flags (including deprecated features)
2024-10-04 16:31:18.758482+02:00 [debug] <0.340.0> Feature flags: application `rabbit` has 1 feature flags (including deprecated features)
2024-10-04 16:31:18.758594+02:00 [debug] <0.340.0> Feature flags: application `rabbit` has 1 feature flags (including deprecated features)
2024-10-04 16:31:18.758669+02:00 [debug] <0.340.0> Feature flags: application `rabbit` has 1 feature flags (including deprecated features)
2024-10-04 16:31:18.758739+02:00 [debug] <0.340.0> Feature flags: application `rabbit` has 1 feature flags (including deprecated features)
2024-10-04 16:31:18.758818+02:00 [debug] <0.340.0> Feature flags: application `rabbit` has 1 feature flags (including deprecated features)
2024-10-04 16:31:18.758961+02:00 [debug] <0.340.0> Feature flags: registry refresh needed: yes, list of feature flags differs
2024-10-04 16:31:18.758973+02:00 [debug] <0.340.0> Feature flags: (re)initialize registry (<0.340.0>)
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags: list of feature flags found:
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [x] detailed_queues_endpoint
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [ ] khepri_db
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [x] message_containers_deaths_v2
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [x] quorum_queue_non_voters
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [x] rabbit_exchange_type_local_random
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [x] rabbitmq_4.0.0
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags: list of deprecated features found:
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [ ] amqp_address_v1
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [ ] global_qos
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [ ] management_metrics_collection
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [ ] queue_master_locator
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [ ] ram_node_type
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags:   [ ] transient_nonexcl_queues
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags: required feature flags not listed above: 23
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags: removed deprecated features not listed above: 1
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags: scanned applications: [rabbit,rabbit_common,rabbitmq_management,rabbitmq_management_agent,rabbitmq_mqtt,rabbitmq_prelaunch,rabbitmq_web_dispatch]
2024-10-04 16:31:18.759041+02:00 [debug] <0.340.0> Feature flags: feature flag states written to disk: yes
2024-10-04 16:31:18.759186+02:00 [debug] <0.340.0> Feature flags: time to regen registry: 201 us
2024-10-04 16:31:18.759194+02:00 [debug] <0.340.0> Feature flags: releasing lock after initializing registry (<0.340.0>)
2024-10-04 16:31:18.759533+02:00 [debug] <0.340.0> Feature flags: synchronizing feature flags on nodes: ['rabbit@Peters-MBP']
2024-10-04 16:31:18.759545+02:00 [debug] <0.340.0> Feature flags: collecting inventory on nodes: ['rabbit@Peters-MBP']
2024-10-04 16:31:18.759750+02:00 [debug] <0.340.0> Feature flags: acquiring lock {feature_flags_state_change,<0.340.0>}
2024-10-04 16:31:18.759772+02:00 [debug] <0.340.0> Feature flags: acquired lock {feature_flags_state_change,<0.340.0>}
2024-10-04 16:31:18.759781+02:00 [debug] <0.340.0> Feature flags: `classic_mirrored_queue_version`: supported; continuing
2024-10-04 16:31:18.759789+02:00 [debug] <0.340.0> Feature flags: `classic_mirrored_queue_version`: already enabled; skipping
2024-10-04 16:31:18.759839+02:00 [debug] <0.340.0> Feature flags: `classic_queue_mirroring`: supported; continuing
2024-10-04 16:31:18.759846+02:00 [debug] <0.340.0> Feature flags: `classic_queue_mirroring`: already enabled; skipping
2024-10-04 16:31:18.759856+02:00 [debug] <0.340.0> Feature flags: `classic_queue_type_delivery_support`: supported; continuing
2024-10-04 16:31:18.759863+02:00 [debug] <0.340.0> Feature flags: `classic_queue_type_delivery_support`: already enabled; skipping
2024-10-04 16:31:18.759871+02:00 [debug] <0.340.0> Feature flags: `delete_ra_cluster_mqtt_node`: supported; continuing
2024-10-04 16:31:18.759881+02:00 [debug] <0.340.0> Feature flags: `delete_ra_cluster_mqtt_node`: already enabled; skipping
2024-10-04 16:31:18.759888+02:00 [debug] <0.340.0> Feature flags: `detailed_queues_endpoint`: supported; continuing
2024-10-04 16:31:18.759896+02:00 [debug] <0.340.0> Feature flags: `detailed_queues_endpoint`: already enabled; skipping
2024-10-04 16:31:18.759903+02:00 [debug] <0.340.0> Feature flags: `direct_exchange_routing_v2`: supported; continuing
2024-10-04 16:31:18.759910+02:00 [debug] <0.340.0> Feature flags: `direct_exchange_routing_v2`: already enabled; skipping
2024-10-04 16:31:18.759919+02:00 [debug] <0.340.0> Feature flags: `drop_unroutable_metric`: supported; continuing
2024-10-04 16:31:18.759925+02:00 [debug] <0.340.0> Feature flags: `drop_unroutable_metric`: already enabled; skipping
2024-10-04 16:31:18.759934+02:00 [debug] <0.340.0> Feature flags: `empty_basic_get_metric`: supported; continuing
2024-10-04 16:31:18.759941+02:00 [debug] <0.340.0> Feature flags: `empty_basic_get_metric`: already enabled; skipping
2024-10-04 16:31:18.759948+02:00 [debug] <0.340.0> Feature flags: `feature_flags_v2`: supported; continuing
2024-10-04 16:31:18.759957+02:00 [debug] <0.340.0> Feature flags: `feature_flags_v2`: already enabled; skipping
2024-10-04 16:31:18.759964+02:00 [debug] <0.340.0> Feature flags: `implicit_default_bindings`: supported; continuing
2024-10-04 16:31:18.759971+02:00 [debug] <0.340.0> Feature flags: `implicit_default_bindings`: already enabled; skipping
2024-10-04 16:31:18.759979+02:00 [debug] <0.340.0> Feature flags: `listener_records_in_ets`: supported; continuing
2024-10-04 16:31:18.759986+02:00 [debug] <0.340.0> Feature flags: `listener_records_in_ets`: already enabled; skipping
2024-10-04 16:31:18.759994+02:00 [debug] <0.340.0> Feature flags: `maintenance_mode_status`: supported; continuing
2024-10-04 16:31:18.760002+02:00 [debug] <0.340.0> Feature flags: `maintenance_mode_status`: already enabled; skipping
2024-10-04 16:31:18.760009+02:00 [debug] <0.340.0> Feature flags: `message_containers`: supported; continuing
2024-10-04 16:31:18.760017+02:00 [debug] <0.340.0> Feature flags: `message_containers`: already enabled; skipping
2024-10-04 16:31:18.760024+02:00 [debug] <0.340.0> Feature flags: `message_containers_deaths_v2`: supported; continuing
2024-10-04 16:31:18.760030+02:00 [debug] <0.340.0> Feature flags: `message_containers_deaths_v2`: already enabled; skipping
2024-10-04 16:31:18.760041+02:00 [debug] <0.340.0> Feature flags: `mqtt_v5`: supported; continuing
2024-10-04 16:31:18.760047+02:00 [debug] <0.340.0> Feature flags: `mqtt_v5`: already enabled; skipping
2024-10-04 16:31:18.760053+02:00 [debug] <0.340.0> Feature flags: `quorum_queue`: supported; continuing
2024-10-04 16:31:18.760060+02:00 [debug] <0.340.0> Feature flags: `quorum_queue`: already enabled; skipping
2024-10-04 16:31:18.760067+02:00 [debug] <0.340.0> Feature flags: `quorum_queue_non_voters`: supported; continuing
2024-10-04 16:31:18.760074+02:00 [debug] <0.340.0> Feature flags: `quorum_queue_non_voters`: already enabled; skipping
2024-10-04 16:31:18.760082+02:00 [debug] <0.340.0> Feature flags: `rabbit_exchange_type_local_random`: supported; continuing
2024-10-04 16:31:18.760090+02:00 [debug] <0.340.0> Feature flags: `rabbit_exchange_type_local_random`: already enabled; skipping
2024-10-04 16:31:18.760099+02:00 [debug] <0.340.0> Feature flags: `rabbit_mqtt_qos0_queue`: supported; continuing
2024-10-04 16:31:18.760105+02:00 [debug] <0.340.0> Feature flags: `rabbit_mqtt_qos0_queue`: already enabled; skipping
2024-10-04 16:31:18.760112+02:00 [debug] <0.340.0> Feature flags: `rabbitmq_4.0.0`: supported; continuing
2024-10-04 16:31:18.760121+02:00 [debug] <0.340.0> Feature flags: `rabbitmq_4.0.0`: already enabled; skipping
2024-10-04 16:31:18.760127+02:00 [debug] <0.340.0> Feature flags: `restart_streams`: supported; continuing
2024-10-04 16:31:18.760134+02:00 [debug] <0.340.0> Feature flags: `restart_streams`: already enabled; skipping
2024-10-04 16:31:18.760143+02:00 [debug] <0.340.0> Feature flags: `stream_filtering`: supported; continuing
2024-10-04 16:31:18.760149+02:00 [debug] <0.340.0> Feature flags: `stream_filtering`: already enabled; skipping
2024-10-04 16:31:18.760156+02:00 [debug] <0.340.0> Feature flags: `stream_queue`: supported; continuing
2024-10-04 16:31:18.760164+02:00 [debug] <0.340.0> Feature flags: `stream_queue`: already enabled; skipping
2024-10-04 16:31:18.760211+02:00 [debug] <0.340.0> Feature flags: `stream_sac_coordinator_unblock_group`: supported; continuing
2024-10-04 16:31:18.760240+02:00 [debug] <0.340.0> Feature flags: `stream_sac_coordinator_unblock_group`: already enabled; skipping
2024-10-04 16:31:18.760289+02:00 [debug] <0.340.0> Feature flags: `stream_single_active_consumer`: supported; continuing
2024-10-04 16:31:18.760297+02:00 [debug] <0.340.0> Feature flags: `stream_single_active_consumer`: already enabled; skipping
2024-10-04 16:31:18.760308+02:00 [debug] <0.340.0> Feature flags: `stream_update_config_command`: supported; continuing
2024-10-04 16:31:18.760323+02:00 [debug] <0.340.0> Feature flags: `stream_update_config_command`: already enabled; skipping
2024-10-04 16:31:18.760339+02:00 [debug] <0.340.0> Feature flags: `tracking_records_in_ets`: supported; continuing
2024-10-04 16:31:18.760347+02:00 [debug] <0.340.0> Feature flags: `tracking_records_in_ets`: already enabled; skipping
2024-10-04 16:31:18.760358+02:00 [debug] <0.340.0> Feature flags: `user_limits`: supported; continuing
2024-10-04 16:31:18.760367+02:00 [debug] <0.340.0> Feature flags: `user_limits`: already enabled; skipping
2024-10-04 16:31:18.760373+02:00 [debug] <0.340.0> Feature flags: `virtual_host_metadata`: supported; continuing
2024-10-04 16:31:18.760380+02:00 [debug] <0.340.0> Feature flags: `virtual_host_metadata`: already enabled; skipping
2024-10-04 16:31:18.760446+02:00 [debug] <0.340.0> Feature flags: releasing lock {feature_flags_state_change,<0.340.0>}
2024-10-04 16:31:18.760494+02:00 [debug] <0.340.0> Feature flags: unregistering controller globally
2024-10-04 16:31:18.760513+02:00 [debug] <0.340.0> Feature flags: [global unregister] @ rabbit@Peters-MBP
2024-10-04 16:31:18.760730+02:00 [debug] <0.862.0> Decoding encrypted config values (if any)
2024-10-04 16:31:18.761212+02:00 [debug] <0.862.0> Boot steps: time to find boot steps: 438 us
2024-10-04 16:31:18.762538+02:00 [info] <0.875.0> Starting MQTT retained message store rabbit_mqtt_retained_msg_store_dets for vhost '/'
2024-10-04 16:31:18.763500+02:00 [info] <0.875.0> Recovered MQTT retained message store rabbit_mqtt_retained_msg_store_dets for vhost '/'
2024-10-04 16:31:18.763830+02:00 [info] <0.895.0> started MQTT TCP listener on [::]:1883
2024-10-04 16:31:18.765763+02:00 [info] <0.862.0> Plugins changed; enabled [rabbitmq_mqtt]

Logs from node 2 (if applicable, with sensitive values edited out)

No response

Logs from node 3 (if applicable, with sensitive values edited out)

No response

rabbitmq.conf

not relevant, advanced.config auto-generated by make run-broker

Steps to deploy RabbitMQ cluster

running from git cloned repo

Steps to reproduce the behavior in question

• start a single node server from the repo, with mqtt plugin NOT enabled

  make run-broker TEST_TMPDIR="/tmp/rabbitmq/" RABBITMQ_ENABLED_PLUGINS=rabbitmq_management
  

• verify mqtt plugin is not enabled and no related feature flags present

  % sbin/rabbitmq-plugins list mqtt -q
  [  ] rabbitmq_mqtt              4.1.0+alpha.285.gbedee25
  [  ] rabbitmq_web_mqtt          4.1.0+alpha.211.gedd8fbc
  [  ] rabbitmq_web_mqtt_examples 4.1.0+alpha.176.gec0d24b
  % sbin/rabbitmqctl list_feature_flags | grep -c mqtt
  0
  % grep -c mqtt /tmp/rabbitmq/rabbit@Peters-MBP/feature_flags
  0
  

• enable mqtt plugin

  % sbin/rabbitmq-plugins enable rabbitmq_mqtt
  Enabling plugins on node rabbit@Peters-MBP:
  rabbitmq_mqtt
  The following plugins have been configured:
    rabbitmq_management
    rabbitmq_management_agent
    rabbitmq_mqtt
    rabbitmq_web_dispatch
  Applying plugin configuration to rabbit@Peters-MBP...
  The following plugins have been enabled:
    rabbitmq_mqtt
  
  started 1 plugins.
  

• observe feature flags of the mqtt plugin are enabled but they are not written to disk

  % sbin/rabbitmq-plugins list mqtt -q
  [E*] rabbitmq_mqtt              4.1.0+alpha.285.gbedee25
  [  ] rabbitmq_web_mqtt          4.1.0+alpha.211.gedd8fbc
  [  ] rabbitmq_web_mqtt_examples 4.1.0+alpha.176.gec0d24b
  % sbin/rabbitmqctl list_feature_flags | grep mqtt
  delete_ra_cluster_mqtt_node	enabled
  mqtt_v5	enabled
  rabbit_mqtt_qos0_queue	enabled
  % grep -c mqtt /tmp/rabbitmq/rabbit@Peters-MBP/feature_flags
  0
  
  
  

advanced.config

No response

Application code

No response

Kubernetes deployment file

No response

[michaelklishin]

@gomoripeti I have updated the discussion template. Feature flags have been updated for 4.1.0. Can you reproduce this behavior with 4.1.0-alpha.0081e1d0 to compare? Or, well, the tip of main when running nodes from source.

@dumbbell this is the question I have mentioned in our internal chat.

[dumbbell]

This is probably a genuine bug in 4.0.x and before because required feature flags are marked as enabled right away instead of going through any "enable" code path. It’s possible that required feature flags from core are added to the file as a by-product of something else: either they were already enabled when the file is created for the first time, or they are added when another non-required feature flag is enabled.

In main, "soft" required feature flags are enabled like any feature flags, so they should be added to the file. "Hard" required ones like all of the 4.0.x (and before) required feature flags will hit this bug. The code will have to explicitly regen the file.

Even though this is not critical, I believe it’s a bug.

[gomoripeti]

thanks for the details. The behaviour is the same on latest main de90cfc

Even though this might be a bug, I could not find a case when this caused a problem in practice eg. upgrading from 4.0.2 to 4.0.3 still worked fine.

[dumbbell]

Yes, I don’t expect this to be fixed in main because the code around "hard" feature flags didn’t change. It should be easy to fix though.