Deprecated features: New module to manage deprecated features (!)

This introduces a way to declare deprecated features in the code, not
only in our communication. The new module allows to disallow the use of
a deprecated feature and/or warn the user when he relies on such a
feature.

[Why]
Currently, we only tell people about deprecated features through blog
posts and the mailing-list. This might be insufficiant for our users
that a feature they use will be removed in a future version:
* They may not read our blog or mailing-list
* They may not understand that they use such a deprecated feature
* They might wait for the big removal before they plan testing
* They might not take it seriously enough

The idea behind this patch is to increase the chance that users notice
that they are using something which is about to be dropped from
RabbitMQ. Anopther benefit is that they should be able to test how
RabbitMQ will behave in the future before the actual removal. This
should allow them to test and plan changes.

[How]
When a feature is deprecated in other large projects (such as FreeBSD
where I took the idea from), it goes through a lifecycle:
1. The feature is still available, but users get a warning somehow when
   they use it. They can disable it to test.
2. The feature is still available, but disabled out-of-the-box. Users
   can re-enable it (and get a warning).
3. The feature is disconnected from the build. Therefore, the code
   behind it is still there, but users have to recompile the thing to be
   able to use it.
4. The feature is removed from the source code. Users have to adapt or
   they can't upgrade anymore.

The solution in this patch offers the same lifecycle. A deprecated
feature will be in one of these deprecation phases:
1. `permitted_by_default`: The feature is available. Users get a warning
   if they use it. They can disable it from the configuration.
2. `denied_by_default`: The feature is available but disabled by
   default. Users get an error if they use it and RabbitMQ behaves like
   the feature is removed. They can re-enable is from the configuration
   and get a warning.
3. `disconnected`: The feature is present in the source code, but is
   disabled and can't be re-enabled without recompiling RabbitMQ. Users
   get the same behavior as if the code was removed.
4. `removed`: The feature's code is gone.

The whole thing is based on the feature flags subsystem, but it has the
following differences with other feature flags:
* The semantic is reversed: the feature flag behind a deprecated feature
  is disabled when the deprecated feature is permitted, or enabled when
  the deprecated feature is denied.
* The feature flag behind a deprecated feature is enabled out-of-the-box
  (meaning the deprecated feature is denied):
    * if the deprecation phase is `permitted_by_default` and the
      configuration denies the deprecated feature
    * if the deprecation phase is `denied_by_default` and the
      configuration doesn't permit the deprecated feature
    * if the deprecation phase is `disconnected` or `removed`
* Feature flags behind deprecated feature don't appear in feature flags
  listings.

Otherwise, deprecated features' feature flags are managed like other
feature flags, in particular inside clusters.

To declare a deprecated feature:

    -rabbit_deprecated_feature(
       {my_deprecated_feature,
        #{deprecation_phase => permitted_by_default,
          msgs => #{when_permitted => "This feature will be removed in RabbitMQ X.0"},
         }}).

Then, to check the state of a deprecated feature in the code:

    case rabbit_deprecated_features:is_permitted(my_deprecated_feature) of
        true ->
            %% The deprecated feature is still permitted.
            ok;
        false ->
            %% The deprecated feature is gone or should be considered
            %% unavailable.
            error
    end.

Warnings and errors are logged automatically. A message is generated
automatically, but it is possible to define a message in the deprecated
feature flag declaration like in the example above.

Here is an example of a logged warning that was generated automatically:

    Feature `my_deprecated_feature` is deprecated.
    By default, this feature can still be used for now.
    Its use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.
    To continue using this feature when it is not permitted by default, set the following parameter in your configuration:
        "deprecated_features.permit.my_deprecated_feature = true"
    To test RabbitMQ as if the feature was removed, set this in your configuration:
        "deprecated_features.permit.my_deprecated_feature = false"

To override the default state of `permitted_by_default` and
`denied_by_default` deprecation phases, users can set the following
configuration:

    # In rabbitmq.conf:
    deprecated_features.permit.my_deprecated_feature = true # or false

The actual behavior protected by a deprecated feature check is out of
scope for this subsystem. It is the repsonsibility of each deprecated
feature code to determine what to do when the deprecated feature is
denied.

V1: Deprecated feature states are initially computed during the
    initialization of the registry, based on their deprecation phase and
    possibly the configuration. They don't go through the `enable/1`
    code at all.

V2: Manage deprecated feature states as any other non-required
    feature flags. This allows to execute an `is_feature_used()`
    callback to determine if a deprecated feature can be denied. This
    also allows to prevent the RabbitMQ node from starting if it
    continues to use a deprecated feature.

V3: Manage deprecated feature states from the registry initialization
    again. This is required because we need to know very early if some
    of them are denied, so that an upgrade to a version of RabbitMQ
    where a deprecated feature is disconnected or removed can be
    performed.

    To still prevent the start of a RabbitMQ node when a denied
    deprecated feature is actively used, we run the `is_feature_used()`
    callback of all denied deprecated features as part of the
    `sync_cluster()` task. This task is executed as part of a feature
    flag refresh executed when RabbitMQ starts or when plugins are
    enabled. So even though a deprecated feature is marked as denied in
    the registry early in the boot process, we will still abort the
    start of a RabbitMQ node if the feature is used.

V4: Support context-dependent warnings. It is now possible to set a
    specific message when deprecated feature is permitted, when it is
    denied and when it is removed. Generic per-context messages are
    still generated.

V5: Improve default warning messages, thanks to @pstack2021.

V6: Rename the configuration variable from `permit_deprecated_features.*`
    to `deprecated_features.permit.*`. As @michaelklishin said, we tend
    to use shorter top-level names.

Author: dumbbell

deps/rabbit/BUILD.bazel

@@ -386,6 +386,14 @@ rabbitmq_integration_suite(
     size = "medium",
 )
 
+rabbitmq_integration_suite(
+    name = "deprecated_features_SUITE",
+    size = "medium",
+    additional_beam = [
+        ":feature_flags_v2_SUITE_beam_files",
+    ],
+)
+
 rabbitmq_integration_suite(
     name = "disconnect_detected_during_alarm_SUITE",
     size = "medium",

deps/rabbit/Makefile

@@ -241,7 +241,7 @@ ct-slow: CT_SUITES = $(SLOW_CT_SUITES)
 # --------------------------------------------------------------------
 
 RMQ_ERLC_OPTS += -I $(DEPS_DIR)/rabbit_common/include
-EDOC_OPTS += {preprocess,true}
+EDOC_OPTS += {preprocess,true},{includes,["."]}
 
 ifdef INSTRUMENT_FOR_QC
 RMQ_ERLC_OPTS += -DINSTR_MOD=gm_qc

deps/rabbit/app.bzl

@@ -95,6 +95,7 @@ def all_beam_files(name = "all_beam_files"):
             "src/rabbit_definitions_hashing.erl",
             "src/rabbit_definitions_import_https.erl",
             "src/rabbit_definitions_import_local_filesystem.erl",
+            "src/rabbit_deprecated_features.erl",
             "src/rabbit_diagnostics.erl",
             "src/rabbit_direct.erl",
             "src/rabbit_direct_reply_to.erl",
@@ -338,6 +339,7 @@ def all_test_beam_files(name = "all_test_beam_files"):
             "src/rabbit_definitions_hashing.erl",
             "src/rabbit_definitions_import_https.erl",
             "src/rabbit_definitions_import_local_filesystem.erl",
+            "src/rabbit_deprecated_features.erl",
             "src/rabbit_diagnostics.erl",
             "src/rabbit_direct.erl",
             "src/rabbit_direct_reply_to.erl",
@@ -512,6 +514,7 @@ def all_srcs(name = "all_srcs"):
     filegroup(
         name = "private_hdrs",
         srcs = [
+            "src/rabbit_feature_flags.hrl",
             "src/rabbit_fifo.hrl",
             "src/rabbit_fifo_dlx.hrl",
             "src/rabbit_fifo_v0.hrl",
@@ -593,6 +596,7 @@ def all_srcs(name = "all_srcs"):
             "src/rabbit_definitions_hashing.erl",
             "src/rabbit_definitions_import_https.erl",
             "src/rabbit_definitions_import_local_filesystem.erl",
+            "src/rabbit_deprecated_features.erl",
             "src/rabbit_diagnostics.erl",
             "src/rabbit_direct.erl",
             "src/rabbit_direct_reply_to.erl",
@@ -878,6 +882,14 @@ def test_suite_beam_files(name = "test_suite_beam_files"):
         erlc_opts = "//:test_erlc_opts",
         deps = ["//deps/rabbit_common:erlang_app"],
     )
+    erlang_bytecode(
+        name = "deprecated_features_SUITE_beam_files",
+        testonly = True,
+        srcs = ["test/deprecated_features_SUITE.erl"],
+        outs = ["test/deprecated_features_SUITE.beam"],
+        app_name = "rabbit",
+        erlc_opts = "//:test_erlc_opts",
+    )
     erlang_bytecode(
         name = "direct_exchange_routing_v2_SUITE_beam_files",
         testonly = True,

deps/rabbit/docs/rabbitmq.conf.example

@@ -548,6 +548,25 @@
 ## NB: Change these only if you understand what you are doing!
 ##
 
+## To permit or deny a deprecated feature when it is in its
+## `permitted_by_default` or `denied_by_default` deprecation phase, the
+## default state can be overriden from the configuration.
+##
+## When a deprecated feature is permitted by default (first phase of the
+## deprecation period), it means the feature is available by default and can
+## be turned off by setting it to false in the configuration.
+##
+## When a deprecated feature is denied by default (second phase of the
+## deprecation period), it means the feature is unavailable by default but can
+## be turned back on by setting it to true in the configuration.
+##
+## When a deprecated feature is "disconnected" or "removed" (last two phases
+## of the deprecation period), it is no longer possible to turn it back on
+## from the configuration.
+##
+# deprecated_features.permit.a_deprecated_feature = true
+# deprecated_features.permit.another_deprecated_feature = false
+
 ## Timeout used when waiting for Mnesia tables in a cluster to
 ## become available.
 ##

deps/rabbit/priv/schema/rabbit.schema

@@ -2131,18 +2131,17 @@ end}.
 %% =====================================
 %%
 
-%% NOTE: `true` is intentionally omitted - add it back when mirrored
-%% queue deprecation is converted to use deprecated features system.
 {mapping,
- "deprecated_features.permit.$name", "rabbit.permitted_deprecated_features",
-    [{datatype, {enum, [false]}}]
+ "deprecated_features.permit.$name", "rabbit.permit_deprecated_features",
+ [{datatype, {enum, [true, false]}}]
 }.
 
 %% This converts:
-%%     deprecated_features.permit.my_feature = false
+%%     deprecated_features.permit.my_feature = true
 %% to:
-%%     {rabbit, [{permitted_deprecated_features, #{my_feature => false}}]}.
-{translation, "rabbit.permitted_deprecated_features",
+%%     {rabbit, [{permit_deprecated_features, #{my_feature => true}}]}.
+
+{translation, "rabbit.permit_deprecated_features",
  fun(Conf) ->
          Settings = cuttlefish_variable:filter_by_prefix(
                       "deprecated_features.permit", Conf),

deps/rabbit/src/rabbit.erl

@@ -512,7 +512,10 @@ start_apps(Apps, RestartTypes) ->
     %% We need to load all applications involved in order to be able to
     %% find new feature flags.
     app_utils:load_applications(Apps),
-    ok = rabbit_feature_flags:refresh_feature_flags_after_app_load(),
+    case rabbit_feature_flags:refresh_feature_flags_after_app_load() of
+        ok    -> ok;
+        Error -> throw(Error)
+    end,
     rabbit_prelaunch_conf:decrypt_config(Apps),
     lists:foreach(
       fun(App) ->
@@ -932,7 +935,10 @@ start(normal, []) ->
         %% once, because it does not involve running code from the
         %% plugins.
         ok = app_utils:load_applications(Plugins),
-        ok = rabbit_feature_flags:refresh_feature_flags_after_app_load(),
+        case rabbit_feature_flags:refresh_feature_flags_after_app_load() of
+            ok     -> ok;
+            Error1 -> throw(Error1)
+        end,
 
         persist_static_configuration(),