Recover bindings for all durable queues including failed to recover.

[hairyhum]

If a queue fails to recover it may still be restarted by the supervisor
and eventually start. After that some bindings may be in rabbit_durable_route
but not rabbit_route. This can cause binding not found errors.

If bindings are recovered for failed queues, the behaviour will be
the same as for the crashed queues. (which is currently broken
but needs to be fixed separately)

Addresses #1873
[#163919158]

This will reqruire a separate PR for 3.7

[hairyhum]

Separate PR for 3.7

[Ayanda-D]

ah, solving this weird case https://groups.google.com/forum/#!topic/rabbitmq-users/-X19dq8cYaY thank you! 👍

[michaelklishin]

@Ayanda-D we don't yet know and would not claim it is sufficient because there was no reliable way to reproduce any of those reports :(

[Ayanda-D]

@michaelklishin ok, this was reported to us as well, but haven't been able to to reproduce it either :/ seemed the error handling could be improved as well? When binding lookup in rabbit_route is done and not found, but exists in rabbit_durable_route, before immediately returning not_found to the channel, maybe do/attempt a re-sync from rabbit_durable_route, to update rabbit_route and rabbit_semi_durable_route? And only return not_found if binding still doesn't exist after a "failed re-synch" attempt (if such a case can ever occur)? coz if its in rabbit_durable_route then surely the binding exists, and returning not_found here doesn't sound right. Some thoughts, guess this PR resolving the issue on recovery then maybe this case may never be hit on runtime. We'll see 😅

[hairyhum]

@Ayanda-D this PR will fix binding not found errors only if there were failed queues. You can check that in logs searching for Queue <...> failed to initialise in the logs.

The option to allow recovery of bindings does also make sense. I was thinking of letting queue.bind to create bindings even if there is an inconsistency, because that's what clients use to recover the topologies.

[Ayanda-D]

Ok, sounds good 👍