Skip to content

Logger exchange: fix race condition during initialisation #9875

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 9, 2023

Conversation

dcorbacho
Copy link
Contributor

The logger exchange needs to declare the exchange during initialisation, which requires the metadata store to be ready.
Metadata store initalisation happens in a rabbit boot step after logger initialisation in the second phase of the prelaunch. The spawned process that declares the exchange, should also wait for the store to be ready. Otherwise it enters a loop trying to decide which store to use which generates a huge log and delays initialisation:

Mnesia->Khepri fallback handling: Mnesia function failed because table
 'rabbit_vhost' is missing or read-only. Migration could be in progress;
 waiting for migration to progress and trying again... 

This commit gives it 60 seconds for the metadata store to boot, and only afterwards tries (and retries if needed) to declare the exchange.

Detected by frequent CI flakes in logging_SUITE:logging_to_exchange_works

Types of Changes

  • Bug fix (non-breaking change which fixes issue #NNNN)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause an observable behavior change in existing systems)
  • Documentation improvements (corrections, new content, etc)
  • Cosmetic change (whitespace, formatting, etc)
  • Build system and/or CI

Checklist

  • I have read the CONTRIBUTING.md document
  • I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
  • I have added tests that prove my fix is effective or that my feature works
  • All tests pass locally with my changes
  • If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
  • If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

@dcorbacho dcorbacho added this to the 3.13.0 milestone Nov 3, 2023
@dcorbacho dcorbacho requested a review from dumbbell November 3, 2023 16:23
@dcorbacho dcorbacho requested a review from dumbbell November 6, 2023 14:08
@dcorbacho dcorbacho force-pushed the logger-exchange-wait-for-init branch from 49198a1 to 5815f2b Compare November 7, 2023 11:42
@dcorbacho dcorbacho requested a review from dumbbell November 7, 2023 11:42
The logger exchange needs to declare the exchange during initialisation,
which requires the metadata store to be ready.
Metadata store initalisation happens in a rabbit boot step after
logger initialisation in the second phase of the prelaunch.
The spawned process that declares the exchange, should also
wait for the store to be ready. Otherwise it enters a loop
trying to decide which store to use which generates a huge log
and delays initialisation:
'Mnesia->Khepri fallback handling: Mnesia function failed because table
 `rabbit_vhost` is missing or read-only. Migration could be in progress;
 waiting for migration to progress and trying again...'

This commit gives it 60 seconds for the metadata store to boot,
and only afterwards tries (and retries if needed) to declare the
exchange.
@dcorbacho dcorbacho force-pushed the logger-exchange-wait-for-init branch from 5815f2b to 55f724a Compare November 8, 2023 11:54
@dcorbacho dcorbacho requested a review from dumbbell November 8, 2023 11:54
@dumbbell dumbbell merged commit 09a95a5 into main Nov 9, 2023
@dumbbell dumbbell deleted the logger-exchange-wait-for-init branch November 9, 2023 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants