Use erlang:system_info(creation) as GUID #3631
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@Mergifyio rebase |
Node GUID allows to differentiate between different incarnations of a node. However, since rabbit may take some time to start (many queues/bindings, etc), there could be a significant difference between Erlang VM being up and responding to RPC requests and the new GUID being announced. During that time, node monitor could incorrectly assume there was a network partition, while in fact a node was simply restarted. With this change, as soon as the Erlang VM is up, we can tell whether it was restarted and avoid false positives. Additionally, we now log if any queues were deleted on behalf of the restarted node. This can take quite a long time if there are many transient queues (eg. auto-delete queues). The longer this takes, the higher were the odds of a restarted node being up again by the time check_partial_partition was called. We may need to reconsider this logic as well but for now - we just log this activity. Co-authored-by: Loïc Hoguin <[email protected]>
✅ Branch has been successfully rebased |
HoloRin
force-pushed
the
creation-as-guid
branch
from
0d9d3d7 to
a92a532
Compare
michaelklishin
added a commit
that referenced
this pull request
Nov 3, 2021
Use erlang:system_info(creation) as GUID (backport #3631)
michaelklishin
added a commit
that referenced
this pull request
Nov 3, 2021
Use erlang:system_info(creation) as GUID (cherry picked from commit 6318a7e) Conflicts: deps/rabbit/src/rabbit_node_monitor.erl
|
Backported to |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of these changes is to prevent a false-positive network partition:
in some cases, in a multi-node cluster, a single node restart could trigger
node monitor to declare a partition and stop the remaining nodes.
Node GUID allows to differentiate between different incarnations of a node.
However, since rabbit may take some time to start (many queues/bindings, etc),
there could be a significant difference between Erlang VM being up and
responding to RPC calls and the new GUID being announced. During that
time, node monitor could incorrectly assume there was a network
partition, while in fact a node was simply restarted. With this change,
as soon as the Erlang VM is up, we can tell whether it was restarted and
avoid false positives.
Additionally, we now log if any queues were deleted on behalf of the
restarted node. This can take quite a long time if there are many transient
queues (eg. auto-delete queues). The longer this takes, the higher were the
odds of a restarted node being up again by the time
check_partial_partition was called. We may need to reconsider this logic
as well but for now - we just log this activity.
Co-authored-by: Loïc Hoguin [email protected]