Handle autoheal winner_waiting state with a chosen winner to avoid crashing node monitor

[Ayanda-D]

Proposed Changes

There are cases where autoheal state can be in winner_waiting state with chosen winner notification received. We want to handle this case like the rest of the {winner_is, Winner} messages, restarting the loser and updating autoheal state to avoid crashing node monitor process as below:

2023-10-08 17:43:20.348800+01:00 [info] <0.1440.0> rabbit on node 'rabbit@host-309' up
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0> ** Generic server rabbit_node_monitor terminating
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0> ** Last message in was {autoheal_msg,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                            {winner_is,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                'rabbit@host-309'}}
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0> ** When Server state == {state,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                             {state,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                 #{{rabbit,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                       'rabbit@host-704'} =>
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                       #Ref<0.64737952.1388052481.133256>,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                   {rabbit,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                       'rabbit@host-309'} =>
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                       #Ref<0.64737952.1388052481.133274>},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                 erlang},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                             ['rabbit@host-704',
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                              'rabbit@host-309'],
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                             {state,#{},erlang},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                             undefined,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                             {erlang,#Ref<0.64737952.1388052481.131475>},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                             {winner_waiting,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                 ['rabbit@host-309'],
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                 ['rabbit@host-309']},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                             1696508188,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                             #{'rabbit@host-704' =>
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                   1696508188,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                               'rabbit@host-309' =>
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                                   1696508188}}
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0> ** Reason for termination ==
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0> ** {function_clause,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>        [{rabbit_autoheal,handle_msg,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>             [{winner_is,'rabbit@host-309'},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>              {winner_waiting,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                  ['rabbit@host-309'],
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>                  ['rabbit@host-309']},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>              ['rabbit@host-704',
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>               'rabbit@host-309']],
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>             [{file,"rabbit_autoheal.erl"},{line,203}]},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>         {rabbit_node_monitor,handle_info,2,
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>             [{file,"rabbit_node_monitor.erl"},{line,657}]},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>         {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1123}]},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>         {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]},
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0>         {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
2023-10-08 17:43:20.356712+01:00 [error] <0.1440.0> 
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>   crasher:
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     initial call: rabbit_node_monitor:init/1
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     pid: <0.1440.0>
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     registered_name: rabbit_node_monitor
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     exception error: no function clause matching
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>                      rabbit_autoheal:handle_msg({winner_is,
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>                                                  'rabbit@host-309'},
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>                                                 {winner_waiting,
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>                                                  ['rabbit@host-309'],
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>                                                  ['rabbit@host-309']},
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>                                                 ['rabbit@host-704',
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>                                                  'rabbit@host-309']) (rabbit_autoheal.erl, line 203)
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>       in function  rabbit_node_monitor:handle_info/2 (rabbit_node_monitor.erl, line 657)
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>       in call from gen_server:try_dispatch/4 (gen_server.erl, line 1123)
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>       in call from gen_server:handle_msg/6 (gen_server.erl, line 1200)
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     ancestors: [rabbit_node_monitor_sup,rabbit_sup,<0.1216.0>]
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     message_queue_len: 0
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     messages: []
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     links: [<0.1439.0>,<0.1258.0>]
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     dictionary: []
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     trap_exit: true
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     status: running
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     heap_size: 17731
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     stack_size: 28
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>     reductions: 121798
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0>   neighbours:
2023-10-08 17:43:20.357292+01:00 [error] <0.1440.0> 
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>     supervisor: {local,rabbit_node_monitor_sup}
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>     errorContext: child_terminated
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>     reason: {function_clause,
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                 [{rabbit_autoheal,handle_msg,
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                      [{winner_is,
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                           'rabbit@host-309'},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                       {winner_waiting,
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                           ['rabbit@host-309'],
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                           ['rabbit@host-309']},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                       ['rabbit@host-704',
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                        'rabbit@host-309']],
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                      [{file,"rabbit_autoheal.erl"},{line,203}]},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                  {rabbit_node_monitor,handle_info,2,
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                      [{file,"rabbit_node_monitor.erl"},{line,657}]},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                  {gen_server,try_dispatch,4,
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                      [{file,"gen_server.erl"},{line,1123}]},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                  {gen_server,handle_msg,6,
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                      [{file,"gen_server.erl"},{line,1200}]},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                  {proc_lib,init_p_do_apply,3,
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                      [{file,"proc_lib.erl"},{line,240}]}]}
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>     offender: [{pid,<0.1440.0>},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                {id,rabbit_node_monitor},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                {mfargs,{rabbit_node_monitor,start_link,[]}},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                {restart_type,transient},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                {shutdown,300000},
2023-10-08 17:43:20.357950+01:00 [error] <0.1439.0>                {child_type,worker}]
2023-10-08 17:43:20.358547+01:00 [info] <0.1853.0> Starting rabbit_node_monitor
2023-10-08 17:43:21.719413+01:00 [notice] <0.60.0> SIGTERM received - shutting down
2023-10-08 17:43:21.719413+01:00 [notice] <0.60.0> 

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)
• Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask on the mailing list.
We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc.

[michaelklishin]

Thanks. All this partition handling strategies will be removed once Khepri becomes the default (in the next few months) but for now, we are willing to accept small incremental improvements like this.