Monitor stream local member in stream queue #4217

acogoluegnes · 2022-03-02T15:53:38Z

This PR restarts an Osiris reader if the local member of a stream queue consumer restarts. This situation can happen during cluster upgrades, when stream leader move between nodes, which also implies restarting the processes of stream replicas.

To react to local member changes, this PR introduces a new "local member" listener type to the stream coordinator. This requires changing the structure of the state and a machine version bump. The new version of the stream coordinator is now 2.

The PR takes care of migrating the v1 state to the v2 state in the appropriate state machine callback.

It also removes the stream listeners data properly when the listener processes go away, which was not done previously.

Fixes #4133

acogoluegnes · 2022-03-08T14:05:53Z

Acceptance Steps

Consumer Keeps Consuming When Leader Is Restarted

We will run an AMQP 0.9.1 consumer app on a stream queue, kill the stream leader process, and make sure the application has not stopped consuming.

Start Broker

cd /tmp
git clone [email protected]:rabbitmq/rabbitmq-server.git rabbitmq-server-4133
cd rabbitmq-server-4133
git checkout rabbitmq-server-4133-monitor-stream-local-member
make run-broker PLUGINS=""

Start Client Application

This steps requires JBang to be installed.

cd /tmp
git clone [email protected]:acogoluegnes/sandbox.git
jbang /tmp/sandbox/src/main/java/StreamQueueMonitorStreamLocalMember4133.java

The application publishes a message every second, it also has a consumer that outputs something for each message:

Received message (offset 1)...
Received message (offset 2)...
Received message (offset 3)...
Received message (offset 4)...
Received message (offset 5)...
...

Restart the Stream Leader Process

We need to locate the PID of the stream leader process. Dump the state of the stream coordinator:

/tmp/rabbitmq-server-4133/sbin/rabbitmqctl eval 'rabbit_stream_coordinator:state().'

Output:

{rabbit_stream_coordinator,
    #{"__sq_1646746859112959129" =>
          {stream,"__sq_1646746859112959129",2,
              {resource,<<"/">>,queue,<<"sq">>},
              #{epoch => 1,
                event_formatter =>
                    {rabbit_stream_queue,format_osiris_event,
                        [{resource,<<"/">>,queue,<<"sq">>}]},
                leader_locator_strategy => <<"client-local">>,
                leader_node => 'rabbit@acogoluegnes-nuc',
                name => "__sq_1646746859112959129",
                nodes => ['rabbit@acogoluegnes-nuc'],
                reference => {resource,<<"/">>,queue,<<"sq">>},
                replica_nodes => [],retention => []},
              ['rabbit@acogoluegnes-nuc'],
              #{'rabbit@acogoluegnes-nuc' =>
                    {member,
                        {running,2,<11591.1008.0>},
                        {writer,2},
                        'rabbit@acogoluegnes-nuc',undefined,
                        #{epoch => 2,
                          event_formatter =>
                              {rabbit_stream_queue,format_osiris_event,
                                  [{resource,<<"/">>,queue,<<"sq">>}]},
                          leader_locator_strategy => <<"client-local">>,
                          leader_node => 'rabbit@acogoluegnes-nuc',
                          name => "__sq_1646746859112959129",
                          nodes => ['rabbit@acogoluegnes-nuc'],
                          reference => {resource,<<"/">>,queue,<<"sq">>},
                          replica_nodes => [],retention => []},
                        running}},
              #{{<11591.1878.0>,leader} => <11591.1008.0>,
                {<11591.1883.0>,leader} => <11591.1008.0>,
                {<11591.1883.0>,member} =>
                    {'rabbit@acogoluegnes-nuc',<11591.1008.0>}},
              undefined,
              {updated,2},
              running}},
    #{<11591.1008.0> => {"__sq_1646746859112959129",member},
      <11591.1878.0> => {#{"__sq_1646746859112959129" => ok},listener},
      <11591.1883.0> => {#{"__sq_1646746859112959129" => ok},listener}},
    #{},undefined,undefined}

Locate the stream leader process, in our case it is <11591.1008.0>:

{running,2,<11591.1008.0>}

Make the PID a local one by replacing the first part by 0, in our case <11591.1008.0> becomes <0.1008.0>. Then run the following command with the local PID to kill the leader process:

/tmp/rabbitmq-server-4133/sbin/rabbitmqctl eval 'exit(list_to_pid("<0.1008.0>"), kill).'

If you dump the stream coordinator state again, the stream leader PID should have changed:

/tmp/rabbitmq-server-4133/sbin/rabbitmqctl eval 'rabbit_stream_coordinator:state().'

If you go back to the client application tab, it should still publish and consume, it should not have stopped.

Stop the client application (Ctrl+C) and the broker.

Cluster Upgrade

The PR changes the structure of the stream coordinator state, this implies a machine version upgrade. The following steps will check that a 3-node cluster can be upgraded.

Create a generic Unix package from the branch and install it:

make -C /tmp/rabbitmq-server-4133 package-generic-unix
mkdir /tmp/rabbitmq_server_4133
tar -xf /tmp/rabbitmq-server-4133/PACKAGES/rabbitmq-server-generic-unix-*.tar.xz --directory /tmp/rabbitmq_server_4133 --strip-components 1
echo "log.file.level = debug" > /tmp/rabbitmq_server_4133/etc/rabbitmq/rabbitmq.conf

Get the generic Unix package of the latest 3.10 alpha and extract it (change [ALPHA] to the appropriate value):

mkdir /tmp/rabbitmq_server_310
tar -xf rabbitmq-server-generic-unix-3.10.0-alpha.[ALPHA].tar.xz --directory rabbitmq_server_310 --strip-components 1
echo "log.file.level = debug" > /tmp/rabbitmq_server_310/etc/rabbitmq/rabbitmq.conf
mkdir /tmp/rabbitmq-4133-mnesia-base

Open 3 different tabs (one for each node) and launch the following commands.

TAB 1:

export RABBITMQ_NODENAME="rabbit1"
export RABBITMQ_NODE_PORT=5672
export RABBITMQ_DIST_PORT=25672
export RABBITMQ_MNESIA_BASE=/tmp/rabbitmq-4133-mnesia-base
alias rctl="/tmp/rabbitmq_server_310/sbin/rabbitmqctl"

TAB 2:

export RABBITMQ_NODENAME="rabbit2"
export RABBITMQ_NODE_PORT=5673
export RABBITMQ_DIST_PORT=25673
export RABBITMQ_MNESIA_BASE=/tmp/rabbitmq-4133-mnesia-base
alias rctl="/tmp/rabbitmq_server_310/sbin/rabbitmqctl"

TAB 3:

export RABBITMQ_NODENAME="rabbit3"
export RABBITMQ_NODE_PORT=5674
export RABBITMQ_DIST_PORT=25674
export RABBITMQ_MNESIA_BASE=/tmp/rabbitmq-4133-mnesia-base
alias rctl="/tmp/rabbitmq_server_310/sbin/rabbitmqctl"

Now start each node.

On each tab:

/tmp/rabbitmq_server_310/sbin/rabbitmq-server -detached

Form the cluster:

TAB 2 & 3:

rctl stop_app
rctl join_cluster rabbit1@$(hostname -s)
rctl start_app

Make sure the 3 nodes joined the cluster.

ANY TAB:

rctl cluster_status

Start the client application (requires the sandbox repository and JBang to be installed, see above for details).

ANY TAB:

jbang /tmp/sandbox/src/main/java/StreamQueueMonitorStreamLocalMember4133.java

Stop the client after a few messages (Ctrl+C). Check the members of the stream coordinator.

ANY TAB:

rctl eval 'ra:members(rabbit_stream_coordinator).'

It should output something like the following (the 3 members and the leader):

{ok,[{rabbit_stream_coordinator,'rabbit1@acogoluegnes-nuc'},
     {rabbit_stream_coordinator,'rabbit2@acogoluegnes-nuc'},
     {rabbit_stream_coordinator,'rabbit3@acogoluegnes-nuc'}],
    {rabbit_stream_coordinator,'rabbit1@acogoluegnes-nuc'}}

We're going now to upgrade the nodes.

The first node.

TAB 1

rctl stop
alias rctl="/tmp/rabbitmq_server_4133/sbin/rabbitmqctl"
/tmp/rabbitmq_server_4133/sbin/rabbitmq-server -detached

Make sure the node joined the cluster again.

ANY TAB:

rctl cluster_status

The second node.

TAB 2

rctl stop
alias rctl="/tmp/rabbitmq_server_4133/sbin/rabbitmqctl"
/tmp/rabbitmq_server_4133/sbin/rabbitmq-server -detached

Make sure the node joined the cluster again.

ANY TAB:

rctl cluster_status

Start the client application again.

TAB 1:

jbang /tmp/sandbox/src/main/java/StreamQueueMonitorStreamLocalMember4133.java

Check the state of the stream coordinator.

TAB 2:

rctl eval 'rabbit_stream_coordinator:state().'

The output should be like the following:

{rabbit_stream_coordinator,
    #{"__sq_1646920629979595000" =>
          {stream,"__sq_1646920629979595000",3,
              {resource,<<"/">>,queue,<<"sq">>},
              #{epoch => 1,
                event_formatter =>
                    {rabbit_stream_queue,format_osiris_event,
                        [{resource,<<"/">>,queue,<<"sq">>}]},
                leader_locator_strategy => <<"client-local">>,
                leader_node => 'rabbit1@acogoluegne-a01',
                name => "__sq_1646920629979595000",
                nodes =>
                    ['rabbit1@acogoluegne-a01','rabbit3@acogoluegne-a01',
                     'rabbit2@acogoluegne-a01'],
                reference => {resource,<<"/">>,queue,<<"sq">>},
                replica_nodes =>
                    ['rabbit3@acogoluegne-a01','rabbit2@acogoluegne-a01'],
                retention => []},
              ['rabbit1@acogoluegne-a01','rabbit3@acogoluegne-a01',
               'rabbit2@acogoluegne-a01'],
              ...
              #{<12133.734.0> => <12133.624.0>,
                 <12133.739.0> => <12133.624.0>,
                 <12133.1196.0> => <12133.624.0>,
                 <12133.1221.0> => <12133.624.0>},
              undefined,
              {updated,3},
              running}},
    #{<11582.585.0> => {"__sq_1646920629979595000",member},
      <12133.624.0> => {"__sq_1646920629979595000",member},
      <12133.734.0> => {"__sq_1646920629979595000",listener},
      <12133.739.0> => {"__sq_1646920629979595000",listener},
      <12134.1614.0> => {"__sq_1646920629979595000",member}},
    #{},undefined,undefined}

In our case, a map of listeners contain terminated PIDs that should have been removed (<12133.1196.0>, <12133.1221.0>). They are the PIDs of the channels of the previous run of the client application. They should go away when all the nodes are upgraded.

Upgrade the third node.

TAB 3

rctl stop
alias rctl="/tmp/rabbitmq_server_4133/sbin/rabbitmqctl"
/tmp/rabbitmq_server_4133/sbin/rabbitmq-server -detached

Make sure the node joined the cluster again.

ANY TAB:

rctl cluster_status

Dump the coordinator state.

ANY TAB:

rctl eval 'rabbit_stream_coordinator:state().'

The output should something like this:

{rabbit_stream_coordinator,
    #{"__sq_1646920629979595000" =>
          {stream,"__sq_1646920629979595000",3,
              {resource,<<"/">>,queue,<<"sq">>},
              #{epoch => 1,
                event_formatter =>
                    {rabbit_stream_queue,format_osiris_event,
                        [{resource,<<"/">>,queue,<<"sq">>}]},
                leader_locator_strategy => <<"client-local">>,
                leader_node => 'rabbit1@acogoluegne-a01',
                name => "__sq_1646920629979595000",
                nodes =>
                    ['rabbit1@acogoluegne-a01','rabbit3@acogoluegne-a01',
                     'rabbit2@acogoluegne-a01'],
                reference => {resource,<<"/">>,queue,<<"sq">>},
                replica_nodes =>
                    ['rabbit3@acogoluegne-a01','rabbit2@acogoluegne-a01'],
                retention => []},
              ['rabbit1@acogoluegne-a01','rabbit3@acogoluegne-a01',
               'rabbit2@acogoluegne-a01'],
             ...
              #{{<12133.734.0>,leader} => <12133.624.0>,
                 {<12133.739.0>,leader} => <12133.624.0>},
              undefined,
              {updated,3},
              running}},
    #{<12134.585.0> => {"__sq_1646920629979595000",member},
      <11582.592.0> => {"__sq_1646920629979595000",member},
      <12133.624.0> => {"__sq_1646920629979595000",member},
      <12133.734.0> => {#{"__sq_1646920629979595000" => ok},listener},
      <12133.739.0> => {#{"__sq_1646920629979595000" => ok},listener}},
    undefined,undefined,undefined}

The listeners should have been converted to the new representation:

#{{<12133.734.0>,leader} => <12133.624.0>,
   {<12133.739.0>,leader} => <12133.624.0>},

The stale PIDs should have been collected.

Even though the client application was launched when 2 nodes were on the last version, the stream coordinator was still running the previous version of the state machine. Let's stop and restart the client application and check out the coordinator state.

Ctrl-C the client application and start it again.

ANY TAB:

jbang /tmp/sandbox/src/main/java/StreamQueueMonitorStreamLocalMember4133.java

Dump the coordinator state.

ANY TAB:

rctl eval 'rabbit_stream_coordinator:state().'

{rabbit_stream_coordinator,
    #{"__sq_1646920629979595000" =>
          {stream,"__sq_1646920629979595000",3,
              {resource,<<"/">>,queue,<<"sq">>},
              #{epoch => 1,
                event_formatter =>
                    {rabbit_stream_queue,format_osiris_event,
                        [{resource,<<"/">>,queue,<<"sq">>}]},
                leader_locator_strategy => <<"client-local">>,
                leader_node => 'rabbit1@acogoluegne-a01',
                name => "__sq_1646920629979595000",
                nodes =>
                    ['rabbit1@acogoluegne-a01','rabbit3@acogoluegne-a01',
                     'rabbit2@acogoluegne-a01'],
                reference => {resource,<<"/">>,queue,<<"sq">>},
                replica_nodes =>
                    ['rabbit3@acogoluegne-a01','rabbit2@acogoluegne-a01'],
                retention => []},
              ['rabbit1@acogoluegne-a01','rabbit3@acogoluegne-a01',
               'rabbit2@acogoluegne-a01'],
              ...
              #{{<12133.1586.0>,leader} => <12133.624.0>,
                 {<12133.1591.0>,leader} => <12133.624.0>,
                 {<12133.1591.0>,member} =>
                    {'rabbit1@acogoluegne-a01',<12133.624.0>}},
              undefined,
              {updated,3},
              running}},
    #{<11582.585.0> => {"__sq_1646920629979595000",member},
      <12134.592.0> => {"__sq_1646920629979595000",member},
      <12133.624.0> => {"__sq_1646920629979595000",member},
      <12133.1586.0> => {#{"__sq_1646920629979595000" => ok},listener},
      <12133.1591.0> => {#{"__sq_1646920629979595000" => ok},listener}},
    undefined,undefined,undefined}

Note the map of listeners, it has now a member listener which will keep a consumer consuming when its local member changes:

#{{<12133.1586.0>,leader} => <12133.624.0>,
   {<12133.1591.0>,leader} => <12133.624.0>,
   {<12133.1591.0>,member} => {'rabbit1@acogoluegne-a01',<12133.624.0>}}

Stop the client application. Shut down the cluster.

On each tab:

rctl stop

Delete the directories:

cd /tmp
rm -rf /tmp/rabbitmq-4133-mnesia-base /tmp/rabbitmq-server-4133 /tmp/rabbitmq_server_310 /tmp/rabbitmq_server_4133 /tmp/sandbox

This commit refactors the stream coordinator listener mechanism to support listeners that get notified when a stream member on the same node of the listening process gets restarted. Local member listeners are useful for AMQP consumers. The change requires a state machine version bump as it changes the state of the stream coordinator. This will be done in a subsequent commit. References #4133 (cherry picked from commit 3db593f8b02e17c23fe6e188ec775961b6f7cf94)

Adapt it to the new structure of stream listener. References #4133

This commit re-creates an Osiris reader when the local member of the stream cluster changes. Consumers can then restart dispatching messages where they left off instead of hanging. References #4133

The stream coordinator have the most up-to-date information. References #4133

Version bump necessary because of the state changes to handle local member listeners. References #4133

These 2 tests can fail in mixed-version, 2-node cluster testing. If the stream coordinator leader ends up on the lower version, it does not contain the fixes and the tests fail. With this commit, the 2 tests are skipped under the appropriate conditions. References #4133

References #4133

These tests requires a level of availability that mixed-version clusters cannot provide, so they are skipped under these conditions. References #4133

Unlikely to pass as it randomly stops a node, which can be the stream coordinator leader. The 2 remaining nodes then cannot elect a new leader because they don't have the same version. References #4133

In stream coordinator. A v2 node, even if running v1, can call the function to register a local member when an AMQP listener is registered, so this commit makes such a call a no-op. References #4133

Monitors were not converted. References #4133

References #4133

kjnilsson

Looks generally good. A few formatting issues and a couple of suggestions.

deps/rabbit/src/rabbit_stream_coordinator.erl

kjnilsson · 2022-03-11T16:14:07Z

deps/rabbit/src/rabbit_stream_coordinator.hrl

-                 listeners = #{} :: #{pid() := LeaderPid :: pid()},
+                 listeners = #{} :: #{pid() | %% v0 & v1
+                                      {pid(), leader | member} %% v2
+                                      := LeaderPid :: pid()} | {node(), LocalPid :: pid()},


Why include a node() here? wont the node always be the one of the listener pid?

deps/rabbit/src/rabbit_stream_coordinator.erl

michaelklishin

All acceptance steps works as expected, although I had to tweak mkdir to use mkdir -p here and there.

michaelklishin · 2022-03-14T13:58:46Z

@acogoluegnes I assumed that most feedback above (seems to be cosmetics and minor code style tweaks) is going to be addressed and focused on the functional part in my review.

michaelklishin · 2022-03-16T17:40:51Z

@kjnilsson ready for another round

(cherry picked from commit 5692272)

(cherry picked from commit d484e04)

(cherry picked from commit 46662ec)

Monitor stream local member in stream queue (backport #4217)

(cherry picked from commit 5692272) (cherry picked from commit 7188374)

(cherry picked from commit d484e04) (cherry picked from commit f6c0d5a)

(cherry picked from commit 46662ec) (cherry picked from commit f1a0bd7)

Monitor stream local member in stream queue (backport #4217) (backport #4282)

acogoluegnes force-pushed the rabbitmq-server-4133-monitor-stream-local-member branch from ae7bcdc to 4f7b7c8 Compare March 8, 2022 16:09

acogoluegnes added 10 commits March 8, 2022 17:40

Fix eol sending for stream queue

9aa7315

Adapt it to the new structure of stream listener. References #4133

Re-create Osiris reader on local member change

db8d940

This commit re-creates an Osiris reader when the local member of the stream cluster changes. Consumers can then restart dispatching messages where they left off instead of hanging. References #4133

Use stream coordinator to check local member presence

c2cfd70

The stream coordinator have the most up-to-date information. References #4133

Delete queue at end of test

ae1684e

Bump machine version from 1 to 2 in stream coordinator

636bb55

Version bump necessary because of the state changes to handle local member listeners. References #4133

Use node with latest machine version to connect

7c47f00

References #4133

Fix a couple of typo

fe7358e

Skip stream queue tests in mixed-version cluster testing

e6a2670

These tests requires a level of availability that mixed-version clusters cannot provide, so they are skipped under these conditions. References #4133

acogoluegnes force-pushed the rabbitmq-server-4133-monitor-stream-local-member branch from 4f7b7c8 to e6a2670 Compare March 8, 2022 16:40

acogoluegnes added 4 commits March 8, 2022 18:20

Filter a stream queue test in mixed-version cluster mode

824bf6a

Unlikely to pass as it randomly stops a node, which can be the stream coordinator leader. The 2 remaining nodes then cannot elect a new leader because they don't have the same version. References #4133

Do not change state for local member listener in v1

6133416

In stream coordinator. A v2 node, even if running v1, can call the function to register a local member when an AMQP listener is registered, so this commit makes such a call a no-op. References #4133

Fix stream coordinator v1-to-v2 machine state change

df61e44

Monitors were not converted. References #4133

Re-issue monitors to clean up stale listeners

6ab1158

References #4133

acogoluegnes marked this pull request as ready for review March 10, 2022 14:14

kjnilsson requested changes Mar 11, 2022

View reviewed changes

michaelklishin approved these changes Mar 14, 2022

View reviewed changes

michaelklishin added backport-v3.9.x labels Mar 14, 2022

michaelklishin added 3 commits March 16, 2022 21:32

Address #4217 QA feedback (cosmetics)

5692272

More #4217 feedback (cosmetics)

d484e04

More #4217 feedback (cosmetics)

46662ec

michaelklishin approved these changes Mar 16, 2022

View reviewed changes

kjnilsson approved these changes Mar 17, 2022

View reviewed changes

michaelklishin merged commit 361f9de into master Mar 17, 2022

michaelklishin deleted the rabbitmq-server-4133-monitor-stream-local-member branch March 17, 2022 11:59

mergify bot mentioned this pull request Mar 17, 2022

Monitor stream local member in stream queue (backport #4217) #4282

Merged

mergify bot pushed a commit that referenced this pull request Mar 17, 2022

Address #4217 QA feedback (cosmetics)

7188374

(cherry picked from commit 5692272)

mergify bot pushed a commit that referenced this pull request Mar 17, 2022

More #4217 feedback (cosmetics)

f6c0d5a

(cherry picked from commit d484e04)

mergify bot pushed a commit that referenced this pull request Mar 17, 2022

More #4217 feedback (cosmetics)

f1a0bd7

(cherry picked from commit 46662ec)

michaelklishin added a commit that referenced this pull request Mar 17, 2022

Merge pull request #4282 from rabbitmq/mergify/bp/v3.10.x/pr-4217

fb96a39

Monitor stream local member in stream queue (backport #4217)

mergify bot pushed a commit that referenced this pull request Mar 17, 2022

Address #4217 QA feedback (cosmetics)

2f3baf6

(cherry picked from commit 5692272) (cherry picked from commit 7188374)

mergify bot pushed a commit that referenced this pull request Mar 17, 2022

More #4217 feedback (cosmetics)

502ad15

(cherry picked from commit d484e04) (cherry picked from commit f6c0d5a)

mergify bot pushed a commit that referenced this pull request Mar 17, 2022

More #4217 feedback (cosmetics)

18f60db

(cherry picked from commit 46662ec) (cherry picked from commit f1a0bd7)

michaelklishin added a commit that referenced this pull request Mar 17, 2022

Merge pull request #4283 from rabbitmq/mergify/bp/v3.9.x/pr-4282

9f61f5f

Monitor stream local member in stream queue (backport #4217) (backport #4282)

michaelklishin added a commit that referenced this pull request Mar 18, 2022

Address #4217 QA feedback (cosmetics)

6e60721

michaelklishin added a commit that referenced this pull request Mar 18, 2022

More #4217 feedback (cosmetics)

7b8bb78

michaelklishin added a commit that referenced this pull request Mar 18, 2022

More #4217 feedback (cosmetics)

eef68fa

acogoluegnes mentioned this pull request Aug 6, 2024

Cancel AMQP stream consumer when local stream member is deleted #11915

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Monitor stream local member in stream queue #4217

Monitor stream local member in stream queue #4217

Uh oh!

acogoluegnes commented Mar 2, 2022 •

edited

Loading

Uh oh!

acogoluegnes commented Mar 8, 2022 •

edited

Loading

Uh oh!

kjnilsson left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kjnilsson Mar 11, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelklishin left a comment

Uh oh!

michaelklishin commented Mar 14, 2022

Uh oh!

michaelklishin commented Mar 16, 2022

Uh oh!

Uh oh!

Monitor stream local member in stream queue #4217

Monitor stream local member in stream queue #4217

Uh oh!

Conversation

acogoluegnes commented Mar 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

acogoluegnes commented Mar 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Acceptance Steps

Consumer Keeps Consuming When Leader Is Restarted

Start Broker

Start Client Application

Restart the Stream Leader Process

Cluster Upgrade

Uh oh!

kjnilsson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kjnilsson Mar 11, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelklishin left a comment

Choose a reason for hiding this comment

Uh oh!

michaelklishin commented Mar 14, 2022

Uh oh!

michaelklishin commented Mar 16, 2022

Uh oh!

Uh oh!

acogoluegnes commented Mar 2, 2022 •

edited

Loading

acogoluegnes commented Mar 8, 2022 •

edited

Loading