Add workaround for rc: -13 #1108
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I've commonly hit this when configuring prometheus:
```
TASK [prometheus : Get container facts] *************************************************************************************************************************************
Monday 24 June 2024 11:09:37 +0000 (0:00:08.528) 0:01:31.707 ***********
fatal: [will-compute-01]: FAILED! => changed=false
module_stderr: ''
module_stdout: ''
msg: |-
MODULE FAILURE
See stdout/stderr for the exact error
rc: -13
fatal: [will-compute-02]: FAILED! => changed=false
module_stderr: ''
module_stdout: ''
msg: |-
MODULE FAILURE
See stdout/stderr for the exact error
rc: -13
```
The ControlPersist workaround is documented in these bug reports:
- ansible/ansible#78344
- ansible/ansible#81777
From the comments, It seems like this does not completely resolve the
issue, but does decrease the frequency that you hit this.
The Prometheus tasks seem particuarly susceptible as they run on
every host.
|
Worth noting that the other related settings have sensible defaults: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ssh_connection.html#parameter-control_path |
|
How well tested is this change? Is it safer to bring it in with Caracal? |
Currently testing with multinode. I will mark as draft for now as I still haven't reached the end of a multinode deployment due to various unrelated issues :D |
|
This seems to have made my multinode deployments more reliable - I've completed several runs end to end now. |
markgoddard
approved these changes
Jun 26, 2024
|
Still hitting this issue when trying to deploy multinode on : Is it worth bumping even more ? I am testing with 2h now |
|
Just hit this in the Groningen Habrok Antelope upgrade |
markgoddard
added a commit
to stackhpc/terraform-kayobe-multinode
that referenced
this pull request
Aug 27, 2024
There is a race condition in Ansible that can result in this failure: msg: |- MODULE FAILURE See stdout/stderr for the exact error rc: -13 See ansible/ansible#78344 and ansible/ansible#81777. In stackhpc/stackhpc-kayobe-config#1108 we applied a workaround to increase the ControlPersist timeout to 1 hour, but this does not always work. Try another workaround of removing the ControlPersist sockets in between Kayobe runs.
markgoddard
added a commit
to stackhpc/terraform-kayobe-multinode
that referenced
this pull request
Aug 27, 2024
There is a race condition in Ansible that can result in this failure: msg: |- MODULE FAILURE See stdout/stderr for the exact error rc: -13 See ansible/ansible#78344 and ansible/ansible#81777. In stackhpc/stackhpc-kayobe-config#1108 we applied a workaround to increase the ControlPersist timeout to 1 hour, but this does not always work. Disabling SSH pipelining prevents the issue at the cost of Ansible execution duration.
markgoddard
pushed a commit
that referenced
this pull request
Aug 27, 2024
I've commonly hit this when configuring prometheus:
```
TASK [prometheus : Get container facts] *************************************************************************************************************************************
Monday 24 June 2024 11:09:37 +0000 (0:00:08.528) 0:01:31.707 ***********
fatal: [will-compute-01]: FAILED! => changed=false
module_stderr: ''
module_stdout: ''
msg: |-
MODULE FAILURE
See stdout/stderr for the exact error
rc: -13
fatal: [will-compute-02]: FAILED! => changed=false
module_stderr: ''
module_stdout: ''
msg: |-
MODULE FAILURE
See stdout/stderr for the exact error
rc: -13
```
The ControlPersist workaround is documented in these bug reports:
- ansible/ansible#78344
- ansible/ansible#81777
From the comments, It seems like this does not completely resolve the
issue, but does decrease the frequency that you hit this.
The Prometheus tasks seem particuarly susceptible as they run on
every host.
(cherry picked from commit 699769c)
markgoddard
added a commit
to stackhpc/terraform-kayobe-multinode
that referenced
this pull request
Aug 29, 2024
There is a race condition in Ansible that can result in this failure: msg: |- MODULE FAILURE See stdout/stderr for the exact error rc: -13 See ansible/ansible#78344 and ansible/ansible#81777. In stackhpc/stackhpc-kayobe-config#1108 we applied a workaround to increase the ControlPersist timeout to 1 hour, but this does not always work. Try another workaround of removing the ControlPersist sockets in between Kayobe runs.
markgoddard
added a commit
to stackhpc/terraform-kayobe-multinode
that referenced
this pull request
Sep 2, 2024
There is a race condition in Ansible that can result in this failure: msg: |- MODULE FAILURE See stdout/stderr for the exact error rc: -13 See ansible/ansible#78344 and ansible/ansible#81777. In stackhpc/stackhpc-kayobe-config#1108 we applied a workaround to increase the ControlPersist timeout to 1 hour, but this does not always work. Here we use a different workaround of disabling SSH pipelining. This has performance implications for Ansible, but is a reasonable trade-off for reliability. We set the config option as an environment variable rather than in ansible.cfg in Kayobe configuration, to avoid a merge conflict on upgrade.
markgoddard
pushed a commit
that referenced
this pull request
Sep 2, 2024
I've commonly hit this when configuring prometheus:
```
TASK [prometheus : Get container facts] *************************************************************************************************************************************
Monday 24 June 2024 11:09:37 +0000 (0:00:08.528) 0:01:31.707 ***********
fatal: [will-compute-01]: FAILED! => changed=false
module_stderr: ''
module_stdout: ''
msg: |-
MODULE FAILURE
See stdout/stderr for the exact error
rc: -13
fatal: [will-compute-02]: FAILED! => changed=false
module_stderr: ''
module_stdout: ''
msg: |-
MODULE FAILURE
See stdout/stderr for the exact error
rc: -13
```
The ControlPersist workaround is documented in these bug reports:
- ansible/ansible#78344
- ansible/ansible#81777
From the comments, It seems like this does not completely resolve the
issue, but does decrease the frequency that you hit this.
The Prometheus tasks seem particuarly susceptible as they run on
every host.
(cherry picked from commit 699769c)
markgoddard
added a commit
that referenced
this pull request
Sep 4, 2024
yoga: Backport Add workaround for rc: -13 (#1108)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've commonly hit this when configuring prometheus:
The ControlPersist workaround is documented in these bug reports:
From the comments, It seems like this does not completely resolve the issue, but does decrease the frequency that you hit this.
The Prometheus tasks seem particuarly susceptible as they run on every host.