Fix libflatccrt race #10918
Merged
Fix libflatccrt race #10918
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10918
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 1 Pending, 1 Unrelated FailureAs of commit 6a12fa8 with merge base d0464f8 ( NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
jathu
added
ciflow/trunk
ciflow/binaries
and removed
CLA Signed
facebook-github-bot
added
the
CLA Signed
jathu
added
module: build/install
kirklandsign
approved these changes
May 15, 2025
jathu
force-pushed
the
jathu/fix-flatccrt
branch
4 times, most recently
from
26e9d75 to
fb6ae9e
Compare
jathu
force-pushed
the
jathu/fix-flatccrt
branch
7 times, most recently
from
8899a8e to
4b3b4ab
Compare
jathu
force-pushed
the
jathu/fix-flatccrt
branch
10 times, most recently
from
ac72cc1 to
4ccdcc6
Compare
Comment on lines
-23
to
-24
| bash backends/qualcomm/scripts/build.sh --skip_aarch64 --job_number 2 --release || \ | ||
| bash backends/qualcomm/scripts/build.sh --skip_aarch64 --job_number 2 --release --no_clean |
There was a problem hiding this comment.
Is this change related?
There was a problem hiding this comment.
Yes, see the comment right above — this was the way the race was mitigated before in #7570
larryliu0820
approved these changes
May 16, 2025
Comment on lines
-100
to
-105
| if(CMAKE_BUILD_TYPE MATCHES "Debug") | ||
| set(FLATCC_LIB flatccrt_d) | ||
| else() | ||
| set(FLATCC_LIB flatccrt) | ||
| endif() | ||
|
|
There was a problem hiding this comment.
Have you confirmed that debug build will also build flatccrt?
There was a problem hiding this comment.
Yes, it seems to have both libs
kirklandsign
approved these changes
May 16, 2025
hinriksnaer
pushed a commit
to hinriksnaer/executorch
that referenced
this pull request
May 19, 2025
### Summary Seems like there is a race in building `libflatccrt.a`. This issue has existed for a while: pytorch#7300. It was temporarily mitigated in pytorch#7570 by just reducing the parallelism. In this diff I attempt to fix it. This is just my assumption of what is wrong. Given flatccrt builds a debug version with a `_d` suffix, if the target isn't depended on (i.e. some target don't use the conditional target name) then the order of how the lib is built causes a race. So for now, always use the non-debug version. Given it's a race, I was never able to repro the issue locally — I can't guarantee this is the problem. However, it seems my recent changes in pytorch#10855 has increased the frequency of the problem in CI. ### Test plan CI cc @larryliu0820
|
@jathu, looks like this causing issue to build qnn backend in debug mode. Looks like the script to build qnn sdk only covers release mode though |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Seems like there is a race in building
libflatccrt.a. This issue has existed for a while: #7300. It was temporarily mitigated in #7570 by just reducing the parallelism.In this diff I attempt to fix it. This is just my assumption of what is wrong. Given flatccrt builds a debug version with a
_dsuffix, if the target isn't depended on (i.e. some target don't use the conditional target name) then the order of how the lib is built causes a race. So for now, always use the non-debug version.Given it's a race, I was never able to repro the issue locally — I can't guarantee this is the problem. However, it seems my recent changes in #10855 has increased the frequency of the problem in CI.
Test plan
CI
cc @larryliu0820