Handle XNNHeader in XNNPACK Runtime #1543
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/1543
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (2 Unrelated Failures)As of commit 5863f26 with merge base 428da4f ( FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D52556131 |
mcr229
force-pushed
the
export-D52556131
branch
from
015f625 to
36a3b01
Compare
mcr229
added a commit
to mcr229/executorch
that referenced
this pull request
Jan 10, 2024
Summary: We introduce XNNHeader on runtime side to handle the newly introduced XNNHeader ahead of time. XNNHeader manages the offsets and sizes of the flatbuffer payload and the constant data payload so that it is accessible by the XNNCompiler It is important to note that on serialization side, we have not yet switched our serialization method to `serialize_xnnpack_binary` so this does not yet use the new serialization format. However, passing tests on this illustrates BC as old models will still be able to run on this new runtime. Passing tests here show that the Header Magic correctly works in discerning between using the XNNHeader and the Flatbuffer header Differential Revision: D52556131
mcr229
added a commit
to mcr229/executorch
that referenced
this pull request
Jan 10, 2024
Summary: We introduce XNNHeader on runtime side to handle the newly introduced XNNHeader ahead of time. XNNHeader manages the offsets and sizes of the flatbuffer payload and the constant data payload so that it is accessible by the XNNCompiler It is important to note that on serialization side, we have not yet switched our serialization method to `serialize_xnnpack_binary` so this does not yet use the new serialization format. However, passing tests on this illustrates BC as old models will still be able to run on this new runtime. Passing tests here show that the Header Magic correctly works in discerning between using the XNNHeader and the Flatbuffer header Differential Revision: D52556131
|
This pull request was exported from Phabricator. Differential Revision: D52556131 |
mcr229
added a commit
to mcr229/executorch
that referenced
this pull request
Jan 10, 2024
Summary: We introduce XNNHeader on runtime side to handle the newly introduced XNNHeader ahead of time. XNNHeader manages the offsets and sizes of the flatbuffer payload and the constant data payload so that it is accessible by the XNNCompiler It is important to note that on serialization side, we have not yet switched our serialization method to `serialize_xnnpack_binary` so this does not yet use the new serialization format. However, passing tests on this illustrates BC as old models will still be able to run on this new runtime. Passing tests here show that the Header Magic correctly works in discerning between using the XNNHeader and the Flatbuffer header Differential Revision: D52556131
mcr229
added a commit
to mcr229/executorch
that referenced
this pull request
Jan 10, 2024
Summary: We introduce XNNHeader on runtime side to handle the newly introduced XNNHeader ahead of time. XNNHeader manages the offsets and sizes of the flatbuffer payload and the constant data payload so that it is accessible by the XNNCompiler It is important to note that on serialization side, we have not yet switched our serialization method to `serialize_xnnpack_binary` so this does not yet use the new serialization format. However, passing tests on this illustrates BC as old models will still be able to run on this new runtime. Passing tests here show that the Header Magic correctly works in discerning between using the XNNHeader and the Flatbuffer header Differential Revision: D52556131
mcr229
force-pushed
the
export-D52556131
branch
from
36a3b01 to
08a6f7a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D52556131 |
mcr229
added a commit
to mcr229/executorch
that referenced
this pull request
Jan 10, 2024
Summary: We introduce XNNHeader on runtime side to handle the newly introduced XNNHeader ahead of time. XNNHeader manages the offsets and sizes of the flatbuffer payload and the constant data payload so that it is accessible by the XNNCompiler It is important to note that on serialization side, we have not yet switched our serialization method to `serialize_xnnpack_binary` so this does not yet use the new serialization format. However, passing tests on this illustrates BC as old models will still be able to run on this new runtime. Passing tests here show that the Header Magic correctly works in discerning between using the XNNHeader and the Flatbuffer header Differential Revision: D52556131
mcr229
force-pushed
the
export-D52556131
branch
from
08a6f7a to
e2520b3
Compare
|
This pull request was exported from Phabricator. Differential Revision: D52556131 |
mcr229
added a commit
to mcr229/executorch
that referenced
this pull request
Jan 10, 2024
Summary: We introduce XNNHeader on runtime side to handle the newly introduced XNNHeader ahead of time. XNNHeader manages the offsets and sizes of the flatbuffer payload and the constant data payload so that it is accessible by the XNNCompiler It is important to note that on serialization side, we have not yet switched our serialization method to `serialize_xnnpack_binary` so this does not yet use the new serialization format. However, passing tests on this illustrates BC as old models will still be able to run on this new runtime. Passing tests here show that the Header Magic correctly works in discerning between using the XNNHeader and the Flatbuffer header Differential Revision: D52556131
mcr229
force-pushed
the
export-D52556131
branch
from
e2520b3 to
383769a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D52556131 |
mcr229
force-pushed
the
export-D52556131
branch
from
383769a to
77defc6
Compare
mcr229
added a commit
to mcr229/executorch
that referenced
this pull request
Jan 11, 2024
Summary: We introduce XNNHeader on runtime side to handle the newly introduced XNNHeader ahead of time. XNNHeader manages the offsets and sizes of the flatbuffer payload and the constant data payload so that it is accessible by the XNNCompiler It is important to note that on serialization side, we have not yet switched our serialization method to `serialize_xnnpack_binary` so this does not yet use the new serialization format. However, passing tests on this illustrates BC as old models will still be able to run on this new runtime. Passing tests here show that the Header Magic correctly works in discerning between using the XNNHeader and the Flatbuffer header Reviewed By: digantdesai Differential Revision: D52556131
|
This pull request was exported from Phabricator. Differential Revision: D52556131 |
…ytorch#1523) Summary: Introducing the XNNPACKHeader to manage the flatbuffer data and constant data. Previously, we have serialized constant data along with flatbuffer. However, with large weights and large tensors in general, this takes a large amount of time and memory converting our dataclass --> json --> flatbuffer. This has become a blocker on some larger models To fix, we circumvent serializing constant tensors via flatbuffer, by appending the constant data after the flatbuffer payload. In order to do this, we need an XNNPACKHeader which will give us the flatbuffer offset, flatbuffer size, constant data offset, and constant data sizes. It will look something like this: ``` ┌───────────────────────────────────┐ │XNNPACK Header │ ├───────────────────────────────────┤ │Padding for 16 byte alignment │ ├───────────────────────────────────┤ │Flatbuffer-serialized payload data │ │ │ │ │ ├───────────────────────────────────┤ │Padding for 16 byte alignment │ ├───────────────────────────────────┤ │Constant Data │ │ │ │ │ └───────────────────────────────────┘ ``` Within the XNNPACK Header, we hold the following: - 4 bytes to offset the header magic - 4 bytes for the header magic - 2 bytes for the header length - 4 bytes for the flatbuffer offset - 4 bytes for the flatbuffer size - 4 bytes for constant data offset - 8 bytes for constant data size Reviewed By: digantdesai Differential Revision: D52497977
Summary: We introduce the `serialize_xnnpack_binary` method which serializees the constant data outside of the flatbuffer. It leverages the xnnheader introduced in the previous diff to store offsets and sizes for both the flatbuffer payload as well as the constant data payload. Note here we have not yet switched the delegate to use the new `serialize_xnnpack_binary` function as this new serialization also requires changes on the runtime side. This will be tested in the diff which follows. Differential Revision: D52498367
Summary: We introduce XNNHeader on runtime side to handle the newly introduced XNNHeader ahead of time. XNNHeader manages the offsets and sizes of the flatbuffer payload and the constant data payload so that it is accessible by the XNNCompiler It is important to note that on serialization side, we have not yet switched our serialization method to `serialize_xnnpack_binary` so this does not yet use the new serialization format. However, passing tests on this illustrates BC as old models will still be able to run on this new runtime. Passing tests here show that the Header Magic correctly works in discerning between using the XNNHeader and the Flatbuffer header Reviewed By: digantdesai Differential Revision: D52556131
mcr229
force-pushed
the
export-D52556131
branch
from
77defc6 to
5863f26
Compare
|
This pull request was exported from Phabricator. Differential Revision: D52556131 |
|
This pull request has been merged in a8da94a. |
SS-JIA
added a commit
that referenced
this pull request
Feb 20, 2024
## Context This changeset is a mirror of #1542 and #1543 for the Vulkan delegate, which enables serialization of constant weight data outside the flatbuffer blob. I decided to package some miscellaneous codebase improvements with this changeset: * Update field names of `VkGraph` table in `schema.fbs`: * In `VulkanBackend.cpp`, use aliases for generated flatbuffer types * Move build targets to a `targets.bzl` file with a `define_common_targets()` method Differential Revision: [D53957852](https://our.internmc.facebook.com/intern/diff/D53957852/) [ghstack-poisoned]
SS-JIA
added a commit
that referenced
this pull request
Feb 20, 2024
## Context This changeset is a mirror of #1542 and #1543 for the Vulkan delegate, which enables serialization of constant weight data outside the flatbuffer blob. I decided to package some miscellaneous codebase improvements with this changeset: * Update field names of `VkGraph` table in `schema.fbs`: * In `VulkanBackend.cpp`, use aliases for generated flatbuffer types * Move build targets to a `targets.bzl` file with a `define_common_targets()` method Differential Revision: [D53957852](https://our.internmc.facebook.com/intern/diff/D53957852/) ghstack-source-id: 215772816 Pull Request resolved: #2014
SS-JIA
added a commit
that referenced
this pull request
Feb 20, 2024
## Context This changeset is a mirror of #1542 and #1543 for the Vulkan delegate, which enables serialization of constant weight data outside the flatbuffer blob. I decided to package some miscellaneous codebase improvements with this changeset: * Update field names of `VkGraph` table in `schema.fbs` and update the namespace to be simply `vkgraph`: * In `VulkanBackend.cpp`, use aliases for generated flatbuffer types * Move build targets to a `targets.bzl` file with a `define_common_targets()` method Differential Revision: [D53957852](https://our.internmc.facebook.com/intern/diff/D53957852/) [ghstack-poisoned]
SS-JIA
added a commit
that referenced
this pull request
Feb 20, 2024
## Context This changeset is a mirror of #1542 and #1543 for the Vulkan delegate, which enables serialization of constant weight data outside the flatbuffer blob. I decided to package some miscellaneous codebase improvements with this changeset: * Update field names of `VkGraph` table in `schema.fbs` and update the namespace to be simply `vkgraph`: * In `VulkanBackend.cpp`, use aliases for generated flatbuffer types * Move build targets to a `targets.bzl` file with a `define_common_targets()` method Differential Revision: [D53957852](https://our.internmc.facebook.com/intern/diff/D53957852/) ghstack-source-id: 215800831 Pull Request resolved: #2016
SS-JIA
added a commit
that referenced
this pull request
Feb 20, 2024
## Context This changeset is a mirror of #1542 and #1543 for the Vulkan delegate, which enables serialization of constant weight data outside the flatbuffer blob. I decided to package some miscellaneous codebase improvements with this changeset: * Update field names of `VkGraph` table in `schema.fbs` and update the namespace to be simply `vkgraph`: * In `VulkanBackend.cpp`, use aliases for generated flatbuffer types * Move build targets to a `targets.bzl` file with a `define_common_targets()` method Differential Revision: [D53957852](https://our.internmc.facebook.com/intern/diff/D53957852/) [ghstack-poisoned]
SS-JIA
added a commit
that referenced
this pull request
Feb 20, 2024
Pull Request resolved: #2016 ## Context This changeset is a mirror of #1542 and #1543 for the Vulkan delegate, which enables serialization of constant weight data outside the flatbuffer blob. I decided to package some miscellaneous codebase improvements with this changeset: * Update field names of `VkGraph` table in `schema.fbs` and update the namespace to be simply `vkgraph`: * In `VulkanBackend.cpp`, use aliases for generated flatbuffer types * Move build targets to a `targets.bzl` file with a `define_common_targets()` method ghstack-source-id: 215803961 @exported-using-ghexport Differential Revision: [D53957852](https://our.internmc.facebook.com/intern/diff/D53957852/)
facebook-github-bot
pushed a commit
that referenced
this pull request
Feb 21, 2024
Summary: Pull Request resolved: #2016 ## Context This changeset is a mirror of #1542 and #1543 for the Vulkan delegate, which enables serialization of constant weight data outside the flatbuffer blob. I decided to package some miscellaneous codebase improvements with this changeset: * Update field names of `VkGraph` table in `schema.fbs` and update the namespace to be simply `vkgraph`: * In `VulkanBackend.cpp`, use aliases for generated flatbuffer types * Move build targets to a `targets.bzl` file with a `define_common_targets()` method ghstack-source-id: 215803961 exported-using-ghexport Reviewed By: mcr229 Differential Revision: D53957852 fbshipit-source-id: 6d6d9c70df6f8cbd9e85e8294208839be84527eb
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
We introduce XNNHeader on runtime side to handle the newly introduced XNNHeader ahead of time. XNNHeader manages the offsets and sizes of the flatbuffer payload and the constant data payload so that it is accessible by the XNNCompiler
It is important to note that on serialization side, we have not yet switched our serialization method to
serialize_xnnpack_binaryso this does not yet use the new serialization format. However, passing tests on this illustrates BC as old models will still be able to run on this new runtime.Passing tests here show that the Header Magic correctly works in discerning between using the XNNHeader and the Flatbuffer header
Differential Revision: D52556131