Skip to content

Qualcomm AI Engine Direct - Add the argument to specify soc model #5211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 10, 2024
Merged

Qualcomm AI Engine Direct - Add the argument to specify soc model #5211

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
560450a
Qualcomm AI Engine Direct - Add the argument to specify soc model
shewu-quic Sep 10, 2024
d34cd1a
address review
shewu-quic Sep 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion examples/models/llama2/export_llama_lib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,14 @@ def build_args_parser() -> argparse.ArgumentParser:
default=False,
help="Generate logits for all inputs.",
)

parser.add_argument(
"--soc_model",
help="[QNN backend] SoC model of current device. e.g. 'SM8650' for Snapdragon 8 Gen 3.",
type=str,
required=False,
default="SM8650",
)
return parser


Expand Down Expand Up @@ -533,7 +541,7 @@ def _export_llama(modelname, args) -> LLMEdgeManager: # noqa: C901

partitioners.append(
get_qnn_partitioner(
args.use_kv_cache, args.pt2e_quantize, args.num_sharding
args.use_kv_cache, args.pt2e_quantize, args.num_sharding, args.soc_model
)
)
# pyre-ignore: Undefined import [21]: Could not find a module corresponding to import `executorch.backends.qualcomm.utils.utils`
Expand Down
9 changes: 1 addition & 8 deletions examples/qualcomm/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,19 +230,12 @@ def build_executorch_binary(
else:
edge_prog = capture_program(model, inputs)

arch_table = {
"SM8650": QcomChipset.SM8650,
"SM8550": QcomChipset.SM8550,
"SM8475": QcomChipset.SM8475,
"SM8450": QcomChipset.SM8450,
}

backend_options = generate_htp_compiler_spec(
use_fp16=False if quant_dtype else True
)
qnn_partitioner = QnnPartitioner(
generate_qnn_executorch_compiler_spec(
soc_model=arch_table[soc_model],
soc_model=getattr(QcomChipset, soc_model),
backend_options=backend_options,
debug=False,
saver=False,
Expand Down
10 changes: 9 additions & 1 deletion extension/llm/export/partitioner_lib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ def get_qnn_partitioner(
use_kv_cache: bool = False,
pt2e_quantize: Optional[str] = None,
num_sharding: int = 0,
soc_model: str = "SM8650", # default to SM8650
):
assert (
use_kv_cache is True
Expand Down Expand Up @@ -138,9 +139,16 @@ def get_qnn_partitioner(
if pt2e_quantize is not None:
use_fp16 = False

soc_chip_table = {
"SM8650": QcomChipset.SM8650,
"SM8550": QcomChipset.SM8550,
"SM8475": QcomChipset.SM8475,
"SM8450": QcomChipset.SM8450,
}
Comment on lines +142 to +147
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need have this table somewhere in the qualcomm backend

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me change it to

    qnn_partitioner = QnnPartitioner(
        generate_qnn_executorch_compiler_spec(
            soc_model=getattr(QcomChipset, soc_model),
            backend_options=backend_options,
            debug=False,
            saver=False,
            shared_buffer=shared_buffer,
            profile=False,
        ),
        skip_node_id_set,
        skip_node_op_set,
    )

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh is it possible to have a function inside qualcomm like

def get_soc_table():
      return soc_chip_table = {
        "SM8650": QcomChipset.SM8650,
        "SM8550": QcomChipset.SM8550,
        "SM8475": QcomChipset.SM8475,
        "SM8450": QcomChipset.SM8450,
    }

and call it here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's actually what I meatn..and the previous version seems good actually

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your advice.
But I think we don't want to add this table for maintenance convenience.
Because if we would like to add new soc, we will need to maintain extra table.


return QnnPartitioner( # pyre-fixme[16]
generate_qnn_executorch_compiler_spec( # pyre-fixme[16]
soc_model=QcomChipset.SM8650, # default to SM8650 # pyre-fixme[16]
soc_model=soc_chip_table[soc_model], # pyre-fixme[16]
# pyre-fixme[16]
backend_options=generate_htp_compiler_spec(
use_fp16=use_fp16,
Expand Down
Loading