Qualcomm AI Engine Direct Backend #490

haowhsu-quic · 2023-09-26T10:46:55Z

Hi there,

This commit adds QNN backend. Please check setup.md and README under backends/qnn for more details.

We add pybind dependency at the top level, because we guess you might need it in the near future. If it's not the case, please feel free to tell us. We'll move it under backends/qnn.

Below is some SoC model mapping:

SM8550 -> Snapdragon 8 Gen 2
SM8475 -> Snapdragon 8 Gen 1+
SM8450 -> Snapdragon 8 Gen 1

Please feel free to contact us if you find any problem.

Co-authored-by: shewu-quic [email protected]
Co-authored-by: chunit-quic [email protected]
Co-authored-by: chiwwang [email protected]
Co-authored-by: harshs-qti [email protected]

netlify · 2023-09-26T10:47:01Z

✅ Deploy Preview for resplendent-gnome-14e531 canceled.

Name	Link
🔨 Latest commit	`6567280`
🔍 Latest deploy log	https://app.netlify.com/sites/resplendent-gnome-14e531/deploys/651d99a329b2c30008e01efe

chiwwang · 2023-09-26T11:15:03Z

@haowhsu-quic , missing co-author harshs-qti [email protected]

Harsh helped us to review some documents.

.gitmodules

backends/qnn/CMakeLists.txt

backends/qnn/README.md

backends/qnn/CMakeLists.txt

kimishpatel · 2023-09-26T14:20:09Z

backends/qnn/builders/node_visitor.py

@@ -0,0 +1,344 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.


no need for meta here either and probably other plcaes as well

Removed, thanks for pointing out.

backends/qnn/examples/README.md

backends/qnn/examples/executor_runner/qnn_executor_runner.cpp

kimishpatel · 2023-09-26T14:51:05Z

backends/qnn/examples/executor_runner/qnn_executor_runner.cpp

+    int inference_index = 0;
+    double elapsed_time = 0;
+    while (std::getline(input_list, file_path)) {
+      auto input_files = split(file_path, " ");


nit: ensure input_files.size() == num_inputs?

Changed, thanks for pointing out.

backends/qnn/examples/executor_runner/qnn_executor_runner.cpp

kimishpatel · 2023-09-26T14:58:26Z

backends/qnn/examples/executor_runner/qnn_executor_runner.cpp

+      // Warm up
+      if (FLAGS_warm_up) {
+        ET_LOG(Info, "Perform 3 inference for warming up");
+        for (int i = 0; i < 3; ++i) {


nit: use FLAG_warmup from arg, or just use some const define instead of directly using 3

Change to integer for user to specify, thank you.

backends/qnn/examples/executor_runner/qnn_executor_runner.cpp

backends/qnn/examples/mobilebert_fine_tune.py

backends/qnn/setup.md

mergennachin

Could you move

backends/qnn/examples to top-level examples/qualcomm instead?

Still reviewing...

.gitmodules

backends/qnn/CMakeLists.txt

backends/qnn/setup.md

.gitmodules

mergennachin · 2023-09-26T15:02:03Z

backends/qnn/setup.md

+
+## Compile a model
+
+### for executorch


We are using ExecuTorch (with E and T capitalized) convention, at least in documentation

Scanned and changed for all related keywords, thanks for pointing out.

mergennachin · 2023-09-26T15:02:52Z

backends/qnn/setup.md

+You can find `qnn_executor_runner` under `build_android/backends/qnn`.
+
+
+## Compile a model


could you continue with the 'Step X' enumeration?

Changed, please take a look at the new version, thanks.

backends/qnn/setup.md

mergennachin · 2023-09-26T15:04:47Z

backends/qnn/setup.md

+
+To gain the best performance, we also employ quantization:
+
+```python


could you just move this code under example, that just takes the model file name?

so that later we can test CI etc and make sure it doesn't go out of date.

Done, add a new example as what examples/export/export_example.py does for generating QNN compatible pte.

mergennachin · 2023-09-26T15:07:16Z

backends/qnn/utils/utils.py

+# LICENSE file in the root directory of this source tree.
+
+import sys
+


could mention what this file is about in the docblock

Thanks for the suggestion, will add docstring incrementally in the future.

backends/qnn/setup.md

backends/qnn/tests/test_qnn_delegate.py

backends/qnn/runtime/QnnExecuTorchBackend.cpp

cccclai

This pr looks great! All codes are well organized and easy to read. Thank you! Just left some comments on the naming probably due to rebase.

cccclai · 2023-09-26T16:22:18Z

backends/qnn/qnn_preprocess.py

+    def preprocess(
+        edge_program: ExportedProgram,
+        compile_specs: List[CompileSpec],
+    ) -> bytes:


Suggested change

) -> bytes:

) -> PreprocessResult:

Changed, thanks for pointing out.

cccclai · 2023-09-26T17:17:37Z

backends/qnn/partition/common_defs.py

+    exir_ops.edge.aten.full.default,
+]
+
+white_list_operator = [


Suggested change

white_list_operator = [

allow_list_operator = [

Changed, thank you.

cccclai · 2023-09-26T17:18:51Z

backends/qnn/partition/qnn_partitioner.py

+                self.partition_tags[delegation_tag] = self.delegation_spec
+
+    # override
+    def partition(self, exported_program: ExportedProgram) -> torch.fx.GraphModule:


Suggested change

def partition(self, exported_program: ExportedProgram) -> torch.fx.GraphModule:

def partition(self, exported_program: ExportedProgram) -> PartitionResult:

Changed, thanks for pointing out.

cccclai · 2023-09-26T17:23:32Z

backends/qnn/passes/fold_qdq.py

+            source_n = n.args[0]
+
+            # To make constant value/tensor be tagged as delegatable during partition
+            if source_n.op == "get_attr":


Is source_n meant to be a call_delegate node or a regular call_function node? Just would like make sure we're using get_attr correctly.

No problem. It is a call delegate node.

It would be a regular call_function node. We use this to bring all constant params from get_attr nodes together.

kimishpatel

some early comments. still reviewing

kimishpatel · 2023-09-26T16:21:51Z

backends/qnn/setup.md

+
+`qnn_executor_runner` is an executable running the compiled model.
+
+You might want to ensure the correct `flatc`. `flatc` can be built along with the above step. For example, we can find `flatc` in `build_x86_64/third-party/flatbuffers/`.


executorch steup reqquires flatbuffers to be installed as listed here https://github.com/pytorch/executorch/blob/main/docs/website/docs/tutorials/00_setting_up_executorch.md. Is that sufficient?

The flatc from the current setup is a bit problematic. I update it in #447

We encountered compilation issue before due to incompatible flatc version(conda installed vs submodule). We might recommend user to build it from source.

kimishpatel · 2023-09-26T16:26:11Z

backends/qnn/setup.md

+example_input = mv2.get_example_inputs()
+model = mv2.get_eager_model().eval()
+
+captured_program = exir.capture(model, example_input)


you probably want to include quantization api calls here

oh I see, it is listed below. You dont need to have "non-delegated" example listed here. Also note that this is setup.md and so you probably want the examples in either examples folder's readme.md or in qnn/readme.md file.

Changed, thanks for the suggestion.

backends/qnn/utils/utils.py

kimishpatel · 2023-09-26T17:05:18Z

backends/qnn/setup.md

@@ -0,0 +1,251 @@
+# Setting up QNN Backend


except some minor nits, instructions here are great

backends/qnn/utils/qnn_constants.py

kimishpatel · 2023-09-26T18:53:57Z

backends/qnn/utils/utils.py

+        inputs,
+        qnn_capture_config(),
+    ).to_edge(qnn_edge_config())
+    return edge_prog.transform(qnn_partitioner_passes)


things of this nature, which is non-standard, in export workflow, is something that should be user visible.

Hence my preference is that you use the utils from examples/export/utils.py directly and add edge_prog.transform(...) in the examples and readmes

Will decompose all this to the user and remove capture_program ultimately.

kimishpatel · 2023-09-26T18:55:49Z

backends/qnn/utils/utils.py

+
+
+def qnn_capture_config():
+    return exir.CaptureConfig(pt2_mode=True, enable_aot=True, _unlift=True)


do you need separate capture_config? I see that you wan _unlift=True. I think this is fine for now, but we should try and make it work default capture, as these options are going away.

This argument means a lot to us. We hope to prefetch known tensors as many as possible instead of treating them as runtime arguments. This would be important for performing graph optimization inside QNN.

So, _unlift=True, doesnt necessarily mean that the parameters of the model actually become arguents to the function. 100% agree with you that otherwise, it becomes a performance issue. If you look at xnnpack delegation example, it actually works with exported_program with "lifted" parameters, but during delegation it consumes all the parameters and only those which are input are supplied.

Lets go over this in the next meeting, as longer term that is the route that we need to take

backends/qnn/utils/utils.py

kimishpatel · 2023-09-26T19:31:56Z

backends/qnn/utils/utils.py

+        "SM8475": PyQnnManager.QcomModel.SM8475,
+        "SM8550": PyQnnManager.QcomModel.SM8550,
+    }
+    backend_type = CompileSpec("backend_type", bytes([2]))


can the constants be named better?

There are other accelerators in QNN, but it is indeed inappropriate to use backend_type in ExecuTorch. It should be changed to accelerator_type in the future. How do you think?

By the way, the constant in bytes([2]) has been changed into meaningful enumeration.

backends/qnn/utils/utils.py

kimishpatel · 2023-09-26T21:12:22Z

backends/qnn/passes/convert_addmm_mm_with_linear.py

@@ -0,0 +1,57 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.


Can you directly use this from backend/transforms. There shouldnt be a need to copy paste this.

Changed, thanks for pointing out.

backends/qnn/passes/insert_io_qdq.py

kimishpatel · 2023-09-26T21:26:24Z

backends/qnn/passes/insert_io_qdq.py

+            name = arg_schema.name
+            value = quant_attrs[name]
+            if type(arg_schema.type) == torch.tensor and type(value) in [int, float]:
+                value = torch.tensor(value)


Is the node added here is also consumed by delegate?

Yes we use it to quantize inputs, or dequantize the output. These added node do exist in our backend IR.

backends/qnn/passes/insert_io_qdq.py

facebook-github-bot · 2023-10-03T22:25:24Z

@kimishpatel has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

kimishpatel · 2023-10-04T03:35:34Z

backends/qualcomm/passes/fold_qdq.py

+
+            # To make constant value/tensor be tagged as delegatable during partition
+            if source_n.op == "get_attr":
+                source_n.meta["source_fn_stack"] = list(n.users.keys())[0].meta.get(


Not necessary to do in this PR, but if you update this PR, please include "TODO: remove this hack as source_fn_stack is internal implementation detail of torch.export". We will follow up on this. There is some issue as to why this metadata is not preserved.

Comment added, will keep tracking on this, too.

- Add TODO to remove the source_fn_stack - Fix the comment of fold_qdq pass - Remove redundant ops in annotate_quant_attrs

- move qnn_executor_runner to examples/backend/qualcomm - move qualcomm/runtime/wrappers to qualcomm/aot/wrappers - move qualcomm/runtime/python to qualcomm/aot/python - change file structure in README accordingly

mergennachin

Thanks for making the suggested changes.

One last minor request.

Would you mind moving examples/backends/qualcomm to just examples/qualcomm?

And move all the relevant scripts (e.g. export_example.py) to examples/qualcomm/scripts/...?

Also, keep the executor_runner in examples/qualcomm/executor_runner/ as is.

Currently we are doing refactoring within in our examples folder, and it'd be great if qualcomm had similar structure as us.

Otherwise, LGTM

cc Will let @kimishpatel take a one last look.

haowhsu-quic · 2023-10-04T11:00:49Z

Thanks for making the suggested changes.

One last minor request.

Would you mind moving examples/backends/qualcomm to just examples/qualcomm?

And move all the relevant scripts (e.g. export_example.py) to examples/qualcomm/scripts/...?

Also, keep the executor_runner in examples/qualcomm/executor_runner/ as is.

Currently we are doing refactoring within in our examples folder, and it'd be great if qualcomm had similar structure as us.

Otherwise, LGTM

cc Will let @kimishpatel take a one last look.

Hi Mergen, thank you for giving us great comments.
For relevant scripts, do you mean we should also put all the examples like mobilenet_v2.py, inception_v4.py, etc. to examples/qualcomm/scripts?

mergennachin · 2023-10-04T11:50:46Z

Yes, any user facing main scripts where they have to run to see the example models e2e is preferred to live in "examples/qualcomm/scripts/*"

mergennachin · 2023-10-04T11:53:19Z

Also don't close the PR yet, someone from Meta will do the merging properly. Most likely Kimish

kimishpatel · 2023-10-04T14:30:48Z

backends/qualcomm/runtime/QnnExecuTorchBackend.cpp

@@ -0,0 +1,205 @@
+/*
+ * Copyright (c) Meta Platforms, Inc. and affiliates.


You dont need Meta here

Removed, thank you.

kimishpatel · 2023-10-04T14:33:55Z

backends/qualcomm/qnn_quantizer.py

@@ -0,0 +1,774 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.


Is this there because of file being copied from somewhere

Removed, things look very different now, thank you.

kimishpatel · 2023-10-04T14:35:06Z

backends/qualcomm/tests/qnn_test_utils.py

@@ -0,0 +1,174 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.


not sure if it is right to include meta here

Removed, thank you.

kimishpatel · 2023-10-04T14:35:22Z

backends/qualcomm/tests/test_qnn_delegate.py

@@ -0,0 +1,1209 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.


also not sure if meta needs to be here

Removed, thank you.

kimishpatel · 2023-10-04T14:44:35Z

Also, please take a look at various copyrights. I have commented on some. There are a few places Meta is in copyright but I am not sure if there is a need for it.

facebook-github-bot · 2023-10-04T14:44:54Z

@kimishpatel has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2023-10-04T15:49:20Z

@kimishpatel has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

- move examples/backend/qualcomm -> example/qualcomm - change documents, CMakeLists.txt accordingly - remove copyright from Meta

haowhsu-quic · 2023-10-04T17:05:10Z

Also don't close the PR yet, someone from Meta will do the merging properly. Most likely Kimish

Sorry, misclicked. Examples are rearranged now.

Also, please take a look at various copyrights. I have commented on some. There are a few places Meta is in copyright but I am not sure if there is a need for it.

Thank you for reminding this, I've scanned for all related files and make sure all the copyright is suitable.

haowhsu-quic · 2023-10-04T17:06:00Z

Also don't close the PR yet, someone from Meta will do the merging properly. Most likely Kimish

Sorry, misclicked. Examples are rearranged now, thank you for the suggestion.

kimishpatel

Looks good. I will try to land this by importing it internally.

facebook-github-bot · 2023-10-04T17:25:03Z

@kimishpatel has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

guangy10 · 2023-10-05T17:32:44Z

examples/qualcomm/README.md

+python mobilenet_v2.py -s <device_serial> -m "SM8550" -b path/to/build_android/ -d /path/to/imagenet-mini/val
+
+python deeplab_v3.py -s <device_serial> -m "SM8550" -b path/to/build_android/ --download


Since we are grouping the scripts under qualcomm/scripts/, it's nice to ask users to cd $EXECUTORCH_ROOT/examples/qualcomm/scripts first so that users can copy/past the commands here and run it hassle-free on their ends

Yes, it's a good siggestion. We'll change this in following PR, thank you.

guangy10 · 2023-10-05T17:35:20Z

examples/models/deeplab_v3/model.py

@@ -7,7 +7,11 @@
 import logging

 import torch
-from torchvision.models.segmentation import deeplabv3, deeplabv3_resnet50  # @manual


@haowhsu-quic Just want to confirm that Qualcomm is not interested in using deeplabc3_resnet50 right? We may want to clean it up later since there is already a resnet50 model in our model portfolio.

Yes, we'll use resnet101 backbone. Resnet50 does not perform well in our experiments.

guangy10

The demos code structure in the examples/ looks good to me!

facebook-github-bot · 2023-10-05T19:01:39Z

@kimishpatel merged this pull request in ef97148.

* Fix runner-et on Mac and Android * Fix typo * Avoid fatal message * Change fatal to warning

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 26, 2023

kimishpatel reviewed Sep 26, 2023

View reviewed changes

.gitmodules Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/CMakeLists.txt Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/README.md Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/CMakeLists.txt Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/examples/README.md Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/examples/executor_runner/qnn_executor_runner.cpp Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/examples/executor_runner/qnn_executor_runner.cpp Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/examples/executor_runner/qnn_executor_runner.cpp Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/examples/executor_runner/qnn_executor_runner.cpp Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/examples/mobilebert_fine_tune.py Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/setup.md Outdated Show resolved Hide resolved

mergennachin reviewed Sep 26, 2023

View reviewed changes

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/setup.md Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/setup.md Outdated Show resolved Hide resolved

mergennachin reviewed Sep 26, 2023

View reviewed changes

backends/qnn/tests/test_qnn_delegate.py Outdated Show resolved Hide resolved

larryliu0820 reviewed Sep 26, 2023

View reviewed changes

backends/qnn/runtime/QnnExecuTorchBackend.cpp Outdated Show resolved Hide resolved

cccclai reviewed Sep 26, 2023

View reviewed changes

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/passes/insert_io_qdq.py Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/passes/insert_io_qdq.py Outdated Show resolved Hide resolved

kimishpatel reviewed Sep 26, 2023

View reviewed changes

backends/qnn/passes/insert_io_qdq.py Outdated Show resolved Hide resolved

kimishpatel requested a review from sxu September 26, 2023 23:03

kimishpatel reviewed Oct 4, 2023

View reviewed changes

Tsai, Chun-I (Joey) and others added 2 commits October 4, 2023 12:30

[Minor fix] (#93)

22bd9fe

- Add TODO to remove the source_fn_stack - Fix the comment of fold_qdq pass - Remove redundant ops in annotate_quant_attrs

[Rearrange source folders] (#94)

52fbd57

- move qnn_executor_runner to examples/backend/qualcomm - move qualcomm/runtime/wrappers to qualcomm/aot/wrappers - move qualcomm/runtime/python to qualcomm/aot/python - change file structure in README accordingly

mergennachin approved these changes Oct 4, 2023

View reviewed changes

haowhsu-quic closed this Oct 4, 2023

haowhsu-quic reopened this Oct 4, 2023

kimishpatel reviewed Oct 4, 2023

View reviewed changes

[Rearrange example folder]

6567280

- move examples/backend/qualcomm -> example/qualcomm - change documents, CMakeLists.txt accordingly - remove copyright from Meta

kimishpatel approved these changes Oct 4, 2023

View reviewed changes

guangy10 reviewed Oct 5, 2023

View reviewed changes

facebook-github-bot closed this in ef97148 Oct 5, 2023

facebook-github-bot added the Merged label Oct 5, 2023

Gasoonjia pushed a commit that referenced this pull request Jul 30, 2024

Fix runner-et on Mac and Android (#490)

7c13f3b

* Fix runner-et on Mac and Android * Fix typo * Avoid fatal message * Change fatal to warning

		@@ -0,0 +1,344 @@
		# Copyright (c) Meta Platforms, Inc. and affiliates.

		You can find `qnn_executor_runner` under `build_android/backends/qnn`.


		## Compile a model


		To gain the best performance, we also employ quantization:

		```python

		# LICENSE file in the root directory of this source tree.

		import sys

	def partition(self, exported_program: ExportedProgram) -> torch.fx.GraphModule:
	def partition(self, exported_program: ExportedProgram) -> PartitionResult:


		`qnn_executor_runner` is an executable running the compiled model.

		You might want to ensure the correct `flatc`. `flatc` can be built along with the above step. For example, we can find `flatc` in `build_x86_64/third-party/flatbuffers/`.



		def qnn_capture_config():
		return exir.CaptureConfig(pt2_mode=True, enable_aot=True, _unlift=True)

		@@ -0,0 +1,57 @@
		# Copyright (c) Meta Platforms, Inc. and affiliates.

		@@ -0,0 +1,205 @@
		/*
		* Copyright (c) Meta Platforms, Inc. and affiliates.

		@@ -0,0 +1,774 @@
		# Copyright (c) Meta Platforms, Inc. and affiliates.

		@@ -0,0 +1,174 @@
		# Copyright (c) Meta Platforms, Inc. and affiliates.

		@@ -0,0 +1,1209 @@
		# Copyright (c) Meta Platforms, Inc. and affiliates.

		python mobilenet_v2.py -s <device_serial> -m "SM8550" -b path/to/build_android/ -d /path/to/imagenet-mini/val

		python deeplab_v3.py -s <device_serial> -m "SM8550" -b path/to/build_android/ --download

Qualcomm AI Engine Direct Backend #490

Qualcomm AI Engine Direct Backend #490

Uh oh!

Conversation

haowhsu-quic commented Sep 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Sep 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for resplendent-gnome-14e531 canceled.

Uh oh!

chiwwang commented Sep 26, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergennachin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cccclai left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

haowhsu-quic commented Sep 26, 2023 •

edited

Loading

netlify bot commented Sep 26, 2023 •

edited

Loading

cccclai left a comment •

edited

Loading