add aoti c/c++ runner to hqq tests; check output for gibberish using spell #824

mikekgfb · 2024-05-18T21:29:00Z

add aoti c/c++ runner to hqq tests
use spell check to detect garbage output

pytorch-bot · 2024-05-18T21:29:02Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/824

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7e67742 with merge base 7c2d949 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

… messages

…ed sequence extraction

…with-aoti

malfet · 2024-05-20T16:13:53Z

.ci/scripts/check_gibberish

+#! /bin/bash
+
+#!/bin/bash


Why do it twice?

malfet · 2024-05-20T16:13:55Z

.ci/scripts/check_gibberish

+cat ${TMPFILE} |  aspell -a -c  | grep '^[\&#]' >/tmp/out.$$
+# Exit with a non-zero status code if there were any spelling errors because:
+# * Finding one or more lines with & or # means we found a spelling error, might be gibberish
+if [ $? -ne 0 ]; then
+    echo "No spelling errors found; likely correct operation. Success."
+    exit 0
+fi
+cat /tmp/out.$$
+echo "Spelling errors found; might indicate garbage output. Failing."
+exit 1


Should we introduce some sort of a tolerance criteria here (i.e. total number of unknown words)

Great idea -- I've had one instance where generate produced an invented non-dictionary work with and 'he calls it "whateversomething".' That being said it only occurred once, so I decided to not overindex on that.

Allowing for some rate of output is a good solution to this. Should we proactively do that, or wait till we see if in actual test runs?

malfet · 2024-05-20T16:14:17Z

quantize.py

+                        # print(
+                        #     f"warning: {name} is padded to satisfy in_features % 1024 == 0"
+                        # )


Sorry, how this change is related to PR in question?

When you output that, it enters the spell checked sequence, and fails! We are too wordy with debug messages, this is primarily my fault in the quantization logic.

malfet · 2024-05-20T16:14:33Z

generate.py

@@ -748,7 +748,7 @@ def callback(x):
        aggregate_metrics["tokens_per_sec"].append(tokens_sec)

        if jit_compile:
-            print(f"JIT compilation time (incl runtime): {compilation_time:.2} seconds")
+            print(f"just-in-time compilation time (incl run time): {compilation_time:.2} seconds")


This is fine, but shouldn't be part of the PR, should it?

malfet · 2024-05-20T16:14:44Z

build/builder.py

@@ -441,6 +441,7 @@ def _initialize_model(

        model.to(dtype=builder_args.precision)

+    print("-----------------------------------------------------------")


Why this is needed?

Looking for a delimiter for our gibberish (load, quantization and other time....). As far as model output it's as much gibberish as a hallucinating model (in terms of utlliity and connectedness to the model).

That being said, there's a point where I strongly believe in dumping performance data -- we had missed regressions when we did not.

malfet · 2024-05-20T16:14:51Z

.github/workflows/run-readme-pr.yml

@@ -244,3 +244,4 @@ jobs:
        echo "tests complete"
        echo "*******************************************"
        echo "::endgroup::"
+


malfet · 2024-05-20T16:15:31Z

.github/workflows/hqq-dtype.yml

+
+          ./cmake-out/aoti_run ${MODEL_DIR}/${MODEL_NAME}.so  -z ${TOKENIZER_PATH} -i "${PROMPT}" > ./output_runner_aoti
+          cat ./output_runner_aoti
+          # .ci/scripts/check_gibberish ./output_runner_aoti --no-extract


Why check is skipped there?

because aoti_runner does not work properly and I can't run proper tests.

Would love to add test here, and add aoti runner everywhere where we call generate --dso-path today as a second test. Pending resolution of beoing able to load CPU model when the aoti_runner was built on a host with cuda.

malfet · 2024-05-20T16:16:19Z

.ci/scripts/extract-sequence.py

+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print("Usage: python scriptname.py filename")


It should have been something like `f"Usage:\n {sys.executable} {sys.argv[0]} filename")

…spell (#824) * add runner to hqq tests * replace cat with a gibberish check * typo * create script to check for gibberish * update gibberish check * update gibberish check * use variable for tokenizer path * aspell dictionaries for english * exclude device name from gibberish check * handle JIT time line * handle Warning: * grep update * fix line exclusion * remove warning which causes gibberish check fail * add sequence extraction for principled handling of perf info and user messages * typo * change output to pass spell check * updates * handle runner which does not have sequence delimiters b/c does not need sequence extraction * add updated workflow yml * typo * native runner weirdness * remove secrets * don't log in for GGUF open_orca model

add runner to hqq tests

c193892

facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label May 18, 2024

replace cat with a gibberish check

be8e226

mikekgfb changed the title ~~add aoti c/c++ runner to hqq tests~~ add aoti c/c++ runner to hqq tests; check output for gibberish using spell May 18, 2024

Michael Gschwind added 2 commits May 18, 2024 14:52

typo

7300c6d

create script to check for gibberish

8378ffc

mikekgfb requested review from metascroy, malfet and larryliu0820 May 18, 2024 22:05

Michael Gschwind added 20 commits May 18, 2024 15:27

update gibberish check

c670f66

update gibberish check

952e001

use variable for tokenizer path

6d8c790

aspell dictionaries for english

14794e1

exclude device name from gibberish check

26b71a3

handle JIT time line

d5feb4f

handle Warning:

cdf359c

grep update

92b097e

fix line exclusion

e480bca

remove warning which causes gibberish check fail

aa1774c

add sequence extraction for principled handling of perf info and user…

3e16ba2

… messages

typo

e907196

change output to pass spell check

94e236a

updates

4dee526

handle runner which does not have sequence delimiters b/c does not ne…

4b367ef

…ed sequence extraction

add updated workflow yml

1360ef9

typo

2bd5dec

native runner weirdness

3246d8e

remove secrets

dbb35cd

don't log in for GGUF open_orca model

3bc7218

Michael Gschwind added 2 commits May 19, 2024 22:30

merge

faae387

Merge branch 'main' of https://github.com/pytorch/torchchat into hqq-…

7e67742

…with-aoti

Gasoonjia approved these changes May 20, 2024

View reviewed changes

mikekgfb merged commit 571841e into main May 20, 2024

mikekgfb deleted the hqq-with-aoti branch May 20, 2024 06:37

malfet reviewed May 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add aoti c/c++ runner to hqq tests; check output for gibberish using spell #824

add aoti c/c++ runner to hqq tests; check output for gibberish using spell #824

Uh oh!

mikekgfb commented May 18, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented May 18, 2024 •

edited

Loading

Uh oh!

malfet May 20, 2024

Uh oh!

malfet May 20, 2024

Uh oh!

mikekgfb May 20, 2024

Uh oh!

malfet May 20, 2024

Uh oh!

mikekgfb May 20, 2024

Uh oh!

malfet May 20, 2024

Uh oh!

malfet May 20, 2024

Uh oh!

mikekgfb May 20, 2024

Uh oh!

malfet May 20, 2024

Uh oh!

malfet May 20, 2024

Uh oh!

mikekgfb May 21, 2024

Uh oh!

malfet May 20, 2024

Uh oh!

mikekgfb May 20, 2024

Uh oh!

Uh oh!

		@@ -441,6 +441,7 @@ def _initialize_model(

		model.to(dtype=builder_args.precision)

		print("-----------------------------------------------------------")

		#! /bin/bash

		#!/bin/bash

add aoti c/c++ runner to hqq tests; check output for gibberish using spell #824

add aoti c/c++ runner to hqq tests; check output for gibberish using spell #824

Uh oh!

Conversation

mikekgfb commented May 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented May 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/824

✅ No Failures

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mikekgfb commented May 18, 2024 •

edited

Loading

pytorch-bot bot commented May 18, 2024 •

edited

Loading