[Benchmark] Generate benchmark record for job failure

[yangw-dev]

Description

compose failure benchmark record

Issue: pytorch/test-infra#6294

Related Query File

https://github.com/pytorch/test-infra/blob/main/torchci/clickhouse_queries/oss_ci_benchmark_llms/query.sql

Details

when a job fails in git_job level, or a device fails in the benchmark test,
we return a benchmark record to indicate the failure, so that we can properly render the HUD UI ti distinguish metris not run and metris run with failures

This pr may introduce Unknown to fields in the HUD execubench table temporarily, will fix it in HUD UI to handle special benchmark value.

for both level of failure, the metric name will be "FAILURE_REPORT".
In HUD, We mainly use the special metric name to identify failure, if more information needed, we get which level the job fails in benchmark.extra_info.

Step Failure

When a failure detected:
we try to extract model info from git_job_name, the step will fail if the model info cannot be extracted.

Example of benchmark record for failure benchmark:

When a job failed at device-job-level,

• device_name: get name from job_report.name . for instance iPhone 15
• device_os: job_report.os with prefix "Android" or "iOS". this should match both android and ios setting
• model.name: extract from git job name
• model.backend: extract from git job name
• metric.name: "FAILURE_REPORT"

{
  "benchmark": {
    "name": "ExecuTorch",
    "mode": "inference",
    "extra_info": {
      "app_type": "IOS_APP",
      "job_conclusion": **"FAILED"**,
      "failure_type": **"DEVICE_JOB"**,
      "job_report": "..."
  },
  "model": {
    "name": **"ic4",**
    "type": "OSS model",
    "backend": **"mps"**
  },
  "metric": {
    "name": **"FAILURE_REPORT"**,
    "benchmark_values": 0,
    "target_value": 0,
    "extra_info": {
      "method": ""
    }
  },
  "runners": [
    {
      "name": **"iPhone 15"**,
      "type": **"iOS 18.0"**,
    }
  ]
}

when a job failed at git-job-level (there is no job_reports)

this happens when a job fails before it runs the benchmark job

• device_name: device_pool_name from git job bane #exmaple: sumsung_galaxy_22
• device_os: "Android" or "iOS"
• model.name: extract from git job name
• model.backend: extract from git job name
• metric.name: "FAILURE_REPORT"

the failure benchmark record looks like:

{
  "benchmark": {
    "name": "ExecuTorch",
    "mode": "inference",
    "extra_info": {
      "app_type": "IOS_APP",
      "job_conclusion": **"FAILURE"**,
      "failure_type": **"GIT_JOB"**,
      "job_report": "{}"
    }
  },
  "model": {
    "name": "ic4",
    "type": "OSS model",
    "backend": "mps"
  },
  "metric": {
    "name": "FAILURE_REPORT",
    ...
  },
  "runners": [
    {
      "name": "samsung_galaxy_s22",
      "type": "Android",
     ...
    }
  ]
}

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9247

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 69c42fa with merge base dd9a85a ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / android / run-emulator (gh) (trunk failure)
  Process completed with exit code 9.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[huydhn]

A word of caution when trying to extract the information about the run from the job name. The job name comes from this line https://github.com/pytorch/executorch/blob/main/.ci/scripts/gather_benchmark_configs.py#L335. So, I think:

• Make the error when failing to parse the job name in extract_model_info clearer by referring to gather_benchmark_configs script. Mostly likely, it has been updated without updating extract_benchmark_results
• Add a comment on both script that they need to be in sync

[yangw-dev]

sounds good!

[yangw-dev]

I also raise exception for get_app_type and get_device_os_type.
Added unittest to those cases too

[yangw-dev]

@huydhn
not urgent: maybe we can add a unit test to check if this works as expected. could crate a common lib to share some configs between this script and another one.

i also added comment in perf.yml for job name step change too.
if this script uses out of those yml files, we can make the regex prefix more flexible to let user pass the step name for chekcing

[huydhn]

While checking the results from your branch on the dashboard, I notice a curious issue where older commits from your branch have a newer timestamps:

It seems like an issue for later.

[huydhn]

LGTM! Let's take an action item to notice the next failure from these workflows after this lands to double check the results on the dashboard before resolving the issue

[yangw-dev]

While checking the results from your branch on the dashboard, I notice a curious issue where older commits from your branch have a newer timestamps:

It seems like an issue for later.

interesting, created a issue here :pytorch/test-infra#6427

[yangw-dev]

LGTM! Let's take an action item to notice the next failure from these workflows after this lands to double check the results on the dashboard before resolving the issue

sounds good, I also need to update UI