Add utility script for profiling CI artifacts

[tmiasko]

For example to compare two builds on ctfe-stress-4-doc benchmark using cachegrind:

$ ./p.py -p cachegrind -b ctfe-stress-4 --builds Doc \
  7611fe438dae91084d17022e705bf64374d5ba4b \
  bcfd3f7e88084850f87b8e34b4dcb9fceb872d00

[Mark-Simulacrum]

I sort of understand the desire for Python scripts but I feel like this ends up redoing some of the collector work, and I'd personally feel better about integrating this directly as a subcommand.

I just pushed up #1025 which does the installation bits automatically (though shells out to rustup toolchain install master).

[Mark-Simulacrum]

I'm experimenting with something similar locally fwiw:

#!/bin/bash

set -euxo pipefail

profiler=$1
commit_a=$2
commit_b=$3
benchmarks=$4
builds=$5
runs=$6

cargo run --release --bin collector -- profile_local $profiler +$commit_a before --include $benchmarks --builds $builds --runs $runs
cargo run --release --bin collector -- profile_local $profiler +$commit_b after --include $benchmarks --builds $builds --runs $runs

if [ "$profiler" = "cachegrind" ]; then
  for a in results/cgout-* ; do
    for b in results/cgout-* ; do
      if [ "$a" = "$b" ]; then
        continue
      fi
      if [ "$a" = "${b//after/before}" ]; then
        echo "planning to write to ${b//cgout-after/cgdiff}"
        cg_diff $a $b | sed "s@$commit_a@@" | sed "s@$commit_b@@" | \
          sed -E "s@llvm\.\d+@@" | \
          rustfilt | \
          cg_annotate /dev/stdin > "${b//cgout-after/cgdiff}"
      fi
    done
  done
fi

[tmiasko]

I just pushed up #1025 which does the installation bits automatically (though shells out to rustup toolchain install master).

Nice!

I'd personally feel better about integrating this directly as a subcommand.

What about extending profile_local to allow multiple toolchains and detecting the special case of profiling with cachegrind and two toolchains for which we would automatically generate the comparison using cg_diff and cg_annotate?

[Mark-Simulacrum]

What about extending profile_local to allow multiple toolchains and detecting the special case of profiling with cachegrind and two toolchains for which we would automatically generate the comparison using cg_diff and cg_annotate?

Yeah, this sounds great. It's a bit of work to pull the pieces through perf -- and it may make sense to switch to a diff_local prefix -- but that general approach seems like the right one. Happy to give suggestions on particulars.

[tmiasko]

Closing in favour of #1028.