Merge branch 'embargoed-builds-preamble'

This topic branch contains preparatory patches for the huge task of
adapting Git for Windows' automation to build embargoed releases.

So what _are_ embargoed releases?

Every once in a while (seemingly on a semi-annual cadence),
security-relevant bugs are discovered in Git, reported responsibly to
the git-security mailing list, and if the core Git developers deem it
important enough of an issue, patches are developed in secret ("under
embargo"). These patches are then reviewed (typically even less
enthusiastically than on the Git mailing list because much more is at
stake than the question whether the lines are indented correctly or
whether the lines have trailing whitespace), and eventually they are
tagged (at least if consensus is reached) and then the task begins for
Git for Windows to provide installers, portable Gits and MinGit versions
well in advance of the publication date ("the lifting of the embargo"),
a date that has to be negotiated between all affected parties (not a
small feat, I can tell ya).

In Git for Windows, I call these artifacts "embargoed build". They are
distributed to the stakeholders such as GitHub Desktop and Visual Studio
in advance of the publication date so that they can test, and integrate
them into their own embargoed builds.

Previously, embargoed builds were built in an Azure Pipeline, in a
private Azure DevOps project. This has become a bit of a problem in the
meantime, for multiple reasons:

- The upstream Git project (which enjoyed very active contributions from
  up to 8 GitHub engineers at its peak) shifted CI builds from Azure
  Pipelines to GitHub Actions a long time ago. Even though investment
  into GitHub Actions seems to have been drastically reduced recently,
  there is too much momentum and the last remnants of Azure Pipelines
  support has been removed, all but forcing Git for Windows to invest in
  a private org on GitHub to continue building confidence in embargoed
  builds by running the CI builds there.

  This has implications also on _building_ the embargoed versions, as it
  is better to build where you test.

- Even though the hoped-for Windows/ARM64 runners have not materialized
  in GitHub Actions yet (Windows/ARM64 runners exist, but they are not
  free, not even for open source), Git for Windows invested substantial
  efforts (with serious, serious time committment of volunteers, most of
  all the never-tiring Dennis Ameling) to side-step this short-coming
  and build a custom infrastructure based on creating custom Azure VMs
  and registering them as Windows/ARM64 runners.

  This support is noticeably lacking from Azure Pipelines, too, and Git
  for Windows would have to invest the same time and work to gain the
  ability to build Windows/ARM64 binaries in Azure Pipelines.

- It is always cumbersome to maintain two totally different things that
  do virtually the same, and Azure Pipelines and GitHub Actions are just
  very different even though they started out using the same technology
  (it does not seem as if a lot of synergy has happened there as a
  consequence, which is a shame).

- For historical reasons, the Azure Pipeline we used was not YAML-based,
  and is therefore very hard to transport to other
  repositories/projects. In contrast, the GitHub workflows we
  established in the Git for Windows projects are very easy to transport
  to forks or even transmogrify them into totally different projects as
  needed. To allow for the same mobility with the Azure Pipeline, we
  would have had to rewrite it in YAML anyway, and it seems a smarter
  investment to extend the already-existing GitHub workflows to that
  end.

The journey to adapt the GitHub workflows to allow for building and
publishing embargoed artifacts is split into these topic branches (with
the current one being the first one):

- embargoed-builds-preamble:

  Some preparatory patches, refactoring, that kind of stuff. This is
  what is actually merged in this here merge commit.

- embargoed-branches:

  Developing a GitHub workflow that initializes "time-traveling"
  branches, i.e. branches that branch off at the time when the version
  preceding the embargoed one were done, in the following repositories
  (which are all used to build Git for Windows' installers, portable
  Gits, MinGits, etc):

    - git
    - build-extra
    - MINGW-packages
    - git-sdk-64
    - git-sdk-arm64
    - git-sdk-32

- embargoed-git-artifacts:

  Adapting the `git-artifacts` workflow to allow for running it in a
  private GitHub org. This requires some special care because access
  tokens are required to access the relevant repositories

- embargoed-tag-git:

  Adapting the `tag-git` workflow to allow for running in a private
  GitHub org. This workflow is actually the one triggered by the
  corresponding slash command, as its build artifacts are required to
  run the `git-artifacts` workflow for all supported CPU architectures.

- embargoed-releases:

  While the regular, non-embargoed release process of Git for Windows
  calls for the build artifacts of the `git-artifacts` workflow runs to
  be gathered and released e.g. to a GitHub release in
  `git-for-windows/git`, the same is not possible in embargoed releases.

  The problem is not even that the build artifacts would have to be
  retrieved from a different org, using different access tokens than the
  GitHub release to which they need to be uploaded, the biggest problem
  is the time factor: While regular releases have a very short time
  window between building the artifacts and releasing them, in embargoed
  releases sometimes months have to pass before the build artifacts can
  be released, and GitHub's retention policies do not allow for such
  long time windows.

  As a consequence, a lot of mechanics are necessary to refactor some of
  the processing that would historically be done during the `release`
  workflow to allow for it to be run waaaaay before the actual release.

  And to counter the retention rules, all embargoed build artifacts are
  uploaded to a GitHub release in the private org, where retention rules
  do not apply.

One step is missing: Actually releasing the embargoed artifacts to a
_public_ GitHub Release in git-for-windows/git, to NuGet.org and to Git
for Windows' Pacman repositories, as well as updating the `build-extra`
and `git-for-windows.github.io` repositories with the information that
is only available once that GitHub Release is available. The reason that
this is missing is that I ran out of steam doing it during the (quite
stressful) release process of v2.47.2 & friends.

Signed-off-by: Johannes Schindelin <[email protected]>

Author: dscho

.github/workflows/break-pacman-upload-lease.yml

@@ -6,6 +6,7 @@ on:
 
 jobs:
   break-lease:
+    if: github.event.repository.owner.login == 'git-for-windows'
     runs-on: ubuntu-latest
     steps:
       - name: Clone build-extra

.github/workflows/build-and-deploy.yml

@@ -36,6 +36,7 @@ env:
 
 jobs:
   build:
+    if: github.event.repository.owner.login == 'git-for-windows'
     runs-on: ${{ github.event.inputs.architecture == 'aarch64' && fromJSON('["Windows", "ARM64"]') || 'windows-latest' }}
     steps:
       - uses: actions/checkout@v4

.github/workflows/open-pr.yml

@@ -24,6 +24,7 @@ env:
 
 jobs:
   open-pr:
+    if: github.event.repository.owner.login == 'git-for-windows'
     runs-on: windows-latest
     steps:
       - name: Determine REPO

.github/workflows/release-git.yml

@@ -25,6 +25,7 @@ env:
 jobs:
   setup:
     runs-on: ubuntu-latest
+    if: github.event.repository.owner.login == 'git-for-windows'
     outputs:
       display-version: ${{ steps.bundle-artifacts.outputs.display-version }}
       tag-name: ${{ steps.bundle-artifacts.outputs.tag-name }}

.github/workflows/updpkgsums.yml

@@ -24,6 +24,7 @@ env:
 
 jobs:
   updpkgsums:
+    if: github.event.repository.owner.login == 'git-for-windows'
     runs-on: windows-latest
     steps:
       - uses: actions/checkout@v4

github-release.js

@@ -58,16 +58,32 @@ const getWorkflowRunArtifactsURLs = async (context, token, owner, repo, workflow
   }, {})
 }
 
-const downloadAndUnZip = async (token, url, name) => {
+const download = async (token, url, outputFile) => {
   const { spawnSync } = require('child_process')
-  const auth = token ? ['-H', `Authorization: Bearer ${token}`] : []
-  const tmpFile = `${process.env.RUNNER_TEMP || process.env.TEMP || '/tmp'}/${name}.zip`
-  const curl = spawnSync('curl', [...auth, '-Lo', tmpFile, url])
+  const headers = token ? ['-H', `Authorization: Bearer ${token}`] : []
+  if (url.match(/^https:\/\/github.com\/[^/]+\/[^/]+\/releases\/assets\/\d+$/)
+    || url.match(/^https:\/\/api\.github.com\/repos\/[^/]+\/[^/]+\/releases\/assets\/\d+$/)) {
+    headers.push('-H', 'Accept: application/octet-stream')
+  }
+  const curl = spawnSync('curl', [...headers, '-fLo', outputFile, url])
   if (curl.error) throw curl.error
-  const { mkdirSync, rmSync } = require('fs')
-  await mkdirSync(name, { recursive: true })
-  const unzip = spawnSync('unzip', ['-d', name, tmpFile])
+}
+
+const unzip = async (zipFile, outputDirectory) => {
+  const { mkdirSync } = require('fs')
+  await mkdirSync(outputDirectory, { recursive: true })
+  const { spawnSync } = require('child_process')
+  const unzip = spawnSync('unzip', ['-d', outputDirectory, zipFile])
   if (unzip.error) throw unzip.error
+}
+
+const getTempFile = (name) => `${process.env.RUNNER_TEMP || process.env.TEMP || '/tmp'}/${name}`
+
+const downloadAndUnZip = async (token, url, name) => {
+  const tmpFile = getTempFile(`${name}.zip`)
+  await download(token, url, tmpFile)
+  await unzip(tmpFile, name)
+  const { rmSync } = require('fs')
   rmSync(tmpFile)
 }
 
@@ -287,17 +303,50 @@ const pushGitTag = (context, setSecret, token, owner, repo, tagName, bundlePath)
   context.log('Done pushing tag')
 }
 
+const downloadReleaseAssets = async (context, setSecret, appId, privateKey, owner, repo, tagName, filenameMatcher) => {
+  const { getAccessTokenForRepo } = require('./repository-updates.js')
+  const token = await getAccessTokenForRepo(context, setSecret, appId, privateKey, owner, repo)
+
+  const githubApiRequest = require('./github-api-request.js')
+  const release = await githubApiRequest(
+    context,
+    token,
+    'GET',
+    `https://api.github.com/repos/${owner}/${repo}/releases/tags/${tagName}`
+  )
+
+  for (const asset of release.assets) {
+    if (!filenameMatcher || filenameMatcher(asset.name)) {
+      context.log(`Downloading ${asset.name}`)
+      await download(token, asset.url, asset.name)
+    }
+  }
+}
+
+const downloadReleaseAssetsFromURL = async (context, setSecret, appId, privateKey, releaseURL, filenameMatcher) => {
+  const [, owner, repo, tagName] = releaseURL.match(
+    /^https:\/\/github.com\/([^/]+)\/([^/]+)\/releases\/tag\/([^/]+)$/
+  )
+  if (!owner || !repo || !tagName) throw new Error(`Invalid release URL: ${releaseURL}`)
+    return await downloadReleaseAssets(context, setSecret, appId, privateKey, owner, repo, tagName, filenameMatcher)
+}
+
 module.exports = {
   createRelease,
   updateRelease,
   uploadReleaseAsset,
   getWorkflowRunArtifactsURLs,
+  download,
+  unzip,
+  getTempFile,
   downloadAndUnZip,
   downloadBundleArtifacts,
   getGitArtifacts,
   sha256sumsFromReleaseNotes,
   calculateSHA256ForFile,
   checkSHA256Sums,
   uploadGitArtifacts,
-  pushGitTag
-}
+  pushGitTag,
+  downloadReleaseAssets,
+  downloadReleaseAssetsFromURL,
+}