Merge remote-tracking branch 'origin/main' into replace-swift-lang-with-swiftlang

shahmishal · shahmishal · commit 6f21c38d4887 · 2021-10-21T09:46:28.000-07:00
diff --git a/README.md b/README.md
@@ -25,6 +25,104 @@ swift-installer-scripts
            └ ...
 ~~~
 
+## Linux Packages (RPM/Deb)
+
+Currently Swift on Linux is distributed via tarball and Docker, and
+we would like to start supporting RPM and Debs officially on swift.org.
+The goal is to provide a seamless install process for Swift on Linux by
+utilizing the platform’s native package manager (RPM/Deb).
+
+
+* Step 1. Develop native packages / installers for the distributions
+* Step 2. Offer the native packages / installers through swift.org
+  * Support all officially supported Linux platforms
+  * Code signed by swift.org certificate
+  * Repository hosted on swift.org
+* Step 3. Offer the native packages / installer through official repositories
+for the various platforms
+  * Work with official repositories to accept package specs
+  * Deprecate swift.org packages / installer repository
+* Step 4. Deprecate swift.org Linux tarballs
+
+### Package Info
+* Package name: swiftlang
+* License: Apache 2.0
+* Maintainer: swift-infrastructure@forums.swift.org
+* URL: https://swift.org
+* Description:
+```
+Swift is a general-purpose programming language built using
+a modern approach to safety, performance, and software design
+patterns.
+
+The goal of the Swift project is to create the best available
+language for uses ranging from systems programming, to mobile
+and desktop apps, scaling up to cloud services. Most
+importantly, Swift is designed to make writing and maintaining
+correct programs easier for the developer.
+```
+
+### RPM Naming Convention:
+
+Package naming convention: `swiftlang-<VERSION>-<RELEASE>.<DIST>.<ARCH>.rpm`  
+Pakcage structure: `/repo/<OS>/releases/<OS_VERSION>/<ARCH>/`  
+Repository configuration: `/repo/<OS>/releases/<OS_VERSION>/swiftlang.repo`  
+
+
+#### Example
+
+* **Package structure:**  
+```
+/repo/centos/releases/8/x86_64/swiftlang-5.5.0-1.el8.x86_64.rpm
+/repo/centos/releases/8/aarch64/swiftlang-5.5.0-1.el8.aarch64.rpm
+```
+
+* **Package URL:**  
+https://download.swift.org/repo/centos/releases/8/aarch64/swiftlang-5.5.0-1.el8.aarch64.rpm
+
+* **Repository configuration file URL:**  
+https://download.swift.org/repo/centos/releases/8/swiftlang.repo
+
+## Tasks
+
+### RPM Package Manager (RPM)*
+
+- [ ] [SR-15325](https://bugs.swift.org/browse/SR-15325) Create RPM spec file
+- [ ] [SR-15326](https://bugs.swift.org/browse/SR-15326) Setup CI job to build the rpm package
+- [ ] [SR-15327](https://bugs.swift.org/browse/SR-15327) Code sign rpm package with swift.org certificate
+- [ ] [SR-15328](https://bugs.swift.org/browse/SR-15328) Host the rpm package on swift.org
+- [ ] [SR-15329](https://bugs.swift.org/browse/SR-15329) Host the rpm repository on swift.org
+- [ ] [SR-15330](https://bugs.swift.org/browse/SR-15330) Verify the rpm package and repository
+- [ ] [SR-15331](https://bugs.swift.org/browse/SR-15331) Update swift.org download / install page
+- [ ] [SR-15332](https://bugs.swift.org/browse/SR-15332) Work with official repositories to accept package specs
+
+*For each platform, we will start with CentOS 8.
+
+### Debian Package (Deb)*
+
+- [ ] [SR-15334](https://bugs.swift.org/browse/SR-15334) Create Debs control file
+- [ ] [SR-15335](https://bugs.swift.org/browse/SR-15335) Setup CI to build the deb package
+- [ ] [SR-15336](https://bugs.swift.org/browse/SR-15336) Code sign package with swift.org certificate
+- [ ] [SR-15337](https://bugs.swift.org/browse/SR-15337) Host the deb package on swift.org
+- [ ] [SR-15338](https://bugs.swift.org/browse/SR-15338) Host the deb repository on swift.org
+- [ ] [SR-15339](https://bugs.swift.org/browse/SR-15339) Verify the deb package and repository
+- [ ] [SR-15340](https://bugs.swift.org/browse/SR-15340) Update swift.org download / install page
+- [ ] [SR-15341](https://bugs.swift.org/browse/SR-15341) Work with official repositories to accept package control files
+
+*For each platform, we will start with Ubuntu 20.04
+
+## Open Questions
+
+* Where should swiftlang be installed on the system?
+	* Option 1: Diverge the install location between platform to best fit the platform requirements.
+		* symlink the toolchain into /usr/ to avoid conflicting with llvm.org binaries.
+	* Option 2: Install in /usr and rename llvm-project binaries (example: swift-lldb/lldb-swift ...)
+	* [GitHub discussion](https://github.com/apple/swift-installer-scripts/pull/37#discussion_r726707320)
+* Should we support multiple swiftlang versions on the system?
+* Multiple packages:
+	* swiftlang
+	* swiftlang-runtime
+
 ## Contributing
 
 Before contributing, please read [CONTRIBUTING.md](CONTRIBUTING.md).
diff --git a/platforms/Linux/amazonlinux/2/Dockerfile b/platforms/Linux/amazonlinux/2/Dockerfile
@@ -12,7 +12,7 @@ LABEL PURPOSE="This image is configured to build Swift for the version of Amazon
 RUN yum -y update
 
 # RPM and yum development tools
-RUN yum install -y rpmdevtools yum-utils
+RUN yum install -y rpmdevtools yum-utils createrepo
 
 # Configure epel
 RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
diff --git a/platforms/Linux/amazonlinux/2/docker-compose.yaml b/platforms/Linux/amazonlinux/2/docker-compose.yaml
@@ -38,6 +38,10 @@ services:
     <<: *common
     command: /bin/bash -cl "./build_rpm.sh"
 
+  createrepo:
+    <<: *common
+    command: /bin/bash -cl "../shared/createrepo_rpm.sh"
+
   shell:
     <<: *common
     entrypoint: /bin/bash -l
diff --git a/platforms/Linux/amazonlinux/2/swiftlang.spec b/platforms/Linux/amazonlinux/2/swiftlang.spec
@@ -29,6 +29,7 @@ Source16:       https://github.com/apple/swift-syntax/archive/swift-%{swift_vers
 Source17:       https://github.com/jpsim/Yams/archive/%{yams_version}.zip#/yams.tar.gz
 Source18:       https://github.com/apple/swift-crypto/archive/refs/tags/%{swift_crypto_version}.tar.gz#/swift-crypto.tar.gz
 Source19:       https://github.com/ninja-build/ninja/archive/refs/tags/v%{ninja_version}.tar.gz#/ninja.tar.gz
+Source20:       https://github.com/KitWare/CMake/archive/refs/tags/v%{cmake_version}.tar.gz#/cmake.tar.gz
 
 Patch0:         patches/swift-api-checker.patch
 Patch1:         patches/hwasan_symbolize.patch
@@ -91,7 +92,7 @@ importantly, Swift is designed to make writing and maintaining
 correct programs easier for the developer.
 
 %prep
-%setup -q -c -n %{swift_source_location} -a 0 -a 1 -a 2 -a 3 -a 4 -a 5 -a 6 -a 7 -a 8 -a 9 -a 10 -a 11 -a 12 -a 13 -a 14 -a 15 -a 16 -a 17 -a 18 -a 19
+%setup -q -c -n %{swift_source_location} -a 0 -a 1 -a 2 -a 3 -a 4 -a 5 -a 6 -a 7 -a 8 -a 9 -a 10 -a 11 -a 12 -a 13 -a 14 -a 15 -a 16 -a 17 -a 18 -a 19 -a 20
 # The Swift build script requires directories to be named
 # in a specific way so renaming the source directories is
 # necessary
@@ -113,6 +114,7 @@ mv swift-argument-parser-%{swift_argument_parser_version} swift-argument-parser
 mv swift-driver-swift-%{swift_version} swift-driver
 mv swift-crypto-%{swift_crypto_version} swift-crypto
 mv ninja-%{ninja_version} ninja
+mv CMake-%{cmake_version} cmake
 
 # ICU
 mv icu-release-%{icu_version} icu
diff --git a/platforms/Linux/centos/7/Dockerfile b/platforms/Linux/centos/7/Dockerfile
@@ -12,7 +12,7 @@ LABEL PURPOSE="This image is configured to build Swift for the version of CentOS
 RUN yum -y update
 
 # RPM and yum development tools
-RUN yum install -y rpmdevtools yum-utils
+RUN yum install -y rpmdevtools yum-utils createrepo
 
 # Configure epel
 RUN yum install -y epel-release centos-release-scl
diff --git a/platforms/Linux/centos/7/build_rpm.sh b/platforms/Linux/centos/7/build_rpm.sh
@@ -34,6 +34,7 @@ pushd $HOME/rpmbuild/SPECS
 yum-builddep -y ./swiftlang.spec
 # Workaround to support clang-3.5 or a later version
 echo -e ". /opt/rh/sclo-git25/enable\n. /opt/rh/llvm-toolset-7/enable\n. /opt/rh/devtoolset-8/enable\n" >> $HOME/.bashrc
+source $HOME/.bashrc
 sed -i -e 's/\*__block/\*__libc_block/g' /usr/include/unistd.h
 # get the sources for Swift as defined in the spec file
 spectool -g -R ./swiftlang.spec
diff --git a/platforms/Linux/centos/7/docker-compose.yaml b/platforms/Linux/centos/7/docker-compose.yaml
@@ -41,6 +41,10 @@ services:
     <<: *common
     command: /bin/bash -cl "./build_rpm.sh"
 
+  createrepo:
+    <<: *common
+    command: /bin/bash -cl "../shared/createrepo_rpm.sh"
+
   shell:
     <<: *common
     entrypoint: /bin/bash -l
diff --git a/platforms/Linux/centos/7/swiftlang.spec b/platforms/Linux/centos/7/swiftlang.spec
@@ -29,6 +29,7 @@ Source16:       https://github.com/apple/swift-syntax/archive/swift-%{swift_vers
 Source17:       https://github.com/jpsim/Yams/archive/%{yams_version}.zip#/yams.tar.gz
 Source18:       https://github.com/apple/swift-crypto/archive/refs/tags/%{swift_crypto_version}.tar.gz#/swift-crypto.tar.gz
 Source19:       https://github.com/ninja-build/ninja/archive/refs/tags/v%{ninja_version}.tar.gz#/ninja.tar.gz
+Source20:       https://github.com/KitWare/CMake/archive/refs/tags/v%{cmake_version}.tar.gz#/cmake.tar.gz
 
 Patch0:         patches/swift-api-checker.patch
 Patch1:         patches/hwasan_symbolize.patch
@@ -92,7 +93,7 @@ importantly, Swift is designed to make writing and maintaining
 correct programs easier for the developer.
 
 %prep
-%setup -q -c -n %{swift_source_location} -a 0 -a 1 -a 2 -a 3 -a 4 -a 5 -a 6 -a 7 -a 8 -a 9 -a 10 -a 11 -a 12 -a 13 -a 14 -a 15 -a 16 -a 17 -a 18 -a 19
+%setup -q -c -n %{swift_source_location} -a 0 -a 1 -a 2 -a 3 -a 4 -a 5 -a 6 -a 7 -a 8 -a 9 -a 10 -a 11 -a 12 -a 13 -a 14 -a 15 -a 16 -a 17 -a 18 -a 19 -a 20
 # The Swift build script requires directories to be named
 # in a specific way so renaming the source directories is
 # necessary
@@ -114,6 +115,7 @@ mv swift-argument-parser-%{swift_argument_parser_version} swift-argument-parser
 mv swift-driver-swift-%{swift_version} swift-driver
 mv swift-crypto-%{swift_crypto_version} swift-crypto
 mv ninja-%{ninja_version} ninja
+mv CMake-%{cmake_version} cmake
 
 # ICU
 mv icu-release-%{icu_version} icu
diff --git a/platforms/Linux/centos/8/Dockerfile b/platforms/Linux/centos/8/Dockerfile
@@ -12,7 +12,7 @@ LABEL PURPOSE="This image is configured to build Swift for the version of CentOS
 RUN yum -y update
 
 # RPM and yum development tools
-RUN yum install -y rpmdevtools yum-utils
+RUN yum install -y rpmdevtools yum-utils createrepo
 
 # Configure epel and powertools
 RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
diff --git a/platforms/Linux/centos/8/docker-compose.yaml b/platforms/Linux/centos/8/docker-compose.yaml
@@ -41,6 +41,10 @@ services:
     <<: *common
     command: /bin/bash -cl "./build_rpm.sh"
 
+  createrepo:
+    <<: *common
+    command: /bin/bash -cl "../shared/createrepo_rpm.sh"
+
   shell:
     <<: *common
     entrypoint: /bin/bash -l
diff --git a/platforms/Linux/centos/8/swiftlang.spec b/platforms/Linux/centos/8/swiftlang.spec
@@ -29,6 +29,7 @@ Source16:       https://github.com/apple/swift-syntax/archive/swift-%{swift_vers
 Source17:       https://github.com/jpsim/Yams/archive/%{yams_version}.zip#/yams.tar.gz
 Source18:       https://github.com/apple/swift-crypto/archive/refs/tags/%{swift_crypto_version}.tar.gz#/swift-crypto.tar.gz
 Source19:       https://github.com/ninja-build/ninja/archive/refs/tags/v%{ninja_version}.tar.gz#/ninja.tar.gz
+Source20:       https://github.com/KitWare/CMake/archive/refs/tags/v%{cmake_version}.tar.gz#/cmake.tar.gz
 
 Patch0:         patches/swift-api-checker.patch
 Patch1:         patches/hwasan_symbolize.patch
@@ -91,7 +92,7 @@ importantly, Swift is designed to make writing and maintaining
 correct programs easier for the developer.
 
 %prep
-%setup -q -c -n %{swift_source_location} -a 0 -a 1 -a 2 -a 3 -a 4 -a 5 -a 6 -a 7 -a 8 -a 9 -a 10 -a 11 -a 12 -a 13 -a 14 -a 15 -a 16 -a 17 -a 18 -a 19
+%setup -q -c -n %{swift_source_location} -a 0 -a 1 -a 2 -a 3 -a 4 -a 5 -a 6 -a 7 -a 8 -a 9 -a 10 -a 11 -a 12 -a 13 -a 14 -a 15 -a 16 -a 17 -a 18 -a 19 -a 20
 # The Swift build script requires directories to be named
 # in a specific way so renaming the source directories is
 # necessary
@@ -113,6 +114,7 @@ mv swift-argument-parser-%{swift_argument_parser_version} swift-argument-parser
 mv swift-driver-swift-%{swift_version} swift-driver
 mv swift-crypto-%{swift_crypto_version} swift-crypto
 mv ninja-%{ninja_version} ninja
+mv CMake-%{cmake_version} cmake
 
 # ICU
 mv icu-release-%{icu_version} icu
diff --git a/platforms/Linux/shared/RPM/createrepo_rpm.sh b/platforms/Linux/shared/RPM/createrepo_rpm.sh
@@ -0,0 +1,35 @@
+# This source file is part of the Swift.org open source project
+#
+# Copyright (c) 2021 Apple Inc. and the Swift project authors
+# Licensed under Apache License v2.0 with Runtime Library Exception
+#
+# See http://swift.org/LICENSE.txt for license information
+# See http://swift.org/CONTRIBUTORS.txt for Swift project authors
+
+#!/usr/bin/env bash
+
+set -eux
+
+output_dir=/output
+if [[ ! -d "$output_dir" ]]; then
+    echo "$output_dir does not exist, so no place to copy the artifacts!"
+    exit 1
+fi
+
+# always make sure we're up to date
+yum update -y
+
+# prepare direcoties
+mkdir -p "$HOME/createrepo"
+
+# Copy rpm file
+cp "$output_dir"/*.rpm "$HOME/createrepo/"
+
+# Create the repodata
+createrepo "$HOME/createrepo/" 2>&1 | tee "$HOME/createrepo-output.txt"
+
+
+# Include the createrepo log which can be used to determine what went
+# wrong if there are no artifacts
+cp "$HOME/createrepo-output.txt" "$output_dir"
+cp -r "$HOME/createrepo/repodata" "$output_dir/repodata"
diff --git a/platforms/Linux/shared/RPM/metadata.inc b/platforms/Linux/shared/RPM/metadata.inc
@@ -14,6 +14,7 @@
 %global swift_argument_parser_version 0.4.3
 %global swift_crypto_version 1.1.5
 %global ninja_version 1.10.2
+%global cmake_version 3.19.6
 
 # locations
 
