Fix python3 six and make issue on Amazon Linux 2 #48

shahmishal · 2021-10-21T18:40:48Z

---------------------------------------------
Error when bootstrapping CMake:
Cannot find appropriate Makefile processor on this system.
Please specify one using environment variable MAKE.
---------------------------------------------

shahmishal · 2021-10-21T19:29:59Z

Started a build with this PR - https://ci.swift.org/job/swift-5.5-rpm-package-amazon-linux-2/5/console

tomerd · 2021-10-21T19:31:39Z

platforms/Linux/amazonlinux/2/build_rpm.sh

@@ -32,6 +32,9 @@ cp patches/*.patch $HOME/rpmbuild/SOURCES/
 pushd $HOME/rpmbuild/SPECS
 # install all the dependencies needed to build Swift from the spec file itself
 yum-builddep -y ./swiftlang.spec
+# fix python six
+mkdir -p /usr/local/lib/python3.7/site-packages/
+easy_install-3.7 six


these workarounds should be done in the Docker file not the build_rpm.sh file. I did't notice that in one of the earlier PRs (centos 7?) so that needs to be moved as well

additionally if we are now building cmake from source, should we remove it from the "BuildRequires" list?

Dockerfile would not have the package installed at this point, we can move it to swiftlang.spec file.

Yes, we should drop CMake from BuildRequires. I will create a new PR for that

dockerfile does install the deps: https://github.com/apple/swift-installer-scripts/blob/main/platforms/Linux/amazonlinux/2/Dockerfile#L23

so it could and should be moved there

futurejones · 2021-10-22T09:58:15Z

@tomerd @shahmishal
Can we get the issues section opened on this repository? Currently we have nowhere to post anything that isn't directly associated with an individual PR.

I have done some testing on the amazon linux rpm - swiftlang-5.5.0-1.amzn2.x86_64.rpm and there are some issues with dependencies(requires) that are preventing the package installing.

I also uploaded the package to a test repository which revealed some strange errors in the package requires and provides lists.

Requires
/bin/sh
/sbin/ldconfig
/sbin/ldconfig
/usr/bin/env
binutils
gcc
git
glibc-static
gzip
ld-linux-x86-64.so.2()(64bit)
ld-linux-x86-64.so.2(GLIBC_2.3)(64bit)
libBlocksRuntime.so()(64bit)
libFoundation.so()(64bit)
libFoundationNetworking.so()(64bit)
libFoundationXML.so()(64bit)
libIndexStore.so.10git()(64bit)
libbsd
libc.so.6(GLIBC_2.17)(64bit)
libcurl
libcurl.so.4()(64bit)
libdispatch.so()(64bit)
libdl.so.2()(64bit)
libdl.so.2(GLIBC_2.2.5)(64bit)
libedit
libedit.so.0()(64bit)
libform.so.6()(64bit)
libgcc_s.so.1()(64bit)
libgcc_s.so.1(GCC_3.0)(64bit)
libgcc_s.so.1(GCC_3.3)(64bit)
libgcc_s.so.1(GCC_3.3.1)(64bit)
libicu
libicudataswift.so.65()(64bit)
libicui18nswift.so.65()(64bit)
libicuucswift.so.65()(64bit)
libllbuild.so()(64bit)
liblldb.so.10git()(64bit)
liblldb.so.10git(LLVM_10)(64bit)
libm.so.6()(64bit)
libm.so.6(GLIBC_2.2.5)(64bit)
libm.so.6(GLIBC_2.23)(64bit)
libncurses.so.6()(64bit)
libpanel.so.6()(64bit)
libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.12)(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit)
libpthread.so.0(GLIBC_2.3.2)(64bit)
libpthread.so.0(GLIBC_2.3.4)(64bit)
libpython3.7m.so.1.0()(64bit)
librt.so.1()(64bit)
librt.so.1(GLIBC_2.2.5)(64bit)
libsqlite
libsqlite3.so.0()(64bit)
libstdc++-static
libstdc++.so.6()(64bit)
libstdc++.so.6(CXXABI_1.3)(64bit)
libstdc++.so.6(CXXABI_1.3.1)(64bit)
libstdc++.so.6(CXXABI_1.3.11)(64bit)
libstdc++.so.6(CXXABI_1.3.3)(64bit)
libstdc++.so.6(CXXABI_1.3.5)(64bit)
libstdc++.so.6(CXXABI_1.3.7)(64bit)
libstdc++.so.6(GLIBCXX_3.4)(64bit)
libstdc++.so.6(GLIBCXX_3.4.11)(64bit)
libstdc++.so.6(GLIBCXX_3.4.14)(64bit)
libstdc++.so.6(GLIBCXX_3.4.15)(64bit)
libstdc++.so.6(GLIBCXX_3.4.17)(64bit)
libstdc++.so.6(GLIBCXX_3.4.18)(64bit)
libstdc++.so.6(GLIBCXX_3.4.19)(64bit)
libstdc++.so.6(GLIBCXX_3.4.20)(64bit)
libstdc++.so.6(GLIBCXX_3.4.21)(64bit)
libstdc++.so.6(GLIBCXX_3.4.22)(64bit)
libstdc++.so.6(GLIBCXX_3.4.9)(64bit)
libswiftCore.so()(64bit)
libswiftDispatch.so()(64bit)
libswiftGlibc.so()(64bit)
libswift_Concurrency.so()(64bit)
libtinfo.so.6()(64bit)
libutil.so.1()(64bit)
libuuid
libuuid.so.1()(64bit)
libuuid.so.1(UUID_1.0)(64bit)
libxml2
libxml2.so.2()(64bit)
libxml2.so.2(LIBXML2_2.4.30)(64bit)
libxml2.so.2(LIBXML2_2.5.7)(64bit)
libxml2.so.2(LIBXML2_2.5.8)(64bit)
libxml2.so.2(LIBXML2_2.5.9)(64bit)
libxml2.so.2(LIBXML2_2.6.0)(64bit)
libxml2.so.2(LIBXML2_2.6.18)(64bit)
libxml2.so.2(LIBXML2_2.6.23)(64bit)
libxml2.so.2(LIBXML2_2.6.8)(64bit)
libxml2.so.2(LIBXML2_2.7.3)(64bit)
libxml2.so.2(LIBXML2_2.9.1)(64bit)
libz.so.1()(64bit)
libz.so.1(ZLIB_1.2.0)(64bit)
tar
tzdata

Provides
libBlocksRuntime.so()(64bit)
libFoundation.so()(64bit)
libFoundationNetworking.so()(64bit)
libFoundationXML.so()(64bit)
libIndexStore.so.10git()(64bit)
libIndexStore.so.10git(LLVM_10)(64bit)
libPackageDescription.so()(64bit)
libPackagePlugin.so()(64bit)
libXCTest.so()(64bit)
lib_InternalSwiftScan.so()(64bit)
lib_InternalSwiftScan.so(LLVM_10)(64bit)
lib_InternalSwiftSyntaxParser.so()(64bit)
lib_InternalSwiftSyntaxParser.so(LLVM_10)(64bit)
libclang_rt.asan-x86_64.so()(64bit)
libclang_rt.dyndd-x86_64.so()(64bit)
libclang_rt.hwasan-x86_64.so()(64bit)
libclang_rt.memprof-x86_64.so()(64bit)
libclang_rt.scudo-x86_64.so()(64bit)
libclang_rt.scudo_minimal-x86_64.so()(64bit)
libclang_rt.ubsan_minimal-x86_64.so()(64bit)
libclang_rt.ubsan_standalone-x86_64.so()(64bit)
libdispatch.so()(64bit)
libicudataswift.so.65()(64bit)
libicui18nswift.so.65()(64bit)
libicuucswift.so.65()(64bit)
libllbuild.so()(64bit)
libllbuildSwift.so()(64bit)
liblldb.so.10git()(64bit)
liblldb.so.10git(LLVM_10)(64bit)
libsourcekitdInProc.so()(64bit)
libsourcekitdInProc.so(LLVM_10)(64bit)
libswiftCore.so()(64bit)
libswiftDemangle.so()(64bit)
libswiftDispatch.so()(64bit)
libswiftGlibc.so()(64bit)
libswiftRemoteMirror.so()(64bit)
libswiftSwiftOnoneSupport.so()(64bit)
libswift_Concurrency.so()(64bit)
libswift_Differentiation.so()(64bit)
swiftlang(x86-64) = 5.5.0-1.amzn2
swiftlang = 5.5.0-1.amzn2

These errors are also present in the centos 7 package and the swift-lang-5.5-1.el8.x86_64.rpm package provided by @tachoknight on the Fedora repository.

It seems to indicate a problem in the spec files setup.

tachoknight · 2021-10-22T10:43:01Z

Sorry, but I'm missing what the errors are.

futurejones · 2021-10-22T11:16:42Z

The Requires list should only list the additional packages (dependencies) that need to be installed.
e.g.

Requires
binutils
gcc
git
glibc-static
gzip
libbsd
libcurl
libedit
libicu
libsqlite3x
libstdc++-static
libuuid
libxml2
python3
tar
tzdata

The Provides list should only list the actual package being installed i.e. swiftlang.
Not all the internal libs e.g libBlocksRuntime.so()(64bit), libFoundation.so()(64bit) etc.

Provides
swiftlang(x86-64) = 5.5.0-1.amzn2
swiftlang = 5.5.0-1.amzn2

tachoknight · 2021-10-22T11:24:36Z

I'm confused; the spec file doesn't explicitly mention any of those files under "provides".

futurejones · 2021-10-22T11:30:57Z

It looks like the requires and provides are automatically added in the build process and it sometimes picks up "private" packages which need to be filtered out.
https://docs.fedoraproject.org/en-US/packaging-guidelines/AutoProvidesAndRequiresFiltering/

shahmishal · 2021-10-22T15:59:07Z

@futurejones

Can we get the issues section opened on this repository? Currently we have nowhere to post anything that isn't directly associated with an individual PR.

Currently for Swift Project, we are using bugs.swift.org. In the future, we are planning on switching over to GitHub Issues.

I have done some testing on the amazon linux rpm - swiftlang-5.5.0-1.amzn2.x86_64.rpm and there are some issues with dependencies(requires) that are preventing the package installing.

Thanks for testing the package! If you already know the fix can you put up a PR to resolve the issue?

tomerd · 2021-10-22T18:16:03Z

I would love to switch to GH issues if possible

tomerd · 2021-10-22T18:17:15Z

thanks for testing this @futurejones, do you know what the issue may be in the spec setup?

compnerd · 2021-10-30T02:52:56Z

The Requires list should only list the additional packages (dependencies) that need to be installed. e.g.
Requires
binutils
gcc
git
glibc-static
gzip
libbsd
libcurl
libedit
libicu
libsqlite3x
libstdc++-static
libuuid
libxml2
python3
tar
tzdata

I think that gcc should be a BuildRequires only. Additionally, the libstdc++-static and glibc-static should be replaced with libstdc++ and glibc. The runtime dynamically links against the C/C++ runtime (and must in order to avoid any ODR). libbsd has not been needed for a while and should be dropped. libicu seems suspect - we provide our own ICU library builds. I wonder if we can tweak the build to statically link against sqlite, libxml2, uuid, and libcurl.

I think that we are including more runtimes than necessary (e.g. hwasan), and simultaneously under including files (e.g. the profile runtime).

tomerd · 2021-11-01T16:45:22Z

I think that gcc should be a BuildRequires only. Additionally, the libstdc++-static and glibc-static should be replaced with libstdc++ and glibc. The runtime dynamically links against the C/C++ runtime (and must in order to avoid any ODR). libbsd has not been needed for a while and should be dropped. libicu seems suspect - we provide our own ICU library builds. I wonder if we can tweak the build to statically link against sqlite, libxml2, uuid, and libcurl.

I think that we are including more runtimes than necessary (e.g. hwasan), and simultaneously under including files (e.g. the profile runtime).

I'd be happy to see this list paired down if possible. less is better in this case. this list comes from our existing docker setup and iirc the -static variants are needed for static linking. @drexin could you chime in here or work with @compnerd to determine if anything should be removed?

@compnerd in any case, I think this PR can move forward?

compnerd · 2021-11-01T16:51:28Z

Just to verify, cmake is being dropped because we build it and use it? If so, doing the rest subsequently SGTM.

shahmishal · 2021-11-01T16:52:33Z

Correct, we are building CMake to make sure we have the latest version required for Swift build system.

tachoknight · 2021-11-01T16:56:53Z

Correct, we are building CMake to make sure we have the latest version required for Swift build system.

Is this a new change in main? I've modified my spec file to build ninja so that BuildDependency is gone, but I don't see a recipe in the build-presets.ini that also builds CMake as well.

shahmishal · 2021-11-01T17:00:45Z

This was added ~2 years back by @compnerd - swiftlang/swift#26301

shahmishal · 2021-11-01T22:17:39Z

It would be a good to review all of the requirements in Dockerfiles and RPM spec file. Created an issue - https://bugs.swift.org/browse/SR-15418 to track the work.

shahmishal added 2 commits October 21, 2021 11:40

Update build_rpm.sh

28826e6

Add make to Amazon Linux 2

e02007e

shahmishal changed the title ~~Fix python3 six issue on Amazon Linux 2~~ Fix python3 six and make issue on Amazon Linux 2 Oct 21, 2021

shahmishal requested a review from tomerd October 21, 2021 19:29

tomerd reviewed Oct 21, 2021

View reviewed changes

shahmishal added 2 commits October 24, 2021 21:37

Move the workaround from spec script to Dockerfile

016b0a1

Drop CMake build requires because it's built by swift build-script

87b32c2

tomerd approved these changes Oct 25, 2021

View reviewed changes

shahmishal merged commit 23ce3ce into main Nov 1, 2021

shahmishal deleted the shahmishal/fix-python3-six-amazonlinux2 branch March 18, 2022 07:00

shahmishal mentioned this pull request Apr 25, 2022

[SR-15418] Review all of the require packages in Dockerfile and RPM spec files swiftlang/swift#57732

Open

Fix python3 six and make issue on Amazon Linux 2 #48

Fix python3 six and make issue on Amazon Linux 2 #48

Uh oh!

Conversation

shahmishal commented Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shahmishal commented Oct 21, 2021

Uh oh!

tomerd Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

tomerd Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

shahmishal Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

shahmishal Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

tomerd Oct 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

futurejones commented Oct 22, 2021

Uh oh!

tachoknight commented Oct 22, 2021

Uh oh!

futurejones commented Oct 22, 2021

Uh oh!

tachoknight commented Oct 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

futurejones commented Oct 22, 2021

Uh oh!

shahmishal commented Oct 22, 2021

Uh oh!

tomerd commented Oct 22, 2021

Uh oh!

tomerd commented Oct 22, 2021

Uh oh!

compnerd commented Oct 30, 2021

Uh oh!

tomerd commented Nov 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

compnerd commented Nov 1, 2021

Uh oh!

shahmishal commented Nov 1, 2021

Uh oh!

tachoknight commented Nov 1, 2021

Uh oh!

shahmishal commented Nov 1, 2021

Uh oh!

shahmishal commented Nov 1, 2021

Uh oh!

Uh oh!

shahmishal commented Oct 21, 2021 •

edited

Loading

tomerd Oct 22, 2021 •

edited

Loading

tachoknight commented Oct 22, 2021 •

edited

Loading

tomerd commented Nov 1, 2021 •

edited

Loading