Add debian bookworm, remove buster #822

sspans-sbp · 2023-04-03T14:19:45Z

Debian bookworm is in hard-freeze - and will probably be released in the coming weeks.
So it makes sense to have a PR ready to add it once released.

https://lists.debian.org/debian-devel-announce/2023/03/msg00004.html

Feel free to close this if it clashes with other plans, just trying to help.

tianon · 2023-04-03T21:56:44Z

As in previous cycles, this will also mean we remove support for Debian Buster (as the compilation of Python is very heavy, so we stick to only maintaining builds for the latest two versions of each supported distribution).

sspans-sbp · 2023-04-07T12:39:27Z

Testing pipeline showed that sqlite3 and gdbm were purged from the slim images.
Which breaks the related python modules.

The usr-merge implemented for bookworm breaks the dpkg-query used for cleanup. The cleanup
attempts to locate packages based on the libraries used. But with usr-merge libraries are now available in /lib and /usr/lib. Resolving this isn't trivial, the linker prefers/lib but packages can install to either /lib or /usr/lib.

I've opted to use realpath to resolve alternate paths for libraries. This seems to work well enough.

root@bf38a4dca3be:/# echo /lib/x86_64-linux-gnu/libsqlite3.so.0 $(realpath /lib/x86_64-linux-gnu/libsqlite3.so.0) | xargs dpkg-query --search
dpkg-query: no path found matching pattern /lib/x86_64-linux-gnu/libsqlite3.so.0
libsqlite3-0:amd64: /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6

An alternative would be to query for filenames only, but dpkg-query will do a wildcard match in that case.
The wildcard match triggers the extraneous inclusion of libc6-dev.

root@705c373582e2:~# dpkg-query --search libc.so.6
libc6-dev:amd64: /usr/share/gdb/auto-load/lib/x86_64-linux-gnu/libc.so.6-gdb.py
libc6:amd64: /lib/x86_64-linux-gnu/libc.so.6

sspans · 2023-05-04T19:36:16Z

FYI: bookwork release date has been announced as 2023-06-10.

https://lists.debian.org/debian-devel-announce/2023/04/msg00007.html

aqeelat · 2023-06-11T06:37:37Z

@sspans-sbp Debian 12 is finally out now. Could we get this PR merged?

sspans-sbp · 2023-06-11T12:22:10Z

I've re-generated the PR now that bookworm has been released.

It looks good to me, but I'll let @tianon judge the usrmerge change

Dockerfile-linux.template

sspans-sbp · 2023-06-13T07:37:05Z

@tianon great suggestion, merged and regenerated.

tianon · 2023-06-13T17:45:45Z

(while reviewing, I noticed this missed versions.sh updates, so while updating those I also squashed ❤️)

Changes: - docker-library/python@9a7a540: Merge pull request docker-library/python#822 from sspans-sbp/master - docker-library/python@5d908b0: add bookworm, remove buster

aqeelat · 2023-06-14T09:29:02Z

@tianon Thank you for merging the PR. How long does it take for the images to be published? The dockerhub repo was updated 11 hours ago and still does not include bookworm.

aqeelat · 2023-06-14T12:45:56Z

@tianon the Jenkins pipeline ran again about 3 hours ago but still no bookworm. Is there something stuck in CI?
https://doi-janky.infosiftr.net/view/flat/job/update.sh/job/python/9897/

sspans-sbp · 2023-06-14T14:37:22Z

@aqeelat The official-images repository has not been updated yet.

I'm not quite sure if that's an automated or manual procedure.
But @tianon is listed as a maintainer - so he will get to it in due time.

tianon · 2023-06-14T16:12:21Z

👀 docker-library/official-images#14854

This pins the docker image used in `src/python/pants/backend/docker/util_rules/docker_build_context_test.py:test_packaged_pex_environment` to https://hub.docker.com/layers/library/python/3.8-buster/images/sha256-bc4b9fb034a871b285bea5418cedfcaa9d2ab5590fb5fb6f0c42aaebb2e2c911?context=explore, which is (at the time of writing) the most recent `3.8-buster` image. This test has recently started failing with an error when building `psutil`: ``` E psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory E 9 | #include <Python.h> E | ^~~~~~~~~~ E compilation terminated. E error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 E ---------------------------------------- E ERROR: Failed building wheel for psutil ``` The failure correlates with the change in base image (from buster to bookworm) docker-library/python#822, which seems to have been published at 2023-06-14T23:05Z (https://hub.docker.com/layers/library/python/3.8/images/sha256-b904ee5365f53d19de5ff11c1ce96a616224a55306d55e3cf29a11e969a3a7e0?context=explore). This change, that pins to the older image, seems to fix the problem. This fixes https://pantsbuild.slack.com/archives/C0D7TNJHL/p1686790197387729

…y-pick of #19324) (#19337) This pins the docker image used in `src/python/pants/backend/docker/util_rules/docker_build_context_test.py:test_packaged_pex_environment` to https://hub.docker.com/layers/library/python/3.8-buster/images/sha256-bc4b9fb034a871b285bea5418cedfcaa9d2ab5590fb5fb6f0c42aaebb2e2c911?context=explore, which is (at the time of writing) the most recent `3.8-buster` image. This test has recently started failing with an error when building `psutil`: ``` E psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory E 9 | #include <Python.h> E | ^~~~~~~~~~ E compilation terminated. E error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 E ---------------------------------------- E ERROR: Failed building wheel for psutil ``` The failure correlates with the change in base image (from buster to bookworm) docker-library/python#822, which seems to have been published at 2023-06-14T23:05Z (https://hub.docker.com/layers/library/python/3.8/images/sha256-b904ee5365f53d19de5ff11c1ce96a616224a55306d55e3cf29a11e969a3a7e0?context=explore). This change, that pins to the older image, seems to fix the problem. This fixes https://pantsbuild.slack.com/archives/C0D7TNJHL/p1686790197387729 Co-authored-by: Huon Wilson <[email protected]>

…y-pick of #19324) (#19335) This pins the docker image used in `src/python/pants/backend/docker/util_rules/docker_build_context_test.py:test_packaged_pex_environment` to https://hub.docker.com/layers/library/python/3.8-buster/images/sha256-bc4b9fb034a871b285bea5418cedfcaa9d2ab5590fb5fb6f0c42aaebb2e2c911?context=explore, which is (at the time of writing) the most recent `3.8-buster` image. This test has recently started failing with an error when building `psutil`: ``` E psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory E 9 | #include <Python.h> E | ^~~~~~~~~~ E compilation terminated. E error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 E ---------------------------------------- E ERROR: Failed building wheel for psutil ``` The failure correlates with the change in base image (from buster to bookworm) docker-library/python#822, which seems to have been published at 2023-06-14T23:05Z (https://hub.docker.com/layers/library/python/3.8/images/sha256-b904ee5365f53d19de5ff11c1ce96a616224a55306d55e3cf29a11e969a3a7e0?context=explore). This change, that pins to the older image, seems to fix the problem. This fixes https://pantsbuild.slack.com/archives/C0D7TNJHL/p1686790197387729 Co-authored-by: Huon Wilson <[email protected]>

…y-pick of #19324) (#19336) This pins the docker image used in `src/python/pants/backend/docker/util_rules/docker_build_context_test.py:test_packaged_pex_environment` to https://hub.docker.com/layers/library/python/3.8-buster/images/sha256-bc4b9fb034a871b285bea5418cedfcaa9d2ab5590fb5fb6f0c42aaebb2e2c911?context=explore, which is (at the time of writing) the most recent `3.8-buster` image. This test has recently started failing with an error when building `psutil`: ``` E psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory E 9 | #include <Python.h> E | ^~~~~~~~~~ E compilation terminated. E error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 E ---------------------------------------- E ERROR: Failed building wheel for psutil ``` The failure correlates with the change in base image (from buster to bookworm) docker-library/python#822, which seems to have been published at 2023-06-14T23:05Z (https://hub.docker.com/layers/library/python/3.8/images/sha256-b904ee5365f53d19de5ff11c1ce96a616224a55306d55e3cf29a11e969a3a7e0?context=explore). This change, that pins to the older image, seems to fix the problem. This fixes https://pantsbuild.slack.com/archives/C0D7TNJHL/p1686790197387729 Co-authored-by: Huon Wilson <[email protected]>

sspans-sbp changed the title ~~Add debian bookworm~~ Add debian bookworm, remove buster Apr 5, 2023

sspans-sbp force-pushed the master branch 3 times, most recently from 342143d to 0ba21b1 Compare April 7, 2023 12:24

sspans-sbp force-pushed the master branch from 28cae4b to 77f1496 Compare April 8, 2023 12:34

sspans-sbp closed this Jun 11, 2023

sspans-sbp force-pushed the master branch 2 times, most recently from 77f1496 to b744d97 Compare June 11, 2023 12:13

sspans-sbp reopened this Jun 11, 2023

tianon reviewed Jun 12, 2023

View reviewed changes

Dockerfile-linux.template Outdated Show resolved Hide resolved

sspans-sbp force-pushed the master branch from dd7a499 to ef169b4 Compare June 13, 2023 07:36

add bookworm, remove buster

5d908b0

tianon force-pushed the master branch from ef169b4 to 5d908b0 Compare June 13, 2023 17:45

tianon mentioned this pull request Jun 13, 2023

Move to Debian 12 (bookworm) docker-library/httpd#235

Merged

yosifkit mentioned this pull request Jun 13, 2023

Add debian:bookworm docker-library/ruby#415

Merged

tianon merged commit 9a7a540 into docker-library:master Jun 13, 2023

tianon mentioned this pull request Jun 14, 2023

Update python docker-library/official-images#14854

Merged

hugo-gomes mentioned this pull request Jun 14, 2023

feat: adding python:slim-bookworm ripe-tech/docker#19

Merged

huonw mentioned this pull request Jun 15, 2023

Pin test_package_pex_environment's image to a particular sha pantsbuild/pants#19324

Merged

WorkerPants mentioned this pull request Jun 15, 2023

Pin test_package_pex_environment's image to a particular sha (Cherry-pick of #19324) pantsbuild/pants#19335

Merged

WorkerPants mentioned this pull request Jun 15, 2023

Pin test_package_pex_environment's image to a particular sha (Cherry-pick of #19324) pantsbuild/pants#19336

Merged

WorkerPants mentioned this pull request Jun 15, 2023

Pin test_package_pex_environment's image to a particular sha (Cherry-pick of #19324) pantsbuild/pants#19337

Merged

TemaSM mentioned this pull request Jun 18, 2023

python:3.9 - Failed run apt update from the last version of the image #837

Closed

yosifkit mentioned this pull request Jun 27, 2023

Update ldd|awk to account for Bookworm's merged /usr nodejs/docker-node#1927

Merged

12 tasks

tianon mentioned this pull request Jul 3, 2023

Update phpmyadmin to latest upstream changes docker-library/official-images#14960

Merged

yosifkit mentioned this pull request Aug 30, 2023

Missing 3.11.5-buster image #863

Closed

oleg-derevenetz mentioned this pull request Dec 12, 2023

Replace Debian Buster (Debian 10) with Debian Bookworm (Debian 12) platisd/clang-tidy-pr-comments#60

Merged

rm03 mentioned this pull request Feb 12, 2024

Backend version updates pennlabs/penn-clubs#605

Merged

dannysauer mentioned this pull request Mar 4, 2024

chore: remove support for Python 3.7; prefer 3.8 determined-ai/determined#7329

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add debian bookworm, remove buster #822

Add debian bookworm, remove buster #822

Uh oh!

sspans-sbp commented Apr 3, 2023

Uh oh!

tianon commented Apr 3, 2023

Uh oh!

sspans-sbp commented Apr 7, 2023 •

edited

Loading

Uh oh!

sspans commented May 4, 2023

Uh oh!

aqeelat commented Jun 11, 2023

Uh oh!

sspans-sbp commented Jun 11, 2023

Uh oh!

Uh oh!

sspans-sbp commented Jun 13, 2023

Uh oh!

tianon commented Jun 13, 2023

Uh oh!

aqeelat commented Jun 14, 2023

Uh oh!

aqeelat commented Jun 14, 2023 •

edited

Loading

Uh oh!

sspans-sbp commented Jun 14, 2023

Uh oh!

tianon commented Jun 14, 2023

Uh oh!

Uh oh!

Add debian bookworm, remove buster #822

Add debian bookworm, remove buster #822

Uh oh!

Conversation

sspans-sbp commented Apr 3, 2023

Uh oh!

tianon commented Apr 3, 2023

Uh oh!

sspans-sbp commented Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sspans commented May 4, 2023

Uh oh!

aqeelat commented Jun 11, 2023

Uh oh!

sspans-sbp commented Jun 11, 2023

Uh oh!

Uh oh!

sspans-sbp commented Jun 13, 2023

Uh oh!

tianon commented Jun 13, 2023

Uh oh!

aqeelat commented Jun 14, 2023

Uh oh!

aqeelat commented Jun 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sspans-sbp commented Jun 14, 2023

Uh oh!

tianon commented Jun 14, 2023

Uh oh!

Uh oh!

sspans-sbp commented Apr 7, 2023 •

edited

Loading

aqeelat commented Jun 14, 2023 •

edited

Loading