Skip to content

TPU upgrades to tf and torch, clean up dead code #1480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 8, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 0 additions & 23 deletions tpu/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -25,36 +25,13 @@ ADD patches/kaggle_session.py /root/.local/lib/${PYTHON_VERSION_PATH}/site-packa
ADD patches/kaggle_web_client.py /root/.local/lib/${PYTHON_VERSION_PATH}/site-packages/kaggle_web_client.py
ADD patches/kaggle_datasets.py /root/.local/lib/${PYTHON_VERSION_PATH}/site-packages/kaggle_datasets.py

# Disable GCP integrations for now
# ADD patches/kaggle_gcp.py /root/.local/lib/${PYTHON_VERSION_PATH}/site-packages/kaggle_gcp.py

# Disable logging to file (why do we need this?)
# ADD patches/log.py /root/.local/lib/${PYTHON_VERSION_PATH}/site-packages/log.py

# sitecustomize adds significant latency to ipython kernel startup and should only be added if needed
# ADD patches/sitecustomize.py /root/.local/lib/${PYTHON_VERSION_PATH}/site-packages/sitecustomize.py

# Prereqs
# This is needed for cv2 (opencv-python):
# https://stackoverflow.com/questions/55313610/importerror-libgl-so-1-cannot-open-shared-object-file-no-such-file-or-directo
RUN apt-get update && apt-get install ffmpeg libsm6 libxext6 -y

# Install all the packages together for maximum compatibility.

# Install Tensorflow.

# Install Pytorch & related packages
# https://cloud.google.com/tpu/docs/pytorch-xla-ug-tpu-vm#changing_pytorch_version
# The URL doesn't include patch version. i.e. must use 1.11 instead of 1.11.0
# We need to keep the numpy version the same as the installed tf one but compatible with other installs.

# Install JAX & related packages
# https://cloud.google.com/tpu/docs/jax-quickstart-tpu-vm#install_jax_on_your_cloud_tpu_vm

# Packages needed by the Notebook editor

# Additional useful packages should be added in the requirements.txt

# Bring in the requirements.txt and replace variables in it:
RUN apt-get install -y gettext
ADD tpu/requirements.txt /kaggle_requirements.txt
Expand Down
6 changes: 3 additions & 3 deletions tpu/config.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ PYTHON_VERSION_PATH=python3.10
TENSORFLOW_VERSION=2.18.0
# gsutil ls gs://pytorch-xla-releases/wheels/tpuvm/* | grep libtpu | grep torch_xla | grep -v -E ".*rc[0-9].*" | sed 's/.*torch_xla-\(.*\)+libtpu.*/\1/' | sort -rV
# Supports nightly
TORCH_VERSION=2.5.0
TORCH_VERSION=2.6.0
# https://github.com/pytorch/audio supports nightly
TORCHAUDIO_VERSION=2.5.0
TORCHAUDIO_VERSION=2.6.0
# https://github.com/pytorch/vision supports nightly
TORCHVISION_VERSION=0.20.0
TORCHVISION_VERSION=0.21.0
TORCH_LINUX_WHEEL_VERSION=manylinux_2_28_x86_64
4 changes: 2 additions & 2 deletions tpu/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# TPU Utils
tpu-info
# Tensorflow packages
tensorflow-tpu>=${TENSORFLOW_VERSION}
tensorflow-tpu==${TENSORFLOW_VERSION}
--find-links https://storage.googleapis.com/libtpu-tf-releases/index.html
tensorflow_hub
tensorflow-io
tensorflow-probability
# Torch packages
torch~=${TORCH_VERSION}
torch==${TORCH_VERSION}
https://storage.googleapis.com/pytorch-xla-releases/wheels/tpuvm/torch_xla-${TORCH_VERSION}+libtpu-${PYTHON_WHEEL_VERSION}-${PYTHON_WHEEL_VERSION}-${TORCH_LINUX_WHEEL_VERSION}.whl
torchaudio==${TORCHAUDIO_VERSION}
torchvision==${TORCHVISION_VERSION}
Expand Down