Skip to content

CI: Micromamba taking too long to resolve the environments in the CI #61531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
datapythonista opened this issue Jun 1, 2025 · 6 comments · May be fixed by #61594
Open

CI: Micromamba taking too long to resolve the environments in the CI #61531

datapythonista opened this issue Jun 1, 2025 · 6 comments · May be fixed by #61594
Assignees
Labels
CI Continuous Integration Dependencies Required and optional dependencies good first issue

Comments

@datapythonista
Copy link
Member

Our CI jobs are frequently failing now as they timeout after 90 minutes of execution. Of those 90 minutes, 25 are spent on micromamba resolving the environment.

In the past we have fixed this by limiting the number of packages to be considered. For example, if the environment just says numpy, maybe there are 200 versions that will be considered. While if we say numpy >= 2 the number can be limited to few.

I'm not sure which packages have lots of options, and we don't want to filter out the versions that make sense to install. But we should have a look and see if by adding few constraints we can get a reasonable time to solve the environment.

@datapythonista datapythonista added CI Continuous Integration good first issue labels Jun 1, 2025
@microslaw
Copy link

take

Ishubhammohole added a commit to Ishubhammohole/pandas that referenced this issue Jun 1, 2025
…time

Added version constraints to packages like ipython, ipywidgets, nbformat, notebook, dask-core, and seaborn-base. This helps reduce environment resolution time and prevents CI timeouts during micromamba setup.

Fixes issue: pandas-dev#61531
@datapythonista
Copy link
Member Author

The failing CI jobs is exactly what we are trying to fix. I'll close the PR for now, as it doesn't really help as it is. The dependencies that need to be fixed are not our development environment (not sure how long it takes, but not 25 minutes the last time I installed it). It's the dependencies for the CI in ci/deps.

Before opening a PR it would be good to share the analysis on what we're planning to do and why, so we can discuss what to pin exactly.

@datapythonista datapythonista added the Dependencies Required and optional dependencies label Jun 2, 2025
@microslaw
Copy link

On a closing note, I've tried to benchmark micromamba on every environment in ci/deps. Nothing came even close to the 25 minutes mentioned, on most environments it finished within two minutes, with few 7-ish minute outliers. Feel free to see the results here

@datapythonista
Copy link
Member Author

Interesting. Looks like it could be the Python 3.13 jobs only, and in some particular cases. I saw the jobs timing out often, but I didn't see in some cases they run fast.

You can see a very slow case here: https://github.com/pandas-dev/pandas/actions/runs/15379937435/job/43269355646?pr=61533

Maybe a bug in micromamba in Python 3.13. If it happens only sometimes, it'll be difficult to find the problem, but maybe we can check in the micromamba repo if there is an issue for this, and open one if it's not.

@microslaw
Copy link

microslaw commented Jun 2, 2025

I looked through their issues, didn't see anything related, I'm not sure if this is micromamba's fault. I will try to reproduce and pinpoint it, so that we'll have something more concrete, unless the issue proves too rare/difficult

@datapythonista
Copy link
Member Author

I opened mamba-org/mamba#3970

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous Integration Dependencies Required and optional dependencies good first issue
Projects
None yet
2 participants