Development conda environment taking >45 minutes to solve

[datapythonista]

When creating a new environment from our environment.yml, it takes more than 45 minutes to solve (one of the steps of the environment creation). This happens both locally, and in the CI of #22854, which uses that file for one of the builds.

2018-11-30T03:02:59.3313236Z [create env]
2018-11-30T03:49:30.3527900Z Solving environment: ...working... done

The number of dependencies in this environment is slightly higher than in the travis jobs (it has the packages for linting and building the docs), but there are not many more.

The environment with those dependencies was creating in a reasonable time until recently (like one week ago). We just added to it couple of small changes, that if undone, don't solve the problem. I also tried to downgrade conda, to see if a new release was doing something different, but the creation of the environment also takes the same with old versions.

I did a bit of research inside conda, and seems like the time is spent in Resolve.solve, in the step of maximizing the timestamps: https://github.com/conda/conda/blob/master/conda/resolve.py#L1031

This receives all the packages candidates as a parameter if I'm not wrong, so I assume the problem comes from a release in one of our dependencies, that makes this optimization very slow. I tried to remove packages from the dependencies file to see if there is a single package that when removed fixes the problem, but after many tests I couldn't identify one, I think it's a combination of packages that causes the issue.

When running the process, one of the cores stay at 100%, confirming (I think) that it's the optimization itself what is taking long (and not a network access or other IO).

@TomAugspurger do you know what the problem can be? And what's the best way to report this to conda? I guess an issue in the conda GitHub, but not sure if there is a better way.

[TomAugspurger]

I know that @msarahan has been doing a lot of work to speed up parts of conda, which will be available in the next conda release.

@msarahan does anything in https://github.com/pandas-dev/pandas/blob/master/environment.yml stand out to you? It's a mix of defaults and conda-forge. I'm timing it now locally.

[TomAugspurger]

Though I can already say that solving didn't take long locally, less than a minute or two (on my Mac).

[datapythonista]

May be it's Linux specific then, both locally and on the azure job takes a lot of time.

[msarahan]

Timestamp maximization is an accurate identification of the issue. PR to disable it at https://github.com/conda/conda/pull/7895/files

That maximization is important at build time, especially locally as you iterate on recipes, but probably not for you here.

The problem as I understand is that conda is not eliminating package choices prior to the timestamp optimization time. Basically, all the solver data is collected at once, when really we should be doing an initial reduction based on all the other constraints, then applying timestamp info only as a tiebreaker.

I don't know why this is only just now popping up for you, but it happens when we have packages with "variants." For example, if you have libnetcdf that has two variants for hdf5 (1.8, 1.10), this triggers the problem.

Please consider applying the PR above in the short term. In the long term, a better filtering scheme will be a better fix than just turning it off.

[datapythonista]

Thanks for the detailed info @msarahan. Applying the PR could be an option for the CI, but new contributors to pandas also create a conda environment with that file to set up the environment, and making them apply the PR doesn't sound reasonable.

If I understand correctly, the problem is that conda can choose among more than one version of a given package. That makes sense to me, as my guess was that the problem started because a new release. So, does it make sense to you, that I try to identify a package or packages with more than one version available, which can be causing the problem, and I just pin them to a specific version in the file?

[msarahan]

Unfortunately, because of the way things work in conda, I don't think that will be a fix. I'm not really suggesting that you incorporate the application of that PR to conda as a standard thing for all pandas devs. I agree that's silly, and we need a fix on the conda side soon. If we merge that PR, there will probably be warnings and/or errors about multiple solutions from conda. That's why we haven't merged it.
I'd like to tell you that there is an immediate solution, but aside from my crummy hack of applying that PR, I don't have one right now. We're working on it, and hope to have a fix quickly, but it will require a new version of conda, and depending on the changes necessary, that new version may need to go through a lengthy QA time.

[datapythonista]

I checked, and the number of candidates the the optimization receives for some packages is huge, like numpy:

numpy                 453
hypothesis            164
setuptools             55
scipy                  51
scikit                 50
notebook               49
cryptography           47
sphinx                 46
widgetsnbextension     45
pandas                 40

I pinned the ones with more candidates, so only the last versions are considered, removed couple of packages that I don't think we need, and the number of packages that the optimization receives decreased from around 2,500 to 800. And the process is not super fast, but I think the solver takes couple of minutes, which is perfectly fine compared to the 45 minutes.

So, looks like this should work as a temporary solution.

And I totally understand that you can't merge that PR. I was more interested in an immediate workaround from our side.