Automate deployment from CircleCI

[oesteban]

Seems like Circle have recently added the tag filtering that was missing in Circle 2.0.

Using that, we could automate deployment to Docker Hub and to Pypi as we currently do with fmriprep and mriqc. Additionally, we would also generate the new conda hash.

Finally this would really simplify the process, since it would be automatically triggered by cutting a new release through github.

[satra]

@oesteban - this is already happening through travis for pypi.

[effigies]

What deployment functionality is currently missing? Is this an actual issue?

[oesteban]

We would need to double check that pushing the docker images is still working with circle 2.0. Also, I don't know if the conda-forge distribution is also updated. If so, you are right this is not an actual issue. Sorry I don't know the status at the moment.

[effigies]

@kaczmarj Can you quickly verify whether the docker images are being pushed correctly? If I'm understanding the current logs, at least non-release commits to master are not producing a nipype/nipype:latest that prevents every CI run from rebuilding that base image. Is that something that's only supposed to happen on releases?

[kaczmarj]

@effigies - the docker images are uploaded to DockerHub only when building the master branch, regardless of release. The latest nipype docker images were uploaded 13 hours ago, which coincides with this CircleCI build. But there is no logic that would upload a release docker image, say nipype/nipype:1.0.0.

What do you mean that every CI is not rebuilding the base image? The base image should only be rebuilt if the base Dockerfile has changed.

[effigies]

@kaczmarj I mean it is rebuilding the image with every CI build. Check out, e.g., https://circleci.com/gh/effigies/nipype/561

Get base image (pull or build) (0)

$ #!/bin/bash -eo pipefail
source /tmp/docker/get_base_image.sh
if [ "$GET_BASE" == "PULL" ]; then
  echo "Pulling base image ..."
  docker pull nipype/nipype:base
elif [ "$GET_BASE" == "BUILD" ]; then
  e=1 && for i in {1..5}; do
    docker build -t nipype/nipype:base - < docker/Dockerfile.base  && e=0 && break || sleep 15
  done && [ "$e" -eq "0" ]
else
  echo "Error: method to get base image not understood"
  exit 1
fi

Sending build context to Docker daemon  10.24kB

Step 1/19 : FROM neurodebian@sha256:7590552afd0e7a481a33314724ae27f76ccedd05ffd7ac06ec38638872427b9b
sha256:7590552afd0e7a481a33314724ae27f76ccedd05ffd7ac06ec38638872427b9b: Pulling from library/neurodebian

Digest: sha256:7590552afd0e7a481a33314724ae27f76ccedd05ffd7ac06ec38638872427b9b
Status: Downloaded newer image for neurodebian@sha256:7590552afd0e7a481a33314724ae27f76ccedd05ffd7ac06ec38638872427b9b
 ---> 5d754f2f51cc
Step 2/19 : ARG DEBIAN_FRONTEND=noninteractive
 ---> Running in 5d7fbe37e4b8
 ---> 21d98a179f36
Step 3/19 : ENV LANG="en_US.UTF-8"     LC_ALL="C.UTF-8"     ND_ENTRYPOINT="/neurodocker/startup.sh"
 ---> Running in e34b0b9c9690
 ---> 9e0385a5429c

So something is happening where it's failing to recognize the Dockerfile as unchanged.

[kaczmarj]

@effigies - yes you're right. It looks like the cache is not found in your circleci account. The cache exists when running on nipy/nipype. I'm not sure how to get around this issue at the moment, but it could involve some clever naming of the cache. Currently, CircleCI is looking for a cache key that is created during the deploy step, which is only run after pushes to master. That cache doesn't exist in most people's forks probably because they don't merge to master.

I specified the cache to be from the master branch because images are only deployed on the master branch.

[effigies]

I see. I wonder if we can push the actual Dockerfile so it shows up in https://hub.docker.com/r/nipype/nipype/~/dockerfile/, and then we could directly compare the current Dockerfile to the deployed one.

[effigies]

(That may be an incredibly stupid thing to suggest. I have next to no knowledge of DockerHub's workings.)

[kaczmarj]

I think that's a good idea. DockerHub doesn't seem to have public documentation of their API, but there are bits and pieces of it in their regular documentation. I can try out some API calls to see if it's possible to upload a Dockerfile.

[oesteban]

Ref. to working cache of layers (without paying): I got circle to use the cache in mriqc with this directive, and then calling docker with the option --cache-from=.

[effigies]

I wonder if we can use contexts to share the cache key from deployment. It would only work for project members, but pretty much the only people using their own Circle allocation to build nipype are members.

[effigies]

(If Docker caching is getting too off topic for this thread, I can create a new issue.)

[effigies]

Potentially useful for automating a deployment step: The hub project can submit pull requests from the command-line, which can be used to initiate the nipype-feedstock PR.

[effigies]

I want to summarize and add some questions to help sharpen the aims of this thread. It may make sense to break this up into separate issues, once we decide whether anything should be done on each sub-issue.

1. @oesteban proposes pushing docker images from CircleCI. Currently every commit to master deploys to nipype/nipype:{base,latest,py2,py3}. We have not pushed version-tagged builds since 0.13.1. Do we want to? Do people use these images, or is it purely for testing on Circle?

2. @oesteban proposes pushing to PyPI from CircleCI. @satra says this is currently done from Travis. I will note that a tagged release causes Travis to push on the first completed job, and all others fail, because PyPI does not permit multiple pushes. In addition to showing misleading failures, real failures will not prevent the package from being uploaded, as long as one job passes. Perhaps the deployment approach in CircleCI is better suited to handle this?

3. @effigies asked whether we can have branch builds on CircleCI reuse cache from master, which would prevent rebuilds on branches besides master. @oesteban suggests using the --cache-from argument. Did this get resolved in your work on fmriprep's CircleCI workflow? Would you be able to patch TST/CI: Split Circle workflow #2386, or submit a PR to it that would enable this? (I had a quick go at playing with this, but my understanding of Circle caches is limited.) (Update: I do not believe this is possible. The easiest approach is to disable personal builds and use the nipy allotment.)

4. @effigies suggested adding a GitHub hub command to the deployment stage of the workflow to automate the nipype-feedstock update. This would make sense to condition on rel/x.y.z branches, so we can begin the test process, as well as tags, to initiate final deployment.

5. In a slack chat, @oesteban described a script for auto-generating the changelog from merge commit messages. This seems useful to me. Any chance of a PR? (Or, recognizing you're busy, just a pointer and any comments that might ease transition to nipype?) (Merged in ENH: Automate updates of CHANGES #2440)