Build a container from a project

[AlbertDeFusco]

It would be useful to have a way to create a docker image for a defined command in a conda-project.yml file.

For reference, the anaconda-project dock subcommand utilized a source-to-image builder image defined in https://github.com/Anaconda-Platform/s2i-anaconda-project.

docs: https://anaconda-project.readthedocs.io/en/latest/user-guide/tasks/docker-image.html

[mattkram]

If we are satisfied with relying on the official Miniconda image, I think we could potentially do something simple with templating very easily.

Here is a stripped example using conda for an API without conda-project that we might consider modifying:

FROM continuumio/miniconda3:latest

# Copy the conda environment config and install dependencies
COPY environment.yml /environment.yml
RUN conda env create -p /conda -f environment.yml

# Link conda to global path and
ENV PATH=/conda/bin:$PATH

WORKDIR /app

# Copy the application code
COPY . .

# Expose the port and run the service
EXPOSE 8000
ENTRYPOINT ["python", "-m", "my_service"]
CMD ["serve"]

[AlbertDeFusco]

here's one I have that works

FROM continuumio/miniconda3 as miniconda
### Install and configure miniconda
RUN conda install conda-forge::conda-project --yes && conda clean --all --yes

FROM miniconda as conda-project

COPY --from=miniconda /opt/conda /opt/conda

### Set timezone
ENV TZ=US/Central
RUN cp /usr/share/zoneinfo/${TZ} /etc/localtime \
    && echo ${TZ} > /etc/timezone

ENV PYTHONDONTWRITEBYTECODE=1
ENV PIP_NO_CACHE_DIR=1

ENV PATH=/opt/conda/bin:$PATH
ENV HOME=/project

COPY . /project
RUN chown -R 1001:1001 /project

USER 1001
WORKDIR /project
RUN ["conda", "project", "install"]

ENTRYPOINT ["conda", "project", "run"]
CMD []

This means from your project directory you can

> docker build -t <img-name> .
> docker run <img-name>                                       # run the first command in conda-project.yml
> docker run <img-name> [command [args ...]]   # run a different defined command or ad-hoc command with arguments

I have caught a bug where a locked project on my desktop decides that it's no longer locked when being installed inside docker build. Maybe there is something with the hashing method that is architecture dependent.

[mattkram]

Oh awesome, that's way better. Sounds like we could add a basic MVP subcommand that just embeds that file and renders it to run the docker build command. Then, if we find any need for more complexity we can explore a templating engine like jinja2 if appropriate.

[mattkram]

ENV TZ=US/Central

What is the purpose of the time zone?

[AlbertDeFusco]

I started putting that in my dockerfiles after working on some timeseries data and realizing that timezone support wasn't something that "just worked" for me. It's strictly not required here.

[AlbertDeFusco]

There is one aspect that needs to be considered here. The blind COPY . /project will bring EVERYTHING even the currently installed env and potentially secret things like the .env file. In anaconda-project dock this is mitigated by first creating an archive of the project, respecting ignores, and then bringing that to the docker build.

[mattkram]

I recommend we respect the same ignore rules as the archive command. We may even get this for free with .dockerignore (if we were to build one when the command runs or if we can override a docker build argument). Something to investigate.

[mattkram]

@AlbertDeFusco removed bug label, felt like a mistake. Feel free to put back if you intended it.

[rmyers]

@AlbertDeFusco you may want to look into using ONBUILD which is a instruction for the container to run when used by an end user. This is basically the same as a s2i build from openshift only supported by the docker file. First you'd add some script to the base image like /usr/local/bin/run_conda_project then the instructions would be:

... 

COPY run_conda_project /usr/local/bin/run_conda_project

ONBUILD ADD . /app/src
ONBUILD RUN /usr/local/bin/run_conda_project

https://docs.docker.com/engine/reference/builder/#onbuild

[AlbertDeFusco]

So you think we might do something like this to publish a builder and runtime image to dockerhub so the project author has a dockerfile with as little as two lines in it?

https://www.busbud.com/blog/going-docker-multi-stage-builds/

[mattkram]

That's really interesting! I was thinking about whether a conda-project-specific image would be worthwhile but I started imagining the combinatorial aspects of "which miniconda + which conda-project version". But maybe that isn't a real concern if it's always "latest bundled with latest"?

I need to grok that ONBUILD thing; I've never seen it before. And how is that different than ENTRYPOINT + CMD? Time to do some reading :)

[AlbertDeFusco]

I was working on this some more and discovered two issues to keep track of:

1. A .dockerignore file should be included with the entry envs/, otherwise weird things happen
2. I keep getting hit with Conda Lock has dropped "optional" #119 during docker build, it doesn't happen on my desktop.

[mattkram]

On 1. We have some question about whether we persist the generated Dockerfile/.dockerignore files. We could create them or make temporary ones. If we create them, we could add them to .gitignore or not.

On 2. Are the channels the same? I'm guessing the channels or OS at least are different and there may be a different version of a dependency based on the availability of a package for arm64? Just a guess.

[mattkram]

I've just come across this project, which may be worth considering integrating: https://github.com/conda-incubator/conda-docker