# How to use the Docker installed libraries in a cwl workflow?

**URL:** <https://cwl.discourse.group/t/how-to-use-the-docker-installed-libraries-in-a-cwl-workflow/944>\
**Category:** CWL Questions\
**Created:** [August 16, 2024, 8:08am UTC](https://cwl.discourse.group/t/how-to-use-the-docker-installed-libraries-in-a-cwl-workflow/944 "2024-08-16T08:08:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rupert\_may](https://avatars.discourse-cdn.com/v4/letter/r/a587f6/32.png) [@rupert\_may](https://cwl.discourse.group/u/rupert_may)\
**Post date:** [August 16, 2024, 8:08am UTC](https://cwl.discourse.group/t/how-to-use-the-docker-installed-libraries-in-a-cwl-workflow/944/1 "2024-08-16T08:08:09Z")

</div>

Hello,

I am creating a cwl workflow, where I am writing the steps to run. The first step is regarding a Docker container, which is used to install some Python libraries. Then, I want the other cwl files to point to the libraries installed within the Docker container, but I am having issues on how to do that.  
This is how the `install_docker_test.cwl` looks like:

```auto
cwlVersion: v1.2
class: CommandLineTool
id: install_requirements
stderr: output_test
requirements:
  InlineJavascriptRequirement: {}
  DockerRequirement:
    dockerPull: test-docker-image
inputs:
  scripts_folder:
    type: Directory
outputs:
  output:
    type: File
    outputBinding:
      glob: output_test

```

And this is the workflow description:

```auto
cwlVersion: v1.2
class: Workflow
id: workflow
requirements:
  EnvVarRequirement:
    envDef:
      TEST_STR: "this_is_my_test"
inputs:
  input_folder:
    type: Directory
  my_script:
    type: string
outputs:
  output:
    type: File
    outputSource: download_files/download_files_output
steps:
  install_requirements:
    run: install_docker_test.cwl
    in:
      input_folder: input_folder
    out: [output]
  download_files:
    run: download_files.cwl
    in:
      my_script: my_script
      input_folder: input_folder
    out: [download_files_output]

```

However, the issue is that the script run inside `download_files.cwl` does not point to the Docker container, even though I have `install_libraries` steps listed before. How to fix this and to make sure that the python script uses the Docker container instead of the generic python installed in the wsl?

---

<div class="post-metadata">

**Author:** ![tetron](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/tetron/32/40_2.png) [@tetron](https://cwl.discourse.group/u/tetron)\
**Post date:** [August 16, 2024, 4:22pm UTC](https://cwl.discourse.group/t/how-to-use-the-docker-installed-libraries-in-a-cwl-workflow/944/2 "2024-08-16T16:22:28Z")

</div>

The `install_requirements` and `download_files` steps are isolated from one another, so installing into the container like that won’t work – the `download_files` step starts in a fresh container environment so any changes made by `install_requirements` are wiped away.

Perhaps you could explain a little more what `install_requirements` needs to do? I suspect what you want to do is put that in a Dockerfile to build your image.

---

<div class="post-metadata">

**Author:** ![rupert\_may](https://avatars.discourse-cdn.com/v4/letter/r/a587f6/32.png) [@rupert\_may](https://cwl.discourse.group/u/rupert_may)\
**Post date:** [August 19, 2024, 10:40am UTC](https://cwl.discourse.group/t/how-to-use-the-docker-installed-libraries-in-a-cwl-workflow/944/3 "2024-08-19T10:40:27Z")

</div>

The `install_requirements` step triggers `install_docker_test.cwl` to run, which has the requirement for the Docker image to exist. I have a Docker file, where I am installing the Python libraries that I need in the other steps (here: download\_files). Here is the example of the Docker file:

```auto
FROM python:3.8
WORKDIR /base
COPY example/ /base/example/
RUN pip install /base/example/custom_lib
COPY requirements.txt /base/example/requirements.txt
RUN --mount=type=cache,target=/root/.cache/pip \
    python -m pip install -r /base/example/requirements.txt
RUN pip install torch
ENV PYTHONUNBUFFERED 1
COPY . .

```

---

<div class="post-metadata">

**Author:** ![rupert\_may](https://avatars.discourse-cdn.com/v4/letter/r/a587f6/32.png) [@rupert\_may](https://cwl.discourse.group/u/rupert_may)\
**Post date:** [August 19, 2024, 12:04pm UTC](https://cwl.discourse.group/t/how-to-use-the-docker-installed-libraries-in-a-cwl-workflow/944/4 "2024-08-19T12:04:20Z")

</div>

The reason why I’ve separated the installation of the libraries with Docker is because the installation step in the workflow (by using a separate cwl file which would have the command as argument to install the Python libraries) would make use of too much memory: `Max memory used: 213MiB`

---

<div class="post-metadata">

**Author:** ![tetron](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/tetron/32/40_2.png) [@tetron](https://cwl.discourse.group/u/tetron)\
**Post date:** [August 19, 2024, 1:40pm UTC](https://cwl.discourse.group/t/how-to-use-the-docker-installed-libraries-in-a-cwl-workflow/944/5 "2024-08-19T13:40:15Z")

</div>

Does `download_files` also have this?

```auto
requirements:
  DockerRequirement:
    dockerPull: test-docker-image

```

But I suspect you have the wrong mental model of the workflow. You can’t install files in step 1 and then find them there in step 2, unless you specified the files as output in step 1 and connected it as in input to step 2.

The reason is that step 1 and step 2 are running in different containers, and could be running on entirely different machines.

You want to build your docker image with all of its software dependencies at the command line (or a separate shell script) and then launch the workflow. CWL has limited support for specifying how to build Docker images.

Does this help?
