Cwl with python and python pip requirements

CWL Questions
1

Hey,
I wrote a nice self-containing python in the cwl. How can I include python’s pip requirements: “pip install” before running the script?
I want to keep everything in one script, see below.
Thank you for your help.
Steffen

cwlVersion: v1.0
class: CommandLineTool
baseCommand: ["python", "convert.py"]
inputs:
  in_file:
    type: File
  out_file:
    type: string
requirements:
  InitialWorkDirRequirement:
    listing:
      - entryname: convert.py
        entry: |-
          import numpy as np
          data = np.loadtxt('$(inputs.in_file.path)')
          with open('$(inputs.out_file)','w') as fOut:
            fOut.write("Sum is: "+str(np.sum(data)))
outputs:
  outfile:
    type: File
    outputBinding:
      glob: $(inputs.out_file)
2

Welcome @SteffenBrinckmann

If your CWL runner support it, then you could add a SoftwareRequirement of numpy

For example, with the CWL reference runner: GitHub - common-workflow-language/cwltool: Common Workflow Language reference implementation you can add the --beta-conda-dependencies flag. If the libraries you need are in PyPI only (or if you prefer PyPI over the Conda registry) then that feature could be extended for a --beta-pip-dependencies flag in the future.

The other option is to add a DockerRequirement either pointing to an image with the dependencies installed, or a dockerFile with an inline recipe using the packages you need.

3

Sorry I did not post my solution:

- baseCommand: ["sh","script.sh"]
- requirements:
  InitialWorkDirRequirement:
    listing:
      - entryname: script.sh
        entry: |-
          pip3 install -r requirements.txt
          python convert.py

and similarly I add requirements.txt and the python-script as initialWorkDirRequirement.

@mrc Thank you for your suggestion:

  • I am on the PyPi train
  • I want to have the cwl applicable for all runners
    I don’t know of an image that would fit and I am not too much into docker-recipes that I know how to solve it that way. I am a scipt-kid who hopes the sh-commands does work.
4

Glad you found a solution! I’ve opened up add --beta-pip-dependencies · Issue #1492 · common-workflow-language/cwltool · GitHub for the pip version as requested

5

For readers looking for a solution with conda (mamba/micromamba):

requirements:
    DockerRequirement: 
      dockerPull: docker.io/mambaorg/micromamba
    InlineJavascriptRequirement: {}
    InitialWorkDirRequirement:
      listing:
        - entryname: environment.yaml
          entry: |- 
            name: env_rio_stac
            channels:
              - conda-forge
            dependencies:
              - python=3.8
              - pip
              - pip:
                - rio_stac
        - entryname: run.sh 
          entry: |- 
            #!/bin/sh
            micromamba create -q -f environment.yaml
            /opt/conda/envs/env_rio_stac/bin/python stac.py
            rm -fr run.sh environment.yaml stac.py
        - entryname: stac.py
          entry: |-
            import rio_stac

Then use:

baseCommand: ["/bin/sh", "run.sh"]