3.2. Frameworks#

Instead of running your code or shell scripts directly on the HPC, it can be easier to use a higher-level framework to handle the job submissions for you. We can organize them by language support:

3.2.1. Multi-language#

You can use any language in any framework by calling a script, but some frameworks make it much more convenient and allow you to set the execution environment.

3.2.1.1. Nextflow#

Any script

3.2.1.2. Apache Spark#

Python/Java/R

3.2.1.3. Apache Airflow#

Any script

3.2.2. Python#

  • Dask

    • very easy to get started and great ecosystem (numpy, pandas, xarray, SpatialData…)

  • Ray

    • a very low level and high performance framework

3.2.2.1. Dask#

3.2.3. R#

3.2.3.1. qsub#