3.2. Frameworks#
Instead of running your code or shell scripts directly on the HPC, it can be easier to use a higher-level framework to handle the job submissions for you. We can organize them by language support:
3.2.1. Multi-language#
You can use any language in any framework by calling a script, but some frameworks make it much more convenient and allow you to set the execution environment.
3.2.1.1. Nextflow#
Any script
3.2.1.2. Apache Spark#
Python/Java/R
3.2.1.3. Apache Airflow#
Any script
3.2.2. Python#
Dask
very easy to get started and great ecosystem (numpy, pandas, xarray, SpatialData…)
Ray
a very low level and high performance framework