Module 4.1 Pipeline development (outline)
Theme 4: Pipeline development and automation
Pipeline development
|
Time |
Activity description |
ILOs
|
|
9.00-9.45 |
Lecture: “Basic
principles of automation”
|
· Understand the basic principles of automation · Be familiar with commonly used workflow management tools (i.e., bash, snakemake, nextflow, cron) · Learn basics about the workload managers and scaling up the computation (i.e., slurm, torque) |
|
10.00-11.45 |
Practical: “Pipeline development – scaling number of tools and samples” |
· Understand the value of using pipelines · Learn how to make a pipeline that runs multiple tools on multiple samples |
|
12.45-15.30 |
Practical: “Pipeline development – from raw reads to data sharing”
|
· Be able to combine multiple tools from previous days into one workflow. · Understand the advantages of automating workflows. · Learn about common issues that must be addressed when automating workflows. |
|
15.30-16.30 |
Group talk: “Setup at SSI and how automation could be introduced in participants’ workflows” |
· Know about setup at SSI · Be able to pinpoint processes that can be automated in their own workflows. |
Details
This module in pipeline development for non-technical people will cover basic principles of automation, workflows, and workload management. Participants will learn about typical tools used for automating bioinformatic workflow and will be introduced to the overall aspects of maintaining an automated system. Participants will get a basic grasp of the advantages of developing and automating pipelines.
In addition to this, participants will learn how to combine multiple different tools used in the course to make a coherent pipeline. They will have to deal with common issues when using automated workflows. Finally, lecturers will help participants understand how automation can be introduced in their processes and provide feedback on how to implement it at their own setup.