From f572333900f8215801935dd709bff61c14e3ad01 Mon Sep 17 00:00:00 2001 From: Mohamed Abuelanin Date: Sun, 23 Jan 2022 13:45:23 +0200 Subject: [PATCH 1/7] Create snakemake_profiles.md --- snakemake_profiles.md | 63 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 snakemake_profiles.md diff --git a/snakemake_profiles.md b/snakemake_profiles.md new file mode 100644 index 0000000..96de364 --- /dev/null +++ b/snakemake_profiles.md @@ -0,0 +1,63 @@ +# Snakemake profiles + +While https://github.com/dib-lab/farm-notes/blob/latest/snakemake-slurm.md works greatly, Snakemake has deprectated the `--cluster-config` with `--profile`. + +Here's a simple example of setting this up globally on a user account. + +```bash +mkdir -p ~/.config/snakemake/slurm +touch ~/.config/snakemake/slurm/{config.yaml,slurm-status.py} +chmod +x ~/.config/snakemake/slurm/slurm-status.py +``` + +**~/.config/snakemake/slurm/config.yaml** + +```yaml +cluster: + mkdir -p logs/{rule}/ && + sbatch + --cpus-per-task={threads} + --mem={resources.mem_mb} + --time={resources.time} + --job-name=smk-{rule} + --output=logs/{rule}/{jobid}.out + --error=logs/{rule}/{jobid}.err + --partition={resources.partition} +default-resources: + - mem_mb=2000 + - time=480 + - partition=med2 +jobs: 100 +latency-wait: 60 +local-cores: 8 +restart-times: 1 +max-jobs-per-second: 20 +keep-going: True +rerun-incomplete: True +printshellcmds: True +scheduler: greedy +use-conda: True +conda-frontend: mamba +cluster-status: ~/.config/snakemake/slurm/slurm-status.py +max-status-checks-per-second: 10 +``` + +**~/.config/snakemake/slurm/slurm-status.py** +```python +#!/usr/bin/env python3 +import subprocess +import sys +jobid = sys.argv[-1] +output = str(subprocess.check_output("sacct -j %s --format State --noheader | head -1 | awk '{print $1}'" % jobid, shell=True).strip()) +running_status=["PENDING", "CONFIGURING", "COMPLETING", "RUNNING", "SUSPENDED", "PREEMPTED"] +if "COMPLETED" in output: + print("success") +elif any(r in output for r in running_status): + print("running") +else: + print("failed") +``` + +
+ +_Thanks to https://fame.flinders.edu.au/blog/2021/08/02/snakemake-profiles-updated_ From a13913796db2d45e7ad0eed4d9ea1b8e7af5f7c0 Mon Sep 17 00:00:00 2001 From: Mohamed Abuelanin Date: Sun, 23 Jan 2022 16:07:01 +0200 Subject: [PATCH 2/7] Update snakemake_profiles.md --- snakemake_profiles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/snakemake_profiles.md b/snakemake_profiles.md index 96de364..607ffa2 100644 --- a/snakemake_profiles.md +++ b/snakemake_profiles.md @@ -1,6 +1,6 @@ # Snakemake profiles -While https://github.com/dib-lab/farm-notes/blob/latest/snakemake-slurm.md works greatly, Snakemake has deprectated the `--cluster-config` with `--profile`. +While https://github.com/dib-lab/farm-notes/blob/latest/snakemake-slurm.md works greatly, Snakemake has deprectated the `--cluster-config` and replaced with `--profile`. Here's a simple example of setting this up globally on a user account. From d163c2d9943f40cafc62ab3119186a692988d60c Mon Sep 17 00:00:00 2001 From: Mohamed Abuelanin Date: Sun, 23 Jan 2022 16:09:48 +0200 Subject: [PATCH 3/7] Update snakemake_profiles.md --- snakemake_profiles.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/snakemake_profiles.md b/snakemake_profiles.md index 607ffa2..47328f8 100644 --- a/snakemake_profiles.md +++ b/snakemake_profiles.md @@ -58,6 +58,15 @@ else: print("failed") ``` +**Run** + +``` +snakemake --profile slurm + +``` + + +
_Thanks to https://fame.flinders.edu.au/blog/2021/08/02/snakemake-profiles-updated_ From b3bc6f036839a7c8dedccf0758cfc318fe6c0b20 Mon Sep 17 00:00:00 2001 From: Mohamed Abuelanin Date: Mon, 24 Jan 2022 10:49:12 +0200 Subject: [PATCH 4/7] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 467d527..8886fef 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ * nodes, tasks, and cpus-per-task [Using Snakemake](snakemake-slurm.md) +[Using Snakemake profiles](snakemake_profiles.md) [Magic commands and informative links](magic-commands.md) From c0bcfeaed7474eb396ef1278000ca91b5fca950a Mon Sep 17 00:00:00 2001 From: Mohamed Abuelanin Date: Mon, 24 Jan 2022 10:49:33 +0200 Subject: [PATCH 5/7] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 8886fef..3895e25 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ * nodes, tasks, and cpus-per-task [Using Snakemake](snakemake-slurm.md) + [Using Snakemake profiles](snakemake_profiles.md) [Magic commands and informative links](magic-commands.md) From 18622fa5e09aecf713c73a91372b9b246eba07b9 Mon Sep 17 00:00:00 2001 From: Mohamed Abuelanin Date: Thu, 3 Mar 2022 08:02:05 +0200 Subject: [PATCH 6/7] Example + custom profile --- snakemake_profiles.md | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/snakemake_profiles.md b/snakemake_profiles.md index 47328f8..9c2617f 100644 --- a/snakemake_profiles.md +++ b/snakemake_profiles.md @@ -26,7 +26,8 @@ cluster: default-resources: - mem_mb=2000 - time=480 - - partition=med2 + - partition=low2 + - threads=1 jobs: 100 latency-wait: 60 local-cores: 8 @@ -61,11 +62,33 @@ else: **Run** ``` -snakemake --profile slurm +snakemake -s --profile slurm ``` +### Create your own profile +The example above is just a usage demo that you can modify. You can create a new parameter in the `sbatch` command as `{resources.new_parameter}` and then use the `new_parameter` in the resources section of a snakemake rule. + +**Example:** + +In `~/.config/snakemake/slurm/config.yaml` append the following parameter to the sbatch command, then use it in the snakemake rule `resources`. + +```yaml +--nodes={resources.nodes} +``` + +```python +rule test_rule: + threads: 64 + resources: + - partition = 'med2' # med2 partition + - mem_mb = 350 * 1024 # 350GB + - time = 2 * 24 * 60 # two days + - nodes = 2 # Two nodes <- new parameter +``` + +Please note that `job name` = `rule name`, and all log files are written in `logs/job_name/job_id`. Multiple runs to the same rule will override the old log file.
From ffe66e488ae0e548dd74a7c1ddb4c534b9ca48ab Mon Sep 17 00:00:00 2001 From: Mohamed Abuelanin Date: Mon, 16 May 2022 23:08:56 -0700 Subject: [PATCH 7/7] Update snakemake_profiles.md Nice! Thanks, Tessa! Co-authored-by: Tessa Pierce Ward --- snakemake_profiles.md | 1 + 1 file changed, 1 insertion(+) diff --git a/snakemake_profiles.md b/snakemake_profiles.md index 9c2617f..e2729e2 100644 --- a/snakemake_profiles.md +++ b/snakemake_profiles.md @@ -23,6 +23,7 @@ cluster: --output=logs/{rule}/{jobid}.out --error=logs/{rule}/{jobid}.err --partition={resources.partition} + --parsable default-resources: - mem_mb=2000 - time=480