Error when running MetaCompass

[pbravakos]

Hi,
I am running the following command:

python3 go_metacompass2.py -1 /home1/pbravakos/Software/MetaCompass/tutorial/thao2000.1.fq -2 /home1/pbravakos/Software/MetaCompass/tutorial/thao2000.2.fq --memory 223 --threads 19 --verbose --mincov 500 -o example2_output

And I get the following error:

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 19
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       all
        1       assemble_unmapped
        1       assembled_references
        1       bam_sort
        1       bowtie2_map
        1       build_contigs
        1       create_tsv
        1       fastq2fasta
        1       join_contigs
        1       kmer_mask
        1       mapping_stats
        1       pilon_contigs
        1       pilon_map
        1       reference_selection
        1       sam_to_bam
        1       stats_all
        1       stats_genome
        17
Resources before job selection: {'_cores': 19, '_nodes': 9223372036854775807}
Ready jobs (1):
        kmer_mask
Selected jobs (1):
        kmer_mask
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Mon Aug  3 13:37:33 2020]
Job 16: ---kmer-mask fastq

ESC[33mJob counts:
        count   jobs
        1       kmer_mask
        1ESC[0m
ESC[33mtouch example2_output/reference_selection/marker.match.1.fastq;kmer-mask -ms 28 -mdb /home1/pbravakos/Software/MetaCompass/refseq/kmer-mask_db/markers.mdb -1 /home1/pbravakos/Software/MetaCompass/tutorial/thao2000.1.fq -clean 0.0 -match 0.01 -nomasking -t 19 -l 103 -o example2_output/reference_selection/markertmp$RANDOM 1>> example2_output/logs/kmermask.log 2>&1ESC[0m
ESC[32m[Mon Aug  3 13:38:08 2020]ESC[0m
ESC[31mError in rule kmer_mask:ESC[0m
ESC[31m    jobid: 0ESC[0m
ESC[31m    output: example2_output/reference_selection/marker.match.1.fastqESC[0m
ESC[31m    log: example2_output/logs/kmermask.log (check log file(s) for error message)ESC[0m
ESC[31mESC[0m
ESC[31mRuleException:
CalledProcessError in line 49 of /home1/pbravakos/Software/MetaCompass/snakemake/metacompass.paired.py:
Command 'set -euo pipefail;  touch example2_output/reference_selection/marker.match.1.fastq;kmer-mask -ms 28 -mdb /home1/pbravakos/Software/MetaCompass/refseq/kmer-mask_db/markers.mdb -1 /home1/pbravakos/Software/MetaCompass/tutorial/thao2000.1.fq -clean 0.0 -match 0.01 -nomasking -t 19 -l 103 -o example2_output/reference_selection/markertmp$RANDOM 1>> example2_output/logs/kmermask.log 2>&1' returned non-zero exit status 1.
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2168, in run_wrapper
  File "/home1/pbravakos/Software/MetaCompass/snakemake/metacompass.paired.py", line 49, in __rule_kmer_mask
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 529, in _callback
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/concurrent/futures/thread.py", line 57, in run
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 515, in cached_or_run
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2199, in run_wrapperESC[0m
ESC[31mExiting because a job execution failed. Look above for error messageESC[0m
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home1/pbravakos/OSD_Metagenome/MetaCompass/.snakemake/log/2020-08-03T133733.155741.snakemake.log
unlocking
removing lock
removing lock
removed all locks

Do you have any idea why this is happening?
Thanks

[vcepeda]

Hi, can you share the example2_output/logs/kmermask.log file?

[pbravakos]

Yes, it is the following:

ERROR: -l too small for reads:
       a = '@NC_018417.1_156282_156681_0:0:0_0:0:0_0/1'
       b = ''

[vcepeda]

Please add the -l 150 parameter. The read length is only required by kmer-mask when performing reference selection.
The example where I used these reads skips the reference selection an only does reference-guided assembly, so I didn't include the length. Also the -mincov 500 (minimum depth of coverage 500) is probably not properly used. Do you wanna make contigs of min length 500?

[pbravakos]

Thanks.
I did add the -l 150 parameter but now it seems I get a new errror:

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 20
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	all
	1	assemble_unmapped
	1	assembled_references
	1	bam_sort
	1	bowtie2_map
	1	build_contigs
	1	create_tsv
	1	fastq2fasta
	1	join_contigs
	1	kmer_mask
	1	mapping_stats
	1	pilon_contigs
	1	pilon_map
	1	reference_selection
	1	sam_to_bam
	1	stats_all
	1	stats_genome
	17
Resources before job selection: {'_cores': 20, '_nodes': 9223372036854775807}
Ready jobs (1):
	kmer_mask
Selected jobs (1):
	kmer_mask
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Thu Aug 13 08:56:39 2020]
Job 16: ---kmer-mask fastq

�[33mJob counts:
	count	jobs
	1	kmer_mask
	1�[0m
�[33mtouch example2_output/reference_selection/marker.match.1.fastq;kmer-mask -ms 28 -mdb /home1/pbravakos/Software/MetaCompass/refseq/kmer-mask_db/markers.mdb -1 /home1/pbravakos/Software/MetaCompass/tutorial/thao2000.1.fq -clean 0.0 -match 0.01 -nomasking -t 20 -l 153 -o example2_output/reference_selection/markertmp$RANDOM 1>> example2_output/logs/kmermask.log 2>&1�[0m
�[33mtouch example2_output/reference_selection/marker.match.1.fastq;kmer-mask -ms 28 -mdb /home1/pbravakos/Software/MetaCompass/refseq/kmer-mask_db/markers.mdb -1 /home1/pbravakos/Software/MetaCompass/tutorial/thao2000.2.fq -clean 0.0 -match 0.01 -nomasking -t 20 -l 153 -o example2_output/reference_selection/markertmp$RANDOM 1>> example2_output/logs/kmermask.log 2>&1�[0m
�[33mcat example2_output/reference_selection/markertmp*.match.1.fastq >example2_output/reference_selection/marker.match.1.fastq && touch example2_output/reference_selection/.run1.ok�[0m
[Thu Aug 13 08:57:46 2020]
Finished job 16.
1 of 17 steps (6%) done
Resources before job selection: {'_cores': 20, '_nodes': 9223372036854775807}
Ready jobs (1):
	fastq2fasta
Selected jobs (1):
	fastq2fasta
Resources after job selection: {'_cores': 19, '_nodes': 9223372036854775806}

[Thu Aug 13 08:57:46 2020]
Job 14: ---Converting fastq to fasta.

if [[ -f example2_output/reference_selection/.run2.ok ]]; then echo 'Previous fastq2fasta rule finished successfully. Skipping fastq2fasta.'; else /home1/pbravakos/Software/MetaCompass/bin/fq2fa -i example2_output/reference_selection/marker.match.1.fastq -o example2_output/reference_selection/masked_reads.fasta && touch example2_output/reference_selection/.run2.ok ;fi
[Thu Aug 13 08:57:46 2020]
Finished job 14.
2 of 17 steps (12%) done
Resources before job selection: {'_cores': 20, '_nodes': 9223372036854775807}
Ready jobs (1):
	reference_selection
Selected jobs (1):
	reference_selection
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Thu Aug 13 08:57:46 2020]
Job 10: ---reference selection.

if [[ -f example2_output/reference_selection/.run3.ok ]]; then echo 'Previous reference_selection rule finished successfully. Skipping reference_selection.'; else python3 /home1/pbravakos/Software/MetaCompass/bin/select_references.py tax example2_output/reference_selection/masked_reads.fasta /home1/pbravakos/Software/MetaCompass/tutorial/thao2000.1.fq,/home1/pbravakos/Software/MetaCompass/tutorial/thao2000.2.fq example2_output/reference_selection 20 1 0.9 1>> example2_output/logs/reference_selection.log 2>&1 && touch example2_output/reference_selection/.run3.ok ;fi
Waiting at most 5 seconds for missing files.
Full Traceback (most recent call last):
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/dag.py", line 470, in check_and_touch_output
    wait_for_files(
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/io.py", line 670, in wait_for_files
    raise IOError(
OSError: Missing files after 5 seconds:
example2_output/reference_selection/mc.refseq.ids

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 455, in _proceed
    self.get_executor(job).handle_job_success(job)
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 544, in handle_job_success
    super().handle_job_success(job)
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 225, in handle_job_success
    job.postprocess(
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/jobs.py", line 991, in postprocess
    self.dag.check_and_touch_output(
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/dag.py", line 477, in check_and_touch_output
    raise MissingOutputException(
snakemake.exceptions.MissingOutputException: Job completed successfully, but some output files are missing. Missing files after 5 seconds:
example2_output/reference_selection/mc.refseq.ids
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.

MissingOutputException in line 63 of /home1/pbravakos/Software/MetaCompass/snakemake/metacompass.paired.py:
Job completed successfully, but some output files are missing. Missing files after 5 seconds:
example2_output/reference_selection/mc.refseq.ids
This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait.
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 544, in handle_job_success
  File "/home1/pbravakos/Software/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 225, in handle_job_success
Removing output files of failed job reference_selection since they might be corrupted:
example2_output/reference_selection/mc.refseq.fna
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home1/pbravakos/OSD_Metagenome/MetaCompass/.snakemake/log/2020-08-13T085638.121155.snakemake.log
unlocking
removing lock
removing lock
removed all locks