Fixing problems with fastq-dump

[ialbert]

Some reader report various errors with fastq-dump that look like:

Failed to create TLS stream for 'www.ncbi.nlm.nih.gov' (130.14.29.110)
item not found while constructing within virtual database module - the path 'SRR21675623' cannot be opened as database or table 

or

( !! The certificate is not correctly signed by the trusted CA !! The certificate is signed with an unacceptable hash. )

In those cases, we found that installing a newer version of sra-tools fixes the problem.

We have an installer that can download and install the native client distributed by NCBI. Run it like so:

curl http://data.biostarhandbook.com/sratools.sh | bash

After running the above, close and reopen the terminal, then test the results with:

fastq-dump --stdout -X 2 SRR390728

I believe the above is a solution to the problems people have been experiencing.

[ChronoWeaver]

I tried again according to your method, and running "fastq-dump --stdout -X 2 SRR390728" succeeded, but when I run "fastq-dump -X 1 -Z SRR1553591" directly, it will report an error, I don't know the reason for the error.(M1 macbook pro)

$ fastq-dump --stdout -X 2 SRR390728

Read 2 spots for SRR390728
Written 2 spots for SRR390728
@SRR390728.1 1 length=72
CATTCTTCACGTAGTTCTCGAGCCTTGGTTTTCAGCGATGGAGAATGACTTTGACAAGCTGAGAGAAGNTNC
+SRR390728.1 1 length=72
;;;;;;;;;;;;;;;;;;;;;;;;;;;9;;665142;;;;;;;;;;;;;;;;;;;;;;;;;;;;;96&&&&(
@SRR390728.2 2 length=72
AAGTAGGTCTCGTCTGTGTTTTCTACGAGCTTGTGTTCCAGCTGACCCACTCCCTGGGTGGGGGGACTGGGT
+SRR390728.2 2 length=72
;;;;;;;;;;;;;;;;;4;;;;3;393.1+4&&5&&;;;;;;;;;;;;;;;;;;;;;<9;<;;;;;464262

$ fastq-dump -X 1 -Z SRR1553591
2022-10-23T02:33:37 fastq-dump.3.0.0 sys: encryption failed while validating token within cryptographic module - Verification issue 0x4008 for this certificate: ( cert. version : 3 serial number : 00 issuer name : C=US, O=The Go Daddy Group, Inc., OU=Go Daddy Class 2 Certification Authority subject name : C=US, O=The Go Daddy Group, Inc., OU=Go Daddy Class 2 Certification Authority issued on : 2004-06-29 17:06:20 expires on : 2034-06-29 17:06:20 signed using : RSA with SHA1 RSA key size : 2048 bits basic constraints : CA=true )
...
2022-10-23T02:33:37 fastq-dump.3.0.0 sys: mbedtls_ssl_get_verify_result for 'locate.ncbi.nlm.nih.gov' returned 0x4008 ( !! The certificate is not correctly signed by the trusted CA !! The certificate is signed with an unacceptable hash. )
Failed to call external services.
(bioinfo)

[ialbert]

oh, just realized that this was the same thread,

this sounds like some sort of network problem frankly - lots of certificate problems

[ialbert]

A workaround would be to

bio search SRR1553591

that prints

[
    {
        "run_accession": "SRR1553591",
        "sample_accession": "SAMN02952040",
        "first_public": "2015-06-05",
        "country": "Sierra Leone",
        "sample_alias": "G3846",
        "fastq_bytes": "50369239;52834496",
        "read_count": "789171",
        "library_name": "G3846_r1.ADXX",
        "library_strategy": "RNA-Seq",
        "library_source": "TRANSCRIPTOMIC",
        "library_layout": "PAIRED",
        "instrument_platform": "ILLUMINA",
        "instrument_model": "Illumina HiSeq 2500",
        "study_title": "Zaire ebolavirus Genome sequencing",
        "fastq_ftp": "ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/001/SRR1553591/SRR1553591_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/001/SRR1553591/SRR1553591_2.fastq.gz"
    

you can download from those links (automate the whole thing with jq)

[ChronoWeaver]

thank you for your reply。
I used your method ，using "$ wget ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/001/SRR1553591/SRR1553591_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq /SRR155/001/SRR1553591/SRR1553591_2.fastq.gz" downloaded the corresponding file, what do I need to do next

[ChronoWeaver]

Can you give me a more detailed explanation, I just started using your book to learn, I don't know how to use the code very well, I am very grateful.

[ChronoWeaver]

#200 (reply in thread)
Thanks again, I succeeded using the manual setup method below in this question.

[cborowski]

I need help with fastq-dump errors after Biostar Handbook software installation. I am not very experienced and frankly I don’t understand the errors. Installation went well so far, but doctor.py returns:
fastq-dump -X 1 -Z SRR1553591 ... ERROR

After running
curl http://data.biostarhandbook.com/sratools.sh | bash

• terminal reopening and
  fastq-dump --stdout -X 2 SRR390728
  I get:

2022-12-22T10:42:21 fastq-dump.2.9.6 sys: connection failed while opening file within cryptographic module - mbedtls_ssl_handshake returned -9984 ( X509 - Certificate verification failed, e.g. CRL, CA or signature check failed )
2022-12-22T10:42:21 fastq-dump.2.9.6 sys: mbedtls_ssl_get_verify_result returned 0x4008 ( !! The certificate is not correctly signed by the trusted CA !! The certificate is signed with an unacceptable hash. )
2022-12-22T10:42:21 fastq-dump.2.9.6 sys: connection failed while opening file within cryptographic module - ktls_handshake failed while accessing '130.14.29.110' from '192.168.2.120'
2022-12-22T10:42:21 fastq-dump.2.9.6 sys: connection failed while opening file within cryptographic module - Failed to create TLS stream for 'www.ncbi.nlm.nih.gov' (130.14.29.110) from '192.168.2.120'
2022-12-22T10:42:21 fastq-dump.2.9.6 err: item not found while constructing within virtual database module - the path 'SRR390728' cannot be opened as database or table
(bioinfo)

Running
bio search SRR1553591
returns
[
{
"run_accession": "SRR1553591",
"sample_accession": "SAMN02952040",
"first_public": "2015-06-05",
"country": "Sierra Leone",
"sample_alias": "G3846",
"fastq_bytes": "50369239;52834496",
"read_count": "789171",
"library_name": "G3846_r1.ADXX",
"library_strategy": "RNA-Seq",
"library_source": "TRANSCRIPTOMIC",
"library_layout": "PAIRED",
"instrument_platform": "ILLUMINA",
"instrument_model": "Illumina HiSeq 2500",
"study_title": "Zaire ebolavirus Genome sequencing",
"fastq_ftp": "ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/001/SRR1553591/SRR1553591_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/001/SRR1553591/SRR1553591_2.fastq.gz"
}
]
(bioinfo)

What does that mean? I understand that the ftp site is the download location for the ebola data, but the fastq-dump tool is still not working. Should i ignore it?

[ialbert]

It is not your fault for being a beginner.

What happens is that the people that run NCBI cannot get their stuff together to the point of absurditity that downloading a file is too complicated ...

Just ponder and think about that for a second ... to truly understand the magnitude of them letting an entire domain of science down

The Biostar Workflows explains the process in more detail on what to do when fastq-dump fails, see the section: When fastq-dump fails.

https://www.biostarhandbook.com/books/workflows/tutorials/fastq-data/#when-fastq-dump-fails

[ialbert]

In a nutshell you can download the links yourself:

wget ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/005/SRR1553425/SRR1553425_1.fastq
wget ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/005/SRR1553425/SRR1553425_2.fastq

[yonglu1]

I had the same problem after run: curl http://data.biostarhandbook.com/sratools.sh | bash
I believe there is something wrong in the Bioinfo environment. I can run fastq-dump without a problem before activate Bioinfo. Please refer to the attached picture for detailed information.

[ialbert]

that means you have two different versions of fastq-dump installed and the system finds the broken one first

uninstall sratools from the bioinfo environment or move the other fastq-dump to be earlier in the PATH variable

mamba uninstall sra-tools

[yonglu1]

Thank you so much for the prompt reply. uninstall sratools from the bioinfo environment fixed my problem. Thanks!

[cborowski]

Man thanks for your quick response, Istvan. I had gone through the When-fastq-dump-fails section several times, but I did not manage to extrapolate the right solution for. At the end I tried and tried and suddenly several steps of the installation and file retrieving process worked that had not done so before. No clue why, but at the moment I am fine. Nevertheless, 1000 thanks for your immediate help - this is really great support! Christian I On 22/12/2022 23:41, Istvan Albert wrote:

It is not your fault for being a beginner. What happens is that the people that run NCBI cannot get their stuff together to the point of absurditity that downloading a file is too complicated ... Just ponder and think about that for a second ... to truly understand the magnitude of letting a entire domain of science down The Biostar Workflows explains the process in more detail on what to do when fastq-dump fails, see the section: When fastq-dump fails. https://www.biostarhandbook.com/books/workflows/tutorials/fastq-data/#when-fastq-dump-fails — Reply to this email directly, view it on GitHub <#239 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHNHEAND2V7UHASSKD4TFRDWOTKKDANCNFSM6AAAAAAQX6BVGY>. You are receiving this because you commented.Message ID: ***@***.***>

-- Dr. Christian Borowski Department of Symbiosis Max Planck Institute for Marine Microbiology Celsiusstr. 1 - D-28359 Bremen - Room 3242 Phone: +49 421 2028 823 eMail: ***@***.***

[nicolemadrid]

I followed the directions in the 'How to fix fastq-dump errors' section but keep receiving this message, I am not familiar with coding and am following this textbook for a college course. I would be grateful for any help.

[ialbert]

uninstall the default sra-tools package

mamba uninstall sra-tools

then close and reopen the terminal

[nicolemadrid]

I uninstalled the default sra-tools package, closed and reopened the terminal and this is the result following the how to fix fastq-dump errors.

[ialbert]

there are a number of similar issues that people experience:

ncbi/sra-tools#598

and

ncbi/sra-tools#467

and a number of others. Perhaps it is a network problem, it sounds like it.

If you keep having problems look at the section "When fastq-dump fails" here:

https://www.biostarhandbook.com/books/workflows/tutorials/fastq-data/#when-fastq-dump-fails

to locate the URLs then download those URLs with wget or curl

[davincios]

I love the doctor.py script -- super helpful.

But I can confirm that I doctory.py has fastq-dump errors for me too.

"curl http://data.biostarhandbook.com/sratools.sh | bash"

does not solve the issue.

I am using a macbook air M1.

[davincios]

Yes it is working now :) thank you

But it does require installing fastq-dump separately

[davincios]

I do want to say that I love the Biostar handbooks so far. It probably has an incredible (indirect) influence on the world by equipping people with bioinformatics know how and enabling them to achieve good things for humanity.

[ialbert]

thank you for the kind words. The handbook has been conceived with many goals and tries to educate on multiple levels, and I am happy to see that come through.

[davincios]

@ialbert

I can confirm that the installation worked cleanly today in one try on a new Ubuntu instance on AWS.

It didn't work cleanly on AWS Linux / CentOS Linux or Mac OS M1. Perhaps indeed due to conflicting / earlier installed software.

[ialbert]

good to know @davincios Thanks for following up.

[ayoelegs]

Hello,

I am receiving some errors. I have followed every prescription in the book. Kindly assist.

ayoelegs@James ~
$ doctor.py
# Doctor! Doctor! Give me the news.
# Checking symptoms ...
# bwa           --> ERROR
# datamash      --> ERROR
# fastqc -h     --> ERROR
# hisat2        --> ERROR
# seqret -h     --> ERROR
# featureCounts --> ERROR
# efetch        ... OK
# esearch       ... OK
# samtools      --> ERROR
# fastq-dump    ... OK
# bowtie2       --> ERROR
# bcftools      ... OK
# seqtk         --> ERROR
# seqkit        --> ERROR
# bio           --> ERROR
# fastq-dump -X 1 -Z SRR1553591 ... OK
#
# Found some problems
#
# Unable to run: bwa, datamash, fastqc -h, hisat2, seqret -h, featureCounts, samtools, bowtie2, seqtk, seqkit, bio
#
# Check the Biostar Handbook for installation help.

[ialbert]

I believe that the problem is that you have not activated the bioinfo environment.

Make sure to do so. Close and open a new terminal, then activate the environment with

micromamba activate bioinfo