Random Issue with Neo

[ohnopleasehelpme]

Im using Neo in conjunction with SpikeInterface - so far I am unable to use Neo to open Neuralynx (.ncs) files:

import spikeinterface.extractors as se
import spikeinterface.toolkit as st
import spikeinterface.sorters as ss
import spikeinterface.comparison as sc
import spikeinterface.widgets as sw
import matplotlib.pylab as plt
import numpy as np
import neo
reader = neo.NeuralynxIO(dirname='~/PycharmProjects/Spike2020/data')
print(reader)
print(reader.segment_count(0))

I get the error:

runfile('/PycharmProjects/Spike2020/Spike.py', wdir='/PycharmProjects/Spike2020')
Traceback (most recent call last):
File "", line 1, in
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/PycharmProjects/Spike2020/Spike.py", line 11, in
reader = neo.NeuralynxIO(dirname='/PycharmProjects/Spike2020/data')
File "/opt/miniconda3/envs/Spike2020/lib/python3.7/site-packages/neo/io/neuralynxio.py", line 51, in init
BaseFromRaw.init(self, dirname)
File "/opt/miniconda3/envs/Spike2020/lib/python3.7/site-packages/neo/io/basefromrawio.py", line 75, in init
self.parse_header()
File "/opt/miniconda3/envs/Spike2020/lib/python3.7/site-packages/neo/rawio/baserawio.py", line 149, in parse_header
self._parse_header()
File "/opt/miniconda3/envs/Spike2020/lib/python3.7/site-packages/neo/rawio/neuralynxrawio.py", line 111, in _parse_header
info = NlxHeader.build_for_file(filename)
File "~/opt/miniconda3/envs/Spike2020/lib/python3.7/site-packages/neo/rawio/neuralynxrawio.py", line 801, in build_for_file
dt1 = re.search(hpd['datetime1_regex'], txt_header).groupdict()
AttributeError: 'NoneType' object has no attribute 'groupdict'

The neuralynx folder has 4 .ncs files and a cheetah log file.txt.

[samuelgarcia]

Hi  @ohnopleasehelpme
thanks for the report.
Could you send us a file that we could test.
if the file is small would you agree to make it public for our test suite ?

@PeterNSteinmetz: could we add something like if 'datetime1_regex' in hpd: in the actual PR you are working on ?

[ohnopleasehelpme]

Sure! Can I send it by email?

[ohnopleasehelpme]

Or - how would you prefer that I send it? Thank you!

[samuelgarcia]

Maybe a drive like file sender would be ok.

[ohnopleasehelpme]

How about through email? Google Drive?

[JuliaSprenger]

Hi @ohnopleasehelpme.
Thanks for reporting the issue.
Best would be to upload the files to some place, e.g. Google Drive, and then send the link via email to samuel or me. See our account details for the addresses.

[samuelgarcia]

I receive privarttle th file.
It is too big (220Mo) to put in the public files.
As the error seeams simple I propose to fix it either directly or in the PR #905 done by @PeterNSteinmetz

[JuliaSprenger]

@samuelgarcia Ok, do you add a commit / comment in #905 for this?

[PeterNSteinmetz]

@samuelgarcia @JuliaSprenger This may already be fixed in PR #905, though I am not sure. That error would arise if a file header does not have a Time Opened entry which matches current patterns, which seems a bit odd. So it would be good to have the test file.

It appears this has to do with the header, so if the larger file is still broken, we can truncate it to a few records and add to the test suite (with permission of the provider of course). If you get me the test file, I can have an immediate look.

[ohnopleasehelpme]

Hi Peter, go ahead! Please let me know if you need anything from my end too

[ohnopleasehelpme]

Hi @JuliaSprenger - thank you for the heads up! I emailed my sample files to Dr. Garcia

[PeterNSteinmetz]

Hi @ohnopleasehelpme . If you have the folder or file on Google Drive, perhaps share it with me. I am user peternsteinmetz@gmail.com there. Alternately DropBox could work. email for larger files over a few MB will often bounce eventually.

[ohnopleasehelpme]

Hi Peter - sent!

[PeterNSteinmetz]

Was able to obtain the test data from Google drive. There are actually two errors in the current code which prevent proper interpretation of these files. Described in more detail below, but the short point is that I have created a test dataset for this issue, fixes that allow correct interpretation, and tests of same.

@ohnopleasehelpme - May we have your permission to use portions of the files you sent as test data for Python Neo which will be publicly available? If so, how would you like the provision of these files to be attributed in the accompanying README.txt file? Normally this would state which lab and investigator the files came from.

In more detail:

The first issue is that these are Cheetah Version 5.4.0 files and the headers for this Cheetah version are slightly different from those previously used for testing (missing a colon in the Time Opened and Time Closed entries).

The second is the these files are sampled at a fairly low rate of 1017.375 Hz, which gives rise to larger variations in Ncs record end times than have been present in prior test files.

I have a branch for committing the test files to gin.g-node ready as soon as permission is available. I also have a branch for the changes to the code and the tests for this file type ready as well. Will commit that once the test data is in gin.g-node so the integration tests will pass on the PR.

Presently these changes are based off PR #905 though could be rebased onto master. Would prefer to just get #905 merged of course.

[samuelgarcia]

Hi Peter.
Thank you for incorporating this fix in your PR.
It is OK to have everything in one big PR.