NeuralynxRawIO - factor parsing of contiguous blocks of Ncs records

doc/source/authors.rst

@@ -52,6 +52,7 @@ and may not be the current affiliation of a contributor.
 * rishidhingra@github
 * Hugo van Kemenade
 * Aitor Morales-Gregorio [13]
+* Peter N Steinmetz [22]
 
 1. Centre de Recherche en Neuroscience de Lyon, CNRS UMR5292 - INSERM U1028 - Universite Claude Bernard Lyon 1
 2. Unité de Neuroscience, Information et Complexité, CNRS UPR 3293, Gif-sur-Yvette, France
@@ -74,6 +75,7 @@ and may not be the current affiliation of a contributor.
 19. IAL Developmental Neurobiology, Kazan Federal University, Kazan, Russia
 20. Harden Technologies, LLC
 21. Institut des Neurosciences Paris-Saclay, CNRS UMR 9197 - Université Paris-Sud, Gif-sur-Yvette, France
+22. Neurtex Brain Research Institute, Dallas, TX, USAs
 
 If we've somehow missed you off the list we're very sorry - please let us know.
 

doc/source/core.rst

@@ -38,7 +38,7 @@ There is a simple hierarchy of containers:
     May contain any of the data objects.
   * :py:class:`Block`: The top-level container gathering all of the data, discrete and continuous,
     for a given recording session.
-    Contains :class:`Segment`, :class:`Unit` and :class:`ChannelIndex` objects.
+    Contains :class:`Segment` and :class:`Group` objects.
 
 
 Grouping/linking objects
@@ -49,18 +49,16 @@ were recorded on which electrodes, which spike trains were obtained from which
 membrane potential signals, etc. They contain references to data objects that
 cut across the simple container hierarchy.
 
-  * :py:class:`ChannelIndex`: A set of indices into :py:class:`AnalogSignal` objects,
-    representing logical and/or physical recording channels. This has two uses:
+  * :py:class:`ChannelView`: A set of indices into :py:class:`AnalogSignal` objects,
+    representing logical and/or physical recording channels.
+    For spike sorting of extracellular signals, where spikes may be recorded on more than one
+    recording channel, the :py:class:`ChannelView` can be used to reference the group of recording channels
+    from which the spikes were obtained.
 
-      1. for linking :py:class:`AnalogSignal` objects recorded from the same (multi)electrode
-         across several :py:class:`Segment`\s.
-      2. for spike sorting of extracellular signals, where spikes may be recorded on more than one
-         recording channel, and the :py:class:`ChannelIndex` can be used to associate each
-         :py:class:`Unit` with the group of recording channels from which it was obtained.
-
-  * :py:class:`Unit`: links the :class:`SpikeTrain` objects within a :class:`Block`,
-    possibly across multiple Segments, that were emitted by the same cell.
-    A :class:`Unit` is linked to the :class:`ChannelIndex` object from which the spikes were detected.
+  * :py:class:`Group`: Can contain any of the data objects, views, or other groups,
+    outside the hierarchy of the segment and block containers.
+    A common use is to link the :class:`SpikeTrain` objects within a :class:`Block`,
+    possibly across multiple Segments, that were emitted by the same neuron.
 
   * :py:class:`CircularRegionOfInterest`, :py:class:`RectangularRegionOfInterest` and :py:class:`PolygonRegionOfInterest`
     are three subclasses that link :class:`ImageSequence` objects to signals (:class:`AnalogSignal` objects)
@@ -105,14 +103,14 @@ In general, an object can access its children with an attribute *childname+s* in
     * :attr:`Block.segments`
     * :attr:`Segments.analogsignals`
     * :attr:`Segments.spiketrains`
-    * :attr:`Block.channel_indexes`
+    * :attr:`Block.groups`
 
 These relationships are bi-directional, i.e. a child object can access its parent:
 
     * :attr:`Segment.block`
     * :attr:`AnalogSignal.segment`
     * :attr:`SpikeTrain.segment`
-    * :attr:`ChannelIndex.block`
+    * :attr:`Group.block`
 
 Here is an example showing these relationships in use::
 
@@ -134,38 +132,39 @@ Here is an example showing these relationships in use::
 
 In some cases, a one-to-many relationship is sufficient. Here is a simple example with tetrodes, in which each tetrode has its own group.::
 
-    from neo import Block, ChannelIndex
+    from neo import Block, Group
     bl = Block()
 
     # the four tetrodes
     for i in range(4):
-        chx = ChannelIndex(name='Tetrode %d' % i,
-                           index=[0, 1, 2, 3])
-        bl.channelindexes.append(chx)
+        group = Group(name='Tetrode %d' % i)
+        bl.groups.append(group)
 
     # now we load the data and associate it with the created channels
     # ...
 
-Now consider a more complex example: a 1x4 silicon probe, with a neuron on channels 0,1,2 and another neuron on channels 1,2,3. We create a group for each neuron to hold the :class:`Unit` object associated with this spike sorting group. Each group also contains the channels on which that neuron spiked. The relationship is many-to-many because channels 1 and 2 occur in multiple groups.::
+Now consider a more complex example: a 1x4 silicon probe, with a neuron on channels 0,1,2 and another neuron on channels 1,2,3.
+We create a group for each neuron to hold the spiketrains for each spike sorting group together with
+the channels on which that neuron spiked::
 
     bl = Block(name='probe data')
 
     # one group for each neuron
-    chx0 = ChannelIndex(name='Group 0',
-                        index=[0, 1, 2])
-    bl.channelindexes.append(chx0)
+    view0 = ChannelView(recorded_signals, index=[0, 1, 2])
+    unit0 = Group(view0, name='Group 0')
+    bl.groups.append(unit0)
 
-    chx1 = ChannelIndex(name='Group 1',
-                        index=[1, 2, 3])
-    bl.channelindexes.append(chx1)
+    view1 = ChannelView(recorded_signals, index=[1, 2, 3])
+    unit1 = Group(view1, name='Group 1')
+    bl.groups.append(unit1)
 
-    # now we add the spiketrain from Unit 0 to chx0
-    # and add the spiketrain from Unit 1 to chx1
+    # now we add the spiketrains from Unit 0 to unit0
+    # and add the spiketrains from Unit 1 to unit1
     # ...
 
-Note that because neurons are sorted from groups of channels in this situation, it is natural that the :py:class:`ChannelIndex` contains a reference to the :py:class:`Unit` object.
-That unit then contains references to its spiketrains. Also note that recording channels can be
-identified by names/labels as well as, or instead of, integer indices.
+
+Now each putative neuron is represented by a :class:`Group` containing the spiktrains of that neuron
+and a view of the signal selecting only those channels from which the spikes were obtained.
 
 
 See :doc:`usecases` for more examples of how the different objects may be used.

doc/source/grouping.rst

@@ -0,0 +1,145 @@
+*************************
+Grouping and linking data
+*************************
+
+
+... to do
+
+
+
+
+Migrating from ChannelIndex/Unit to ChannelView/Group
+==============================================
+
+
+Examples
+--------
+
+A simple example with two tetrodes. Here the :class:`ChannelIndex` was not being
+used for grouping, simply to associate a name with each channel.
+
+Using :class:`ChannelIndex`::
+
+    import numpy as np
+    from quantities import kHz, mV
+    from neo import Block, Segment, ChannelIndex, AnalogSignal
+
+    block = Block()
+    segment = Segment()
+    segment.block = block
+    block.segments.append(segment)
+
+    for i in (0, 1):
+        signal = AnalogSignal(np.random.rand(1000, 4) * mV,
+                              sampling_rate=1 * kHz,)
+        segment.analogsignals.append(signal)
+        chx = ChannelIndex(name=f"Tetrode #{i + 1}",
+                           index=[0, 1, 2, 3],
+                           channel_names=["A", "B", "C", "D"])
+        chx.analogsignals.append(signal)
+        block.channel_indexes.append(chx)
+
+Using array annotations, we annotate the channels of the :class:`AnalogSignal` directly::
+
+    import numpy as np
+    from quantities import kHz, mV
+    from neo import Block, Segment, AnalogSignal
+
+    block = Block()
+    segment = Segment()
+    segment.block = block
+    block.segments.append(segment)
+
+    for i in (0, 1):
+        signal = AnalogSignal(np.random.rand(1000, 4) * mV,
+                              sampling_rate=1 * kHz,
+                              channel_names=["A", "B", "C", "D"])
+        segment.analogsignals.append(signal)
+
+
+Now a more complex example: a 1x4 silicon probe, with a neuron on channels 0,1,2 and another neuron on channels 1,2,3.
+We create a :class:`ChannelIndex` for each neuron to hold the :class:`Unit` object associated with this spike sorting group.
+Each :class:`ChannelIndex` also contains the list of channels on which that neuron spiked.
+
+::
+
+    import numpy as np
+    from quantities import ms, mV, kHz
+    from neo import Block, Segment, ChannelIndex, Unit, SpikeTrain, AnalogSignal
+
+    block = Block(name="probe data")
+    segment = Segment()
+    segment.block = block
+    block.segments.append(segment)
+
+    # create 4-channel AnalogSignal with dummy data
+    signal = AnalogSignal(np.random.rand(1000, 4) * mV,
+                          sampling_rate=10 * kHz)
+    # create spike trains with dummy data
+    # we will pretend the spikes have been extracted from the dummy signal
+    spiketrains = [
+        SpikeTrain(np.arange(5, 100) * ms, t_stop=100 * ms),
+        SpikeTrain(np.arange(7, 100) * ms, t_stop=100 * ms)
+    ]
+    segment.analogsignals.append(signal)
+    segment.spiketrains.extend(spiketrains)
+    # assign each spiketrain to a neuron (Unit)
+    units = []
+    for i, spiketrain in enumerate(spiketrains):
+        unit = Unit(name=f"Neuron #{i + 1}")
+        unit.spiketrains.append(spiketrain)
+        units.append(unit)
+
+    # create a ChannelIndex for each unit, to show which channels the spikes come from
+    chx0 = ChannelIndex(name="Channel Group 1", index=[0, 1, 2])
+    chx0.units.append(units[0])
+    chx0.analogsignals.append(signal)
+    units[0].channel_index = chx0
+    chx1 = ChannelIndex(name="Channel Group 2", index=[1, 2, 3])
+    chx1.units.append(units[1])
+    chx1.analogsignals.append(signal)
+    units[1].channel_index = chx1
+
+    block.channel_indexes.extend((chx0, chx1))
+
+
+Using :class:`ChannelView` and :class`Group`::
+
+    import numpy as np
+    from quantities import ms, mV, kHz
+    from neo import Block, Segment, ChannelView, Group, SpikeTrain, AnalogSignal
+
+    block = Block(name="probe data")
+    segment = Segment()
+    segment.block = block
+    block.segments.append(segment)
+
+    # create 4-channel AnalogSignal with dummy data
+    signal = AnalogSignal(np.random.rand(1000, 4) * mV,
+                          sampling_rate=10 * kHz)
+    # create spike trains with dummy data
+    # we will pretend the spikes have been extracted from the dummy signal
+    spiketrains = [
+        SpikeTrain(np.arange(5, 100) * ms, t_stop=100 * ms),
+        SpikeTrain(np.arange(7, 100) * ms, t_stop=100 * ms)
+    ]
+    segment.analogsignals.append(signal)
+    segment.spiketrains.extend(spiketrains)
+    # assign each spiketrain to a neuron (now using Group)
+    units = []
+    for i, spiketrain in enumerate(spiketrains):
+        unit = Group(spiketrain, name=f"Neuron #{i + 1}")
+        units.append(unit)
+
+    # create a ChannelView of the signal for each unit, to show which channels the spikes come from
+    # and add it to the relevant Group
+    view0 = ChannelView(signal, index=[0, 1, 2], name="Channel Group 1")
+    units[0].add(view0)
+    view1 = ChannelView(signal, index=[1, 2, 3], name="Channel Group 2")
+    units[1].add(view1)
+
+    block.groups.extend(units)
+
+
+Now each putative neuron is represented by a :class:`Group` containing the spiktrains of that neuron
+and a view of the signal selecting only those channels from which the spikes were obtained.