{ "cells": [ { "cell_type": "markdown", "id": "591a5673", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "#

Introduction to\n", "![Neo](./neo_material/neo_logo.png)\n", "

Representing electrophysiology data in Python" ] }, { "cell_type": "markdown", "id": "d02e9275", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Data sources in electrophysiology\n", "\n", "![Data sources](./neo_material/ephys_data_sources.png)\n" ] }, { "cell_type": "markdown", "id": "dee2a771", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Data modalities in electrophysiology\n", "\n", "![Data sources](./neo_material/ephys_data_modalities.png)\n" ] }, { "cell_type": "markdown", "id": "4c7b4932", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "How can these diverse data types and formats by treated in a common framework to \n", "- allow combined analyses\n", "- facilitate reproducibility\n", "- simplify scientific workflows" ] }, { "cell_type": "markdown", "id": "b80bdd60", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "...without inventing yet another file format standard?" ] }, { "cell_type": "markdown", "id": "c09c4aac", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Neo - Generic representation of common ephys modalities\n", "- Standardized representation of ephys data during runtime\n", "- Efficient handling of large data arrays thanks to `numpy`\n", "- Description of physical units and unit conversion thanks to `quantities`" ] }, { "cell_type": "code", "execution_count": 19, "id": "6e5df85c", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "import neo\n", "import numpy as np\n", "import quantities as pq" ] }, { "cell_type": "markdown", "id": "9242b2de", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Data Classes\n", "- `AnalogSignal`: Continuous data sampled in **regular** intervals" ] }, { "cell_type": "markdown", "id": "7f8c47d0", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "*essential metadata*: physical unit of samples, time stamps of the samples (first timestamp & sampling interval)" ] }, { "cell_type": "code", "execution_count": 34, "id": "ecfbe500", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "anasig = neo.AnalogSignal(np.zeros((50,2)), units='uV',\n", " sampling_rate=10000*pq.Hz,\n", " t_start=120*pq.ms)" ] }, { "cell_type": "markdown", "id": "fa04abd6", "metadata": { "slideshow": { "slide_type": "-" } }, "source": [ "Note the dimensions of `Analogsignal` object: (`time`, `channel`)" ] }, { "cell_type": "markdown", "id": "710a0eb7", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Accessing metadata" ] }, { "cell_type": "code", "execution_count": 37, "id": "9ba68d73", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "units: 1.0 uV\n", "sampling_rate: 10000.0 Hz\n", "sampling_period: 0.0001 s\n", "t_start & t_stop: (array(120.) * ms, array(125.) * ms)\n", "times: [120. 120.1 120.2 120.3 120.4 120.5 120.6 120.7 120.8 120.9 121. 121.1\n", " 121.2 121.3 121.4 121.5 121.6 121.7 121.8 121.9 122. 122.1 122.2 122.3\n", " 122.4 122.5 122.6 122.7 122.8 122.9 123. 123.1 123.2 123.3 123.4 123.5\n", " 123.6 123.7 123.8 123.9 124. 124.1 124.2 124.3 124.4 124.5 124.6 124.7\n", " 124.8 124.9] ms\n" ] } ], "source": [ "print(f'units: {anasig.units}')\n", "print(f'sampling_rate: {anasig.sampling_rate}')\n", "print(f'sampling_period: {anasig.sampling_period.simplified}')\n", "print(f't_start & t_stop: {anasig.t_start, anasig.t_stop}')\n", "print(f'times: {anasig.times}')" ] }, { "cell_type": "markdown", "id": "afdf47c2", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "\n", "![AnalogSignal](./neo_material/base_schematic_0.svg)\n" ] }, { "cell_type": "markdown", "id": "9171b1f5", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Data Classes\n", "- `AnalogSignal`: Continuous data sampled in **regular** intervals\n", "- `IrregularlySampledSignal`: Continuous data sampled in **irregular** intervals\n", "- `ImageSequence`: Continuous 2D **frames** sampled in regular intervals" ] }, { "cell_type": "markdown", "id": "cec7e8c0", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![ImageSequence](./neo_material/base_schematic_2.svg)" ] }, { "cell_type": "markdown", "id": "0bfb4ac8", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Data Classes\n", "- `AnalogSignal`: Continuous data sampled in **regular** intervals\n", "- `IrregularlySampledSignal`: Continuous data sampled in **irregular** intervals\n", "- `ImageSequence`: Continuous 2D **image frames** sampled in regular intervals\n", "- `SpikeTrain`: Time point data (& optional waveform snippet)" ] }, { "cell_type": "markdown", "id": "0534eb6b", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "*essential metadata* time values, physical units of times, (& waveform sampling rate, waveform offset to corresponding time value)" ] }, { "cell_type": "code", "execution_count": 29, "id": "d7efe06e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "spiketrain: [1. 4. 5.7] ms\n", "t_start & t_stop: (array(0.) * ms, array(300.) * ms)\n" ] } ], "source": [ "st = neo.SpikeTrain([1, 4, 5.7], units='ms', name='#001', t_start=0*pq.ms, t_stop=300*pq.ms)\n", "print(f'spiketrain: {st}')\n", "print(f't_start & t_stop: {st.t_start, st.t_stop}')" ] }, { "cell_type": "markdown", "id": "d5ad23e7", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![SpikeTrain](./neo_material/base_schematic_3.svg)" ] }, { "cell_type": "markdown", "id": "d0456d48", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Data Classes\n", "- `AnalogSignal`: Continuous data sampled in **regular** intervals\n", "- `IrregularlySampledSignal`: Continuous data sampled in **irregular** intervals\n", "- `ImageSequence`: Continuous 2D **image frames** sampled in regular intervals\n", "- `SpikeTrain`: Time point data (& optional waveform snippet)\n", "- `Event`: Experiment reference time points (e.g. trigger, trial start, ...)\n", "- `Epoch`: Experiment reference time ranges (e.g. trial, stimulation, ...)" ] }, { "cell_type": "markdown", "id": "759eee37", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![Epoch](./neo_material/base_schematic_5.svg)" ] }, { "cell_type": "markdown", "id": "b1d5a21d", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Additional metadata attributes\n", "- human readable label of objects via `name` attribute\n", "- custom metadata annotations via `annotation` and `array_annotation` attributes\n", "- `Event` and `Epoch` can be used to `label` each time point / time period" ] }, { "cell_type": "code", "execution_count": 42, "id": "321dc012", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "name: electrode 1A\n", "annotations: {'signal_quality': 'good'}\n", "number of channels: 2\n", "array_annotations: {'channel_id': array([1, 2])}\n" ] } ], "source": [ "anasig.name = 'electrode 1A'\n", "anasig.annotate(signal_quality='good')\n", "anasig.array_annotate(channel_id=[1,2])\n", "print(f'name: {anasig.name}')\n", "print(f'annotations: {anasig.annotations}')\n", "print(f'number of channels: {anasig.shape[-1]}')\n", "print(f'array_annotations: {anasig.array_annotations}')" ] }, { "cell_type": "markdown", "id": "c2062418", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Neo objects provide utility functions\n", "Some usefull utility attributes and methods of neo data objects are\n", "- `.times` to get array of corresponding time values\n", "- `.time_slice()` to crop to a specific time range\n", "- `.merge()` to combine multiple objects of the same type\n", "- `.concatenate()` to append multiple signal objects\n", "- `.downsample()` to create a new signal with a different sampling rate\n", "- `.magnitude` to extract the underlying numpy array\n", "- check out the [documentation](https://neo.readthedocs.io/en/latest/core.html) to discover more!" ] }, { "cell_type": "markdown", "id": "0b7a117b", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Relations between data objects\n", "- `ChannelView`: select a subset of channels of a signal, e.g. all even channels of an `AnalogSignal`\n", "- **`Segment`**: contains data objects with a shared clock, e.g. a trial\n", "- `Group`: groups data objects logically (no common clock required, e.g. `SpikeTrain`s of a neuronal unit)\n", "- **`Block`**: contains all objects of a recording" ] }, { "cell_type": "markdown", "id": "635213d6", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![Block](./neo_material/base_schematic.svg)" ] }, { "cell_type": "markdown", "id": "63624385", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Neo structure\n", "\n", "```\n", "Block 0\n", " .segments\n", " Segment 0\n", " .analogsignals\n", " AnalogSignal 0\n", " AnalogSignal 1\n", " .spiketrains\n", " SpikeTrain 0\n", " SpikeTrain 1\n", " SpikeTrain 2\n", " Segment 1\n", " .analogsignals\n", " AnalogSignal 0\n", " AnalogSignal 1\n", " .spiketrains\n", " SpikeTrain 0\n", " SpikeTrain 1\n", " SpikeTrain 2\n", "Block 1\n", " .segments\n", " Segment 0\n", " ...\n", "```" ] }, { "cell_type": "markdown", "id": "1130f965", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Neo Class Overview\n", "![neo_uml](./neo_material/simple_generated_diagram_with_channelview.svg)" ] }, { "cell_type": "markdown", "id": "d46c3064", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Automatic generation of Neo objects\n", "\"IODiagram\"" ] }, { "cell_type": "markdown", "id": "9afd13ea", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Loading a recording session\n", "- generation of a complete neo structure requires only **2 lines of code** and the name of the original recording system." ] }, { "cell_type": "code", "execution_count": 51, "id": "27a022ab", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "Block with 1 segments, 1 groups\n", "annotations: {'openephys_version': '0.4'}\n", "file_origin: './neo_material/example_data/OpenEphys_SampleData_1'\n", "# segments (N=1)\n", "0: Segment with 1 analogsignals, 1 events, 1 spiketrains\n", " annotations: {'openephys_version': '0.4',\n", " 'date_created': \"'3-Oct-2018 131650'\",\n", " 'openephys_segment_index': 1}\n", " # analogsignals (N=1)\n", " 0: AnalogSignal with 2 channels of length 423936; units V; datatype float32 \n", " name: 'Signals CH'\n", " annotations: {'stream_id': 'CH'}\n", " sampling rate: 40000.0 Hz\n", " time: 1.2992 s to 11.8976 s" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "recording_folder = './neo_material/example_data/OpenEphys_SampleData_1'\n", "io = neo.io.OpenEphysIO(recording_folder)\n", "block = io.read_block()\n", "block" ] }, { "cell_type": "markdown", "id": "82d2f42a", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Accessing data objects\n", "- child objects can be accessed the corresponding attribute based on the class name (plural!). E.g. `.segments`, `.analogsignals`, `.spiketrains`\n", "- child objects are stored in *lists*" ] }, { "cell_type": "code", "execution_count": 60, "id": "da4d5648", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "AnalogSignal with 2 channels of length 423936; units V; datatype float32 \n", "name: 'Signals CH'\n", "annotations: {'stream_id': 'CH'}\n", "sampling rate: 40000.0 Hz\n", "time: 1.2992 s to 11.8976 s" ] }, "execution_count": 60, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# accessing the list of Segments\n", "block.segments\n", "# accessing the first AnalogSignal of a single Segment\n", "segment = block.segments[0]\n", "segment.analogsignals[0]" ] }, { "cell_type": "markdown", "id": "d8faf977", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Inspecting a spiketrain object" ] }, { "cell_type": "code", "execution_count": 67, "id": "18359d77", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "SpikeTrain name: 'STp106.0n0#0' annotations: {'id': 'STp106.0n0#0'}" ] }, "execution_count": 67, "metadata": {}, "output_type": "execute_result" } ], "source": [ "spiketrain = segment.spiketrains[0]\n", "spiketrain" ] }, { "cell_type": "code", "execution_count": 70, "id": "3eed0a74", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of spikes: 454\n" ] }, { "data": { "text/plain": [ "array([1.30545 , 1.403875, 1.415325, 1.422425, 1.428775, 1.45465 ,\n", " 1.4658 , 1.5147 , 1.51905 , 1.521725, 1.546 , 1.563975,\n", " 1.566525, 1.568325, 1.573275, 1.58005 , 1.588125, 1.59505 ,\n", " 1.599125, 1.654425]) * s" ] }, "execution_count": 70, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(f'Number of spikes: {len(spiketrain)}')\n", "spiketrain.times[:20]" ] }, { "cell_type": "markdown", "id": "b338dc98", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Dealing with large datasets\n", "How to load only required data\n", "\n", "- some IOs are based on `RawIO` concept for efficient reading of data\n", "- `RawIO`s require additional symmetries in the dataset for efficient loading\n", "- *lazy* data objects (ProxyObjects) can be loaded using `io.read_block(`**`lazy=True`**`)`\n", "- ProxyObjects provide a `.load(t_start, t_stop)` method that loads requested data in memory and returns complete neo data object." ] }, { "cell_type": "code", "execution_count": 109, "id": "e516de90", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "AnalogSignalProxy name: 'Signals CH' annotations: {'stream_id': 'CH'}" ] }, "execution_count": 109, "metadata": {}, "output_type": "execute_result" } ], "source": [ "recording_folder = './neo_material/example_data/OpenEphys_SampleData_1'\n", "io = neo.io.OpenEphysIO(recording_folder)\n", "block = io.read_block(lazy=True)\n", "lazy_anasig = block.segments[0].analogsignals[0]\n", "lazy_anasig" ] }, { "cell_type": "markdown", "id": "9a6a885a", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Loading data from a Proxy object\n", "- ProxyObjects contain metadata and shape information\n" ] }, { "cell_type": "code", "execution_count": 90, "id": "f50bc7a3", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "signal shape: (423936, 2)\n", "signal sampling rate: 40000.0 Hz\n", "signal annotations: {'stream_id': 'CH'}\n" ] } ], "source": [ "print(f'signal shape: {lazy_anasig.shape}')\n", "print(f'signal sampling rate: {lazy_anasig.sampling_rate}')\n", "print(f'signal annotations: {lazy_anasig.annotations}')" ] }, { "cell_type": "markdown", "id": "e4ef7c39", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "- data of a specific channel and time range can be loaded selectively into a new neo object" ] }, { "cell_type": "code", "execution_count": 85, "id": "2aa4d7e9", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "anasig = lazy_anasig.load(time_slice=(5*pq.s,6*pq.s), channel_indexes=[0])" ] }, { "cell_type": "code", "execution_count": 93, "id": "7a4dca6d", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "signal shape: (40000, 1)\n", "signal sampling rate: 40000.0 Hz\n", "signal annotations: [5. 5.000025 5.00005 ... 5.999925 5.99995 5.999975] s\n", "signal values: [[13.45]\n", " [13.1 ]\n", " [10.8 ]\n", " ...\n", " [12.3 ]\n", " [13.05]\n", " [14.15]]\n" ] } ], "source": [ "print(f'signal shape: {anasig.shape}')\n", "print(f'signal sampling rate: {anasig.sampling_rate}')\n", "print(f'signal annotations: {anasig.times}')\n", "print(f'signal values: {anasig.magnitude}')" ] }, { "cell_type": "markdown", "id": "1f2d38e9", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### How to save data?" ] }, { "cell_type": "markdown", "id": "32c21898", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "- selected open formats are supported for writing\n", " - **NIX**[1](#fn1)\n", " - NWB[1](#fn1)\n", " - Matlab[2](#fn2)\n", " - Ascii[2](#fn2)\n", " - Numpy Pickle[3](#fn3)\n", " \n", "1 Support of neo-compatible format aspects\n", "\n", "2 Does not capture complete set of metadata\n", "\n", "3 Strong dependency on Numpy and Neo version" ] }, { "cell_type": "code", "execution_count": 119, "id": "18593599", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[]\n" ] } ], "source": [ "filename = 'my_first_neo_dataset.nix'\n", "with neo.io.NixIO(filename, 'ow') as io:\n", " io.write_block(block)" ] }, { "cell_type": "markdown", "id": "ffaaba4d", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Why two NixIOs in Neo?\n", "`neo.io.NixIO`\n", "- non-RawIO implementation\n", "- can read and write arbitrary Neo object structures\n", "\n", "`neo.io.NixIOFr`\n", "- RawIO implementation\n", "- can very efficiently read specific Neo-Nix files that contain symmetric objects" ] }, { "cell_type": "markdown", "id": "6f7915d3", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Time for ...\n", "\n" ] }, { "cell_type": "markdown", "id": "78a0901c", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "- Questions" ] }, { "cell_type": "markdown", "id": "3283b41c", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "- Break\n" ] }, { "cell_type": "markdown", "id": "a66e0e79", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "- Interactive exploration of neo structures!" ] }, { "cell_type": "markdown", "id": "d9e8db03", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "\n", "\n", "\n", "
" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" } }, "livereveal":{ "autolaunch": true, "scroll": true} }, "nbformat": 4, "nbformat_minor": 5 }