|
@@ -4,18 +4,11 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "## Demonstration of validation testing with NetworkUnit by replicating the results in Gutzen et al. (sub.)\n",
|
|
|
- "This notebook showcases the basic validation workflow with the [SciUnit](https://github.com/scidash/sciunit) package and the test repository [NetworkUnit](https://github.com/INM-6/NetworkUnit).\n",
|
|
|
+ "# Demonstration of validation testing with NetworkUnit by replicating the results in Gutzen et al. (2018)\n",
|
|
|
+ "This notebook should showcase the basic validation workflow with the [SciUnit](https://github.com/scidash/sciunit) package and the test repository [NetworkUnit](https://github.com/INM-6/NetworkUnit). Here, the results are replicated reported in\n",
|
|
|
"\n",
|
|
|
- "Requirements (>= Python 2.7.14):\n",
|
|
|
- "\n",
|
|
|
- " elephant == 0.5.0\n",
|
|
|
- " sciunit == 0.2.0.2\n",
|
|
|
- " jupyter == 1.0.0\n",
|
|
|
- " tabulate == 0.8.2\n",
|
|
|
- " networkx == 2.1\n",
|
|
|
- " fastcluster == 1.1.24\n",
|
|
|
- " seaborn == 0.8.1\n",
|
|
|
+ "Gutzen, R., von Papen, M., Trensch, G., Quaglio, P., Grün, S., and Denker, M. (2018). \n",
|
|
|
+ "*Reproducible neural network simulations: statistical methods for model validation on the level of network activity data*\n",
|
|
|
" \n",
|
|
|
"**Note:** because of the large number of neurons, the computations involving the calculation of correlation coefficients for all pairs of neurons are very expensive, i.e.very time consuming."
|
|
|
]
|
|
@@ -24,7 +17,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "### Table of Contents\n",
|
|
|
+ "## Table of Contents\n",
|
|
|
"- [Setup](#setup)\n",
|
|
|
"1. [Polychrony model class](#poly_model)\n",
|
|
|
"1. [Iteration I](#it1)\n",
|
|
@@ -47,8 +40,19 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "### Setup <a id='setup'></a>\n",
|
|
|
- "importing the required packages"
|
|
|
+ "# Setup <a id='setup'></a>\n",
|
|
|
+ "## Requirements\n",
|
|
|
+ "The notebook is executable by \n",
|
|
|
+ "* cloning this [repository](https://web.gin.g-node.org/INM-6/network_validation) \n",
|
|
|
+ "* and installing the necessary packages defined in the [requirements file](https://web.gin.g-node.org/INM-6/network_validation/src/master/requirements.txt).\n",
|
|
|
+ "* It runs with Python 2.7 and Python 3."
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "Importing the required packages"
|
|
|
]
|
|
|
},
|
|
|
{
|
|
@@ -76,7 +80,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Load NetworkUnit from GitHub (only required once)\n",
|
|
|
+ "### Load NetworkUnit from GitHub (only required once)\n",
|
|
|
"Since the module is not yet pip-installable it needs to be cloned from GitHub."
|
|
|
]
|
|
|
},
|
|
@@ -164,7 +168,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "### Define the polychronization model class<a id='poly_model'></a>\n",
|
|
|
+ "## Define the polychronization model class<a id='poly_model'></a>\n",
|
|
|
"Within the SciUnit framework the models are represented as class objects. These model classes inherit from a capability class (here `ProducesSpikeTrains`) and possibly other model classes. NetworkUnit provides an abstract model class `spiketrain_data` which already has the capability of `ProducesSpikeTrains` and several other utility functions for spiketrain data such as aligning the spiketrains to 0 ms.\n",
|
|
|
"\n",
|
|
|
"The essential property of the model class is to generate a simulation outcome, either by executing the simulation or by loading the data of a previously run simulation. Here we do the latter and thus equip the class with a load() function and the variable file_path."
|
|
@@ -224,14 +228,14 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "### Iteration I<a id='it1'></a>"
|
|
|
+ "## Iteration I<a id='it1'></a>"
|
|
|
]
|
|
|
},
|
|
|
{
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Define the model instances<a id='model1'></a>\n",
|
|
|
+ "### Define the model instances<a id='model1'></a>\n",
|
|
|
"The abstract parent class polychrony_data provides the loading routine for the simulation data. As this is equivalent for both simulators, this way the same routine is simply inherited by simulation specific child classes.\n",
|
|
|
"The child classes thus only need to specify the file path to the corresponding simulation outcome."
|
|
|
]
|
|
@@ -281,7 +285,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Show rasterplots\n",
|
|
|
+ "### Show rasterplots\n",
|
|
|
"This is equivalent to the spiking activity displayed in Fig. 5 in Gutzen et al."
|
|
|
]
|
|
|
},
|
|
@@ -316,7 +320,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Define test classes<a id='test1'></a>\n",
|
|
|
+ "### Define test classes<a id='test1'></a>\n",
|
|
|
"The abstract base classes for these tests are implemented in NetworkUnit so that here the tests can be easily defined by inherting from these test classes and setting the parameters and pair the test with a score class. To perfom a hypothesis test the `effect_size` score can be replaced for example with the `ks_distance` or the `mwu_statistic`.\n",
|
|
|
"The inhereted `TestM2M` class adapts the test such that the tests don't need to be initiliazed with experimental data. "
|
|
|
]
|
|
@@ -353,7 +357,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### How to generate the test prediction\n",
|
|
|
+ "### How to generate the test prediction\n",
|
|
|
"As the essential attribute of a model is to be able to generate predictions, the main functionality of the test class if the generate_prediction() function. This function calculates the measure(s), corresponding to the specific test (here firing rates), given by the simulation or experimental data set."
|
|
|
]
|
|
|
},
|
|
@@ -372,7 +376,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### How to generate the test score\n",
|
|
|
+ "### How to generate the test score\n",
|
|
|
"In order to not only generate the predictions but to also compare them, i.e. perform the validation test, the judge function is called. \n",
|
|
|
"The `judge()` function does:\n",
|
|
|
" 1. checks if the model has all the required capabilities. If a model \n",
|
|
@@ -565,7 +569,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Visualization<a id='viz'></a>\n",
|
|
|
+ "### Visualization<a id='viz'></a>\n",
|
|
|
"The visual inspection of the samples of comparison measures is a very relevant part of the validation workflow. Thus, all tests have the inherent functionality to visualize their predictions by the function `visualize_samples()`.\n",
|
|
|
"\n",
|
|
|
"These plots are equivalent to those displayed in Fig. 5 in Gutzen et al.."
|
|
@@ -758,14 +762,14 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "### Iteration II<a id='poly_model'></a>"
|
|
|
+ "## Iteration II<a id='poly_model'></a>"
|
|
|
]
|
|
|
},
|
|
|
{
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Define the model (-instances)<a id='model2'></a>\n",
|
|
|
+ "### Define the model (-instances)<a id='model2'></a>\n",
|
|
|
"\n",
|
|
|
"As before the specific simulations are represented by child classes of `polychrony_data` with the corresponding file paths to the simulation results. Here, we even another generation for the simulations of every network state."
|
|
|
]
|
|
@@ -939,14 +943,14 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "### Iteration III<a id='it3'></a>"
|
|
|
+ "## Iteration III<a id='it3'></a>"
|
|
|
]
|
|
|
},
|
|
|
{
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Define the model (-instances)<a id='model3'></a>"
|
|
|
+ "### Define the model (-instances)<a id='model3'></a>"
|
|
|
]
|
|
|
},
|
|
|
{
|
|
@@ -1017,7 +1021,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Define additional tests<a id='test3'></a>\n",
|
|
|
+ "### Define additional tests<a id='test3'></a>\n",
|
|
|
"We add additional validation tests in order to get a more comprehensive assesment of the network dynamics, quantify the overall similarity/difference more accurately, and ideally increase the confidence we have in ther (qualitative) equivalence."
|
|
|
]
|
|
|
},
|
|
@@ -1053,7 +1057,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### perform tests and average over the 5 network states<a id='test3_avg'></a>"
|
|
|
+ "### perform tests and average over the 5 network states<a id='test3_avg'></a>"
|
|
|
]
|
|
|
},
|
|
|
{
|
|
@@ -1168,7 +1172,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Performing further testing of the correlation structure<a id='test3_struct'></a>\n",
|
|
|
+ "### Performing further testing of the correlation structure<a id='test3_struct'></a>\n",
|
|
|
"To also take into account the organisation or correlations in the two simulations, we compare the rate correlation matrices by their normed scalar product and assess the significance by comparison with surrogate data. This score is not implemented in NetworkUnit, but we use the NetworkUnit test formalism to generate and visualize the predictions (the correlation matrices). This figure is equivalent to Fig. 8 in Gutzen et al."
|
|
|
]
|
|
|
},
|
|
@@ -1308,7 +1312,7 @@
|
|
|
"cell_type": "markdown",
|
|
|
"metadata": {},
|
|
|
"source": [
|
|
|
- "#### Calculate the power spectrum<a id='test3_power'></a>"
|
|
|
+ "### Calculate the power spectrum<a id='test3_power'></a>"
|
|
|
]
|
|
|
},
|
|
|
{
|