update organization

DATASETS.json

@@ -1,44 +0,0 @@
-{
-    "tracking": {
-        "description": "whisker and nose tracking, based on high-speed videos taken at 200 Hz, and analyzed using the python 'videobatch' program.",
-        "reference": [
-            {
-                "role": "is-analyzed-using",
-                "name": "videobatch",
-                "uri":  "https://github.com/gwappa/python-videobatch"
-            },
-            {
-                "role": "is-performed-by",
-                "name": "Sina E. Dominiak",
-                "uri":  null
-            },
-            {
-                "role": "is-performed-by",
-                "name": "Keisuke Sehara",
-                "uri":  "https://orcid.org/0000-0003-4368-8143"
-            }
-        ],
-        "domains": {
-            "projection": {
-                "description": "per-trial or per-session maximal projection images.",
-                "mimetype":    "image/png",
-                "suffix":      ".png"
-            },
-            "ROI": {
-                "description": "B/W masks for ROIs to track colors from.",
-                "mimetype":    "image/png",
-                "suffix":      ".png"
-            },
-            "tracked": {
-                "description": "the CSV files containing tracked object positions in pixels. 'Slice' represents the ZERO-STARTING frame index in the video of origin.",
-                "mimetype":    "text/csv",
-                "suffix":      ".csv"
-            },
-            "states": {
-                "description": "the CSV files containing information about correspondence between trial numbers, video frames (ONE-STARTING) and behavioral states.",
-                "mimetype":    "text/csv",
-                "suffix":      ".csv"
-            }
-        }
-    }
-}

README.md

@@ -8,6 +8,7 @@ to the Human Brain Project Collab.
 - [How this dataset was acquired](#how-this-dataset-was-acquired)
   - [Setup](#setup)
   - [Task](#task)
+  - [Experimental conditions](#experimental-conditions)
   - [Acquisition and basic analysis](#acquisition-and-basic-analysis)
     - [Raw videos](#raw-videos)
     - [Behavioral states](#behavioral-states)
@@ -25,7 +26,7 @@ to the Human Brain Project Collab.
 
 <a link="#image-of-setup" />
 
-<img src="images/airtrack_platform.jpg" alt="(An image of the Airtrack platform, images/airtrack_platform.jpg)" style="zoom:33%;" />
+![(An image of the Airtrack platform, images/airtrack_platform.jpg)](images/airtrack_platform.jpg)
 
 We used the Airtrack platform ([Nashaat et al., 2016](https://doi.org/10.1152/jn.00088.2016)) (above) to engage a head-fixed mouse in a freely-behaving environment. Pumped air flow lets the plus-maze environment float on the table so that the head-fixed mouse (only the head-post holder is shown at the center of the plus maze) can easily propel the environment.
 
@@ -35,7 +36,7 @@ The position of the floating plus maze was monitored using an Arduno microcontro
 
 ### Task
 
-<img src="images/task.png" alt="(A schematic of the task, images/task.png)" style="zoom:40%;" />
+![(A schematic of the task, images/task.png)](images/task.png)
 
 The water-restricted, head-fixed mouse performed a simple plus-maze task:
 
@@ -45,14 +46,18 @@ The water-restricted, head-fixed mouse performed a simple plus-maze task:
    - If the lane in front of the animal was the correct one, the LED was _off_.
    - If the lane in front of the animal was not the correct one, the LED was _on_.
 3. When the mouse locomoted to the end of the correct lane, the mouse was allowed an access to the lick port for short (~2 s) amount of time (**Right** image in the figure):
-   - The lick port was attached to a linear stage (i.e. the apparatus on the red mechanical arm in [the setup figure](README.md#image-of-setup)), and normally was in the protracted position to keep the mouse out of access.
+   - The lick port was attached to a linear stage (i.e. the apparatus on the red mechanical arm in [the setup figure](#image-of-setup)), and normally was in the protracted position to keep the mouse out of access.
    - The microcontroller for Airtrack detected the position of the animal, and moved the linear stage to push the lick port right in front of the animal.
 
 
+### Experimental conditions
+
+- Subject-specific information can be found in [SUBJECTS.json](datasets/SUBJECTS.json).
+- Session-specific information can be found in [SESSIONS.json](datasets/SESSIONS.json).
 
 ### Acquisition and basic analysis
 
-<img src="images/acquisition.png" alt="(Sample images from high-speed video acquisition, images/acquisition.png)" style="zoom:50%;" />
+![(Sample images from high-speed video acquisition, images/acquisition.png)](images/acquisition.png)
 
 A high-speed camera acquired whisking behavior of the animal.
 
@@ -72,7 +77,7 @@ For some videos of the dataset, the behavioral states of the animal were manuall
 - **Expect**: the animal was standing at the end of the correct, rewarding lane, waiting for the lick port to come forward.
 - **Lick**: the lick port was in front of the animal.
 
-The data was stored as CSV files (the `states` domain of the `tracking` dataset).
+The data was stored as CSV files (the `states` domain of the `raw` dataset).
 
 Note that the frame numbers in the videos are indicated as _one_-starting numbers, i.e. the first frame of the video was referred to as the frame 1.
 
@@ -96,7 +101,8 @@ In brief, the procedures were as follows:
 
 ## How this dataset can be read
 
-We prepared the [helper.py](helper.py) python module for easing the dataset reading. Please check the [walkthrough.ipynb](walkthrough.ipynb) Jupyter notebook to see how to use it to read the dataset.
+We prepared the [helper.py](helper.py) python module for easing the dataset reading.
+Please check the [walkthrough.ipynb](walkthrough.ipynb) Jupyter notebook to see how to use it to read the dataset.
 
 General information about the Jupyter program can be found [here](https://jupyter.org/).
 

SESSIONS.csv

@@ -1,11 +0,0 @@
-subject,date,index,whiskers,leftC1,rightC1,leftC2,rightC2,nose
-green,2016-11-13,1,intact,Left-Green,Right-Magenta,NA,NA,NA
-green,2016-11-19,1,intact,Left-Green,Right-Magenta,NA,NA,NA
-red,2016-11-15,1,intact,Left-Green,Right-Magenta,NA,NA,NA
-zero,2016-11-15,1,intact,LeftRight-Green,LeftRight-Magenta,NA,NA,NA
-tlx,2017-08-19,1,intact,Left-Magenta,Right-Magenta,Left-Green,Right-Green,Nose-Green
-tlx,2017-08-20,1,intact,Left-Magenta,Right-Magenta,Left-Green,Right-Green,Nose-Green
-tlx,2017-08-31,1,trimmed,Left-Magenta,Right-Magenta,Left-Green,Right-Green,Nose-Green
-ndnf,2017-08-19,1,intact,Left-Magenta,Right-Magenta,Left-Green,Right-Green,Nose-Green
-ndnf,2017-08-20,1,intact,Left-Magenta,Right-Magenta,Left-Green,Right-Green,Nose-Green
-ndnf,2017-09-02,1,trimmed,Left-Magenta,Right-Magenta,Left-Green,Right-Green,Nose-Green

SUBJECTS.csv

@@ -1,6 +0,0 @@
-name,ID,sex,date_of_birth,date_of_surgery,turning_habit
-green,NA,NA,NA,NA,Right
-red,NA,NA,NA,NA,Right
-zero,NA,NA,NA,NA,Left
-ndnf,MLA-002380,Female,2017-01-06,2017-04-21,Left
-tlx,MLA-002464,Male,2017-01-12,NA,Left

batch_scripts/projection.py

@@ -0,0 +1 @@
+/annex/objects/MD5-s688--f8c12aa6c9653a380e393e9428940ac1

batch_scripts/tracking.py

@@ -0,0 +1 @@
+/annex/objects/MD5-s1677--68d7f88ee3d7ed52744bc50454c502d6

datasets/DATASETS.json

@@ -0,0 +1 @@
+/annex/objects/MD5-s2445--4be382aa41f94b3331b4008bb09c5bd1

datasets/SESSIONS.csv

@@ -0,0 +1 @@
+/annex/objects/MD5-s826--3674b6b275c4168b701fc1cbe94fee00

datasets/SUBJECTS.csv

@@ -0,0 +1 @@
+/annex/objects/MD5-s219--92648b90efc325b0be997ca938882552

helper.py

@@ -32,19 +32,19 @@ from warnings import warn as _warn
 import numpy as _np
 import pandas as _pd
 
-DATASETS_METADATA_FILE = 'datasets_metadata.json'
+DATASETS_METADATA_FILE = 'DATASETS.json'
 
-HOW_TO_USE = """
+HOW_TO_USE = f"""
 
 ------
 
-This 'helper.py' is written to work at the **root directory of the dataset**.
+This 'helper.py' is written to work at the **root repository directory**.
 
-1. please make sure that the directory structure of the dataset remains unchanged
+1. please make sure that the directory structure of the datasets remain unchanged
   (you can miss data files, though).
 2. please reposition this file inside the root directory (where you can find
-  '{DATASETS_METADATA_FILE}').
-3. change the current directory to the root directory of the dataset.
+  'REPOSITORY.json').
+3. change the current directory to the root repository directory.
 4. from a Python session, run `import helper`.
 """
 
@@ -52,20 +52,21 @@ SESSION_PATTERN   = _re.compile(r'([a-zA-Z]+)([0-9]{4}-[0-9]{2}-[0-9]{2})-([0-9]
 SUBDOMAIN_PATTERN = _re.compile(r'-([a-zA-Z0-9-]+)$')
 RUN_PATTERN       = _re.compile(r'_run([0-9]+)_')
 
-rootdir = _Path(__file__).parent
+rootdir    = _Path(__file__).parent
+datasetdir = rootdir / "datasets"
 
-def __read_root_metadata(rootdir):
-    rootdir = _Path(rootdir)
-    if not rootdir.is_dir():
+def __read_root_metadata(datasetdir):
+    rootdir = _Path(datasetdir)
+    if not datasetdir.is_dir():
         raise RuntimeError(f"not a directory: {rootdir}")
-    metadata_file = rootdir / DATASETS_METADATA_FILE
+    metadata_file = datasetdir / DATASETS_METADATA_FILE
     if not metadata_file.is_file():
         raise RuntimeError(f"not a file: {metadata_file}")
     with open(metadata_file, 'r') as src:
         return _js.load(src, object_hook=_cl.OrderedDict)
 
 def __read_csv_metadata(filename):
-    metadata_file = rootdir / filename
+    metadata_file = datasetdir / filename
     if not metadata_file.is_file():
         print(f"***cannot read from: {filename}", file=_sys.stderr)
         return None
@@ -80,8 +81,8 @@ try:
 except RuntimeError as e:
     __errormsg(f"failed to read from '{DATASETS_METADATA_FILE}' ({e})")
 
-subjects_metadata = __read_csv_metadata("subjects_metadata.csv")
-sessions_metadata = __read_csv_metadata("sessions_metadata.csv")
+subjects_metadata = __read_csv_metadata("SUBJECTS.csv")
+sessions_metadata = __read_csv_metadata("SESSIONS.csv")
 session_params    = ('session_name', 'session_type', 'date', 'session_index')
 base_params       = tuple(set(('dataset', 'subject', 'domain', 'file', 'subdomain') \
                     + session_params))
@@ -182,7 +183,7 @@ class predicate:
         if ('datasets' not in self.__cached.keys()) or (recalculate == True):
             dss = []
             for ds_name in root_metadata.keys():
-                spec = dataspec(dict(dataset=ds_name), rootdir / ds_name)
+                spec = dataspec(dict(dataset=ds_name), datasetdir / ds_name)
                 if spec.data.is_dir():
                     if self.__validate__('dataset', spec.context):
                         # print(f"adding dataset: {ds_name}")

tracking/green/session2016-11-13-001/states/green_session2016-11-13-001_allruns_states.csv

@@ -1 +0,0 @@
-Trial,Video,Start,Stop,State,Whisker,Notes
1,1,103620,104604,AtEnd,,
1,1,104605,105144,Backward,,
1,1,105145,105357,Right,,
1,1,105358,105556,Forward,,
1,1,105557,105661,Expect,,
1,1,105662,106135,Lick,,
2,1,108802,108859,AtEnd,,
2,1,108860,109210,Backward,,
2,1,109211,109405,Right,,
2,1,109406,109535,Forward,,
2,1,109536,109689,Backward,,
2,1,109690,109888,Right,,
2,1,109889,110035,Forward,,
2,1,110036,110175,Expect,,
2,1,110176,110647,Lick,,
3,1,110798,112131,AtEnd,,
3,1,112132,112410,Backward,,
3,1,112411,112644,Right,,
3,1,112645,112777,Forward,,
3,1,112778,112918,Expect,,
3,1,112919,113393,Lick,,
4,1,117300,117372,AtEnd,,
4,1,117373,117760,Backward,,
4,1,117761,117981,Right,,
4,1,117982,118335,Right,,
4,1,118336,118503,Forward,,
4,1,118504,118638,Expect,,
4,1,118639,119111,Lick,,
5,1,123910,123969,AtEnd,,
5,1,123970,124271,Backward,,
5,1,124272,124466,Right,,
5,1,124467,124617,Forward,,
5,1,124618,124763,Expect,,
5,1,124764,125234,Lick,,

tracking/green/session2016-11-19-001/states/green_session2016-11-19-001_allruns_states.csv

@@ -1 +0,0 @@
-Trial,Video,Start,Stop,State,Whisker,Notes
1,1,1982,2072,AtEnd,,
1,1,2073,2407,Backward,,
1,1,2408,2606,Right,,
1,1,2607,2743,Forward,,
1,1,2744,2901,Backward,,
1,1,2902,3049,Right,,
1,1,3050,3295,Forward,,
1,1,3296,3441,Expect,,
1,1,3442,3913,Lick,,
2,1,8810,9173,AtEnd,,
2,1,9174,9729,Backward,,
2,1,9730,9944,Right,,
2,1,9945,10020,Forward,,
2,1,10021,10184,Backward,,
2,1,10185,10339,Right,,
2,1,10340,10542,Forward,,
2,1,10543,10688,Expect,,
2,1,10689,11159,Lick,,
3,1,11435,12366,AtEnd,,
3,1,12367,12833,Backward,,
3,1,12834,13042,Right,,
3,1,13043,13290,Forward,,
3,1,13291,13953,AtEnd,,
3,1,13954,14200,Backward,,
3,1,14201,14330,Right,,
3,1,14331,14414,Forward,,
3,1,14415,14454,Backward,,
3,1,14455,14585,Right,,
3,1,14586,14780,Forward,,
3,1,14781,14914,Expect,,
3,1,14915,15385,Lick,,
4,1,88688,89808,AtEnd,,
4,1,89809,90099,Backward,,
4,1,90100,90230,Right,,
4,1,90231,90271,Forward,,
4,1,90271,90599,Midway,,
4,1,90600,90601,Backward,,
4,1,90642,90839,Right,,
4,1,90840,90923,Forward,,
4,1,90924,91063,Backward,,
4,1,91064,91065,Right,,
4,1,91066,91372,Forward,,
4,1,91373,91374,Expect,,
4,1,91375,91995,Lick,,