.. _io_dev_guide:

********************
IO developers' guide
********************


.. _io_guiline:

Guidelines for IO implementation
================================

There are two ways to add a new IO module:
  * By directly adding a new IO class in a module within :mod:`neo.io`: the reader/writer will deal directly with Neo objects
  * By adding a RawIO class in a module within :mod:`neo.rawio`: the reader should work with raw buffers from the file and provide
    some internal headers for the scale/units/name/...
    You can then generate an IO module simply by inheriting from your RawIO class and from :class:`neo.io.BaseFromRaw`

For read only classes, we encourage you to write a :class:`RawIO` class because it allows slice reading,
and is generally much quicker and easier (although only for reading) than implementing a full IO class.
For read/write classes you can mix the two levels neo.rawio for reading and neo.io for writing.

Recipe to develop an IO module for a new data format:
    1. Fully understand the object model. See :doc:`core`. If in doubt ask the `mailing list`_.
    2. Fully understand :mod:`neo.rawio.examplerawio`, It is a fake IO to explain the API. If in doubt ask the list.
    3. Copy/paste ``examplerawio.py`` and choose clear file and class names for your IO.
    4. implement all methods that **raise(NotImplementedError)** in :mod:`neo.rawio.baserawio`. Return None when the object is not supported (spike/waveform)
    5. Write good docstrings. List dependencies, including minimum version numbers.
    6. Add your class to :mod:`neo.rawio.__init__`. Keep imports inside ``try/except`` for dependency reasons.
    7. Create a class in :file:`neo/io/`
    8. Add your class to :mod:`neo.io.__init__`. Keep imports inside ``try/except`` for dependency reasons.
    9. Create an account at https://gin.g-node.org and deposit files in :file:`NeuralEnsemble/ephy_testing_data`.
    10. Write tests in :file:`neo/rawio/test_xxxxxrawio.py`. You must at least pass the standard tests (inherited from :class:`BaseTestRawIO`). See :file:`test_examplerawio.py`
    11. Write a similar test in :file:`neo.tests/iotests/test_xxxxxio.py`. See :file:`test_exampleio.py`
    12. Make a pull request when all tests pass.

Miscellaneous
=============

    * If your IO supports several versions of a format (like ABF1, ABF2), upload to the gin.g-node.org test file repository all file versions possible. (for test coverage).
    * :py:func:`neo.core.Block.create_many_to_one_relationship` offers a utility to complete the hierachy when all one-to-many relationships have been created.
    * In the docstring, explain where you obtained the file format specification if it is a closed one.
    * If your IO is based on a database mapper, keep in mind that the returned object MUST be detached,
      because this object can be written to another url for copying.


Tests
=====

:py:class:`neo.rawio.tests.common_rawio_test.BaseTestRawIO` and :py:class:`neo.test.io.commun_io_test.BaseTestIO` provide standard tests.
To use these you need to upload some sample data files at `gin-gnode`_. They will be publicly accessible for testing Neo.
These tests:

  * check the compliance with the schema: hierachy, attribute types, ...
  * For IO modules able to both write and read data, it compares a generated dataset with the same data after a write/read cycle.

The test scripts download all files from `gin-gnode`_ and stores them locally in ``/tmp/files_for_tests/``.
Subsequent test runs use the previously downloaded files, rather than trying to download them each time.

Each test must have at least one class that inherits ``BaseTestRawIO`` and that has 3 attributes:
  * ``rawioclass``: the class
  * ``entities_to_test``: a list of files (or directories) to be tested one by one
  * ``files_to_download``: a list of files to download (sometimes bigger than ``entities_to_test``)

Here is an example test script taken from the distribution: :file:`test_axonrawio.py`:

.. literalinclude:: ../../neo/test/rawiotest/test_axonrawio.py


Logging
=======

All IO classes by default have logging using the standard :mod:`logging` module: already set up.
The logger name is the same as the fully qualified class name, e.g. :class:`neo.io.hdf5io.NeoHdf5IO`.
The :attr:`class.logger` attribute holds the logger for easy access.

There are generally 3 types of situations in which an IO class should use a logger

  * Recoverable errors with the file that the users need to be notified about.
    In this case, please use :meth:`logger.warning` or :meth:`logger.error`.
    If there is an exception associated with the issue, you can use :meth:`logger.exception` in the exception handler to automatically include a backtrace with the log.
    By default, all users will see messages at this level, so please restrict it only to problems the user absolutely needs to know about.
  * Informational messages that advanced users might want to see in order to get some insight into the file.
    In this case, please use :meth:`logger.info`.
  * Messages useful to developers to fix problems with the io class.
    In this case, please use :meth:`logger.debug`.

A log handler is automatically added to :mod:`neo`, so please do not user your own handler.
Please use the :attr:`class.logger` attribute for accessing the logger inside the class rather than :meth:`logging.getLogger`.
Please do not log directly to the root logger (e.g. :meth:`logging.warning`), use the class's logger instead (:meth:`class.logger.warning`).
In the tests for the io class, if you intentionally test broken files, please disable logs by setting the logging level to `100`.


ExampleIO
=========

.. autoclass:: neo.rawio.ExampleRawIO

.. autoclass:: neo.io.ExampleIO

Here are the entire files:

.. literalinclude:: ../../neo/rawio/examplerawio.py

.. literalinclude:: ../../neo/io/exampleio.py


.. _`mailing list`: https://groups.google.com/forum/#!forum/neuralensemble
.. _gin-gnode: https://gin.g-node.org/NeuralEnsemble/ephy_testing_data