io.rst 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278
  1. ******
  2. Neo IO
  3. ******
  4. .. currentmodule:: neo
  5. Preamble
  6. ========
  7. The Neo :mod:`io` module aims to provide an exhaustive way of loading and saving several widely used data formats in electrophysiology.
  8. The more these heterogeneous formats are supported, the easier it will be to manipulate them as Neo objects in a similar way.
  9. Therefore the IO set of classes propose a simple and flexible IO API that fits many format specifications.
  10. It is not only file-oriented, it can also read/write objects from a database.
  11. At the moment, there are 3 families of IO modules:
  12. 1. for reading closed manufacturers' formats (Spike2, Plexon, AlphaOmega, BlackRock, Axon, ...)
  13. 2. for reading(/writing) formats from open source tools (KlustaKwik, Elan, WinEdr, WinWcp, PyNN, ...)
  14. 3. for reading/writing Neo structure in neutral formats (HDF5, .mat, ...) but with Neo structure inside (NeoHDF5, NeoMatlab, ...)
  15. Combining **1** for reading and **3** for writing is a good example of use: converting your datasets
  16. to a more standard format when you want to share/collaborate.
  17. Introduction
  18. ============
  19. There is an intrinsic structure in the different Neo objects, that could be seen as a hierachy with cross-links. See :doc:`core`.
  20. The highest level object is the :class:`Block` object, which is the high level container able to encapsulate all the others.
  21. A :class:`Block` has therefore a list of :class:`Segment` objects, that can, in some file formats, be accessed individually.
  22. Depending on the file format, i.e. if it is streamable or not, the whole :class:`Block` may need to be loaded, but sometimes
  23. particular :class:`Segment` objects can be accessed individually.
  24. Within a :class:`Segment`, the same hierarchical organisation applies.
  25. A :class:`Segment` embeds several objects, such as :class:`SpikeTrain`,
  26. :class:`AnalogSignal`, :class:`AnaloSignalArray`, :class:`EpochArray`, :class:`EventArray`
  27. (basically, all the different Neo objects).
  28. Depending on the file format, these objects can sometimes be loaded separately, without the need to load the whole file.
  29. If possible, a file IO therefore provides distinct methods allowing to load only particular objects that may be present in the file.
  30. The basic idea of each IO file format is to have, as much as possible, read/write methods for the individual encapsulated objects,
  31. and otherwise to provide a read/write method that will return the object at the highest level of hierarchy
  32. (by default, a :class:`Block` or a :class:`Segment`).
  33. The :mod:`neo.io` API is a balance between full flexibility for the user (all :meth:`read_XXX` methods are enabled)
  34. and simple, clean and understandable code for the developer (few :meth:`read_XXX` methods are enabled).
  35. This means that not all IOs offer the full flexibility for partial reading of data files.
  36. One format = one class
  37. ======================
  38. The basic syntax is as follows. If you want to load a file format that is implemented in a generic :class:`MyFormatIO` class::
  39. >>> from neo.io import MyFormatIO
  40. >>> reader = MyFormatIO(filename="myfile.dat")
  41. you can replace :class:`MyFormatIO` by any implemented class, see :ref:`list_of_io`
  42. Modes
  43. ======
  44. IO can be based on a single file, a directory containing files, or a database.
  45. This is described in the :attr:`mode` attribute of the IO class.
  46. >>> from neo.io import MyFormatIO
  47. >>> print MyFormatIO.mode
  48. 'file'
  49. For *file* mode the *filename* keyword argument is necessary.
  50. For *directory* mode the *dirname* keyword argument is necessary.
  51. Ex:
  52. >>> reader = io.PlexonIO(filename='File_plexon_1.plx')
  53. >>> reader = io.TdtIO(dirname='aep_05')
  54. Supported objects/readable objects
  55. ==================================
  56. To know what types of object are supported by a given IO interface::
  57. >>> MyFormatIO.supported_objects
  58. [Segment , AnalogSignal , SpikeTrain, Event, Spike]
  59. Supported objects does not mean objects that you can read directly. For instance, many formats support :class:`AnalogSignal`
  60. but don't allow them to be loaded directly, rather to access the :class:`AnalogSignal` objects, you must read a :class:`Segment`::
  61. >>> seg = reader.read_segment()
  62. >>> print(seg.analogsignals)
  63. >>> print(seg.analogsignals[0])
  64. To get a list of directly readable objects ::
  65. >>> MyFormatIO.readable_objects
  66. [Segment]
  67. The first element of the previous list is the highest level for reading the file. This mean that the IO has a :meth:`read_segment` method::
  68. >>> seg = reader.read_segment()
  69. >>> type(seg)
  70. neo.core.Segment
  71. All IOs have a read() method that returns a list of :class:`Block` objects (representing the whole content of the file)::
  72. >>> bl = reader.read()
  73. >>> print bl[0].segments[0]
  74. neo.core.Segment
  75. Lazy and cascade options
  76. ========================
  77. In some cases you may not want to load everything in memory because it could be too big.
  78. For this scenario, two options are available:
  79. * ``lazy=True/False``. With ``lazy=True`` all arrays will have a size of zero, but all the metadata will be loaded. lazy_shape attribute is added to all object that
  80. inheritate Quantitities or numpy.ndarray (AnalogSignal, AnalogSignalArray, SpikeTrain) and to object that have array like attributes (EpochArray, EventArray)
  81. In that cases, lazy_shape is a tuple that have the same shape with lazy=False.
  82. * ``cascade=True/False``. With ``cascade=False`` only one object is read (and *one_to_many* and *many_to_many* relationship are not read).
  83. By default (if they are not specified), ``lazy=False`` and ``cascade=True``, i.e. all data is loaded.
  84. Example cascade::
  85. >>> seg = reader.read_segment( cascade=True)
  86. >>> print(len(seg.analogsignals)) # this is N
  87. >>> seg = reader.read_segment(cascade=False)
  88. >>> print(len(seg.analogsignals)) # this is zero
  89. Example lazy::
  90. >>> seg = reader.read_segment(lazy=False)
  91. >>> print(seg.analogsignals[0].shape) # this is N
  92. >>> seg = reader.read_segment(lazy=True)
  93. >>> print(seg.analogsignals[0].shape) # this is zero, the AnalogSignal is empty
  94. >>> print(seg.analogsignals[0].lazy_shape) # this is N
  95. Some IOs support advanced forms of lazy loading, cascading or both (these features are currently limited to the HDF5 IO, which supports both forms).
  96. * For lazy loading, these IOs have a :meth:`load_lazy_object` method that takes a single parameter: a data object previously loaded by the same IO
  97. in lazy mode. It returns the fully loaded object, without links to container objects (Segment etc.). Continuing the lazy example above::
  98. >>> lazy_sig = seg.analogsignals[0] # Empty signal
  99. >>> full_sig = reader.load_lazy_object(lazy_sig)
  100. >>> print(lazy_sig.lazy_shape, full_sig.shape) # Identical
  101. >>> print(lazy_sig.segment) # Has the link to the object "seg"
  102. >>> print(full_sig.segment) # Does not have the link: None
  103. * For lazy cascading, IOs have a :meth:`load_lazy_cascade` method. This method is not called directly when interacting with the IO, but its
  104. presence can be used to check if an IO supports lazy cascading. To use lazy cascading, the cascade parameter is set to ``'lazy'``::
  105. >>> block = reader.read(cascade='lazy')
  106. You do not have to do anything else, lazy cascading is now active for the object you just loaded. You can interact with the object in the same way
  107. as if it was loaded with ``cascade=True``. However, only the objects that are actually accessed are loaded as soon as they are needed::
  108. >>> print(block.channelindexes[0].name) # The first ChannelIndex is loaded
  109. >>> print(block.segments[0].analogsignals[1]) # The first Segment and its second AnalogSignal are loaded
  110. Once an object has been loaded with lazy cascading, it stays in memory::
  111. >>> print(block.segments[0].analogsignals[0]) # The first Segment is already in memory, its first AnalogSignal is loaded
  112. .. _neo_io_API:
  113. Details of API
  114. ==============
  115. The :mod:`neo.io` API is designed to be simple and intuitive:
  116. - each file format has an IO class (for example for Spike2 files you have a :class:`Spike2IO` class).
  117. - each IO class inherits from the :class:`BaseIO` class.
  118. - each IO class can read or write directly one or several Neo objects (for example :class:`Segment`, :class:`Block`, ...): see the :attr:`readable_objects` and :attr:`writable_objects` attributes of the IO class.
  119. - each IO class supports part of the :mod:`neo.core` hierachy, though not necessarily all of it (see :attr:`supported_objects`).
  120. - each IO class has a :meth:`read()` method that returns a list of :class:`Block` objects. If the IO only supports :class:`Segment` reading, the list will contain one block with all segments from the file.
  121. - each IO class that supports writing has a :meth:`write()` method that takes as a parameter a list of blocks, a single block or a single segment, depending on the IO's :attr:`writable_objects`.
  122. - each IO is able to do a *lazy* load: all metadata (e.g. :attr:`sampling_rate`) are read, but not the actual numerical data. lazy_shape attribute is added to provide information on real size.
  123. - each IO is able to do a *cascade* load: if ``True`` (default) all child objects are loaded, otherwise only the top level object is loaded.
  124. - each IO is able to save and load all required attributes (metadata) of the objects it supports.
  125. - each IO can freely add user-defined or manufacturer-defined metadata to the :attr:`annotations` attribute of an object.
  126. If you want to develop your own IO
  127. ==================================
  128. See :doc:`io_developers_guide` for information on how to implement a new IO.
  129. .. _list_of_io:
  130. List of implemented formats
  131. ===========================
  132. .. automodule:: neo.io
  133. Logging
  134. =======
  135. :mod:`neo` uses the standard Python :mod:`logging` module for logging.
  136. All :mod:`neo.io` classes have logging set up by default, although not all classes produce log messages.
  137. The logger name is the same as the full qualified class name, e.g. :class:`neo.io.hdf5io.NeoHdf5IO`.
  138. By default, only log messages that are critically important for users are displayed, so users should not disable log messages unless they are sure they know what they are doing.
  139. However, if you wish to disable the messages, you can do so::
  140. >>> import logging
  141. >>>
  142. >>> logger = logging.getLogger('neo')
  143. >>> logger.setLevel(100)
  144. Some io classes provide additional information that might be interesting to advanced users.
  145. To enable these messages, do the following::
  146. >>> import logging
  147. >>>
  148. >>> logger = logging.getLogger('neo')
  149. >>> logger.setLevel(logging.INFO)
  150. It is also possible to log to a file in addition to the terminal::
  151. >>> import logging
  152. >>>
  153. >>> logger = logging.getLogger('neo')
  154. >>> handler = logging.FileHandler('filename.log')
  155. >>> logger.addHandler(handler)
  156. To only log to the terminal::
  157. >>> import logging
  158. >>> from neo import logging_handler
  159. >>>
  160. >>> logger = logging.getLogger('neo')
  161. >>> handler = logging.FileHandler('filename.log')
  162. >>> logger.addHandler(handler)
  163. >>>
  164. >>> logging_handler.setLevel(100)
  165. This can also be done for individual IO classes::
  166. >>> import logging
  167. >>>
  168. >>> logger = logging.getLogger('neo.io.hdf5io.NeoHdf5IO')
  169. >>> handler = logging.FileHandler('filename.log')
  170. >>> logger.addHandler(handler)
  171. Individual IO classes can have their loggers disabled as well::
  172. >>> import logging
  173. >>>
  174. >>> logger = logging.getLogger('neo.io.hdf5io.NeoHdf5IO')
  175. >>> logger.setLevel(100)
  176. And more detailed logging messages can be enabled for individual IO classes::
  177. >>> import logging
  178. >>>
  179. >>> logger = logging.getLogger('neo.io.hdf5io.NeoHdf5IO')
  180. >>> logger.setLevel(logging.INFO)
  181. The default handler, which is used to print logs to the command line, is stored in :attr:`neo.logging_handler`.
  182. This example changes how the log text is displayed::
  183. >>> import logging
  184. >>> from neo import logging_handler
  185. >>>
  186. >>> formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
  187. >>> logging_handler.setFormatter(formatter)
  188. For more complex logging, please see the documentation for the logging_ module.
  189. .. note:: If you wish to implement more advanced logging as describe in the documentation for the logging_ module or elsewhere on the internet, please do so before calling any :mod:`neo` functions or initializing any :mod:`neo` classes.
  190. This is because the default handler is created when :mod:`neo` is imported, but it is not attached to the :mod:`neo` logger until a class that uses logging is initialized or a function that uses logging is called.
  191. Further, the handler is only attached if there are no handlers already attached to the root logger or the :mod:`neo` logger, so adding your own logger will override the default one.
  192. Additional functions and/or classes may get logging during bugfix releases, so code relying on particular modules not having logging may break at any time without warning.
  193. .. _`logging`: http://docs.python.org/library/logging.html