|
@@ -1,963 +0,0 @@
|
|
|
-
|
|
|
-=============
|
|
|
-odML Tutorial
|
|
|
-=============
|
|
|
-
|
|
|
-:Author:
|
|
|
- Lyuba Zehl;
|
|
|
- based on work by Hagen Fritsch
|
|
|
-:Release:
|
|
|
- 1.4
|
|
|
-:License:
|
|
|
- Creative Commons Attribution-ShareAlike 4.0 International
|
|
|
- `License <http://creativecommons.org/licenses/by-sa/4.0/>`_
|
|
|
-
|
|
|
--------------------------------------------------------------------------------
|
|
|
-
|
|
|
-odML (open metadata Markup Language)
|
|
|
-====================================
|
|
|
-
|
|
|
-odML (open metadata Markup Language) is a framework, proposed by `Grewe et al.
|
|
|
-(2011) <http://journal.frontiersin.org/article/10.3389/fninf.2011.00016/full>`_,
|
|
|
-to organize and store experimental metadata in a human- and machine-readable,
|
|
|
-XML based format (odml). In this tutorial we will illustrate the conceptual
|
|
|
-design of the odML framework and show hands-on how you can generate your own
|
|
|
-odML metadata file collection. A well organized metadata management of your
|
|
|
-experiment is a key component to guarantee the reproducibility of your research
|
|
|
-and facilitate the provenance tracking of your analysis projects.
|
|
|
-
|
|
|
-What are metadata and why are they needed?
|
|
|
- Metadata are data about data. They describe the conditions under which the
|
|
|
- actual raw-data of an experimental study were acquired. The organization of
|
|
|
- such metadata and their accessibility may sound like a trivial task, and
|
|
|
- most laboratories developed their home-made solutions to keep track of
|
|
|
- their metadata. Most of these solutions, however, break down if data and
|
|
|
- metadata need to be shared within a collaboration, because implicit
|
|
|
- knowledge of what is important and how it is organized is often
|
|
|
- underestimated.
|
|
|
-
|
|
|
- While maintaining the relation to the actual raw-data, odML can help to
|
|
|
- collect all metadata which are usually distributed over several files and
|
|
|
- formats, and to store them unitedly which facilitates sharing data and
|
|
|
- metadata.
|
|
|
-
|
|
|
-Key features of odML
|
|
|
- - open, XML based language, to collect, store and share metadata
|
|
|
- - Machine- and human-readable
|
|
|
- - Python-odML library
|
|
|
- - Interactive odML-Editor
|
|
|
-
|
|
|
--------------------------------------------------------------------------------
|
|
|
-
|
|
|
-Structure of this tutorial
|
|
|
-==========================
|
|
|
-
|
|
|
-The scientific background of the possible user community of odML varies
|
|
|
-enormously (e.g. physics, informatics, mathematics, biology, medicine,
|
|
|
-psychology). Some users will be trained programmers, others probably have never
|
|
|
-learned a programming language.
|
|
|
-
|
|
|
-To cover the different demands of all users, we provide a slow introduction to
|
|
|
-the odML framework that even allows programming beginners to learn the basic
|
|
|
-concepts. We will demonstrate how to generate an odML file and present more
|
|
|
-advanced possibilities of the Python-odML library (e.g., how to search for
|
|
|
-certain metadata or how to integrate existing terminologies).
|
|
|
-
|
|
|
-At the end of this tutorial we will provide a few guidelines that will help you
|
|
|
-to create an odML file structure that is optimised for your individual
|
|
|
-experimental project and complements the special needs of your laboratory.
|
|
|
-
|
|
|
-The code for the example odML files, which we use within this tutorial is part
|
|
|
-of the documentation package (see doc/example_odMLs/).
|
|
|
-
|
|
|
-A summary of available odML terminologies and templates can be found `here
|
|
|
-<http://portal.g-node.org/odml/terminologies/v1.1/terminologies.xml>`_.
|
|
|
-
|
|
|
--------------------------------------------------------------------------------
|
|
|
-
|
|
|
-Download and Installation
|
|
|
-=========================
|
|
|
-
|
|
|
-The odML framework is an open source project of the German Neuroinformatics
|
|
|
-Node (`G-Node <http://www.g-node.org/>`_, `odML project website
|
|
|
-<http://www.g-node.org/projects/odml>`_) of the International Neuroinformatics
|
|
|
-Coordination Facility (`INCF <http://www.g-node.org/>`_). The source code for
|
|
|
-the Python-odML library is available on `GitHub <https://github.com/>`_ under
|
|
|
-the project name `python-odml <https://github.com/G-Node/python-odml>`_.
|
|
|
-
|
|
|
-Dependencies
|
|
|
-------------
|
|
|
-
|
|
|
-The Python-odML library (version 1.4) runs under Python 2.7 or 3.5.
|
|
|
-
|
|
|
-Additionally, the Python-odML library depends on Enum, lxml, pyyaml and rdflib.
|
|
|
-
|
|
|
-When the odML-Python library is installed via pip or the setup.py, these
|
|
|
-packages will be automatically downloaded and installed. Alternatively, they
|
|
|
-can be installed from the OS package manager.
|
|
|
-
|
|
|
-On Ubuntu, the dependency packages are available as ``python-enum`` and
|
|
|
-``python-lxml``.
|
|
|
-
|
|
|
-Note that on Ubuntu 14.04, the latter package additionally requires the
|
|
|
-installation of ``libxml2-dev``, ``libxslt1-dev``, and ``lib32z1-dev``.
|
|
|
-
|
|
|
-
|
|
|
-Installation...
|
|
|
----------------
|
|
|
-
|
|
|
-... via pip:
|
|
|
-************
|
|
|
-
|
|
|
-The simplest way to install the Python-odML library is from `PyPI
|
|
|
-<https://pypi.python.org/>`_ using `pip <https://pip.pypa.io/en/stable/>`_::
|
|
|
-
|
|
|
- $ pip install odml
|
|
|
-
|
|
|
-The appropriate Python dependencies will be automatically
|
|
|
-downloaded and installed.
|
|
|
-
|
|
|
-If you are not familiar with PyPI and pip, please have a look at the available
|
|
|
-online documentation.
|
|
|
-
|
|
|
-Installation
|
|
|
-------------
|
|
|
-
|
|
|
-To download the Python-odML library please either use git and clone the
|
|
|
-repository from GitHub::
|
|
|
-
|
|
|
- $ cd /home/usr/toolbox/
|
|
|
- $ git clone https://github.com/G-Node/python-odml.git
|
|
|
-
|
|
|
-... or if you don't want to use git, download the ZIP file also provided on
|
|
|
-GitHub to your computer (e.g. as above on your home directory under a "toolbox"
|
|
|
-folder).
|
|
|
-
|
|
|
-To install the Python-odML library, enter the corresponding directory and run::
|
|
|
-
|
|
|
- $ cd /home/usr/toolbox/python-odml/
|
|
|
- $ python setup.py install
|
|
|
-
|
|
|
-
|
|
|
-Bugs & Questions
|
|
|
-----------------
|
|
|
-
|
|
|
-Should you find a behaviour that is likely a bug, please file a bug report at
|
|
|
-`the github bug tracker <https://github.com/G-Node/python-odml/issues>`_.
|
|
|
-
|
|
|
-If you have questions regarding the use of the library or the editor, ask
|
|
|
-the question on `Stack Overflow <http://stackoverflow.com/>`_, be sure to tag
|
|
|
-it with `odml` and we'll do our best to quickly solve the problem.
|
|
|
-
|
|
|
--------------------------------------------------------------------------------
|
|
|
-
|
|
|
-Basic knowledge on odML
|
|
|
-=======================
|
|
|
-
|
|
|
-Before we start, it is important to know the basic structure of an odML
|
|
|
-file. Within an odML file metadata are grouped and stored in a
|
|
|
-hierarchical tree structure which consists of three basic odML
|
|
|
-objects.
|
|
|
-
|
|
|
-Document:
|
|
|
- - description: *root of the tree*
|
|
|
- - parent: *no parent*
|
|
|
- - children: *Section*
|
|
|
-
|
|
|
-Section:
|
|
|
- - description: *branches of the tree*
|
|
|
- - parent: *Document or Section*
|
|
|
- - children: *Section and/or Property*
|
|
|
-
|
|
|
-Property:
|
|
|
- - description: *leafs of the tree (contains metadata values)*
|
|
|
- - parent: *Section*
|
|
|
- - children: *none*
|
|
|
-
|
|
|
-
|
|
|
-Each of these odML objects has a certain set of attributes where the
|
|
|
-user can describe the object and its contents. Which attribute belongs
|
|
|
-to which object and what the attributes are used for, is better explained
|
|
|
-in an example odML file (e.g., "THGTTG.odml").
|
|
|
-
|
|
|
-
|
|
|
-A first look
|
|
|
-============
|
|
|
-
|
|
|
-If you want to get familiar with the concept behind the odML framework and how
|
|
|
-to handle odML files in Python, you can have a first look at the example odML
|
|
|
-file provided in the Python-odML library. For this you first need to run the
|
|
|
-python code ("thgttg.py") to generate the example odML file ("THGTTG.odml").
|
|
|
-When using the following commands, make sure you adapt the paths to the
|
|
|
-python-odml module to your owns!::
|
|
|
-
|
|
|
- $ cd /home/usr/.../python-odml
|
|
|
- $ ls doc/example_odMLs
|
|
|
- thgttg.py
|
|
|
- $ python doc/example_odMLs/example_odMLs.py "/home/usr/.../python-odml"
|
|
|
- $ ls doc/example_odMLs
|
|
|
- THGTTG.odml thgttg.py
|
|
|
-
|
|
|
-Now open a Python shell within the Python-odML library directory, e.g. with
|
|
|
-IPython::
|
|
|
-
|
|
|
- $ ipython
|
|
|
-
|
|
|
-In the IPython shell, first import the odml package::
|
|
|
-
|
|
|
- >>> import odml
|
|
|
-
|
|
|
-Second, load the example odML file with the following command lines::
|
|
|
-
|
|
|
- >>> to_load = './doc/example_odMLs/THGTTG.odml'
|
|
|
- >>> odmlEX = odml.load(to_load)
|
|
|
-
|
|
|
-If you open a Python shell outside of the Python-odML library directory, please
|
|
|
-adapt your Python-Path and the path to the "THGTTG.odml" file accordingly.
|
|
|
-
|
|
|
-How you can access the different odML objects and their attributes once you
|
|
|
-loaded an odML file and how you can make use of the attributes is described in
|
|
|
-more detail in the following chapters for each odML object type (Document,
|
|
|
-Section, Property).
|
|
|
-
|
|
|
-How you can create the different odML objects on your own and how to connect
|
|
|
-them to build your own metadata odML file will be described in later chapters.
|
|
|
-Further advanced functions you can use to navigate through your odML files, or to
|
|
|
-create an odML template file, or to make use of common odML terminologies
|
|
|
-provided via `the G-Node repository
|
|
|
-<http://portal.g-node.org/odml/terminologies/v1.1/terminologies.xml>`_ can also
|
|
|
-be found later on in this tutorial.
|
|
|
-
|
|
|
-But now, let us first have a look at the example odML file (THGTTG.odml)!
|
|
|
-
|
|
|
-
|
|
|
-The Document
|
|
|
-------------
|
|
|
-
|
|
|
-If you loaded the example odML file, let's have a first look at the Document::
|
|
|
-
|
|
|
- >>> print odmlEX
|
|
|
- Document 42 {author = D. N. Adams, 2 sections}
|
|
|
-
|
|
|
-As you can see, the printout gives you a short summary of the Document of the
|
|
|
-loaded example odML file.
|
|
|
-
|
|
|
-The print out gives you already the follwing information about the odML file:
|
|
|
-
|
|
|
-- ``Document`` tells you that you are looking at an odML Document
|
|
|
-- ``42`` is the user defined version of this odML file
|
|
|
-- ``{...}`` provides ``author`` and number of attached sections
|
|
|
-- ``author`` states the author of the odML file, "D. N. Adams" in the example case
|
|
|
-- ``2 sections`` tells you that this odML Document has 2 Section directly
|
|
|
- appended
|
|
|
-
|
|
|
-Note that the Document printout tells you nothing about the depth of the
|
|
|
-complete tree structure, because it is not displaying the children of its
|
|
|
-directly attached Sections. It also does not display all Document attributes.
|
|
|
-In total, a Document has the following 4 attributes:
|
|
|
-
|
|
|
-author
|
|
|
- - Returns the author (returned as string) of this odML file.
|
|
|
-
|
|
|
-date
|
|
|
- - Returns ta user defined date. Could for example be used to state
|
|
|
- the date of first creation or the date of last changes.
|
|
|
-
|
|
|
-document
|
|
|
- - Returns the current Document object.
|
|
|
-
|
|
|
-parent
|
|
|
- - Returns the parent object (which is ``None`` for a Document).
|
|
|
-
|
|
|
-repository
|
|
|
- - Returns the URL (returned as string) to a user defined repository of
|
|
|
- terminologies used in this Document. Could be the URL to the G-Node
|
|
|
- terminologies or to a user defined template.
|
|
|
-
|
|
|
-version
|
|
|
- - Returns the user defined version (returned as string) of this odML file.
|
|
|
-
|
|
|
-id
|
|
|
- - id is a UUID (universally unique identifiers) that uniquely identifies
|
|
|
- the current document. If not otherwise specified, this id is automatically
|
|
|
- created and assigned.
|
|
|
-
|
|
|
-Let's check out all attributes with the following commands::
|
|
|
-
|
|
|
- >>> print(odmlEX.author)
|
|
|
- D. N. Adams
|
|
|
- >>> print(odmlEX.date)
|
|
|
- 1979-10-12
|
|
|
- >>> print(odmlEX.document)
|
|
|
- Document 42 {author = D. N. Adams, 2 sections}
|
|
|
- >>> print(odmlEX.parent)
|
|
|
- None
|
|
|
- >>> print(odmlEX.repository)
|
|
|
- http://portal.g-node.org/odml/terminologies/v1.1/terminologies.xml
|
|
|
- >>> print(odmlEX.version)
|
|
|
- 42
|
|
|
-
|
|
|
-As expected for a Document, the attributes author and version match the
|
|
|
-information given in the Document printout, the document attribute just returns
|
|
|
-the Document, and the parent attribute is ``None``.
|
|
|
-
|
|
|
-As you learned in the beginning, Sections can be attached to a Document. They
|
|
|
-represent the next hierarchy level of an odML file. Let's have a look which
|
|
|
-Sections were attached to the Document of our example odML file using the
|
|
|
-following command::
|
|
|
-
|
|
|
- >>> print(odmlEX.sections)
|
|
|
- [Section[4|2] {name = TheCrew, type = crew, id = ...},
|
|
|
- Section[1|7] {name = TheStarship, type = starship, id = ...}]
|
|
|
-
|
|
|
-As expected from the Document printout our example contains two Sections. The
|
|
|
-printout and attributes of a Section are explained in the next chapter.
|
|
|
-
|
|
|
-
|
|
|
-The Sections
|
|
|
-------------
|
|
|
-
|
|
|
-There are several ways to access Sections. You can either call them by name or
|
|
|
-by index using either explicitly the function that returns the list of
|
|
|
-Sections (see last part of `The Document`_ chapter) or using again a short cut
|
|
|
-notation. Let's test all the different ways to access a Section, by having a
|
|
|
-look at the first Section in the sections list attached to the Document in our
|
|
|
-example odML file::
|
|
|
-
|
|
|
- >>> print(odmlEX.sections['TheCrew'])
|
|
|
- Section[4|2] {name = TheCrew, type = crew, id = ...}
|
|
|
- >>> print(odmlEX.sections[0])
|
|
|
- Section[4|2] {name = TheCrew, type = crew, id = ...}
|
|
|
- >>> print(odmlEX['TheCrew'])
|
|
|
- Section[4|2] {name = TheCrew, type = crew, id = ...}
|
|
|
- >>> print(odmlEX[0])
|
|
|
- Section[4|2] {name = TheCrew, type = crew, id = ...}
|
|
|
-
|
|
|
-In the following we will call Sections explicitly by their name using the
|
|
|
-short cut notation.
|
|
|
-
|
|
|
-The printout of a Section is similar to the Document printout and gives you
|
|
|
-already the following information:
|
|
|
-
|
|
|
-- ``Section`` tells you that you are looking at an odML Section
|
|
|
-- ``[4|2]`` states that this Section has four Sections and two Properties directly attached to it
|
|
|
-- ``{...}`` provides ``name``, ``type`` and ``id`` of the Section
|
|
|
-- ``name`` is the name of this Section, 'TheCrew' in the example case
|
|
|
-- ``type`` provides the type of the Section, 'crew' in the example case
|
|
|
-- ``id`` provides the uuid of the Section, the actual value has been omitted in the example to improve readability.
|
|
|
-
|
|
|
-
|
|
|
-Note that the Section printout tells you nothing about the depth of a possible
|
|
|
-sub-Section tree below the directly attached ones. It also only list the type
|
|
|
-of the Section as one of the Section attributes. In total, a Section can be
|
|
|
-defined by the following 5 attributes:
|
|
|
-
|
|
|
-name
|
|
|
- - Returns the name of this Section. Should indicate what kind of
|
|
|
- information can be found in this Section.
|
|
|
-
|
|
|
-definition
|
|
|
- - Returns the definition of the content within this Section. Should
|
|
|
- describe what kind of information can be found in this Section.
|
|
|
-
|
|
|
-document
|
|
|
- - Returns the Document to which this Section belongs to. Note that this
|
|
|
- attribute is set automatically for a Section and all its children when
|
|
|
- it is attached to a Document.
|
|
|
-
|
|
|
-parent
|
|
|
- - Returns the parent to which this Section was directly attached to. Can be
|
|
|
- either a Document or another Section.
|
|
|
-
|
|
|
-type
|
|
|
- - Returns the classification type which allows to connect related Sections
|
|
|
- due to a superior semantic context.
|
|
|
-
|
|
|
-reference
|
|
|
- - Returns a reference that can be used to state the origin or source file
|
|
|
- of the metadata stored in the Properties that are grouped by this
|
|
|
- Section.
|
|
|
-
|
|
|
-repository
|
|
|
- - Returns the URL (returned as string) to a user defined repository of
|
|
|
- terminologies used in this Document. Could be the URL to the G-Node
|
|
|
- terminologies or to a user defined template.
|
|
|
-
|
|
|
-id
|
|
|
- - id is a UUID (universally unique identifiers) that uniquely identifies
|
|
|
- the current section. If not otherwise specified, this id is automatically
|
|
|
- created and assigned.
|
|
|
-
|
|
|
-Let's have a look at the attributes for the Section 'TheCrew'::
|
|
|
-
|
|
|
- >>> print(odmlEX['TheCrew'].name)
|
|
|
- TheCrew
|
|
|
- >>> print(odmlEX['TheCrew'].definition)
|
|
|
- Information on the crew
|
|
|
- >>> print(odmlEX['TheCrew'].document)
|
|
|
- Document 42 {author = D. N. Adams, 2 sections}
|
|
|
- >>> print(odmlEX['TheCrew'].parent)
|
|
|
- Document 42 {author = D. N. Adams, 2 sections}
|
|
|
- >>> print(odmlEX['TheCrew'].type)
|
|
|
- crew
|
|
|
- >>> print(odmlEX['TheCrew'].reference)
|
|
|
- None
|
|
|
- >>> print(odmlEX['TheCrew'].repository)
|
|
|
- None
|
|
|
- >>> print(odmlEX['TheCrew'].id)
|
|
|
- None
|
|
|
-
|
|
|
-As expected for this Section, the name and type attribute match the information
|
|
|
-given in the Section printout, and the document and parent attribute return the
|
|
|
-same object, namely our example Document.
|
|
|
-
|
|
|
-To see which Sections are directly attached to the Section 'TheCrew' again use
|
|
|
-the following command::
|
|
|
-
|
|
|
- >>> print(odmlEX['TheCrew'].sections)
|
|
|
- [Section[0|5] {name = Arthur Philip Dent, type = crew/person, id = ...},
|
|
|
- Section[0|5] {name = Zaphod Beeblebrox, type = crew/person, id = ...},
|
|
|
- Section[0|5] {name = Tricia Marie McMillan, type = crew/person, id = ...},
|
|
|
- Section[0|5] {name = Ford Prefect, type = crew/person, id = ...}]
|
|
|
-
|
|
|
-Or, for accessing these sub-Sections::
|
|
|
-
|
|
|
- >>> print(odmlEX['TheCrew'].sections['Ford Prefect'])
|
|
|
- Section[0|5] {name = Ford Prefect, type = crew/person, id = ...}
|
|
|
- >>> print(odmlEX['TheCrew'].sections[3])
|
|
|
- Section[0|5] {name = Ford Prefect, type = crew/person, id = ...}
|
|
|
- >>> print(odmlEX['TheCrew']['Ford Prefect'])
|
|
|
- Section[0|5] {name = Ford Prefect, type = crew/person, id = ...}
|
|
|
- >>> print(odmlEX['TheCrew'][3])
|
|
|
- Section[0|5] {name = Ford Prefect, type = crew/person, id = ...}
|
|
|
-
|
|
|
-As you learned, besides sub-Sections, a Section can also have Properties
|
|
|
-attached. Let's see which Properties are attached to the Section 'TheCrew'::
|
|
|
-
|
|
|
- >>> print(odmlEX['TheCrew'].properties)
|
|
|
- [Property: {name = NameCrewMembers},
|
|
|
- Property: {name = NoCrewMembers}]
|
|
|
-
|
|
|
-The printout and attributes of a Property are explained in the next chapter.
|
|
|
-
|
|
|
-
|
|
|
-The Properties
|
|
|
---------------
|
|
|
-
|
|
|
-Properties need to be called explicitly via the properties function of a
|
|
|
-Section. You can then either call a Property by name or by index::
|
|
|
-
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'])
|
|
|
- Property: {name = NoCrewMembers}
|
|
|
- >>> print(odmlEX['TheCrew'].properties[1])
|
|
|
- Property: {name = NoCrewMembers}
|
|
|
-
|
|
|
-In the following we will only call Properties explicitly by their name.
|
|
|
-
|
|
|
-The Property printout is reduced and only gives you information about the
|
|
|
-following:
|
|
|
-
|
|
|
-- ``Property`` tells you that you are looking at an odML Property
|
|
|
-- ``{...}`` provides the ``name`` of the Property
|
|
|
-- ``NoCrewMembers`` is the name of this Property
|
|
|
-
|
|
|
-Note that the Property printout tells you nothing about the number of Values,
|
|
|
-and very little about the Property attributes. In total, a Property can be
|
|
|
-defined by the following 9 attributes:
|
|
|
-
|
|
|
-name
|
|
|
- - Returns the name of the Property. Should indicate what kind of metadata
|
|
|
- are stored in this Property.
|
|
|
-
|
|
|
-definition
|
|
|
- - Returns the definition of this Property. Should describe what kind of
|
|
|
- metadata are stored in this Property.
|
|
|
-
|
|
|
-document
|
|
|
- - Returns the Document to which the parent Section of this Property belongs
|
|
|
- to. Note that this attribute is set automatically for a Section and all
|
|
|
- its children when it is attached to a Document.
|
|
|
-
|
|
|
-parent
|
|
|
- - Returns the parent Section to which this Property was attached to.
|
|
|
-
|
|
|
-values
|
|
|
- - Returns the metadata of this Property. Can be either a single metadata or
|
|
|
- multiple, but homogeneous metadata (all with same dtype and unit). For
|
|
|
- this reason, the output is always provided as a list.
|
|
|
-
|
|
|
-dtype
|
|
|
- - Returns the odml data type of the stored metadata.
|
|
|
-
|
|
|
-unit
|
|
|
- - Returns the unit of the stored metadata.
|
|
|
-
|
|
|
-uncertainty
|
|
|
- - recommended
|
|
|
- - Can be used to specify the uncertainty of the given metadata value.
|
|
|
-
|
|
|
-reference
|
|
|
- - Returns a reference that can be used to state an external definition
|
|
|
- of the metadata value.
|
|
|
-
|
|
|
-dependency
|
|
|
- - optional
|
|
|
- - A name of another Property within the same section, which this property
|
|
|
- depends on.
|
|
|
-
|
|
|
-dependency_value
|
|
|
- - optional
|
|
|
- - Value of the other Property specified in the 'dependency' attribute on
|
|
|
- which this Property depends on.
|
|
|
-
|
|
|
-value_origin
|
|
|
- - A reference to state the origin of the metadata value e.g. a file name.
|
|
|
-
|
|
|
-Let's check which attributes were defined for the Property 'NoCrewMembers'::
|
|
|
-
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].name)
|
|
|
- NoCrewMembers
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].definition)
|
|
|
- Number of crew members
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].document)
|
|
|
- Document 42 {author = D. N. Adams, 2 sections}
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].values)
|
|
|
- [4]
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].dtype)
|
|
|
- int
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].unit)
|
|
|
- None
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].uncertainty)
|
|
|
- 1
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].reference)
|
|
|
- The Hitchhiker's guide to the Galaxy (novel)
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].dependency)
|
|
|
- None
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NoCrewMembers'].dependency_value)
|
|
|
- None
|
|
|
-
|
|
|
-As mentioned the values attribute of a Property can only contain multiple
|
|
|
-metadata when they have the same ``dtype`` and ``unit``, as it is the case for
|
|
|
-the Property 'NameCrewMembers'::
|
|
|
-
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NameCrewMembers'].values)
|
|
|
- ['Arthur Philip Dent',
|
|
|
- 'Zaphod Beeblebrox',
|
|
|
- 'Tricia Marie McMillan',
|
|
|
- 'Ford Prefect']
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NameCrewMembers'].dtype)
|
|
|
- person
|
|
|
- >>> print(odmlEX['TheCrew'].properties['NameCrewMembers'].unit)
|
|
|
- None
|
|
|
-
|
|
|
-NOTE: 'property.values' will always return a copy! Any direct changes to the
|
|
|
-returned list will have no affect on the actual property values. If you want to
|
|
|
-make changes to a property value, either use the 'append', 'extend' and 'remove'
|
|
|
-methods or assign a new value list to the property.
|
|
|
-
|
|
|
-
|
|
|
--------------------------------------------------------------------------------
|
|
|
-
|
|
|
-Generating an odML-file
|
|
|
-=======================
|
|
|
-
|
|
|
-After getting familiar with the different odML objects and their attributes,
|
|
|
-you will now learn how to generate your own odML file by reproducing some parts
|
|
|
-of the example THGTTG.odml.
|
|
|
-
|
|
|
-We will show you first how to create the different odML objects with their
|
|
|
-attributes. Please note that some attributes are obligatory, some are
|
|
|
-recommended and others are optional when creating the corresponding odML
|
|
|
-objects. A few are automatically generated in the process of creating an odML
|
|
|
-file. Furthermore, all attributes of an odml object can be edited at any time.
|
|
|
-
|
|
|
-If you opened a new IPython shell, please import first again the odml package::
|
|
|
-
|
|
|
- >>> import odml
|
|
|
-
|
|
|
-
|
|
|
-Create a document
|
|
|
------------------
|
|
|
-
|
|
|
-Let's start by creating the Document. Note that none of the Document attributes
|
|
|
-are obligatory::
|
|
|
-
|
|
|
- >>> MYodML = odml.Document()
|
|
|
-
|
|
|
-You can check if your new Document contains actually what you created by using
|
|
|
-some of the commands you learned before::
|
|
|
-
|
|
|
- >>> MYodML
|
|
|
- >>> Document None {author = None, 0 sections}
|
|
|
-
|
|
|
-As you can see, we created an "empty" Document where the version and the author
|
|
|
-attributes are not defined and no section is yet attached. You will learn how to create
|
|
|
-and add a Section to a Document in the next chapter. Let's focus here on defining
|
|
|
-the Document attributes::
|
|
|
-
|
|
|
- >>> MYodML.author = 'D. N. Adams'
|
|
|
- >>> MYodML.version = 42
|
|
|
-
|
|
|
-For the date attribute you require a datetime object as entry. For this reason,
|
|
|
-you need to first import the Python package datetime::
|
|
|
-
|
|
|
- >>> import datetime as dt
|
|
|
-
|
|
|
-Now, let's define the date attribute of the Document::
|
|
|
-
|
|
|
- >>> MYodML.date = dt.date(1979, 10, 12)
|
|
|
-
|
|
|
-Next, let us also add a repository attribute. Exemplary, we can import the
|
|
|
-Python package os to extract the absolute path to our previously used example
|
|
|
-odML file and add this as repository::
|
|
|
-
|
|
|
- >>> import os
|
|
|
- >>> url2odmlEX = 'file:///' + os.path.abspath(to_load)
|
|
|
- >>> MYodML.repository = url2odmlEX
|
|
|
-
|
|
|
-The document and parent attribute are automatically set and should not be
|
|
|
-fiddled with.
|
|
|
-
|
|
|
-Check if your new Document contains actually all attributes now::
|
|
|
-
|
|
|
- >>> print(MYodML.author)
|
|
|
- D. N. Adams
|
|
|
- >>> print(MYodML.date)
|
|
|
- 1979-10-12
|
|
|
- >>> print(MYodML.document)
|
|
|
- Document 42 {author = D. N. Adams, 0 sections}
|
|
|
- >>> print(MYodML.parent)
|
|
|
- None
|
|
|
- >>> print(MYodML.repository)
|
|
|
- file:///home/usr/.../python-odml/doc/example_odMLs/THGTTG.odml
|
|
|
- >>> print(MYodML.version)
|
|
|
- 42
|
|
|
-
|
|
|
-Note that you can also define all attributes when first creating a Document::
|
|
|
-
|
|
|
- >>> MYodML = odml.Document(author='D. N. Adams',
|
|
|
- version=42,
|
|
|
- date=dt.date(1979, 10, 12),
|
|
|
- repository=url2odmlEX)
|
|
|
-
|
|
|
-Our new created Document is, though, still "empty", because it does not contain
|
|
|
-yet Sections. Let's change this!
|
|
|
-
|
|
|
-
|
|
|
-Create a section
|
|
|
-----------------
|
|
|
-
|
|
|
-We now create a Section by reproducing the Section "TheCrew" of the example
|
|
|
-odml file from the beginning::
|
|
|
-
|
|
|
- >>> sec1 = odml.Section(name="TheCrew",
|
|
|
- definition="Information on the crew",
|
|
|
- type="crew")
|
|
|
-
|
|
|
-Note that only the attribute name is obligatory. The attributes definition and
|
|
|
-type are recommended, but could be either not defined at all or defined later
|
|
|
-on.
|
|
|
-
|
|
|
-Let us now attach this Section to our previously generated Document. With this,
|
|
|
-the attribute document and parent of our new Section are automatically
|
|
|
-updated::
|
|
|
-
|
|
|
- >>> MYodML.append(sec1)
|
|
|
-
|
|
|
- >>> print(MYodML)
|
|
|
- Document 42 {author = D. N. Adams, 1 sections}
|
|
|
- >>> print(MYodML.sections)
|
|
|
- [Section[0|0] {name = TheCrew, type = crew, id = ...}]
|
|
|
-
|
|
|
- >>> print(sec1.document)
|
|
|
- Document 42 {author = D. N. Adams, 1 sections}
|
|
|
- >>> print(sec1.parent)
|
|
|
- Document 42 {author = D. N. Adams, 1 sections}
|
|
|
-
|
|
|
-It is also possible to connect a Section directly to a parent object.
|
|
|
-Let's try this with the next Section we create::
|
|
|
-
|
|
|
- >>> sec2 = odml.Section(name="Arthur Philip Dent",
|
|
|
- definition="Information on Arthur Dent",
|
|
|
- type="crew/person",
|
|
|
- parent=sec1)
|
|
|
-
|
|
|
- >>> print(sec2)
|
|
|
- Section[0|0] {name = Arthur Philip Dent, type = crew/person, id = ...}
|
|
|
-
|
|
|
- >>> print(sec2.document)
|
|
|
- Document 42 {author = D. N. Adams, 1 sections}
|
|
|
- >>> print(sec2.parent)
|
|
|
- [Section[1|0] {name = TheCrew, type = crew, id = ...}
|
|
|
-
|
|
|
-Note that all of our created Sections do not contain any Properties yet. Let's
|
|
|
-see if we can change this...
|
|
|
-
|
|
|
-
|
|
|
-Create a Property:
|
|
|
-------------------
|
|
|
-
|
|
|
-Let's create our first Property::
|
|
|
-
|
|
|
- >>> prop1 = odml.Property(name="Gender",
|
|
|
- definition="Sex of the subject",
|
|
|
- values="male")
|
|
|
-
|
|
|
-Note that again, only the name attribute is obligatory for creating a Property.
|
|
|
-The remaining attributes can be defined later on, or are automatically
|
|
|
-generated in the process.
|
|
|
-
|
|
|
-If a value is defined, but the dtype is not, as it is the case for our example
|
|
|
-above, the dtype is deduced automatically::
|
|
|
-
|
|
|
- >>> print(prop1.dtype)
|
|
|
- string
|
|
|
-
|
|
|
-Generally, you can use the following odML data types to describe the format of
|
|
|
-the stored metadata:
|
|
|
-
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| dtype | required data examples |
|
|
|
-+===================================+=======================================+
|
|
|
-| odml.DType.int or 'int' | 42 |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.float or 'float' | 42.0 |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.boolean or 'boolean' | True or False |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.string or 'string' | 'Earth' |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.date or 'date' | dt.date(1979, 10, 12) |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.datetime or 'datetime' | dt.datetime(1979, 10, 12, 11, 11, 11) |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.time or 'time' | dt.time(11, 11, 11) |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.person or 'person' | 'Zaphod Beeblebrox' |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.text or 'text' | 'any text containing \n linebreaks' |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.url or 'url' | "https://en.wikipedia.org/wiki/Earth" |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-| odml.DType.tuple | "(39.12; 67.19)" |
|
|
|
-+-----------------------------------+---------------------------------------+
|
|
|
-
|
|
|
-The available types are implemented in the 'odml.dtypes' Module. Note that the
|
|
|
-last four data types, if not defined, cannot be deduced, but are instead
|
|
|
-always interpreted as string.
|
|
|
-
|
|
|
-If we append now our new Property to the previously created sub-Section
|
|
|
-'Arthur Philip Dent', the Property will also inherit the document attribute and
|
|
|
-automatically update its parent attribute::
|
|
|
-
|
|
|
- >>> MYodML['TheCrew']['Arthur Philip Dent'].append(prop1)
|
|
|
-
|
|
|
- >>> print(prop1.document)
|
|
|
- Document 42 {author = D. N. Adams, 1 sections}
|
|
|
- >>> print(prop1.parent)
|
|
|
- Section[0|1] {name = Arthur Philip Dent, type = crew/person, id = ...}
|
|
|
-
|
|
|
-Next, let us create a Property with multiple metadata entries::
|
|
|
-
|
|
|
- >>> prop2 = odml.Property(name="NameCrewMembers",
|
|
|
- definition="List of crew members names",
|
|
|
- values=["Arthur Philip Dent",
|
|
|
- "Zaphod Beeblebrox",
|
|
|
- "Tricia Marie McMillan",
|
|
|
- "Ford Prefect"],
|
|
|
- dtype=odml.DType.person)
|
|
|
-
|
|
|
-As you learned before, in such a case, the metadata entries must be
|
|
|
-homogeneous! That means they have to be of the same dtype, unit, and
|
|
|
-uncertainty (here ``odml.DType.person``, None, and None, respectively).
|
|
|
-
|
|
|
-To further build up our odML file, let us attach now this new Property to the
|
|
|
-previously created Section 'TheCrew'::
|
|
|
-
|
|
|
- >>> MYodML['TheCrew'].append(prop2)
|
|
|
-
|
|
|
-Note that it is also possible to add a metadata entry later on::
|
|
|
-
|
|
|
- >>> prop2.append("Blind Passenger")
|
|
|
- >>> print(MYodML['TheCrew'].properties['NameCrewMembers'].values)
|
|
|
- ['Arthur Philip Dent',
|
|
|
- 'Zaphod Beeblebrox',
|
|
|
- 'Tricia Marie McMillan',
|
|
|
- 'Ford Prefect',
|
|
|
- 'Blind Passenger']
|
|
|
-
|
|
|
-
|
|
|
-The tuple datatype you might have noticed in the dtype table above has to be
|
|
|
-specially handled. It is intended to enforce a specific number of datapoints
|
|
|
-for each value entry. This is useful in case of 2D or 3D data, where all
|
|
|
-datapoints always have to be present for each entry.
|
|
|
-The dtype itself has to contain the number corresponding to the required value
|
|
|
-data points. For the value data points themselves, they have to be enclosed
|
|
|
-by brackets and separated by a semicolon.
|
|
|
-
|
|
|
- >>> pixel_prop = odml.Property(name="pixel map")
|
|
|
- >>> pixel_prop.dtype = "2-tuple"
|
|
|
- >>> pixel_prop.values = ["(1; 2)", "(3; 4)"]
|
|
|
-
|
|
|
- >>> voxel_prop = odml.Property(name="voxel map")
|
|
|
- >>> voxel_prop.dtype = "3-tuple"
|
|
|
- >>> voxel_prop.values = "(1; 2; 3)"
|
|
|
-
|
|
|
-Please note, that inconsistent tuple values will raise an error:
|
|
|
-
|
|
|
- >>> tprop = odml.Property(name="tuple fail")
|
|
|
- >>> tprop.dtype = "3-tuple"
|
|
|
- >>> tprop.values = ["(1; 2)"]
|
|
|
-
|
|
|
-
|
|
|
-Printing XML-representation of an odML file:
|
|
|
---------------------------------------------
|
|
|
-
|
|
|
-Although the XML-representation of an odML file is a bit hard to read, it is
|
|
|
-sometimes helpful to check, especially during a generation process, how the
|
|
|
-hierarchical structure of the odML file looks like.
|
|
|
-
|
|
|
-Let's have a look at the XML-representation of our small odML file we just
|
|
|
-generated::
|
|
|
-
|
|
|
- >>> print(odml.tools.xmlparser.XMLWriter(MYodML))
|
|
|
- <odML version="1.1">
|
|
|
- <date>1979-10-12</date>
|
|
|
- <section>
|
|
|
- <definition>Information on the crew</definition>
|
|
|
- <property>
|
|
|
- <definition>List of crew members names</definition>
|
|
|
- <name>NameCrewMembers</name>
|
|
|
- <type>person</type>
|
|
|
- <value>[Arthur Philip Dent,Zaphod Beeblebrox,Tricia Marie McMillan,Ford Prefect,Blind Passenger ]</value>
|
|
|
- </property>
|
|
|
- <name>TheCrew</name>
|
|
|
- <section>
|
|
|
- <definition>Information on Arthur Dent</definition>
|
|
|
- <property>
|
|
|
- <definition>Sex of the subject</definition>
|
|
|
- <name>Gender</name>
|
|
|
- <type>string</type>
|
|
|
- <value>[male ]</value>
|
|
|
- </property>
|
|
|
- <name>Arthur Philip Dent</name>
|
|
|
- <type>crew/person</type>
|
|
|
- </section>
|
|
|
- <type>crew</type>
|
|
|
- </section>
|
|
|
- <version>42</version>
|
|
|
- <repository>file:///home/zehl/Projects/toolbox/python-odml/doc/example_odMLs/THGTTG.odml</repository>
|
|
|
- <author>D. N. Adams</author>
|
|
|
- </odML>
|
|
|
-
|
|
|
-
|
|
|
-Saving an odML file:
|
|
|
---------------------
|
|
|
-
|
|
|
-You can save your odML file using the following command::
|
|
|
-
|
|
|
- >>> save_to = '/home/usr/toolbox/python-odml/doc/example_odMLs/myodml.odml'
|
|
|
- >>> odml.save(MYodML, save_to)
|
|
|
-
|
|
|
-
|
|
|
-Loading an odML file:
|
|
|
----------------------
|
|
|
-
|
|
|
-You already learned how to load the example odML file. Here just as a reminder
|
|
|
-you can try to reload your own saved odML file::
|
|
|
-
|
|
|
- >>> my_reloaded_odml = odml.load(save_to)
|
|
|
-
|
|
|
-
|
|
|
--------------------------------------------------------------------------------
|
|
|
-
|
|
|
-
|
|
|
-Advanced odML-Features
|
|
|
-======================
|
|
|
-
|
|
|
-
|
|
|
-Advanced knowledge on Values
|
|
|
-----------------------------
|
|
|
-
|
|
|
-Data type conversions
|
|
|
-*********************
|
|
|
-
|
|
|
-After creating a Property with metadata the data type can be changed and the
|
|
|
-format of the corresponding entry will converted to the new data type, if the
|
|
|
-new format is valid for the given metadata::
|
|
|
-
|
|
|
- >>> test_dtype_conv = odml.Property('p', values=1.0)
|
|
|
- >>> print(test_dtype_conv.values)
|
|
|
- [1.0]
|
|
|
- >>> print(test_dtype_conv.dtype)
|
|
|
- float
|
|
|
- >>> test_dtype_conv.dtype = odml.DType.int
|
|
|
- >>> print(test_dtype_conv.values)
|
|
|
- [1]
|
|
|
- >>> print(test_dtype_conv.dtype)
|
|
|
- int
|
|
|
-
|
|
|
-If the conversion is invalid a ValueError is raised.
|
|
|
-
|
|
|
-Also note, that during such a process metadata loss may occur if a float is
|
|
|
-converted to an integer and then back to a float::
|
|
|
-
|
|
|
- >>> test_dtype_conv = odml.Property('p', values=42.42)
|
|
|
- >>> print(test_dtype_conv.values)
|
|
|
- [42.42]
|
|
|
- >>> test_dtype_conv.dtype = odml.DType.int
|
|
|
- >>> test_dtype_conv.dtype = odml.DType.float
|
|
|
- >>> print(test_dtype_conv.values)
|
|
|
- [42.0]
|
|
|
-
|
|
|
-
|
|
|
-Advanced knowledge on Properties
|
|
|
---------------------------------
|
|
|
-
|
|
|
-Advanced knowledge on Sections
|
|
|
-------------------------------
|
|
|
-
|
|
|
-Links & Includes
|
|
|
-****************
|
|
|
-(DEPRECATED; new version coming soon)
|
|
|
-
|
|
|
-Sections can be linked to other Sections, so that they include their defined
|
|
|
-attributes. A link can be within the document (``link`` property) or to an
|
|
|
-external one (``include`` property).
|
|
|
-
|
|
|
-After parsing a document, these links are not yet resolved, but can be using
|
|
|
-the :py:meth:`odml.doc.BaseDocument.finalize` method::
|
|
|
-
|
|
|
- >>> d = xmlparser.load("sample.odml")
|
|
|
- >>> d.finalize()
|
|
|
-
|
|
|
-Note: Only the parser does not automatically resolve link properties, as the referenced
|
|
|
-sections may not yet be available.
|
|
|
-However, when manually setting the ``link`` (or ``include``) attribute, it will
|
|
|
-be immediately resolved. To avoid this behaviour, set the ``_link`` (or ``_include``)
|
|
|
-attribute instead.
|
|
|
-The object remembers to which one it is linked in its ``_merged`` attribute.
|
|
|
-The link can be unresolved manually using :py:meth:`odml.section.BaseSection.unmerge`
|
|
|
-and merged again using :py:meth:`odml.section.BaseSection.merge`.
|
|
|
-
|
|
|
-Unresolving means to remove sections and properties that do not differ from their
|
|
|
-linked equivalents. This should be done globally before saving using the
|
|
|
-:py:meth:`odml.doc.BaseDocument.clean` method::
|
|
|
-
|
|
|
- >>> d.clean()
|
|
|
- >>> xmlparser.XMLWriter(d).write_file('sample.odml')
|
|
|
-
|
|
|
-Changing a ``link`` (or ``include``) attribute will first unmerge the section and
|
|
|
-then set merge with the new object.
|
|
|
-
|
|
|
-Terminologies
|
|
|
-*************
|
|
|
-(deprecated; new version coming soon)
|
|
|
-odML supports terminologies that are data structure templates for typical use cases.
|
|
|
-Sections can have a ``repository`` attribute. As repositories can be inherited,
|
|
|
-the current applicable one can be obtained using the
|
|
|
-:py:meth:`odml.section.BaseSection.get_repository` method.
|
|
|
-
|
|
|
-To see whether an object has a terminology equivalent, use the
|
|
|
-:py:meth:`odml.property.BaseProperty.get_terminology_equivalent`
|
|
|
-method, which returns the corresponding object of the terminology.
|