.. _run:

DataLad, Run!
-------------

In previous examples, all changes that happened to the dataset or
the files it contains were
saved to the dataset's history by hand. We added larger and smaller
files and saved them, and we also modified smaller file contents and
saved these modifications.

Often, however, files get changed by shell commands
or by scripts.
Consider a `data scientist <https://xkcd.com/1838/>`_.
She has data files with numeric data,
and code scripts in Python, R, Matlab or any other programming language
that will use the data to compute results or figures. Such output is
stored in new files, or modifies existing files.

But only a few weeks after these scripts were executed she finds it hard
to remember which script was modified for which reason or created which
output. How did this result came to be? Which script would she need
to run again on which data to produce this particular figure?

In this section we will experience how DataLad can help
to record the changes in a dataset after execution of a script
from the shell.

Let's say, for example, that you enjoyed the longnow podcasts a lot,
and you start a podcast-night with friends to wind down from all of
the exciting DataLad lectures. They propose to make a
list of speakers and titles to cross out what they've already listened
to, and ask you to prepare such a list.

"Mhh... probably there is a DataLad way to do this... wasn't there also
a note about metadata extraction at some point?" But as we're not that
far into the lectures, you decide to write a simple shell script
to generate a text file that lists speaker and title
name instead.

To do this, we're following a best practice that will reappear in the
later section on `YODA principles <101-123-yoda.html>`_ : Collecting all
additional scripts that work with content of a subdataset *outside*
of this subdataset, in a dedicated ``code/`` directory,
and collating the output of the execution of these scripts
*outside* of the subdataset as well -- and
therefore not modifying the subdataset.

The motivation behind this will become clear in later sections,
but for now we'll start with best-practice building.
Therefore, create a subdirectory ``code/`` in the ``DataLad-101``
superdataset:

.. runrecord:: _examples/DL-101-108-101
   :language: console
   :workdir: dl-101/DataLad-101

   $ mkdir code
   $ tree -d

Inside of ``Datalad-101/code``, create a simple shell script ``list_titles.sh``.
This script will carry out a simple task:
It will loop through the file names of the ``.mp3`` files and
write out speaker names and talk titles in a very simple fashion.
The content of this script is written below -- the ``cat`` command
will write it into the script.

.. runrecord:: _examples/DL-101-108-102
   :language: console
   :workdir: dl-101/DataLad-101

   $ cat << EOT > code/list_titles.sh
   for i in recordings/longnow/Long_Now__Seminars*/*.mp3; do
      # get the filename
      base=\$(basename "\$i");
      # strip the extension
      base=\${base%.mp3};
      # date as yyyy-mm-dd
      printf "\${base%%__*}\t" | tr '_' '-';
      # name and title without underscores
      printf "\${base#*__}\n" | tr '_' ' ';
   done
   EOT

Save this script to the dataset.

.. runrecord:: _examples/DL-101-108-103
   :language: console
   :workdir: dl-101/DataLad-101

   $ datalad status

.. runrecord:: _examples/DL-101-108-104
   :language: console
   :workdir: dl-101/DataLad-101

   $ datalad save -m "Add simple script to write a list of podcast speakers and titles"

Once we run this script, it will simply print dates, names and titles to
your terminal. We can save its outputs to a new file
``recordings/podcasts.tsv`` in the superdataset by redirecting these
outputs with ``bash code/list_titles.sh > recordings/podcasts.tsv``.

Obviously, we could create this file, and subsequently save it to the superdataset.
However, just as in the example about the data scientist,
in a bit of time, we will forget how this file came into existence, or
that the script ``code/list_titles.sh`` is associated with this file, and
can be used to update it later on.

.. index:: ! datalad command; run

The :command:`datalad run` command (:manpage:`datalad-run` manual)
can help with this. It records a command's impact on a dataset.

Let's try the simplest way to use this command: :command:`datalad run`,
followed by a commit message (``-m "a concise summary"``), and the
command that executes the script from the shell: ``bash code/list_titles.sh``.
It is helpful to enclose the command in quotation marks.

Note that we execute the command from the root of the superdataset.
It is recommended to use :command:`datalad run` in the root of the dataset
you want to record the changes in, so make sure to run this
command from the root of ``DataLad-101``.

.. runrecord:: _examples/DL-101-108-105
   :language: console
   :workdir: dl-101/DataLad-101

   $ datalad run -m "create a list of podcast titles" "bash code/list_titles.sh > recordings/podcasts.tsv"

.. findoutmore:: Why is there a "notneeded" in the command summary?

   If you have stumbled across the command execution summary
   ``save (notneeded: 1, ok: 1)`` and wondered what is "notneeded":
   the :command:`datalad save` at the end of a :command:`datalad run` will query all potential
   subdatasets *recursively* for modifications, and as there are no
   modifications in the ``longnow`` subdataset, this part of ``save`` returns
   a "notneeded" summary. Thus, after a :command:`datalad run`, you'll get a
   "notneeded" for every subdataset with no modifications in the execution
   summary.

Let's take a look into the history:

.. runrecord:: _examples/DL-101-108-106
   :language: console
   :workdir: dl-101/DataLad-101
   :lines: 1-30
   :emphasize-lines: 6, 11, 25

   $ git log -p

The commit message we have supplied with ``-m`` directly after :command:`datalad run` appears
in our history as a short summary.
Additionally, the output of the command, ``recordings/podcasts.tsv``,
was saved right away.

But there is more in this log entry, a section in between the markers


``=== Do not change lines below ===`` and

``^^^ Do not change lines above ^^^``.

This is the so-called ``run record`` -- a recording of all of the
information in the :command:`datalad run` command, generated by DataLad.
In this case, it is a very simple summary. One informative
part is highlighted:
``"cmd": "bash code/list_titles.sh"`` is the command that was run
in the terminal.
This information therefore maps the command, and with it the script,
to the output file, in one commit. Nice, isn't it?

Arguably, the :term:`run record` is not the most human-readable way to display information.
This representation however is less for the human user (the human user should
rely on their informative commit message), but for DataLad, in particular for the
:command:`datalad rerun` command, which you will see in action shortly.

Note that any :command:`datalad run` command that does not result in any changes
in a dataset (no modification of existing content; no additional files)
will not produce any record in the dataset's history. Try to run the exact same
command as before, and check whether anything in your log changes:

.. runrecord:: _examples/DL-101-108-107
   :language: console
   :workdir: dl-101/DataLad-101

   $ datalad run -m "Try again to create a list of podcast titles" "bash code/list_titles.sh > recordings/podcasts.tsv"

.. runrecord:: _examples/DL-101-108-108
   :language: console
   :workdir: dl-101/DataLad-101
   :lines: 1-5
   :emphasize-lines: 2

   $ git log --oneline

The most recent commit is still the :command:`datalad run` command from before,
and there was no second :command:`datalad run` commit created.

