.. _config2:

Tuning datasets to your needs: Configurations [2]
-------------------------------------------------

As the last section already suggest, within a Git repository,
``.git/config`` is not the only configuration file.
There are also ``.gitmodules`` and ``.gitattributes``, and in DataLad datasets
there also is a ``.datalad/config`` file.

All of these files store configurations, but have an important difference:
They are version controlled, and upon sharing a dataset these configurations
will be shared as well. An example for a shared configuration
is the one that the ``text2git`` configuration template applied:
In the shared copy of your dataset, text files are also saved with Git,
and not Git-annex (see section :ref:`sibling`). The configuration responsible
for this behavior is in a ``.gitattributes`` file, and we'll start this
section by looking into it.

``.gitattributes``
^^^^^^^^^^^^^^^^^^

This file lies right in the root of your superdataset:

.. runrecord:: _examples/DL-101-125-104
   :language: console
   :workdir: dl-101/DataLad-101

   $ cat .gitattributes

This looks neither spectacular nor pretty. Also, it does not follow the ``section-option-value``
organization of the ``.git/config`` file anymore. Instead, there are three lines,
and all of these seem to have something to do with the configuration of Git-annex.
There even is one key word that you recognize: MD5E.
If you have read the hidden section in :ref:`symlink`
you will recognize it as a reference to the type of
key used by Git-annex to identify and store file content in the object-tree.
The first row, ``* annex.backend=MD5E``, therefore translates to "Everything in this
directory should be hashed with a MD5E hash function".
But what is the rest? We'll start with the last row::

   * annex.largefiles=(not(mimetype=text/*))

Uhhh, cryptic. The lecturer explains:

"Git-annex will *annex*, that is, *store in the object-tree*,
anything it considers to be a "large file". By default, anything
in your dataset would be a "large file", that means anything would be annexed.
However, in section :ref:`symlink` I already mentioned that exceptions to this
behavior can be defined based on

#. file size

#. and/or path/pattern, and thus for example file extensions,
   or names, or file types (e.g. text files, as with the
   ``text2git`` configuration template).

"In ``.gitattributes``, you can define what a large file and what is not
by simply telling Git-annex by writing such rules."

What you can see in this ``.gitattribute`` file is a rule based on **file types**:
With ``(not(mimetype=text/*))`` [#f1]_, the ``text2git`` configuration template
configured Git-annex to regard all files of type text **not** as a large file.
Thanks to this little line, your text files are not annexed, but stored
directly in Git.

The patterns ``*`` and ``**`` are so-called "wildcards" used in :term:`globbing`.
``*`` matches any file or directory in the current directory, and ``**`` matches
all files and directories in the current directory *and subdirectories*. In technical
terms, ``**`` matches *recursively*. The third row therefore
translates to "Don't annex anything that is a text file in this directory" for Git-annex.

However, rules can be even simpler. The second row simply takes a complete directory
(``.git``) and instructs Git-annex to regard nothing in it as a "large file".
The second row, ``**/.git* annex.largefiles=nothing`` therefore means that no
``.git`` repository in this directory or a subdirectory should be considered
a "large file". This way, the ``.git`` repositories are protected from being annexed.
If you had a single file (``myfile.pdf``) you wouldn't want annexed, specifying a rule such as::

   myfile.pdf annex.largefiles=nothing

will keep it stored in Git. To see an example of this, navigate into the longnow subdataset,
and view this dataset's ``.gitattributes`` file:

.. runrecord:: _examples/DL-101-125-105
   :language: console
   :workdir: dl-101/DataLad-101

   $ cat recordings/longnow/.gitattributes

The relevant part is ``README.md annex.largefiles=nothing``.
This instructs Git-annex to specifically not annex ``README.md``.

Lastly, if you wanted to configure a rule based on **size**, you could add a row such as::

   ** annex.largefiles(largerthan=20kb)

to store only files exceeding 20KB in size in Git-annex [#f2]_.

As you may have noticed, unlike ``.git/config`` files,
there can be multiple ``.gitattributes`` files within a dataset. So far, you have seen one
in the root of the superdataset, and in the root of the ``longnow`` subdataset.
In principle, you can add one to every directory-level of your dataset.
For example, there is another ``.gitattributes`` file within the
``.datalad`` directory:

.. runrecord:: _examples/DL-101-125-107
   :language: console
   :workdir: dl-101/DataLad-101

   $ cat .datalad/.gitattributes

As with Git configuration files, more specific or lower-level configurations take precedence
over more general or higher-level configurations. Specifications in a subdirectory can
therefore overrule specifications made in the ``.gitattributes`` file of the parent
directory.

In summary, the ``.gitattributes`` files will give you the possibility to configure
what should be annexed and what shouldn't be annexed up to individual file level.
This can be very handy, and allows you to tune your dataset to your custom needs.
For example, files you will often edit by hand could be stored in Git if they are
not too large to ease modifying them [#f3]_.
Once you know the basics of this type of configuration syntax, writing
your own rules is easy. For more tips on how configure Git-annex's content
management in ``.gitattributes``, take a look at `this <https://git-annex.branchable.com/tips/largefiles/>`_
page of the Git-annex documentation.
Later however you will see preconfigured DataLad *procedures* such as ``text2git`` that
can apply useful configurations for you, just as ``text2git`` added the last line
in the root ``.gitattributes`` file.

``.gitmodules``
^^^^^^^^^^^^^^^

On last configuration file we want to look at is the ``.gitmodules`` file.
There is one right in the root of your dataset:

.. runrecord:: _examples/DL-101-125-108
   :language: console
   :workdir: dl-101/DataLad-101

   $ cat .gitmodules

Based on these contents, you might have already guessed what this file
stores. ``.gitmodules`` is a configuration file that stores the mapping between
your own dataset and any subdatasets you have installed in it.
There will be an entry for each submodule (subdataset) in your dataset.
The name *submodule* is Git terminology, and describes a Git repository inside of
another Git repository, i.e. the super- and subdataset principles.
Upon sharing your dataset, the information about subdatasets and where to retrieve
them from is stored and shared with this file.



``.datalad/config``
^^^^^^^^^^^^^^^^^^^

DataLad adds a repository-specific configuration file as well.
It can be found in the ``.datalad`` directory, and just like ``.gitattributes``
and ``.gitmodules`` it is version controlled and is thus shared together with
the dataset. One can configure
`many options <http://docs.datalad.org/en/latest/generated/datalad.config.html>`_,
but currently, our ``.datalad/config`` file only stores a dataset ID:

.. runrecord:: _examples/DL-101-125-106
   :language: console
   :workdir: dl-101/DataLad-101

   $ cat .datalad/config

Note, though, that local configurations within a Git configuration file
will take precedence over configurations that can be distributed with a dataset.
Otherwise, dataset updates with :command:`datalad update` (or, for Git-users,
:command:`git pull`) could suddenly and unintentionally alter local DataLad
behavior that was specifically configured.


Writing to configuration files other than ``.git/config``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

"Didn't you say that knowing git :command:`git config` command is already
half of what I need to know?" you ask. "Now there are three other configuration
files, and I don't know with which command I can write into these files."

"Excellent question", you hear in return, "but in reality, you **do** know:
it's also the :command:`git config` command. The only part of it you need to
adjust is the ``-f``, ``--file`` parameter. By default, the command writes to
a Git config file. But it can write to a different file if you specify it
appropriately. For example

   ``git config --file=.gitmodules --replace-all submodule."name".url "new URL"``

will update your submodule's URL. Keep in mind though that you would need
to commit this change, as ``.gitmodules`` is version controlled".

Let's try this:

.. runrecord:: _examples/DL-101-125-109
   :workdir: dl-101/DataLad-101
   :language: console

   $ git config --file=.gitmodules --replace-all submodule."recordings/longnow".url "git@github.com:datalad-datasets/longnow-podcasts.git"

This command will replace the submodule's https URL with an SSH URL.
The later is often used if someone has an *SSH key pair* and added the
public key to their Github account (you can read more about this
`here <https://help.github.com/en/articles/which-remote-url-should-i-use>`_).
We will revert this change shortly, but use it to show the difference between
a :command:`git config` on a ``.git/config`` file and on a version controlled file:

.. runrecord:: _examples/DL-101-125-110
   :workdir: dl-101/DataLad-101
   :language: console

   $ datalad status

.. runrecord:: _examples/DL-101-125-111
   :workdir: dl-101/DataLad-101
   :language: console

   $ git diff

As these two commands show, the ``.gitmodules`` file is modified. The https URL
has been deleted (note the ``-``, and a SSH URL has been added. To keep these
changes, we would need to :command:`datalad save` them. However, as we want to stay with
https URLs, we will just *checkout* this change -- using a Git tool to undo an
unstaged modification.

.. runrecord:: _examples/DL-101-125-112
   :workdir: dl-101/DataLad-101
   :language: console

   $ git checkout .gitmodules
   $ datalad status

Note, though, that the ``.gitattributes`` file can not be modified with a :command:`git config`
command. This is due to its different format that does not comply to the
``section.variable.value`` structure of all other configuration files. This file, therefore,
has to be edited by hand, with an editor of your choice.

Summary
^^^^^^^

This has been an intense lecture, you have to admit. One definite
take-away from it has been that you now know a second reason why the hidden
``.git`` and ``.datalad`` directory contents and also the contents of ``.gitmodules`` and
``.gitattributes`` shouldn't be carelessly tempered with -- they contain all of
the repositories configurations.

But you now also know how to modify these configurations with enough
care and background knowledge such that nothing should go wrong once you
want to work with and change them. You can use the :command:`git config` command
for Git configuration files on different scopes, and even the ``.gitmodules`` or ``datalad/config``
files. Of course you don't yet know all of the available configuration options. However,
you already know some core Git configurations such as name, email, and editor. Even more
important, you know how to configure Git-annex's content management based on ``largefile``
rules, and you understand the majority of variables within ``.gitmodules`` or the sections
in ``.git/config``. Slowly, you realize with pride,
you're more and more becoming a DataLad power-user.

Write a note about configurations in datasets into ``notes.txt``.

.. runrecord:: _examples/DL-101-125-113
   :workdir: dl-101/DataLad-101
   :language: console

   $ cat << EOT >> notes.txt
   Configurations for datasets exist on different levels
   (systemwide, global, and local), and in different types
   of files (not version controlled (git)config files, or
   version controlled .datalad/config, .gitattributes, or
   gitmodules files). With the exception of .gitattributes,
   all these configuration files share a common structure,
   and can be modified with the git config command, but also
   with an editor by hand.

   Depending on whether a configuration file is version
   controlled or not, the configurations will be shared together
   with the dataset. More specific configurations and not-shared
   configurations will always take precedence over more global or
   shared configurations.

   The git config --list --show-origin command is a useful tool
   to give an overview over existing configurations. Particularly
   important may be the .gitattributes file, in which one can set
   rules for Git-annex about which files should be version-controlled
   with Git instead of being annexed.


   EOT

.. rubric:: Footnotes

.. [#f1] When opening any file on a UNIX system, the file does not need to have a file
         extension (such as ``.txt``, ``.pdf``, ``.jpg``) for the operating system to know
         how to open or use this file (in contrast to Windows, which does not know how to
         open a file without an extension). To do this, Unix systems rely on a file's
         MIME type -- an information about a file's content. A ``.txt`` file for example
         has MIME type ``text/plain`` as does a bash script (``.sh``), a Python
         script has MIME type ``text/x-python``, a ``.jpg`` file is ``image/jpg``, and
         a ``.pdf`` file has MIME type ``application/pdf``. You can find out the MIME type
         of a file by running:

            file --mime-type path/to/file

.. [#f2] Specifying annex.largefiles in your .gitattributes file will make the configuration
         "portable" -- shared copies of your dataset will retain these configurations.
         You could however also set largefiles rules in your ``.git/config`` file. Rules
         specified in there take precendence over rules in ``.gitattributes``. You can set
         them using the :command:`git config` command::

            git config annex.largefiles 'largerthan=100kb and not (include=*.c or include=*.h)'

         The above command annexes files larger than 100KB, and will never annex files with a
         ``.c`` or ``.h`` extension.

.. [#f3] Should you ever need to, this file is also where one would change the Git-annex
         backend in order to store new files with a new backend. Switching the backend of
         *all* files (new as well as existing ones) requires the :command:`git annex migrate`
         (see `the documentation <https://git-annex.branchable.com/git-annex-migrate/>`_ for
         more information on this command).