実験パッケージを準備

.gitignore

@@ -0,0 +1,18 @@
+
+/.cache/
+/.conda/
+/.config/
+/.ipython/
+/.local/
+/.tmp/
+/.bashrc
+/.bash_logout
+/.profile
+/.netrc
+.ipynb_checkpoints/
+.fonts/
+.jupyter/
+.npm/
+.ssh/
+.jupyter-server-log.txt
+

WORKFLOW/enter_metadata.ipynb

@@ -0,0 +1,180 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 実験メタデータを入力する\n",
+    "\n",
+    "実験日や実験者などのメタデータを実験記録に追加します。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. メタデータを入力する"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "1. 以下のセルを実行して、実験日を入力してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ipywidgets import Text, Button\n",
+    "from IPython.display import clear_output\n",
+    "import datetime\n",
+    "\n",
+    "d_today = datetime.date.today()\n",
+    "def on_click_callback(clicked_button: Button) -> None:\n",
+    "    global experiment_date\n",
+    "    experiment_date = text.value\n",
+    "    clear_output()\n",
+    "    print(\"登録しました：\", experiment_date)\n",
+    "\n",
+    "# テキストボックス\n",
+    "text = Text(\n",
+    "    value=str(d_today),\n",
+    "    description='実験日：'\n",
+    ")\n",
+    "button = Button(description='入力完了')\n",
+    "button.on_click(on_click_callback)\n",
+    "display(text, button)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "2. 以下のセルを実行して、実験者を入力してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ipywidgets import Text, Button\n",
+    "from IPython.display import clear_output\n",
+    "import datetime\n",
+    "\n",
+    "def on_click_callback(clicked_button: Button) -> None:\n",
+    "    global experimenter\n",
+    "    experimenter = text.value\n",
+    "    clear_output()\n",
+    "    print(\"登録しました：\", experimenter)\n",
+    "\n",
+    "# テキストボックス\n",
+    "text = Text(\n",
+    "    description='実験者：'\n",
+    ")\n",
+    "button = Button(description='入力完了')\n",
+    "button.on_click(on_click_callback)\n",
+    "display(text, button)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "heading_collapsed": true
+   },
+   "source": [
+    "## 2.メタ情報をファイルに保存する"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "hidden": true
+   },
+   "source": [
+    "### - 2.1 メタ情報をmeta_data.jsonに書き込む"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "hidden": true
+   },
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from collections import OrderedDict\n",
+    "\n",
+    "meta_data = {\n",
+    "    \"experiment_date\": experiment_date,\n",
+    "    \"experimenter\": experimenter\n",
+    "}\n",
+    "with open('../meta_data.json', 'w') as jf:\n",
+    "    json.dump(meta_data, jf, ensure_ascii=False, indent=2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. 実行結果をデータガバナンス機能に同期する\n",
+    "\n",
+    "ここまでの内容を保存し、データガバナンス機能に同期します。  \n",
+    "以下のセルを実行してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from IPython.display import display, Javascript\n",
+    "display(Javascript('IPython.notebook.save_checkpoint();'))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "hidden": true
+   },
+   "outputs": [],
+   "source": [
+    "# FIXME：papermill実装に修正\n",
+    "%cd ~/\n",
+    "!datalad save -m \"メタデータ入力\" SECTIONS/enter_metadata.ipynb\n",
+    "!datalad save -m \"メタデータ入力\" meta_data.json\n",
+    "!datalad unlock .\n",
+    "!datalad push --to gin"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

WORKFLOW/finish.ipynb

@@ -0,0 +1,121 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 実験を終了する\n",
+    "\n",
+    "実験記録をデータガバナンス機能に保存して、実験を終了します。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "heading_collapsed": true
+   },
+   "source": [
+    "## 1.実行環境構成を記録する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "hidden": true
+   },
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "conda env export -n base > ~/environment.yml\n",
+    "pip freeze > ~/requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. 高性能実験環境から実験記録を取得する\n",
+    "\n",
+    "高性能実験環境を利用している場合には、以下のセルを実行してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "tmp = os.environ[\"BINDER_REF_URL\"]\n",
+    "tmp = tmp.split(\"/\")\n",
+    "tmp = tmp[len(tmp) - 1]\n",
+    "repo_name = tmp[:tmp.find(\".\")]\n",
+    "\n",
+    "!rm ~/input_data/*\n",
+    "!rm ~/output_data/*\n",
+    "!rm ~/source/*\n",
+    "\n",
+    "!scp mdx:~/$repo_name/input_data/* ~/input_data/\n",
+    "!scp mdx:~/$repo_name/output_data/* ~/output_data/\n",
+    "!scp mdx:~/$repo_name/source/* ~/source/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. 実験記録をデータガバナンス機能に同期する\n",
+    "\n",
+    "実験記録をデータガバナンス機能に同期します。  \n",
+    "以下のセルを実行してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from IPython.display import display, Javascript\n",
+    "display(Javascript('IPython.notebook.save_checkpoint();'))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "hidden": true
+   },
+   "outputs": [],
+   "source": [
+    "# FIXME：papermill実装に修正する\n",
+    "%cd ~/\n",
+    "!datalad save -m \"実験終了\"\n",
+    "!datalad unlock .\n",
+    "!datalad push --to gin"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

WORKFLOW/images/notebooks.diag

@@ -0,0 +1,42 @@
+
+   blockdiag {
+    node_width = 230;
+    node_height = 145;
+
+        group {
+            orientation = portrait;
+            shape = line;
+            style = none;
+
+            group {
+                shape = line;
+                style = none;
+                orientation = portrait;
+                group {
+                    orientation = portrait;
+                    color = "#ffeed9";
+                    "0. 実験ワークフロー実行準備"[label = "0. 実験ワークフロー実行準備", fontsize = 14];
+                    "required_every_time";
+                }
+            }
+
+            group {
+                shape = line;
+                style = none;
+                orientation = portrait;
+                group {
+                    orientation = portrait;
+                    color = "#fcdcb1";
+                    "1. 実験ワークフロー"[label = "1. 実験ワークフロー", fontsize = 14];
+                    "enter_metadata";
+                    "save";
+                    "finish";
+                    "experiment_on_mdx";
+                }
+            }
+        }
+
+        "0. 実験ワークフロー実行準備" -> "1. 実験ワークフロー";
+        "enter_metadata","save","experiment_on_mdx" -> "finish";
+
+    }

WORKFLOW/param_files/params.json

@@ -0,0 +1,10 @@
+{
+    "siblings": {
+        "ginHttp": "http://dg02.dg.rcos.nii.ac.jp/",
+        "ginSsh": "ssh://root@dg02.dg.rcos.nii.ac.jp:3001/",
+        "gitHugibHttp": "https://github.com/",
+        "gitHubSsh": "git@github.com:"
+    },
+    "rcosBinderUrl": "https://jupyter.cs.rcos.nii.ac.jp"
+}
+

WORKFLOW/save.ipynb

@@ -0,0 +1,113 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 実験を途中保存する\n",
+    "\n",
+    "実験終了前の段階で実験をデータガバナンス機能に途中保存します。  "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. コミットメッセージを変数に入力する\n",
+    "\n",
+    "以下のセルを実行して、実験作業のログとして残す短いメッセージを入力してください。  \n",
+    "※入力値に誤りがある場合、もう一度実行することで訂正ができます。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "message = input()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. 高性能実験環境から実験記録を取得する\n",
+    "\n",
+    "高性能実験環境を利用している場合には、以下のセルを実行してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "tmp = os.environ[\"BINDER_REF_URL\"]\n",
+    "tmp = tmp.split(\"/\")\n",
+    "tmp = tmp[len(tmp) - 1]\n",
+    "repo_name = tmp[:tmp.find(\".\")]\n",
+    "\n",
+    "!rm ~/input_data/*\n",
+    "!rm ~/output_data/*\n",
+    "!rm ~/source/*\n",
+    "\n",
+    "!scp mdx:~/$repo_name/input_data/* ~/input_data/\n",
+    "!scp mdx:~/$repo_name/output_data/* ~/output_data/\n",
+    "!scp mdx:~/$repo_name/source/* ~/source/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. 途中保存する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from IPython.display import display, Javascript\n",
+    "display(Javascript('IPython.notebook.save_checkpoint();'))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# FIXME：papermill実装に修正する\n",
+    "%cd ~/\n",
+    "!datalad save -m $message\n",
+    "!datalad unlock .\n",
+    "!datalad push --to gin"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

WORKFLOW/util/base_datalad_save_push.ipynb

@@ -0,0 +1,91 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# ワークフロー実行結果を書き戻す\n",
+    "\n",
+    "このノートブックでは、変更内容をGINリポジトリに書き戻します。  \n",
+    "※各セクションのセルでpapermillによって自動実行されるノートブックなので手作業での実施は非推奨です。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "parameters"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "SIBLING_NAME = 'gin'\n",
+    "SAVE_MESSAGE = '[GIN] modify on binder'\n",
+    "PATH = '/home/jovyan'\n",
+    "IS_RECURSIVE = True\n",
+    "TO_GIT = False\n",
+    "RESULT_RENDERER = 'default'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1.書き戻しの準備を行う"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datalad import api\n",
+    "\n",
+    "api.save(message=SAVE_MESSAGE, path=PATH, recursive=IS_RECURSIVE, to_git=TO_GIT)\n",
+    "api.unlock(PATH)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2.変更内容を書き戻す"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datalad import api\n",
+    "\n",
+    "api.push(to=SIBLING_NAME, result_renderer=RESULT_RENDERER, path=PATH, recursive=IS_RECURSIVE)"
+   ]
+  }
+ ],
+ "metadata": {
+  "celltoolbar": "Tags",
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

WORKFLOW/util/required_every_time.ipynb

@@ -0,0 +1,424 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "4719bc46",
+   "metadata": {},
+   "source": [
+    "# 必要な準備を行う\n",
+    "\n",
+    "実験中のワークフロー実行のために必要な準備を行います。  \n",
+    "上から順番に実行してください。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "87401770",
+   "metadata": {},
+   "source": [
+    "## 1. Git管理対象外ファイルを.gitignoreで設定する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f2cc773c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "cd ~/\n",
+    "echo \"\n",
+    "/.cache/\n",
+    "/.conda/\n",
+    "/.config/\n",
+    "/.ipython/\n",
+    "/.local/\n",
+    "/.tmp/\n",
+    "/.bashrc\n",
+    "/.bash_logout\n",
+    "/.profile\n",
+    "/.netrc\n",
+    ".ipynb_checkpoints/\n",
+    ".fonts/\n",
+    ".jupyter/\n",
+    ".npm/\n",
+    ".ssh/\n",
+    ".jupyter-server-log.txt\n",
+    "\" > ~/.gitignore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "05ec67b8",
+   "metadata": {},
+   "source": [
+    "## 2. READMEに実験実行環境へのリンクを追加する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2eb51aba",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "%cd ~/\n",
+    "with open('README.md', 'a', newline='\\n') as f:\n",
+    "    f.write(\"\\n## 実験実行環境にアクセスしたい場合\\n以下のリンクをクリックしてください<br>https://jupyter.cs.rcos.nii.ac.jp\" + os.environ[\"JUPYTERHUB_SERVICE_PREFIX\"] + \"notebooks/experiment.ipynb\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dd0a038d",
+   "metadata": {},
+   "source": [
+    "## 3. ユーザー認証を行う\n",
+    "\n",
+    "この手順では、あなたのユーザ情報をシステムに認証させる手続きを行います。  \n",
+    "以下のセルを実行行し、画面の表示に沿ってデータガバナンス機能に登録したユーザー名、パスワード、メールアドレスを入力してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bf28557d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%cd ~/WORKFLOW/scripts\n",
+    "import json\n",
+    "from scripts import utils\n",
+    "\n",
+    "# 以下の認証の手順で用いる、\n",
+    "# GINのドメイン名等をパラメタファイルから取得する\n",
+    "params = {}\n",
+    "with open(utils.fetch_param_file_path(), mode='r') as f:\n",
+    "    params = json.load(f)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "61c68c98",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import time\n",
+    "import getpass\n",
+    "import requests\n",
+    "\n",
+    "from IPython.display import clear_output\n",
+    "from requests.auth import HTTPBasicAuth\n",
+    "from http import HTTPStatus\n",
+    "\n",
+    "# 正常に認証が終わるまで繰り返し\n",
+    "while True:\n",
+    "    name = input(\"ユーザー名：\")\n",
+    "    password = getpass.getpass(\"パスワード：\")\n",
+    "    email = input(\"メールアドレス：\")\n",
+    "    clear_output()\n",
+    "    \n",
+    "    # GIN API Basic Authentication\n",
+    "    # refs: https://docs.python-requests.org/en/master/user/authentication/\n",
+    "    \n",
+    "    # 既存のトークンがあるか確認する\n",
+    "    response = requests.get(params['siblings']['ginHttp']+'api/v1/users/' + name + '/tokens', auth=(name, password))\n",
+    "    tokens = response.json()\n",
+    "\n",
+    "    # 既存のトークンがなければ作成する\n",
+    "    if len(tokens) < 1:\n",
+    "        response = requests.post(params['siblings']['ginHttp']+'api/v1/users/' + name + '/tokens', data={\"name\": \"system-generated\"} ,auth=(name, password))\n",
+    "\n",
+    "    if response.status_code == HTTPStatus.OK or HTTPStatus.CREATED:\n",
+    "        tokens = response.json()\n",
+    "        clear_output()\n",
+    "        print(\"認証が正常に完了しました。次の手順へお進みください。\")\n",
+    "        break\n",
+    "    else:\n",
+    "        clear_output()\n",
+    "        print(\"ユーザ名、またはパスワードが間違っています。\\n恐れ入りますがもう一度ご入力ください。\")\n",
+    "\n",
+    "!git config --global user.name $name\n",
+    "!git config --global user.email $email"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fce82ba7",
+   "metadata": {},
+   "source": [
+    "## 4. データ同期のための設定をする\n",
+    "\n",
+    "この手順では、今の実行環境とデータガバナンス機能のリポジトリでデータの同期をとるための準備をします。  \n",
+    "以下を実行することで、システムがデータ同期の準備の手続きを行います。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e277f2c2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "#!/bin/bash\n",
+    "if [ ! -e ~/.ssh/id_ed25519 ]; then\n",
+    "    # 鍵ペアが無ければ作成\n",
+    "    ssh-keygen -t ed25519 -N \"\" -f ~/.ssh/id_ed25519\n",
+    "fi\n",
+    "\n",
+    "# dataset化されなかったので下のセルに暫定的に修正\n",
+    "#if [ ! -d ~/.datalad/ ]; then\n",
+    "    # Dataladのデータセットでなければデータセット化する\n",
+    "#    datalad create --force /home/jovyan\n",
+    "#fi"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "03dc3f22",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%cd ~/\n",
+    "!datalad create --force ."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8181ce0f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# 公開鍵アップロード\n",
+    "# refs: https://github.com/gogs/docs-api/blob/master/Users/Public%20Keys.md#create-a-public-key\n",
+    "import os\n",
+    "import requests\n",
+    "import time\n",
+    "from http import HTTPStatus\n",
+    "\n",
+    "import json\n",
+    "from scripts import utils\n",
+    "\n",
+    "pubkey = !cat ~/.ssh/id_ed25519.pub\n",
+    "\n",
+    "# 認証時に取得したトークンを使ってPOSTリクエスト\n",
+    "response = requests.post(\n",
+    "                params['siblings']['ginHttp']+'api/v1/user/keys?token=' + tokens[0]['sha1'],\n",
+    "                data={\n",
+    "                    \"title\": \"system-generated-\"+str(time.time()),\n",
+    "                    \"key\": pubkey[0]\n",
+    "                })\n",
+    "msg = response.json()\n",
+    "\n",
+    "# コンテナを消す際にコンテナとつなぐための公開鍵も削除のため、\n",
+    "# パラメータとしてGINから発行された鍵IDを保存\n",
+    "if response.status_code == HTTPStatus.CREATED:\n",
+    "    # params.jsonへの追記（鍵ID）\n",
+    "    params['ginKeyId'] = str(response.json()['id'])\n",
+    "    with open(utils.fetch_param_file_path(), mode='w') as f:\n",
+    "        json.dump(params, f, indent=4)\n",
+    "    print('Public key is ready.')\n",
+    "elif msg['message'] == 'Key content has been used as non-deploy key':\n",
+    "    print('Public key is ready before time.')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "13f1918f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# sibling url をsshに変更する\n",
+    "import os\n",
+    "import tempfile\n",
+    "from datalad import api\n",
+    "from IPython.display import clear_output\n",
+    "\n",
+    "remote_info = 'remote.txt'\n",
+    "\n",
+    "with tempfile.TemporaryDirectory() as tmpdir:\n",
+    "    # 使い終わったらPython側で消してくれるTemporaryDirectoryを利用\n",
+    "    # refs: https://qiita.com/everylittle/items/aa7c6f612ff0a9db7f01\n",
+    "    tmppath = os.path.join(tmpdir, remote_info)\n",
+    "    !git remote show origin > $tmppath\n",
+    "    with open(tmppath, 'r') as f:\n",
+    "        lines = [s.strip() for s in f.readlines()]\n",
+    "        \n",
+    "        # FIXME: GitHub等リポジトリにも対応するためハードコーディングを解消\n",
+    "        ssh_push_url = lines[2].replace('Push  URL: http://dg02.dg.rcos.nii.ac.jp/', 'ssh://root@dg02.dg.rcos.nii.ac.jp:3001/') \n",
+    "\n",
+    "api.siblings(action='add', name='gin', url=ssh_push_url)\n",
+    "clear_output()\n",
+    "print('SSH connection is ready.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4bd90923",
+   "metadata": {},
+   "source": [
+    "## 5. 高性能実験環境利用のために必要な設定をする\n",
+    "\n",
+    "高性能実験環境を利用している場合には、以降のセルを実行してください。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c13e1f3c",
+   "metadata": {},
+   "source": [
+    "### 5.1 アカウント情報の入力\n",
+    "\n",
+    "以下のセルを実行し、表示されるフォームにアカウント情報を入力してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "652fbc4a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from IPython.display import clear_output\n",
+    "import getpass\n",
+    "name_mdx = input(\"高性能実験環境におけるSSHユーザ名：\")\n",
+    "clear_output()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eebec153",
+   "metadata": {},
+   "source": [
+    "### 5.2. SSH秘密鍵の設置\n",
+    "\n",
+    "左上のバナーを押下し、ファイル一覧画面に遷移してください。  \n",
+    "遷移後、id_rsaファイルをドラッグアンドドロップによりアップロードしてください。  \n",
+    "アップロード後、以下のセルを実行してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2b3c02b6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!mkdir -p /home/jovyan/.ssh/\n",
+    "!mv ~/id_rsa ~/.ssh/\n",
+    "!chmod 600 ~/.ssh/id_rsa\n",
+    "\n",
+    "with open(\"/home/jovyan/.ssh/config\", \"a\") as f:\n",
+    "     f.write(\"Host mdx\\n\")\n",
+    "     f.write(\"  Hostname 163.220.176.51\\n\")\n",
+    "     f.write(\"  User \" + name_mdx + \"\\n\")\n",
+    "     f.write(\"  Port 22\\n\")\n",
+    "     f.write(\"  IdentityFile ~/.ssh/id_rsa\\n\")\n",
+    "     f.write(\"  StrictHostKeyChecking no\\n\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0cf0dff5",
+   "metadata": {},
+   "source": [
+    "### 5.3. 実験活動のためのディレクトリ初期化"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "078c6bc2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "tmp = os.environ[\"BINDER_REF_URL\"]\n",
+    "tmp = tmp.split(\"/\")\n",
+    "tmp = tmp[len(tmp) - 1]\n",
+    "repo_name = tmp[:tmp.find(\".\")]\n",
+    "\n",
+    "cmd = \"mkdir ./\" + repo_name + \"/\"\n",
+    "!ssh mdx $cmd\n",
+    "\n",
+    "!scp -r ~/input_data/ mdx:~/$repo_name/input_data\n",
+    "!scp -r ~/output_data/ mdx:~/$repo_name/output_data\n",
+    "!scp -r ~/source/ mdx:~/$repo_name/source"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ed3fb5e4",
+   "metadata": {},
+   "source": [
+    "## 6. 実行結果を実験リポジトリに同期する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d12d4375",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from IPython.display import display, Javascript\n",
+    "display(Javascript('IPython.notebook.save_checkpoint();'))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7b237228",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# SSHホスト（＝GIN）を信頼する設定\n",
+    "# ドメイン名がハードコーディングにつき要修正\n",
+    "with open('/home/jovyan/.ssh/config', mode='w') as f:\n",
+    "    f.write('host dg02.dg.rcos.nii.ac.jp\\n\\tStrictHostKeyChecking no\\n\\tUserKnownHostsFile=/dev/null\\n')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "daa0c2cd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#FIXME：papermill実装に修正する。TO_GIT=true\n",
+    "%cd ~/\n",
+    "!git add .\n",
+    "!git commit -m \"prepare_for_workflow\"\n",
+    "!git push gin master"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

WORKFLOW/util/scripts/nb_utils.py

@@ -0,0 +1,259 @@
+# -*- coding: utf-8 -*-
+import os
+import re
+import sys
+import shutil
+from subprocess import run, CalledProcessError
+from tempfile import TemporaryDirectory
+from pathlib import Path
+from lxml import etree
+from nbformat import read, NO_CONVERT
+from itertools import chain, zip_longest
+from jinja2 import Template
+from datetime import datetime
+
+title_font_size = 11
+item_font_size = 9
+head_margin = 3
+text_margin = 2
+
+SVG_TEXT = '{http://www.w3.org/2000/svg}text'
+SVG_RECT = '{http://www.w3.org/2000/svg}rect'
+
+
+def parse_headers(nb_path):
+    nb = read(str(nb_path), as_version=NO_CONVERT)
+
+    # Notebookのセルからmarkdownの部分を取り出し、行ごとのリストにする
+    lines = [
+        line.strip()
+        for line in chain.from_iterable(
+            cell['source'].split('\n')
+            for cell in nb.cells
+            if cell['cell_type'] == 'markdown'
+        )
+        if len(line.strip()) > 0 and not line.startswith('---')
+    ]
+
+    # h1, h2 の行とその次行の最初の１文を取り出す
+    headers = [
+        (' '.join(line0.split()[1:]),
+         line1.split("。")[0] if line1 is not None else '')
+        for (line0, line1) in zip_longest(lines, lines[1:])
+        if line0.startswith('# ') or line0.startswith('## ')
+    ]
+    # 最初の見出しはtitle, 残りはheadersとして返す
+    return {
+        'path': nb_path.as_posix(),
+        'title': {
+            'text': _to_title_text(nb_path, headers[0][0]),
+            'summary': headers[0][1],
+        },
+        'headers': [
+            {
+                'text': text,
+                'summary': (
+                    summary if not re.match(r'(?:#|!\[)', summary) else ''),
+            }
+            for (text, summary) in headers[1:]
+        ],
+    }
+
+def _to_title_text(nb_path, text):
+    no = nb_path.name.split('-')[0]
+    title = text if not text.startswith('About:') else text[6:]
+    return f'{title}'
+
+def _get_notebook_headers(nb_dir):
+    return dict([
+        (nb.name, parse_headers(nb))
+        for nb in nb_dir.glob("*.ipynb")
+    ])
+
+def notebooks_toc(nb_dir):
+    nb_headers = sorted(
+        _get_notebook_headers(Path(nb_dir)).items(),
+        key=lambda x: x[0])
+
+    return "\n".join(chain.from_iterable([
+        [
+            f'* [{headers["title"]["text"]}]({nb_dir}/{str(nb)})'
+        ] + list(chain.from_iterable([
+            [
+                f'    - {header["text"]}',
+                (f'      - {header["summary"]}'
+                 if len(header["summary"]) > 0 else ''),
+            ]
+            for header in headers['headers']
+        ]))
+        for nb, headers in nb_headers
+    ]))
+
+import json
+
+JSON = ""
+def load_json(PATH):
+    with open(PATH) as f:
+        JSON = json.load(f)
+        return JSON
+
+
+def generate_svg_diag(
+        output='WORKFLOW/images/notebooks.svg',
+        diag='WORKFLOW/images/notebooks.diag',
+        dir_util='WORKFLOW/FLOW/util',
+        dir_01='WORKFLOW/FLOW/01_preparation_phase',
+        dir_02='WORKFLOW/FLOW/02_experimental_phase',
+        dir_03='WORKFLOW/FLOW/03_after_research_phase',
+        font='.fonts/ipag.ttf',
+):
+    with TemporaryDirectory() as workdir:
+        skeleton = Path(workdir) / 'skeleton.svg'
+        _generate_skeleton(skeleton, Path(diag), Path(font))
+        _embed_detail_information(Path(output), skeleton, Path(dir_util), Path(dir_01), Path(dir_02), Path(dir_03))
+        return output
+
+def _generate_skeleton(output, diag, font):
+    run(['blockdiag', '-f', font, '-Tsvg', '-o', output, diag], check=True)
+
+def setup_python_path():
+    ver = sys.version_info
+    lib_path = f'~/.local/lib/python{ver.major}.{ver.minor}/site-packages'
+    lib_path = str(Path(lib_path).expanduser())
+    if lib_path not in sys.path:
+        sys.path.append(lib_path)
+
+def _embed_detail_information(output, skeleton, dir_util, dir_01, dir_02, dir_03):
+    # Notebookのヘッダ取得
+    nb_headers = _get_notebook_headers(dir_util)
+    nb_headers.update(_get_notebook_headers(dir_01))
+    nb_headers.update(_get_notebook_headers(dir_02))
+    nb_headers.update(_get_notebook_headers(dir_03))
+
+    # 雛形の読み込み
+    tree = etree.parse(str(skeleton))
+
+    # 雛形をNotebook情報で置き換え
+    for elem in list(tree.findall(SVG_TEXT)):
+        if _is_target_rect(elem, nb_headers.keys()):
+            nb_name = _find_matching_notebook(nb_headers.keys(), elem.text)
+            _embed_info_in_one_rect(elem, nb_headers, Path('WORKFLOW/FLOW'), nb_name)
+
+    # SVGの保存
+    output.parent.mkdir(parents=True, exist_ok=True)
+    with output.open(mode='wb') as f:
+        f.write(etree.tostring(tree, method='xml', pretty_print=True))
+
+def _is_target_rect(elem, notebooks):
+    return (
+        elem.getprevious() is not None and
+        elem.getprevious().tag == SVG_RECT and
+        len(elem.text) > 0 and
+        _find_matching_notebook(notebooks, elem.text) is not None)
+
+def _find_matching_notebook(notebooks, title):
+    for nb in notebooks:
+        if nb.startswith(title):
+            return nb
+
+def _embed_info_in_one_rect(elem, nb_headers, nb_dir, nb_name):
+    headers = nb_headers[nb_name]
+    nb_file = nb_headers[nb_name]['path']
+    nb_file = nb_file.replace('WORKFLOW/FLOW/', '')
+    rect_elem = elem.getprevious()
+    rect = (
+        (int(rect_elem.attrib['x']), int(rect_elem.attrib['y'])),
+        (int(rect_elem.attrib['width']), int(rect_elem.attrib['height'])))
+    childpos = elem.getparent().index(elem)
+    parent_elem = elem.getparent()
+    remove_texts(elem)
+    title = headers['title']['text']
+    if elem.text.find(':') >= 0:
+        title = title + ' - ' + elem.text.split(':')[1]
+    line_num = insert_title(parent_elem, childpos, rect, title, str(nb_file))
+    insert_headers(parent_elem, childpos, rect, headers['headers'], line_num)
+
+def remove_texts(elem):
+    old_text = elem
+    while old_text is not None:
+        if (old_text.getnext() is not None and
+                old_text.getnext().tag == SVG_TEXT):
+            next_text = old_text.getnext()
+        else:
+            next_text = None
+        old_text.getparent().remove(old_text)
+        old_text = next_text
+
+def insert_title(parent_elem, childpos, rect, title, link):
+    height_title = (
+        text_margin + (title_font_size + text_margin) * 2 + head_margin * 2)
+    lines = split_title(title)
+    if len(lines) == 2:
+        text_elem = create_text(rect, title_font_size, font_weight='bold')
+        text_elem.text = lines[0]
+        text_elem.attrib['y'] = str(
+                rect[0][1] + head_margin + text_margin + title_font_size)
+        text_elems = [text_elem]
+
+        text_elem = create_text(rect, title_font_size, font_weight='bold')
+        text_elem.text = lines[1]
+        text_elem.attrib['y'] = str(
+                rect[0][1] + height_title - text_margin - head_margin)
+        text_elems.append(text_elem)
+    else:
+        text_elem = create_text(rect, title_font_size, font_weight='bold')
+        text_elem.text = title
+        text_elem.attrib['y'] = str(
+                rect[0][1] + height_title // 2 + title_font_size // 2)
+        text_elems = [text_elem]
+
+    parent_elem.insert(childpos, create_anchor(text_elems, link))
+    return len(lines)
+
+def insert_headers(parent_elem, childpos, rect, headers, title_lines):
+    offset_y = (
+        text_margin +
+        (title_font_size + text_margin) * (title_lines + 1) +
+        head_margin * 2 + text_margin)
+    for i, header in enumerate(headers):
+        text_elem = create_text(rect, item_font_size)
+        text_elem.text = header['text']
+        text_elem.attrib['y'] = str(
+                rect[0][1] + offset_y + (item_font_size + text_margin) * i +
+                item_font_size)
+        parent_elem.insert(childpos, text_elem)
+
+def split_title(title):
+    if u'：' in title:
+        return [title[:title.index(u'：') + 1], title[title.index(u'：') + 1:]]
+    elif len(title) >= 15:
+        words = re.split(r'([-(（])', title, 1)
+        ret = words[0:1] + [''.join(x) for x in zip(words[1::2], words[2::2])]
+        return [re.sub(r'^--', '- ', x) for x in ret]
+    else:
+        return [title]
+
+def create_text(rect, font_size, font_weight='normal', font_style='normal'):
+    text_elem = etree.Element(SVG_TEXT)
+    text_elem.attrib['fill'] = 'rgb(0,0,0)'
+    text_elem.attrib['font-family'] = 'sans-serif'
+    text_elem.attrib['font-size'] = str(font_size)
+    text_elem.attrib['font-style'] = font_style
+    text_elem.attrib['font-weight'] = font_weight
+    text_elem.attrib['font-anchor'] = 'middle'
+    text_elem.attrib['x'] = str(rect[0][0] + text_margin)
+    text_elem.attrib['width'] = str(rect[1][0] - text_margin * 2)
+    return text_elem
+
+def create_anchor(elems, link):
+    a_elem = etree.Element('a')
+    a_elem.attrib['{http://www.w3.org/1999/xlink}href'] = link
+    for elem in elems:
+        a_elem.append(elem)
+    return a_elem
+
+
+# refs: https://note.nkmk.me/python-if-name-main/
+# maDMP.ipynbからコマンドライン引数でdiagファイルのパスが渡されてくる
+if __name__ == '__main__':
+    generate_svg_diag(diag=sys.argv[1])

WORKFLOW/util/scripts/utils.py

@@ -0,0 +1,3 @@
+def fetch_param_file_path():
+    param_file_path = '/home/jovyan/WORKFLOW/param_files/params.json'
+    return param_file_path

environment.yml

@@ -0,0 +1,8 @@
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - datalad
+  - pip:
+    - lxml
+    - blockdiag

output_data/.gitkeep

@@ -0,0 +1 @@
+

source/main.ipynb

@@ -0,0 +1,32 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}