Merge pull request #1 from ivis-yoshida/develop/workflow-template

各セクションの実装

PACKAGE/SECTIONS/enter_metadata.ipynb

@@ -85,13 +85,17 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import papermill as pm\n",
+    "%cd ~/\n",
+    "!git add ~/\n",
+    "!git commit -m 'メタデータ入力'\n",
     "\n",
-    "pm.execute_notebook(\n",
-    "    './util/datalad_save_push.ipynb',\n",
-    "    '-',\n",
-    "    parameters = dict(SAVE_MESSAGE = 'メタデータ入力', PATH = '/home/jovyan/meta_data.json')\n",
-    ")"
+    "# import papermill as pm\n",
+    "# \n",
+    "# pm.execute_notebook(\n",
+    "#     './util/datalad_save_push.ipynb',\n",
+    "#     '-',\n",
+    "#     parameters = dict(SAVE_MESSAGE = 'メタデータ入力', PATH = '/home/jovyan/meta_data.json')\n",
+    "# )"
    ]
   }
  ],
@@ -111,7 +115,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.8"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

PACKAGE/SECTIONS/save.ipynb

@@ -39,9 +39,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from datalad import api\n",
+    "%cd ~/\n",
+    "!git add ~/\n",
+    "!git commit -m $save_message\n",
     "\n",
-    "api.save(message=save_message)"
+    "# from datalad import api\n",
+    "# api.save(message=save_message)"
    ]
   }
  ],
@@ -61,7 +64,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.8"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

PACKAGE/SECTIONS/util/datalad_save_push.ipynb

@@ -97,7 +97,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.8"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

PACKAGE/SECTIONS/util/prepare_for_workflow_every_time.ipynb

@@ -0,0 +1,71 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "4719bc46",
+   "metadata": {},
+   "source": [
+    "# ワークフロー実行準備（binder環境起動毎）\n",
+    "\n",
+    "- このノートブックでは、binder環境が起動するたびに必要なワークフロー実行のための処理を行います。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3d968f7c",
+   "metadata": {},
+   "source": [
+    "## データ書き戻しのための.netrcファイルの設定\n",
+    "\n",
+    "- 以下のセルを実行し、表示されたURLに遷移して、【ユーザ名】と【パスワード】を書き換えてください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "178dfe28",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "echo \"\n",
+    "machine dg02.dg.rcos.nii.ac.jp\n",
+    "login 【ユーザ名】\n",
+    "password 【パスワード】\n",
+    "\" > ~/.netrc"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ec0051a9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "print(\"https://jupyter.cs.rcos.nii.ac.jp\" + os.environ[\"JUPYTERHUB_SERVICE_PREFIX\"] + \"edit/.netrc\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

PACKAGE/environment.yaml

@@ -1,6 +0,0 @@
-name: package environment
-channels:
-  - conda-forge
-  - defaults
-dependencies:
-  - datalad

PACKAGE/experiment.ipynb

@@ -8,16 +8,23 @@
     "\n",
     "これは、実験中のワークフローを支援するためのノートブックです。  \n",
     "リンクに遷移して必要なワークフローを実行してください。  \n",
-    "※実験は、sourceフォルダ内のmain.ipynb に記述してください。"
+    "※実験は、sourceフォルダ内の [main.ipynb](./source/main.ipynb) に記述してください。"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "##  0. 準備\n",
-    "\n",
-    "実験中ワークフローを実行する前に、準備として「」を実行してください。"
+    "##  0. 準備"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "|||\n",
+    "|:---:|:---:|\n",
+    "|[書き戻しの準備](./SECTIONS/util/prepare_for_workflow_every_time.ipynb)|"
    ]
   },
   {
@@ -33,7 +40,7 @@
    "source": [
     "||||\n",
     "|:---:|:---:|:---:|\n",
-    "|[実験メタデータを入力する](./SECTIONS/enter_metadata.ipynb)|[実験を途中保存する]()|[実験を終了する]()"
+    "|[実験メタデータを入力する](./SECTIONS/enter_metadata.ipynb)|[実験を途中保存する](./SECTIONS/save.ipynb)|[実験を終了する](./SECTIONS/finish.ipynb)"
    ]
   }
  ],
@@ -53,7 +60,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.8"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

WORKFLOW/FLOWS/base_FLOW.ipynb

@@ -27,7 +27,7 @@
    "source": [
     "|||\n",
     "|:---:|:---:|\n",
-    "|[ワークフロー実行準備をする](../SECTIONS/util/prepare_for_datalad_push.ipynb)|"
+    "|[binder環境立ち上げ毎に実行](../SECTIONS/base/util/prepare_for_workflow_every_time.ipynb)|[初回立ち上げ時に実行](../SECTIONS/base/util/prepare_for_workflow.ipynb)|"
    ]
   },
   {
@@ -45,7 +45,7 @@
    "source": [
     "||||||\n",
     "|:---:|:---:|:---:|:---:|:---:|\n",
-    "|[GakuNin RDMと連携する](../SECTIONS/01_preparation_phase/work_with_GakuNinRDM.ipynb)|[研究用のフォルダ構成を初期化する](../SECTIONS/01_preparation_phase/init_research_repo.ipynb)|[データクリーニングツールを準備する](../SECTIONS/01_preparation_phase/setup_data_cleaning_tools.ipynb)|[データ分析ツールを準備する](../SECTIONS/01_preparation_phase/setup_data_analysis_tools.ipynb)|[解析基盤でR言語を使う準備をする](../SECTIONS/01_preparation_phase/setup_r_kernel.ipynb)|"
+    "|[GakuNin RDMと連携する](../SECTIONS/base/01_preparation_phase/work_with_GakuNinRDM.ipynb)|[データクリーニングツールを準備する](../SECTIONS/base/01_preparation_phase/setup_data_cleaning_tools.ipynb)|[データ分析ツールを準備する](../SECTIONS/base/01_preparation_phase/setup_data_analysis_tools.ipynb)|"
    ]
   },
   {
@@ -63,7 +63,7 @@
    "source": [
     "||||||\n",
     "|:---:|:---:|:---:|:---:|:---:|\n",
-    "|[実験を実行する](../SECTIONS/02_experimental_phase/launch_an_experiment.ipynb)|[研究リポジトリをモニタリングする](../SECTIONS/02_experimental_phase/monitor_repository.ipynb)|[実験に依存関係を定義する](../SECTIONS/02_experimental_phase/define_experimental_dependencies.ipynb)|[パイプラインを構築する](../SECTIONS/02_experimental_phase/support_making_pipeline.ipynb)|[研究データを追加する](../SECTIONS/02_experimental_phase/add_research_data.ipynb)|"
+    "|[実験を実行する](../SECTIONS/base/02_experimental_phase/launch_an_experiment.ipynb)|[実験パイプラインをチェックする](../SECTIONS/base/02_experimental_phase/check_pipeline.ipynb)|[実験パイプラインを実行する](../SECTIONS/base/02_experimental_phase/run_pipeline.ipynb)|[研究リポジトリをモニタリングする](../SECTIONS/base/02_experimental_phase/monitor_repository.ipynb)|"
    ]
   },
   {
@@ -72,7 +72,7 @@
    "source": [
     "## 3. 実験終了後フェーズ\n",
     "\n",
-    "実験終了後フェーズではキュレーション機能との連携やデータや論文の公開をサポートします。"
+    "実験終了後フェーズでは研究成果の公開をサポートします。"
    ]
   },
   {
@@ -81,7 +81,7 @@
    "source": [
     "|||\n",
     "|:---:|:---:|\n",
-    "|[キュレーション機能と連携する](../SECTIONS/03_after_research_phase/work_with_data_curation.ipynb)|[WEKO3と連携する](../SECTIONS/03_after_research_phase/work_with_weko3.ipynb)|"
+    "|[研究成果を公開する](../SECTIONS/base/03_after_research_phase/publish.ipynb)|"
    ]
   }
  ],
@@ -101,7 +101,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.8"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

WORKFLOW/SECTIONS/base/02_experimental_phase/check_pipeline.ipynb

@@ -0,0 +1,105 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 実験パイプラインのチェック\n",
+    "\n",
+    "これは実験パイプライン構成をチェックするためのノートブックです。  "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. パイプライン定義ファイルのチェック\n",
+    "\n",
+    "- このステップでは、パイプライン定義ファイルとパッケージ群の情報に相違がないか確認します。\n",
+    "- 以下のセルを実行し、エラーが起きないことを確認してください。\n",
+    "- エラーが起きた場合は、表示されたリンクに遷移してパイプライン定義ファイルを修正＆エラーが起きなくなるまで再実行を繰り返してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import glob\n",
+    "\n",
+    "# 作業ディレクトリに移動\n",
+    "%cd ~/experiments/\n",
+    "\n",
+    "# 実験パイプライン定義ファイルの読み込み\n",
+    "pipeline = json.load(open(\"pipeline.json\"))\n",
+    "print(\"実験パイプライン：\" + str(pipeline))\n",
+    "\n",
+    "# 実験パッケージ群の読み込み\n",
+    "files = glob.glob(\"./*/\")\n",
+    "\n",
+    "experiments = []\n",
+    "for file in files:\n",
+    "    experiments += [os.path.basename(file[:-1])]\n",
+    "print(\"実験パッケージ　：\" + str(experiments))\n",
+    "\n",
+    "# 元のディレクトリに移動\n",
+    "%cd -\n",
+    "\n",
+    "# 過不足の確認\n",
+    "if set(pipeline) != set(experiments):\n",
+    "    link = \"https://jupyter.cs.rcos.nii.ac.jp\" + os.environ[\"JUPYTERHUB_SERVICE_PREFIX\"] + \"edit/experiments/pipeline.json\"\n",
+    "    raise Exception(\"実験パイプラインと実験パッケージ間で相違があります\\n\\n以下のURLに遷移して修正してください\\n\" + link)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. 変更内容を書き戻す"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%cd ~/\n",
+    "!git add ~/\n",
+    "!git commit -m '実験パイプラインのチェック'\n",
+    "\n",
+    "# import papermill as pm\n",
+    "# \n",
+    "# pm.execute_notebook(\n",
+    "#     '../util/datalad_save_push.ipynb',\n",
+    "#     '-',\n",
+    "#     parameters = dict(SAVE_MESSAGE = 'パイプライン構築', PATH = [未定])\n",
+    "# )"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

WORKFLOW/SECTIONS/base/02_experimental_phase/launch_an_experiment.ipynb

@@ -9,16 +9,6 @@
     "これは、実験パッケージを用意して実験を行うためのノートブックです。  "
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## ◆◆◆開発メモ◆◆◆\n",
-    "\n",
-    "- 検証中\n",
-    "- launch binderのリンク作成が出来ていない"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -45,22 +35,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "1. 以下のセルを実行すると入力フォームが現れます。ステップ1で作成した空リポジトリの名前とSSH URLを入力してください。"
+    "1. 以下のセルを実行すると入力フォームが現れます。ステップ1で作成した空リポジトリの名前を入力してください。"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 85,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "ex100\n"
-     ]
-    }
-   ],
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
    "source": [
     "repository_title = input()"
    ]
@@ -71,7 +53,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ssh_url = input()"
+    "%cd ~/\n",
+    "tmp = !git remote -v\n",
+    "tmp = tmp[0]\n",
+    "tmp = tmp[tmp.find(\"http\"):]\n",
+    "tmp = tmp.split(\"/\")\n",
+    "tmp = tmp[0] + \"//\" + tmp[2] + \"/\" + tmp[3]\n",
+    "\n",
+    "ssh_url = tmp + \"/\" + repository_title + \".git\"\n",
+    "print(ssh_url)"
    ]
   },
   {
@@ -87,21 +77,18 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "%%sh\n",
-    "cd ~/"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
+    "!mkdir ~/.tmp/\n",
+    "!git init ~/.tmp/$repository_title\n",
+    "# !datalad create ~/.tmp/$repository_title\n",
+    "!cp -r ~/WORKFLOW/PACKAGE/. ~/.tmp/$repository_title/\n",
     "\n",
-    "experiment_path = \"experiments/\" + repository_title\n",
-    "os.mkdir(experiment_path)\n",
-    "os.system(\"cp -r PACKAGE/* \" + experiment_path)"
+    "%cd ~/.tmp/$repository_title\n",
+    "!git add ~/.tmp/$repository_title\n",
+    "!git commit -m '1st commit'\n",
+    "!git remote add $repository_title $ssh_url\n",
+    "!git push $repository_title master\n",
+    "# !datalad save --to-git -m \"1st commit\"\n",
+    "# !datalad push"
    ]
   },
   {
@@ -117,21 +104,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "%%sh\n",
-    "cd ~/"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "\n",
-    "os.system(\"datalad create --force -d . \" + experiment_path)\n",
-    "os.chdir(experiment_path)\n",
-    "os.system(\"datalad siblings add -s gin --url \" + ssh_url)"
+    "%cd ~/\n",
+    "!git submodule add $ssh_url ./experiments/$repository_title"
    ]
   },
   {
@@ -143,30 +117,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 93,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "['ex100']\n"
-     ]
-    }
-   ],
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
    "source": [
     "import json\n",
-    "save_data = \"\"\n",
+    "\n",
     "with open('/home/jovyan/experiments/pipeline.json', 'r') as f:\n",
-    "    read_data = json.load(f)\n",
-    "    print(read_data)\n",
-    "if read_data == []:\n",
-    "    save_data = [repository_title]\n",
-    "else:\n",
-    "    read_data.append(repository_title)\n",
-    "    save_data = read_data\n",
+    "    pipeline = json.load(f)\n",
+    "\n",
+    "pipeline.append(repository_title)\n",
+    "\n",
     "with open('/home/jovyan/experiments/pipeline.json', 'w') as f:\n",
-    "    json.dump(save_data, f, indent = 4)"
+    "    json.dump(pipeline, f, indent = 4)"
    ]
   },
   {
@@ -176,27 +139,27 @@
     "## 5. 変更内容を書き戻す"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "1. 書き戻しメッセージを変更したい場合は以下のセルでsave_messageに代入する値を編集してください。"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "save_message = '実験の作成'"
+    "%cd ~/\n",
+    "!git add ~/\n",
+    "!git commit -m '実験の作成'\n",
+    "!git push origin master\n",
+    "# !datalad save --to-git -m \"実験の作成\"\n",
+    "# !datalad push"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "2. 変更内容を書き戻す"
+    "## 6. 複製したパッケージのコンテナを立ち上げて、ソースコードのJupyterNotebookに遷移する\n",
+    "\n",
+    "以下のセルを実行して表示されるリンクを押下してください。"
    ]
   },
   {
@@ -205,30 +168,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import papermill as pm\n",
+    "import urllib\n",
     "\n",
-    "pm.execute_notebook(\n",
-    "    '../util/datalad_save_push.ipynb',\n",
-    "    '-',\n",
-    "    parameters = dict(SAVE_MESSAGE = save_message)\n",
+    "print(\n",
+    "    \"https://binder.cs.rcos.nii.ac.jp/v2/git/\" + urllib.parse.quote(ssh_url, safe='') + \"/HEAD?filepath=experiment.ipynb\"\n",
     ")"
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 6. 複製したパッケージのコンテナを立ち上げて、ソースコードのJupyterNotebookに遷移する\n",
-    "\n",
-    "※下のリンクは未完成"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "[![Binder](https://binder.cs.rcos.nii.ac.jp/badge_logo.svg)](https://binder.cs.rcos.nii.ac.jp/v2/gh/[ユーザー名]/['/home/jovyan/[project-title]/[wxperiment-title]']/main)"
-   ]
   }
  ],
  "metadata": {
@@ -247,7 +192,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.8"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

WORKFLOW/SECTIONS/base/02_experimental_phase/run_pipeline.ipynb

@@ -0,0 +1,142 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "8db76987",
+   "metadata": {},
+   "source": [
+    "# 実験パイプラインの実行\n",
+    "\n",
+    "これは実験パイプラインを実行するためのノートブックです。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "71da3526",
+   "metadata": {},
+   "source": [
+    "## 1. 実験パイプライン情報の取得\n",
+    "\n",
+    "- このステップでは、実行する実験パイプラインの定義情報を取得します。\n",
+    "- 以下のセルを実行してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1d8a854",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "# 作業ディレクトリに移動\n",
+    "%cd ~/experiments/\n",
+    "\n",
+    "# 実験パイプライン定義ファイルの読み込み\n",
+    "pipeline = json.load(open(\"pipeline.json\"))\n",
+    "print(\"実験パイプライン：\" + str(pipeline))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "77ea8bce",
+   "metadata": {},
+   "source": [
+    "## 2. 実験パッケージの最新化"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b4ac08ea",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for experiment in pipeline:\n",
+    "    %cd $experiment\n",
+    "    !git pull\n",
+    "    %cd ../"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "96b52333",
+   "metadata": {},
+   "source": [
+    "## 3. 実験パイプラインの実行\n",
+    "\n",
+    "- このステップでは、定義情報に基づいて実験パイプラインを実行します。\n",
+    "- 以下のセルを実行してください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8dd4ffaf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%cd ~/experiments/\n",
+    "\n",
+    "import papermill\n",
+    "\n",
+    "outputs = []\n",
+    "\n",
+    "for experiment in pipeline:\n",
+    "    print(\"\\n============================================================ \" + experiment)\n",
+    "\n",
+    "    print(\"\\n# 出力ディレクトリの初期化\")\n",
+    "    cmd = \"rm -rf \" + experiment + \"/output_data/*\"\n",
+    "    print(\"$ \" + cmd)\n",
+    "    !$cmd\n",
+    "    \n",
+    "    print(\"\\n前実験の出力データを当実験の入力データとして複製\")\n",
+    "    for output in outputs:\n",
+    "        cmd = \"cp -rf \" + output + \"/output_data/* \" + experiment + \"/input_data/\" \n",
+    "        print(\"$ \" + cmd)\n",
+    "        !$cmd\n",
+    "\n",
+    "    print(\"\\n実験再現のための環境構築\")\n",
+    "    cmd = \"pip install -r \" + experiment + \"/requirements.txt\"\n",
+    "    print(\"$ \" + cmd)\n",
+    "    !$cmd\n",
+    "\n",
+    "    print(\"\\n実験ソースコードの実行\")\n",
+    "    cmd = experiment + \"/source/\"\n",
+    "    print(\"$ cd \" + cmd)\n",
+    "    %cd $cmd\n",
+    "\n",
+    "    papermill.execute_notebook(\"main.ipynb\", \"log.ipynb\")\n",
+    "    \n",
+    "    cmd = \"-\"\n",
+    "    print(\"$ cd \" + cmd)\n",
+    "    %cd $cmd\n",
+    "\n",
+    "    print(\"\\n次実験に備えて本実験の出力データを登録\")\n",
+    "    outputs += [experiment]"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

WORKFLOW/SECTIONS/base/util/prepare_for_workflow.ipynb

@@ -14,151 +14,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 1. SSH鍵を準備する"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "1. SSH鍵を生成し、秘密鍵を.ssh/id_rsaに配置します。"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%sh\n",
-    "cd ~/\n",
-    "mkdir .ssh\n",
-    "ssh-keygen -f .ssh/id_rsa -t rsa -N \"\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "2. 以下のセルで出力された公開鍵を[こちら](http://dg02.dg.rcos.nii.ac.jp/user/settings/ssh)からGINリポジトリに登録してください。"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%sh\n",
-    "cd ~/\n",
-    "cat .ssh/id_rsa.pub"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 2. SSH接続で書き戻しを行うためにssh鍵に権限を与える（ssh鍵は.sshディレクトリの中にid_rsaとして保存する）"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%sh\n",
-    "cd ~/\n",
-    "chmod 600 .ssh/id_rsa"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 3. SSH鍵の利用確認をオフにする"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%sh\n",
-    "echo \"host *\\n\\tStrictHostKeyChecking no\\n\\tUserKnownHostsFile /dev/null\" >> ~/.ssh/config"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 4. Dataladとpapermillをインストールする"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "pip install datalad"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "pip install papermill"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 5. environment.yamlを作成する"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "1. pyyamlをインストールする"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "pip install pyyaml"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "2. environment.yamlを作成する"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%sh\n",
-    "echo \"name: Binder environment\\nchannels:\\n  - conda-forge\\n  - defaults\\ndependencies:\\n  - pip\\n    - datalad\\n    - papermill\\n\" >> ~/environment.yaml"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 6. .gitconfigにユーザー名とメールアドレスを登録する"
+    "## 1. .gitconfigにユーザー名とメールアドレスを登録する"
    ]
   },
   {
@@ -210,7 +66,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 7. Dataladを使用するため、このリポジトリをデータセットとする"
+    "## 2. Dataladを使用するため、このリポジトリをデータセットとする"
    ]
   },
   {
@@ -228,7 +84,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 8. ワークフロー実行結果をGINリポジトリに反映させるため、GINリポジトリをリモートリポジトリに設定する"
+    "## 3. Git管理対象外ファイルを.gitignoreで設定する"
    ]
   },
   {
@@ -238,81 +94,27 @@
    "outputs": [],
    "source": [
     "%%sh\n",
-    "mkdir ./tmp\n",
-    "git remote show origin > ./tmp/remote.txt"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import json\n",
-    "json_open = open('/home/jovyan/param_files/param_for_dataladsiblings.json', 'r')\n",
-    "json_load = json.load(json_open)\n",
-    "http = json_load['http']\n",
-    "ssh = json_load['ssh']"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "with open('./tmp/remote.txt', 'r') as f:\n",
-    "    lines = [s.strip() for s in f.readlines()]\n",
-    "    ssh_push_url = lines[2].replace(http, ssh) "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "\n",
-    "os.system(\"rm -r ./tmp\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from datalad import api\n",
     "\n",
-    "try:\n",
-    "    api.siblings(action='add', name='gin', url=ssh_push_url)\n",
-    "except:\n",
-    "    pass"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 9. .local配下をgit管理しないように.gitignoreで設定する"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%sh\n",
-    "echo \".local/*\" >> ~/.gitignore"
+    "echo \"\n",
+    "/.cache/\n",
+    "/.conda/\n",
+    "/.config/\n",
+    "/.ipython/\n",
+    "/.local/\n",
+    "/.tmp/\n",
+    "/.bashrc\n",
+    "/.bash_logout\n",
+    "/.profile\n",
+    "/.netrc\n",
+    ".ipynb_checkpoints/\n",
+    "\" > ~/.gitignore"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 10. 実験用フォルダを作成し、その中にpipeline.jsonを用意する"
+    "## 4. 実験用フォルダを作成し、その中にpipeline.jsonを用意する"
    ]
   },
   {
@@ -323,14 +125,14 @@
    "source": [
     "%%sh\n",
     "mkdir ~/experiments\n",
-    "echo \"[]\" >> ~/experiments/pipeline.json"
+    "echo \"[]\" > ~/experiments/pipeline.json"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 11. 必要なファイルをPACKAGEテンプレートに複製する"
+    "## 5. 必要なファイルをPACKAGEテンプレートに複製する"
    ]
   },
   {
@@ -341,19 +143,16 @@
    "source": [
     "%%sh\n",
     "cd ~/\n",
-    "mkdir WORKFLOW/PACKAGE/.ssh\n",
-    "cp .ssh/id_rsa WORKFLOW/PACKAGE/.ssh/id_rsa\n",
-    "cp .ssh/config WORKFLOW/PACKAGE/.ssh/config\n",
     "cp .gitconfig WORKFLOW/PACKAGE/.gitconfig\n",
-    "cp .gitconfig WORKFLOW/PACKAGE/.gitconfig\n",
-    "cp environment.yaml WORKFLOW/PACKAGE/environment.yaml"
+    "cp .gitignore WORKFLOW/PACKAGE/.gitignore\n",
+    "cp environment.yml WORKFLOW/PACKAGE/environment.yml"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 12. 変更内容を書き戻す"
+    "## 6. 変更内容を書き戻す"
    ]
   },
   {
@@ -362,13 +161,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import papermill as pm\n",
-    "\n",
-    "pm.execute_notebook(\n",
-    "    '../util/datalad_save_push.ipynb',\n",
-    "    '-',\n",
-    "    parameters = dict(SAVE_MESSAGE = 'ワークフロー実行準備')\n",
-    ")"
+    "%cd ~/\n",
+    "!datalad save --to-git -m \"save\"\n",
+    "!datalad push"
    ]
   }
  ],
@@ -388,7 +183,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.8"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

WORKFLOW/SECTIONS/base/util/prepare_for_workflow_every_time.ipynb

@@ -0,0 +1,71 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0a0f3e94",
+   "metadata": {},
+   "source": [
+    "# ワークフロー実行準備（binder環境起動毎）\n",
+    "\n",
+    "- このノートブックでは、binder環境が起動するたびに必要なワークフロー実行のための処理を行います。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cf9dcf69",
+   "metadata": {},
+   "source": [
+    "## データ書き戻しのための.netrcファイルの設定\n",
+    "\n",
+    "- 以下のセルを実行し、表示されたURLに遷移して、【ユーザ名】と【パスワード】を書き換えてください。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "083c2d1e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "echo \"\n",
+    "machine dg02.dg.rcos.nii.ac.jp\n",
+    "login 【ユーザ名】\n",
+    "password 【パスワード】\n",
+    "\" > ~/.netrc"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c123aadd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "print(\"https://jupyter.cs.rcos.nii.ac.jp\" + os.environ[\"JUPYTERHUB_SERVICE_PREFIX\"] + \"edit/.netrc\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}