Merge commit 'bbeb007d927d4ef3c9e64ddf9a90b8d2061a764a' into main

PACKAGE/experiment.ipynb

@@ -20,24 +20,40 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "    1. 変数にメタデータ情報を入力し、入力後セルを実行する"
+    "1. 変数にメタデータ情報を入力し、入力後セルを実行する"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 16,
    "metadata": {},
    "outputs": [],
    "source": [
-    "Experiment_date = \"2021/12/15\" \n",
-    "Experimenter = \"yamada taro\""
+    "import json\n",
+    "meta_data = {\n",
+    "    \"experiment_date\": \"2021/12/16\",\n",
+    "    \"experimenter\": \"tanaka taro\"\n",
+    "}"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "    2. meta_data.jsonに保存する"
+    "2. meta_data.jsonに保存する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from collections import OrderedDict\n",
+    "\n",
+    "with open('./meta_data.json', 'w') as jf:\n",
+    "    json.dump(meta_data, jf, ensure_ascii=False, indent=2)"
    ]
   },
   {
@@ -51,23 +67,23 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "    1. コミットメッセージを変数に入力する"
+    "1. コミットメッセージを変数に入力する"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [],
    "source": [
-    "save_message = \"\""
+    "save_message = \" \""
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "    2. 一部のファイルのみ保存したい場合は、変数に保存したいファイル名を入力する"
+    "2. 途中保存する"
    ]
   },
   {
@@ -76,14 +92,73 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "save_files = \"\""
+    "from datalad import api\n",
+    "\n",
+    "api.save(message=save_message)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "- ## 実験を終了する\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "1. 実行環境構成を記録する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "conda env export > environment.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "pip freeze > requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "2. パイプラインを作成する（検討中）"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "3. 実験単体の再現性をモニタリングする（コード付帯機能で行うか検討中）"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "    3. 途中保存する"
+    "4. コミットメッセージを変数に入力し、実験記録の書き戻し準備をする"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "SAVE_MESSAGE = ''"
    ]
   },
   {
@@ -94,27 +169,74 @@
    "source": [
     "from datalad import api\n",
     "\n",
-    "api.save(message=save_message, path=save_files)"
+    "api.save(message=SAVE_MESSAGE)"
    ]
   },
   {
-   "cell_type": "markdown",
+   "cell_type": "code",
+   "execution_count": null,
    "metadata": {},
+   "outputs": [],
    "source": [
-    "- ## 実験を終了する\n"
+    "import os\n",
+    "\n",
+    "os.chdir(\"/home/jovyan\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "chmod 600 /home/jovyan/.ssh/id_rsa"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "mkdir ./tmp\n",
+    "git remote show origin > ./tmp/remote.txt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open('./tmp/remote.txt', 'r') as f:\n",
+    "    lines = [s.strip() for s in f.readlines()]\n",
+    "    ssh_push_url = lines[2].replace('Push  URL: https://github.com/', 'ssh://root@dg02.dg.rcos.nii.ac.jp:3001/')"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "option: 実験メタデータを入力する\n",
-    "1. 実行環境構成を記録する\n",
-    "2. パイプラインを作成する（検討中）\n",
-    "3. 実験単体の再現性をモニタリングする（コード付帯機能で行うか検討中）\n",
-    "4. 実験記録の書き戻し準備\n",
     "5. 実験記録を書き戻す"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datalad import api\n",
+    "\n",
+    "try:\n",
+    "    api.siblings(action='add', name=SIBLING_NAME, url=SSH_PUSH_URL)\n",
+    "except:\n",
+    "    pass\n",
+    "\n",
+    "api.push(to=SIBLING_NAME, result_renderer='default')"
+   ]
   }
  ],
  "metadata": {

WORKFLOW/FLOWS/base_FLOW.ipynb

@@ -10,6 +10,20 @@
     "初期フェーズから中期フェーズ、後期フェーズの順に進み、各フェーズで必要なセクションを研究者自身が実行する想定"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. データガバナンス機能と連携する"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "[書き戻しの準備をする](../SECTIONS/util/create_params_for_push.ipynb)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

WORKFLOW/SECTIONS/01_preparation_phase/init_research_repo.ipynb

@@ -13,30 +13,86 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 1. dmp.jsonからプロジェクト名を取得する"
+    "## 1. DMP情報からプロジェクト名を取得する"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 2. 取得したプロジェクト名で研究用のフォルダ構成を初期化する"
+    " - ## データガバナンス機能にDMP情報がある場合は以下を実行して2. に進む"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The Project\n"
+     ]
+    }
+   ],
+   "source": [
+    "import sys\n",
+    "import os\n",
+    "\n",
+    "sys.path.append(os.path.abspath(\"../util\"))\n",
+    "\n",
+    "import load\n",
+    "\n",
+    "dmp = load.get_dmp()\n",
+    "project_name = dmp['project']['title']"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 3. 変更内容を書き戻す"
+    "- ## システム外にDMP情報がある場合は以下を実行して2. に進む"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# [dummy] 本リポジトリのDMP情報\n",
+    "DMP_PATHにjson形式のDMPファイルのパスを入力する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "DMP_PATH = \"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys\n",
+    "import os\n",
+    "\n",
+    "sys.path.append(os.path.abspath(\"../util\"))\n",
     "\n",
-    "※以下は、ワークフロー機能で利用する本リポジトリのDMP情報の一覧です。入力いただいたdmp.jsonの内容に基づいてシステムが定義しているため、**手動で変更・実行する必要はありません**。"
+    "import load\n",
+    "\n",
+    "dmp = load.get_dmp_from_outsystem(DMP_PATH)\n",
+    "project_name = dmp['project']['title']"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. 取得したプロジェクト名で研究用のフォルダ構成を初期化する"
    ]
   },
   {
@@ -45,13 +101,27 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "######################\n",
-    "### parameterized by system ###\n",
-    "######################\n",
+    "import os\n",
     "\n",
-    "PROJECT_NAME = \"The project\"\n",
-    "LEADER = \"John Doe\"\n",
-    "RESEARCHES = .... "
+    "experiment_folder_pass = \"/home/jovyan/\" + project_name\n",
+    "os.makedirs(experiment_folder_pass)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. 変更内容を書き戻す"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "papermill ../util/datalad_save_push.ipynb - -f ../../params.yaml -p SAVE_MESSAGE = '研究用のフォルダ構成を初期化'"
    ]
   }
  ],

WORKFLOW/SECTIONS/02_experimental_phase/add_research_data.ipynb

@@ -6,7 +6,7 @@
    "source": [
     "# 研究データを追加する\n",
     "\n",
-    "これはDMP情報に基づき、研究リポジトリを更新するためのノートブックです。"
+    "研究リポジトリを更新するためのノートブックです。"
    ]
   },
   {
@@ -22,15 +22,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "data_name = ''\n",
-    "..."
+    "data_title = ''"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 2. 研究データを研究リポジトリに追加する"
+    "## 2. 研究データを研究フォルダに追加する"
    ]
   },
   {

WORKFLOW/SECTIONS/02_experimental_phase/launch_an_experiment.ipynb

@@ -11,23 +11,65 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 1. パッケージを研究用フォルダに複製する"
+    "## 1. 実験名を入力する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "experiment_title = ''"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. パッケージを研究用フォルダに複製する"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "cp -r /home/jovyan/PACKAGE/* /home/jovyan/[project_title]/[experiment_title]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. 複製したパッケージを書き戻す"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "papermill ../util/datalad_save_push.ipynb ./init_research_repo_output.ipynb -f ../../params.yaml -p SAVE_MESSAGE = '実験を作成'"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 2. 複製したパッケージを書き戻す"
+    "## 4. 複製したパッケージのコンテナを立ち上げて、ソースコードのJupyterNotebookに遷移する\n",
+    "※下のリンクは未完成"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 3. 複製したパッケージのコンテナを立ち上げて、ソースコードのJupyterNotebookに遷移する\n",
-    "\n",
-    "※binderコンテナを立ち上げるボタンを配置する  "
+    "[![Binder](https://binder.cs.rcos.nii.ac.jp/badge_logo.svg)](https://binder.cs.rcos.nii.ac.jp/v2/gh/[ユーザー名]/['/home/jovyan/[project-title]/[wxperiment-title]']/main)"
    ]
   }
  ],

WORKFLOW/SECTIONS/util/create_params_for_push.ipynb

@@ -0,0 +1,78 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# datalad save&push用params.yaml作成"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%sh\n",
+    "mkdir ./tmp\n",
+    "git remote show origin > ./tmp/remote.txt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open('./tmp/remote.txt', 'r') as f:\n",
+    "    lines = [s.strip() for s in f.readlines()]\n",
+    "    ssh_push_url = lines[2].replace('Push  URL: https://github.com/', 'ssh://root@dg02.dg.rcos.nii.ac.jp:3001/') # ssh_remote = lines[2].replace('Push  URL: http://dg02.dg.rcos.nii.ac.jp/', 'ssh://root@dg02.dg.rcos.nii.ac.jp:3001/')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'DEFAULT_BRANCH': 'origin', 'SIBLING_NAME': 'gin', 'SSH_PUSH_URL': 'ssh://root@dg02.dg.rcos.nii.ac.jp:3001/ivis-mizuguchi/workflow-template'}\n"
+     ]
+    }
+   ],
+   "source": [
+    "import yaml\n",
+    "\n",
+    "with open('../../params.yaml', 'w') as yf:\n",
+    "    yaml.dump({\n",
+    "        \"SSH_PUSH_URL\": ssh_push_url,\n",
+    "        \"SIBLING_NAME\": \"gin\",\n",
+    "        \"DEFAULT_BRANCH\": \"origin\"\n",
+    "    }, yf, default_flow_style=False)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

WORKFLOW/SECTIONS/util/datalad_save_push.ipynb

@@ -23,7 +23,7 @@
    "source": [
     "from datalad import api\n",
     "\n",
-    "api.save(path=SAVE_PATH, message=SAVE_MESSAGE"
+    "api.save(message=SAVE_MESSAGE)"
    ]
   },
   {
@@ -34,7 +34,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.chdir(CWD)"
+    "os.chdir(\"/home/jovyan\")"
    ]
   },
   {
@@ -47,17 +47,6 @@
     "chmod 600 /home/jovyan/.ssh/id_rsa"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%sh\n",
-    "mkdir ./tmp\n",
-    "git remote show origin > ./tmp/remote.txt"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -73,13 +62,8 @@
    "source": [
     "from datalad import api\n",
     "\n",
-    "# SSHでの接続URLを生成\n",
-    "with open('./tmp/remote.txt', 'r') as f:\n",
-    "    lines = [s.strip() for s in f.readlines()]\n",
-    "    ssh_remote = lines[2].replace('Push  URL: '+HTTP_HOST, SSH_HOST)\n",
-    "\n",
     "try:\n",
-    "    api.siblings(action='add', name=SIBLING_NAME, url=ssh_remote)\n",
+    "    api.siblings(action='add', name=SIBLING_NAME, url=SSH_PUSH_URL)\n",
     "except:\n",
     "    pass\n",
     "\n",

WORKFLOW/SECTIONS/util/load.py

@@ -0,0 +1,12 @@
+import json
+
+DMP = ""
+def get_dmp():
+    with open('../../../dmp.json') as f:
+        DMP = json.load(f)
+        return DMP
+
+def get_dmp_from_outsystem(DMP_PATH):
+    with open(DMP_PATH) as f:
+        DMP = json.load(f)
+        return DMP

WORKFLOW/SECTIONS/util/load_dmp.ipynb

@@ -1,43 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# DMP情報を読み込む"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import json\n",
-    "with open('../../../dmp.json') as f:\n",
-    "    dmp = json.load(f)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.7.8"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}