{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4719bc46",
   "metadata": {},
   "source": [
    "# 必要な準備を行う\n",
    "\n",
    "実験中のワークフロー実行のために必要な準備を行います。  \n",
    "上から順番に実行してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd0a038d",
   "metadata": {},
   "source": [
    "## 1. ユーザー認証を行う\n",
    "\n",
    "この手順では、あなたのユーザ情報をシステムに認証させる手続きを行います。  \n",
    "以下のセルを実行行し、画面の表示に沿ってデータガバナンス機能に登録したユーザー名、パスワード、メールアドレスを入力してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf28557d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/EX-WORKFLOW/util/scripts\n",
    "import json\n",
    "from scripts import utils\n",
    "\n",
    "# 以下の認証の手順で用いる、\n",
    "# GINのドメイン名等をパラメタファイルから取得する\n",
    "params = {}\n",
    "with open(utils.fetch_param_file_path(), mode='r') as f:\n",
    "    params = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61c68c98",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import getpass\n",
    "import requests\n",
    "\n",
    "from IPython.display import clear_output\n",
    "from requests.auth import HTTPBasicAuth\n",
    "from http import HTTPStatus\n",
    "\n",
    "# 正常に認証が終わるまで繰り返し\n",
    "while True:\n",
    "    name = input(\"ユーザー名：\")\n",
    "    password = getpass.getpass(\"パスワード：\")\n",
    "    email = input(\"メールアドレス：\")\n",
    "    clear_output()\n",
    "    \n",
    "    # GIN API Basic Authentication\n",
    "    # refs: https://docs.python-requests.org/en/master/user/authentication/\n",
    "    \n",
    "    # 既存のトークンがあるか確認する\n",
    "    response = requests.get(params['siblings']['ginHttp']+'api/v1/users/' + name + '/tokens', auth=(name, password))\n",
    "    tokens = response.json()\n",
    "\n",
    "    # 既存のトークンがなければ作成する\n",
    "    if len(tokens) < 1:\n",
    "        response = requests.post(params['siblings']['ginHttp']+'api/v1/users/' + name + '/tokens', data={\"name\": \"system-generated\"} ,auth=(name, password))\n",
    "\n",
    "    if response.status_code == HTTPStatus.OK or HTTPStatus.CREATED:\n",
    "        tokens = response.json()\n",
    "        clear_output()\n",
    "        print(\"認証が正常に完了しました。次の手順へお進みください。\")\n",
    "        break\n",
    "    else:\n",
    "        clear_output()\n",
    "        print(\"ユーザ名、またはパスワードが間違っています。\\n恐れ入りますがもう一度ご入力ください。\")\n",
    "\n",
    "!git config --global user.name $name\n",
    "!git config --global user.email $email"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fce82ba7",
   "metadata": {},
   "source": [
    "## 2. データ同期のための設定をする\n",
    "\n",
    "この手順では、今の実行環境とデータガバナンス機能のリポジトリでデータの同期をとるための準備をします。  \n",
    "以下を実行することで、システムがデータ同期の準備の手続きを行います。\n",
    "\n",
    "※実データの保存先として、必要であればAWS S3準拠のオブジェクトストレージを利用することもできます。  \n",
    "もし当該ストレージを利用する場合は、このNotebookと合わせて[こちら](operate_S3_annex.ipynb)も実行してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e277f2c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "#!/bin/bash\n",
    "if [ ! -e ~/.ssh/id_ed25519 ]; then\n",
    "    # 鍵ペアが無ければ作成\n",
    "    ssh-keygen -t ed25519 -N \"\" -f ~/.ssh/id_ed25519\n",
    "fi\n",
    "\n",
    "if [ ! -d ~/.datalad/ ]; then\n",
    "    # Dataladのデータセットでなければデータセット化する\n",
    "    datalad create --force /home/jovyan\n",
    "fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8181ce0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 公開鍵アップロード\n",
    "# refs: https://github.com/gogs/docs-api/blob/master/Users/Public%20Keys.md#create-a-public-key\n",
    "import os\n",
    "import requests\n",
    "import time\n",
    "from http import HTTPStatus\n",
    "\n",
    "import json\n",
    "from scripts import utils\n",
    "\n",
    "pubkey = !cat ~/.ssh/id_ed25519.pub\n",
    "\n",
    "# 認証時に取得したトークンを使ってPOSTリクエスト\n",
    "response = requests.post(\n",
    "                params['siblings']['ginHttp']+'api/v1/user/keys?token=' + tokens[0]['sha1'],\n",
    "                data={\n",
    "                    \"title\": \"system-generated-\"+str(time.time()),\n",
    "                    \"key\": pubkey[0]\n",
    "                })\n",
    "msg = response.json()\n",
    "\n",
    "# コンテナを消す際にコンテナとつなぐための公開鍵も削除のため、\n",
    "# パラメータとしてGINから発行された鍵IDを保存\n",
    "if response.status_code == HTTPStatus.CREATED:\n",
    "    # params.jsonへの追記（鍵ID）\n",
    "    params['ginKeyId'] = str(response.json()['id'])\n",
    "    with open(utils.fetch_param_file_path(), mode='w') as f:\n",
    "        json.dump(params, f, indent=4)\n",
    "    print('Public key is ready.')\n",
    "elif msg['message'] == 'Key content has been used as non-deploy key':\n",
    "    print('Public key is ready before time.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13f1918f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datalad import api\n",
    "from IPython.display import clear_output\n",
    "\n",
    "# sibling url をsshに変更する\n",
    "%cd ~/\n",
    "http_url = !git config --get remote.origin.url\n",
    "for item in http_url:\n",
    "    http_url = item\n",
    "    ssh_url = item.replace(params['siblings']['ginHttp'], params['siblings']['ginSsh'])\n",
    "    \n",
    "# siblingsにGINを登録する\n",
    "sibling = !datalad siblings -s gin\n",
    "for item in sibling:\n",
    "    if 'unknown sibling name' in item:\n",
    "        api.siblings(action='add', name='gin', url=ssh_url)\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "clear_output()\n",
    "print('SSH connection is ready.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4264f9f0",
   "metadata": {},
   "source": [
    "## 3. READMEに実験実行環境へのリンクを追加する\n",
    "\n",
    "当実行環境へアクセスするためのリンクを、データガバナンス機能の当実験リポジトリのREADMEに追記します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11e7f6fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, urllib\n",
    "from IPython.display import clear_output\n",
    "\n",
    "print('当画面のURLをブラウザからコピーし、以下のフォームに入力してください。')\n",
    "url = input()\n",
    "print(url.find(\"/notebooks/\"))\n",
    "url = url[:url.find(\"/notebooks/\")] + \"/notebooks/experiment.ipynb\"\n",
    "\n",
    "%cd ~/\n",
    "with open('README.md', 'w', newline='\\n') as f:\n",
    "    f.write(\"\\n## 実験実行環境にアクセスしたい場合\\n以下のリンクをクリックしてください<br>\" + url + \"  \")\n",
    "    f.write(\"\\n\\n上記リンクからアクセスできない場合は以下のリンクから実験実行環境を再度立ち上げてください  \")\n",
    "    f.write(\"\\n※「他リポジトリから実験の入力データを用意する」で入力データを用意した場合は、再度必要なファイルの実体をdatalad get　'ファイルパス'で取得する必要があります。  \")\n",
    "    f.write(\"\\nhttps://binder.cs.rcos.nii.ac.jp/v2/git/\" + urllib.parse.quote(http_url, safe='') + \"/HEAD?filepath=experiment.ipynb\")\n",
    "\n",
    "clear_output()\n",
    "print('READMEに実行環境へのリンクを追加しました。')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed3fb5e4",
   "metadata": {},
   "source": [
    "## 4. 実行結果を実験リポジトリに同期する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d12d4375",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Javascript\n",
    "display(Javascript('IPython.notebook.save_checkpoint();'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b237228",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/EX-WORKFLOW/util/scripts\n",
    "from scripts import utils\n",
    "\n",
    "# SSHホスト（＝GIN）を信頼する設定\n",
    "utils.config_GIN(ginHttp = params['siblings']['ginHttp'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f506e02",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "\n",
    "# Git管理のパスのリストを作成する\n",
    "%cd ~/\n",
    "files = os.listdir()\n",
    "# ディレクトリ一覧からGit-annex管理するディレクトリ(input_dataとoutput_data)を排除する\n",
    "dirs = [f for f in files if os.path.isdir(f)]\n",
    "dirs.remove('input_data')\n",
    "dirs.remove('output_data')\n",
    "# HOME直下のファイルを取得\n",
    "files = [f for f in files if os.path.isfile(f)]\n",
    "# Git管理するパスの配列を作成する\n",
    "files.extend(dirs)\n",
    "save_path = files\n",
    "\n",
    "# Git-annex管理するパスの配列を作成する\n",
    "annexed_save_path = ['input_data', 'output_data']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d44ccef",
   "metadata": {},
   "outputs": [],
   "source": [
    "import papermill as pm\n",
    "\n",
    "%cd ~/\n",
    "# Git-annex管理ファイルを保存\n",
    "pm.execute_notebook(\n",
    "    'EX-WORKFLOW/util/base_datalad_save_push.ipynb',\n",
    "    '/home/jovyan/.local/push_log.ipynb',\n",
    "    parameters = dict(SAVE_MESSAGE = 'ワークフロー実行準備 (1/2)', PATH = annexed_save_path, IS_RECURSIVE = False)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9201ef94",
   "metadata": {},
   "outputs": [],
   "source": [
    "import papermill as pm\n",
    "\n",
    "%cd ~/\n",
    "# Git管理ファイルを保存\n",
    "pm.execute_notebook(\n",
    "    'EX-WORKFLOW/util/base_datalad_save_push.ipynb',\n",
    "    '/home/jovyan/.local/push_log.ipynb',\n",
    "    parameters = dict(SAVE_MESSAGE = 'ワークフロー実行準備 (2/2)', TO_GIT = True, PATH = save_path, IS_RECURSIVE = False)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ff4751e",
   "metadata": {},
   "source": [
    "## 5. ワークフロー図を更新する\n",
    "\n",
    "ワークフロー図にこのワークフローが実行済みであることを反映します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2e74fd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/\n",
    "path = 'EX-WORKFLOW/images/notebooks.diag'\n",
    "\n",
    "with open('.gitignore', 'r') as f:\n",
    "    text = f.read()\n",
    "    if text.find(path) == -1:\n",
    "        !echo \"/\" + $path >> ./.gitignore\n",
    "\n",
    "# notebooks.diagのgit管理を外す\n",
    "!git update-index --skip-worktree $path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa6250de",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/\n",
    "find = '\"required_every_time\"[fontsize = 14];'\n",
    "replace = '\"required_every_time\"[numbered = 済, fontsize = 14];'\n",
    "\n",
    "with open(path, 'r') as f:\n",
    "    s = f.read()\n",
    "\n",
    "with open(path, 'w') as f:\n",
    "    s = s.replace(find, replace)\n",
    "    f.write(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc2e4eab",
   "metadata": {},
   "source": [
    "## 6. 実験中ワークフロー機能トップページに遷移する\n",
    "\n",
    "続けてワークフロー機能を実行する場合は、[こちら](../../experiment.ipynb)からトップページに遷移できます。  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}