{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4b6b7c2a",
   "metadata": {},
   "source": [
    "# 初期セットアップを行う\n",
    "\n",
    "ワークフロー実行のための準備を行います。\n",
    "上から順番にセルを実行してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "396fd7ef",
   "metadata": {},
   "source": [
    "## １．ユーザーの認証を行う\n",
    "\n",
    "この手順では、あなたのユーザ情報をシステムに認証させる手続きを行います。  \n",
    "以下のセルを実行し、画面の表示に沿ってデータガバナンス機能に登録したユーザー名、パスワード、メールアドレスを入力してください。\n",
    "\n",
    "◆◆◆開発メモ◆◆◆  \n",
    "ゆくゆくはSSOに対応し、コンテナ立ち上げ毎のユーザー名、パスワード、メールアドレスの入力を避けたい"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b5bb6bcc",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_through_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from scripts import utils\n",
    "\n",
    "# 以下の認証の手順で用いる、\n",
    "# GINのドメイン名等をパラメタファイルから取得する\n",
    "params = {}\n",
    "with open(utils.fetch_param_file_path(), mode='r') as f:\n",
    "    params = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b8192cc3",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_through_control": {
     "frozen": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "認証が正常に完了しました。次の手順へお進みください。\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import time\n",
    "import getpass\n",
    "import requests\n",
    "\n",
    "from IPython.display import clear_output\n",
    "from requests.auth import HTTPBasicAuth\n",
    "from http import HTTPStatus\n",
    "\n",
    "tokens = []\n",
    "access_token = {}\n",
    "\n",
    "# 正常に認証が終わるまで繰り返し\n",
    "while True:\n",
    "    name = input(\"ユーザー名：\")\n",
    "    password = getpass.getpass(\"パスワード：\")\n",
    "    email = input(\"メールアドレス：\")\n",
    "    clear_output()\n",
    "    \n",
    "    # GIN API Basic Authentication\n",
    "    # refs: https://docs.python-requests.org/en/master/user/authentication/\n",
    "    \n",
    "    # 既存のトークンがあるか確認する\n",
    "    response = requests.get(params['siblings']['ginHttp']+'api/v1/users/' + name + '/tokens', auth=(name, password))\n",
    "    if response.status_code == HTTPStatus.UNAUTHORIZED:\n",
    "        print(\"ユーザ名、またはパスワードが間違っています。\\n恐れ入りますがもう一度ご入力ください。\")\n",
    "        continue\n",
    "    \n",
    "    tokens = response.json()\n",
    "    if len(tokens) >= 1:\n",
    "        access_token = response.json()[-1]\n",
    "        clear_output()\n",
    "        break\n",
    "    elif len(tokens) < 1:\n",
    "        # 既存のトークンがなければ作成する\n",
    "        response = requests.post(params['siblings']['ginHttp']+'api/v1/users/' + name + '/tokens', data={\"name\": \"system-generated\"} ,auth=(name, password))\n",
    "        if response.status_code == HTTPStatus.CREATED:\n",
    "            access_token = response.json()\n",
    "            clear_output()\n",
    "            break\n",
    "    \n",
    "!git config --global user.name $name\n",
    "!git config --global user.email $email\n",
    "!cp ~/.gitconfig ~/WORKFLOW/PACKAGE/.gitconfig\n",
    "\n",
    "print(\"認証が正常に完了しました。次の手順へお進みください。\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47ab2ef7",
   "metadata": {},
   "source": [
    "## 2. データ同期のための設定をする\n",
    "\n",
    "この手順では、今の実行環境とデータガバナンス機能のリポジトリでデータの同期をとるための準備をします。  \n",
    "以下を実行することで、システムがデータ同期の準備の手続きを行います。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "312e1feb",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_through_control": {
     "frozen": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating public/private ed25519 key pair.\n",
      "Your identification has been saved in /home/jovyan/.ssh/id_ed25519\n",
      "Your public key has been saved in /home/jovyan/.ssh/id_ed25519.pub\n",
      "The key fingerprint is:\n",
      "SHA256:rKx4RuQCrB46t5e9ZrJlMrQhmbdtQWZglNuWaog7gqc jovyan@jupyter-ivis-5frino-5fmizuguchi-40rdm-2enii-2eac-2ejp--ivis-2dm\n",
      "The key's randomart image is:\n",
      "+--[ED25519 256]--+\n",
      "|   .+.           |\n",
      "|   ...           |\n",
      "|.    o+.         |\n",
      "|.. oo++.         |\n",
      "|.o+++o. S        |\n",
      "|o.o+=* o         |\n",
      "|+..+=oB          |\n",
      "|B.o.*B+          |\n",
      "|E*o=o=..         |\n",
      "+----[SHA256]-----+\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "#!/bin/bash\n",
    "if [ ! -e ~/.ssh/id_ed25519 ]; then\n",
    "    # 鍵ペアが無ければ作成\n",
    "    ssh-keygen -t ed25519 -N \"\" -f ~/.ssh/id_ed25519\n",
    "fi\n",
    "if [ ! -d ~/.datalad/ ]; then\n",
    "    # Dataladのデータセットでなければデータセット化する\n",
    "    datalad create --force /home/jovyan\n",
    "fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "55a949e7",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_through_control": {
     "frozen": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Public key is ready.\n"
     ]
    }
   ],
   "source": [
    "# 公開鍵アップロード\n",
    "# refs: https://github.com/gogs/docs-api/blob/master/Users/Public%20Keys.md#create-a-public-key\n",
    "import os\n",
    "import requests\n",
    "import time\n",
    "from http import HTTPStatus\n",
    "\n",
    "import json\n",
    "from scripts import utils\n",
    "\n",
    "pubkey = !cat ~/.ssh/id_ed25519.pub\n",
    "\n",
    "# 認証時に取得したトークンを使ってPOSTリクエスト\n",
    "response = requests.post(\n",
    "                params['siblings']['ginHttp']+'api/v1/user/keys?token=' + access_token['sha1'],\n",
    "                data={\n",
    "                    \"title\": \"system-generated-\"+str(time.time()),\n",
    "                    \"key\": pubkey[0]\n",
    "                })\n",
    "msg = response.json()\n",
    "\n",
    "# コンテナを消す際にコンテナとつなぐための公開鍵も削除のため、\n",
    "# パラメータとしてGINから発行された鍵IDを保存\n",
    "if response.status_code == HTTPStatus.CREATED:\n",
    "    # params.jsonへの追記（鍵ID）\n",
    "    params['ginKeyId'] = str(response.json()['id'])\n",
    "    with open(utils.fetch_param_file_path(), mode='w') as f:\n",
    "        json.dump(params, f, indent=4)\n",
    "    print('Public key is ready.')\n",
    "elif msg['message'] == 'Key content has been used as non-deploy key':\n",
    "    print('Public key is ready before time.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e9d09497",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_through_control": {
     "frozen": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SSH connection is ready.\n"
     ]
    }
   ],
   "source": [
    "from datalad import api\n",
    "from IPython.display import clear_output\n",
    "\n",
    "# sibling url をsshに変更する\n",
    "%cd ~/\n",
    "http_url = !git config --get remote.origin.url\n",
    "for item in http_url:\n",
    "    ssh_url = item.replace(params['siblings']['ginHttp'], params['siblings']['ginSsh'])\n",
    "    \n",
    "# siblingsにGINを登録する\n",
    "sibling = !datalad siblings -s gin\n",
    "for item in sibling:\n",
    "    if 'unknown sibling name' in item:\n",
    "        api.siblings(action='add', name='gin', url=ssh_url)\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "clear_output()\n",
    "print('SSH connection is ready.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88c7b474",
   "metadata": {},
   "source": [
    "## 3. リポジトリ内のファイルを更新する"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e18d033c-54af-4e79-be20-d6feaf0b7d8e",
   "metadata": {},
   "source": [
    "### 3-1. Git管理対象外ファイルを.gitignoreで設定する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "baa5f1a7-4097-4b38-9dfa-0126f784fea5",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_through_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "%%sh\n",
    "#!/bin/bash\n",
    "\n",
    "if [ ! -f ~/.gitignore ]; then  # .gitignoreファイルがなければ作成\n",
    "\n",
    "echo \"\n",
    "/.cache/\n",
    "/.conda/\n",
    "/.config/\n",
    "/.ipython/\n",
    "/.local/\n",
    "/.tmp/\n",
    "/.bashrc\n",
    "/.bash_logout\n",
    "/.profile\n",
    "/.netrc\n",
    ".ipynb_checkpoints/\n",
    ".fonts/\n",
    ".jupyter/\n",
    ".npm/\n",
    ".ssh/\n",
    ".jupyter-server-log.txt\n",
    ".gitconfig\n",
    "\" >> ~/.gitignore\n",
    "\n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "781cf0ad-13ff-4bf3-938d-69ad7030fa18",
   "metadata": {},
   "source": [
    "###  3-2. 実験記録管理ファイルを作成する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "75fec939-2f46-45cf-b0c7-d4c00c36f2e3",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_through_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "%%sh\n",
    "#!/bin/bash\n",
    "if [ ! -d ~/experiments/ ]; then\n",
    "    # 実験配置用ディレクトリがなければ作成\n",
    "    mkdir ~/experiments\n",
    "fi\n",
    "\n",
    "if [ ! -f ~/experiments/pipeline.json ]; then\n",
    "    # pipeline.jsonがなければ初期化\n",
    "    echo \"[]\" > ~/experiments/pipeline.json\n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86cc20a7-cc8c-4e80-b892-47de45f07e76",
   "metadata": {},
   "source": [
    "### 3-3 実験パッケージに必要なファイルを追加する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e05b378c-b2cb-47cd-99ff-ea911bd7d10e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan\n"
     ]
    }
   ],
   "source": [
    "%cd ~/\n",
    "\n",
    "path_base_package = 'WORKFLOW/PACKAGE/base'\n",
    "\n",
    "if not os.path.exists(path_base_package + '/.gitignore'):\n",
    "    !cp .gitignore $path_base_package/.gitignore\n",
    "\n",
    "if not os.path.exists(path_base_package + '/EX-WORKFLOW/param_files'):\n",
    "    !mkdir -p $path_base_package/EX-WORKFLOW/param_files\n",
    "    !cp WORKFLOW/FLOW/param_files/params.json $path_base_package/EX-WORKFLOW/param_files/params.json\n",
    "\n",
    "if not os.path.exists(path_base_package + '/EX-WORKFLOW/util/base_datalad_save_push.ipynb'):\n",
    "    !cp WORKFLOW/FLOW/util/base_datalad_save_push.ipynb $path_base_package/EX-WORKFLOW/util/base_datalad_save_push.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b0d26fe-3a6a-4e5e-9975-af37ec2125f5",
   "metadata": {},
   "source": [
    "### 3-4. READMEに実行環境へのリンクを追加する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "42e2a369",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "%cd ~/\n",
    "\n",
    "# READMEに遷移リンクがあれば削除してから、新しく実行環境のリンクを追記する\n",
    "path = 'README.md'\n",
    "s = ''\n",
    "with open(path, 'r') as f:\n",
    "    s = f.read()\n",
    "    s = s[:s.find('## ワークフロー実行が2回目以降の場合')]\n",
    "    \n",
    "with open(path, 'w') as f:\n",
    "    f.write(s)\n",
    "\n",
    "with open(path, 'a', newline='\\n') as f:\n",
    "    f.write(\"\\n## ワークフロー実行が2回目以降の場合\\nワークフロー実行環境へ遷移する場合は以下のリンクをクリックしてください<br>https://jupyter.cs.rcos.nii.ac.jp\" + os.environ[\"JUPYTERHUB_SERVICE_PREFIX\"] + \"notebooks/WORKFLOW/FLOW/base_FLOW.ipynb\")\n",
    "    f.write(\"<br>※上記のリンクからワークフロー機能トップページに遷移できない場合は、「初期設定をする」のmaDMP遷移ボタンからワークフロー機能トップページに遷移してください。その場合maDMPの実行は不要です。\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "771fbbaf",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 4. 実行結果をデータガバナンス機能に同期する\n",
    "\n",
    "ここまでの内容を保存し、データガバナンス機能に同期します。  \n",
    "以下のセルを実行してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94d13667",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ノートブックを保存する\n",
    "from IPython.display import display, Javascript\n",
    "display(Javascript('IPython.notebook.save_checkpoint();'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7eafffd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SSHホスト（＝GIN）を信頼する設定\n",
    "# ドメイン名がハードコーディングにつき要修正\n",
    "with open('/home/jovyan/.ssh/config', mode='w') as f:\n",
    "    f.write('host dg02.dg.rcos.nii.ac.jp\\n\\tStrictHostKeyChecking no\\n\\tUserKnownHostsFile=/dev/null\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5abba0fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import papermill as pm\n",
    "\n",
    "%cd ~/WORKFLOW/FLOW/util\n",
    "pm.execute_notebook(\n",
    "    './base_datalad_save_push.ipynb',\n",
    "    '/home/jovyan/.local/push_log.ipynb',\n",
    "    parameters = dict(SAVE_MESSAGE = '[GIN] 研究リポジトリ初期設定を完了', TO_GIT=True)\n",
    ")\n",
    "print('データ同期が完了しました。')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab25d6d7",
   "metadata": {},
   "source": [
    "## 5. ワークフロー図を更新する\n",
    "\n",
    "ワークフロー図にこのワークフローが実行済みであることを反映します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc2d2061",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/\n",
    "path = 'WORKFLOW/images/notebooks.diag'\n",
    "\n",
    "with open('.gitignore', 'r') as f:\n",
    "    text = f.read()\n",
    "    if text.find(path) == -1:\n",
    "        !echo \"/\" + $path >> ./.gitignore\n",
    "\n",
    "# notebooks.diagのgit管理を外す\n",
    "!git update-index --skip-worktree $path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ccde93b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/\n",
    "find = '\"base_required_every_time\"[fontsize = 14];'\n",
    "replace = '\"base_required_every_time\"[numbered = 済, fontsize = 14];'\n",
    "\n",
    "with open(path, 'r') as f:\n",
    "    s = f.read()\n",
    "\n",
    "with open(path, 'w') as f:\n",
    "    s = s.replace(find, replace)\n",
    "    f.write(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "845ce791",
   "metadata": {},
   "source": [
    "## 6. ワークフロー機能トップページに遷移する\n",
    "\n",
    "続けてワークフロー機能を実行する場合は、[こちら](../base_FLOW.ipynb)からトップページに遷移できます。  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}