{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# モニタリング（研究データ容量）\n",
    "\n",
    "研究データ容量について、DMPに記載された上限を超過していないかモニタリングします。  \n",
    "以下のセルを上から実行してください。2回目以降の実行では、画面上部に表示される以下のボタンをクリックしてから実行して下さい。  \n",
    "![UnfreezeBotton](https://raw.githubusercontent.com/NII-DG/workflow-template/develop/sections/images/unfreeze_button.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 基準値となるデータ容量上限値を取得する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "\n",
    "os.chdir('..')\n",
    "from util.scripts import utils \n",
    "\n",
    "path_params = utils.fetch_monitoring_param_file_path()\n",
    "limit = \"\"\n",
    "with open(path_params, \"r\") as f:\n",
    "    dmp = json.load(f)\n",
    "    limit = dmp[\"dataSize\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"input : \" + str(limit))\n",
    "\n",
    "if isinstance(limit, str):  # \"100\", \"100B\", \"100b\", \"100KB\", \"100kb\"\n",
    "    limit = limit.lower()\n",
    "\n",
    "    if limit[-1] == \"b\":\n",
    "        limit = limit[:-1]\n",
    "\n",
    "    units = [\"k\", \"m\", \"g\", \"t\", \"p\", \"e\", \"z\"]\n",
    "    if limit[-1] in units:\n",
    "        limit = 1024 ** (units.index(limit[-1]) + 1) * int(limit[:-1])\n",
    "\n",
    "print(\"output: \" + str(limit))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 実験パッケージ情報を取得する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import glob\n",
    "\n",
    "# 作業ディレクトリに移動\n",
    "%cd ~/experiments/\n",
    "\n",
    "# 実験パッケージ群の読み込み\n",
    "dirs = glob.glob(\"./*/\")\n",
    "\n",
    "experiments = []\n",
    "for dir_name in dirs:\n",
    "    experiments += [os.path.basename(dir_name[:-1])]\n",
    "print(\"実験パッケージ　：\" + str(experiments))\n",
    "\n",
    "# 元のディレクトリに移動\n",
    "%cd -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 各実験パッケージについて実データを取得する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for experiment in experiments:\n",
    "    %cd ~/experiments/$experiment\n",
    "    !git pull\n",
    "    !datalad get -r .\n",
    "    %cd ../"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. データ容量をチェックする"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "isSucceeded = True\n",
    "\n",
    "result = !du -s ~/experiments/\n",
    "result = result[0].split(\"\\t\")[0]\n",
    "result = int(result)\n",
    "print(\"datasize: \" + str(result))\n",
    "\n",
    "if result > limit:\n",
    "    isSucceeded = False\n",
    "    raise Exception(\"データ容量を超過しています\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 4. モニタリング結果をREADMEに反映する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from util.scripts import utils\n",
    "\n",
    "utils.reflect_monitoring_results('dataAmount', isSucceeded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 実行結果をデータガバナンス機能に同期する\n",
    "\n",
    "ここまでの内容を保存し、データガバナンス機能に同期します。  \n",
    "以下のセルを実行してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Javascript\n",
    "display(Javascript('IPython.notebook.save_checkpoint();'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import papermill as pm\n",
    "\n",
    "%cd ~/\n",
    "\n",
    "pm.execute_notebook(\n",
    "    'WORKFLOW/FLOW/util/base_datalad_save_push.ipynb',\n",
    "    '/home/jovyan/.local/push_log.ipynb',\n",
    "    parameters = dict(SAVE_MESSAGE = 'モニタリング（研究データ容量）', IS_RECURSIVE = False, TO_GIT = True)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 6. FLOWに遷移する\n",
    "\n",
    "続けてワークフローを実行する場合は、[こちら](../base_FLOW.ipynb)からFLOWに遷移できます。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}