demo1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a723089-b919-4aac-b5a9-350e5ac9d29f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import astropy.io.fits as fitsio\n",
    "\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "def setBG_incell(color):    \n",
    "    script = (\n",
    "        \"var cell = this.closest('.jp-CodeCell');\"\n",
    "        \"var editor = cell.querySelector('.jp-Editor');\"\n",
    "        \"editor.style.background='{}';\"\n",
    "        \"this.parentNode.removeChild(this)\"\n",
    "    ).format(color)\n",
    "    \n",
    "    display(HTML('<img src onerror=\"{}\" style=\"display:none\">'.format(script)))\n",
    "\n",
    "def setBG_outcell(color):    \n",
    "    script = (\n",
    "        \"var cell = this.closest('.jp-CodeCell');\"\n",
    "        \"var editor = cell.querySelector('.jp-OutputArea');\"\n",
    "        \"editor.style.background='{}';\"\n",
    "        \"this.parentNode.removeChild(this)\"\n",
    "    ).format(color)\n",
    "    \n",
    "    display(HTML('<img src onerror=\"{}\" style=\"display:none\">'.format(script)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87788753-5d20-4c2d-9633-f67d645b18d2",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfa294a2-fe05-4344-a715-aa45adb708c5",
   "metadata": {},
   "source": [
    "# CSST主巡天仿真\n",
    "+ 仿真软件git仓库: [https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim](https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim)\n",
    "\n",
    "  <img src=\"pics/csstsim_git.png\" alt=\"csst_sim_git.png\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbccb71f-1932-4f91-8c01-c3f367399ae6",
   "metadata": {},
   "source": [
    "## 1. 安装"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33813cc5-8343-445e-8968-ae681185e5da",
   "metadata": {},
   "source": [
    "## 1.1 [**CSST仿真v3.0.0 版本软件使用手册** ](https://www.kdocs.cn/l/cjyiU0SXGyn2)\n",
    "  + 获取代码: **git clone** https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim.git\n",
    "  + 配置环境: anaconda\n",
    "    + **install.sh**, **requirements.txt**\n",
    "    + python3.11\n",
    "    + numpy, astropy, scipy, galsim, healpy, h5py, toml, lmfit, psutil, pyyaml, sep, numba, cython, cfitsio, mpi4py\n",
    "  + 安装软件: **pip install -e .**\n",
    "  + 测试数据: [**data_test2024**](https://pan.baidu.com/s/1WazLj3qCXZqgU7ngw0VKbw?pwd=kbsm)\n",
    "    + PSF数据库\n",
    "      + 多色成像: data_test2024/set1_dynamic\n",
    "      + 无缝光谱: data_test2024/SLS_PSF_PCA_fp\n",
    "    + 测试星表数据\n",
    "      + 恒星星表: data_test2024/starcat_C9\n",
    "      + 星系星表: data_test2024/galcat_C9/cat2CSSTSim_bundle-50sqDeg\n",
    "      + 类星体星表: data_test2024/galcat_C9/qsosed\n",
    "      + fits数据:  data_test2024/stampCatsIndex.hdf5 (stampCats)\n",
    "      + 曝光指向数据: data_test2024/pointing50_C9\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a443dbb-64c3-41c0-818a-e9fa4ef464cf",
   "metadata": {},
   "source": [
    "### 1.2 运行\n",
    "\n",
    "\n",
    "+ <font color=\"grey\"> [mpirun -np N] </font> **python3** LOCALPATH/csst_msc_sim/run_sim.py  **--catalog** C9_Catalog_fits **--config_file** config_overall_test2024.yaml  **-c** LOCALPATH/data_test2024/run_test2024/config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0d31aa9-76ce-412f-8992-5588148a9b25",
   "metadata": {},
   "outputs": [],
   "source": [
    "#配置conda环境\n",
    "!conda info --envs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fcceeca-8720-48aa-abdf-e3fdc90610ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "#查看python版本\n",
    "!python --version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "343e430e-9c51-4f88-95bb-f614ea57dfe6",
   "metadata": {},
   "outputs": [],
   "source": [
    "LOCALPATH=\"/public/home/chengliang/csstsimTest2024\"\n",
    "!ls --color $LOCALPATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3b06b32-4ac2-4b7d-bb29-9e2ae22d2402",
   "metadata": {},
   "outputs": [],
   "source": [
    "#准备测试数据\n",
    "!tree -C -L 2 $LOCALPATH/data_test2024/run_test2024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6210ffd7-389e-4c43-a3c0-b14a461d3c14",
   "metadata": {},
   "outputs": [],
   "source": [
    "#下载CSST仿真软件\n",
    "!tree -C -L 1 $LOCALPATH/csst_msc_sim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f31f465c-8bc8-4133-afaa-fede73b0af3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#进入CSST仿真软件目录并完成安装\n",
    "%cd $LOCALPATH/csst_msc_sim\n",
    "!pip install -e ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d02e07d-8b2a-4f45-8e3d-cd4f79607c2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#检查CSST仿真软件模块\n",
    "import observation_sim\n",
    "observation_sim.__file__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97504b20-40b5-4f8e-a56b-69855ece6d2f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7f8285c4-90a1-43e9-bb5c-d268906963e1",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bfd2585-f7bb-4cd3-8b6e-91ebe7728aa2",
   "metadata": {},
   "source": [
    "## 2.配置"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d39722c-d24a-48e0-b070-e0e11176f5e6",
   "metadata": {},
   "source": [
    "+ data_test2024/run_test2024/config\n",
    "  + 全局配置: config_overall_test2024.yaml，主要是路径信息\n",
    "  + 观测配置: obs_config_SCI_WIDE_phot.yaml，主要是成像chip设置和各类效应开关\n",
    "\n",
    "\n",
    "<img src=\"pics/fp.png\" alt=\"fp.png\" width=\"400\">\n",
    "\n",
    "\n",
    "  + 为了便于检查，可以先按2*8格式输出图像: **format_output: NO**\n",
    "    \n",
    "<img src=\"pics/img_out2x8.png\" alt=\"img_out2x8.png\" width=\"400\"> **$\\rightarrow$** <img src=\"pics/img_out1x16.png\" alt=\"img_out1x16.png\" width=\"500\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95b16f37-7a35-4a14-8807-9c405c0ba41f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd $LOCALPATH/data_test2024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "042f05fe-4252-489c-9bb8-5682a2ef7934",
   "metadata": {},
   "outputs": [],
   "source": [
    "!bat --theme=\"Monokai Extended Bright\" --plain run_test2024/config/config_overall_test2024.yaml \n",
    "setBG_outcell('black')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7f6ec2a-abc5-4be1-84ad-e2c959cae26f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!bat --theme=\"Monokai Extended Bright\" --plain run_test2024/config/obs_config_SCI_WIDE_phot.yaml\n",
    "setBG_outcell('black')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b68c16a-2fd9-4930-bc68-f6df9d6e9584",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ccab997f-9566-4f27-8c9f-5e2531e8b72d",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63e67d64-ef29-4f47-b9e2-7ae7fc2d0061",
   "metadata": {},
   "source": [
    "## 3.多色成像"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47e51d31-5f88-4ffc-bcb0-695e4ebaa718",
   "metadata": {},
   "source": [
    "+ data_test2024/run_test2024/catalog/Catalog_example_test2024.py\n",
    "  + **ra, dec, z**: [e.g., 27.87411627, -47.86911686, 0.]\n",
    "  + **star**: [e.g., 0-galaxy, 1-star, 2-quasar, 3-stamp, 4-calib]\n",
    "  + **mag_use_normal**: [e.g., 20.]\n",
    "  + **sed**: [e.g., wave, flux]\n",
    "\n",
    "<img src=\"pics/img_catalogExample.png\" alt=\"img_catalogExample.png\" width=\"600\"> <img src=\"pics/img_star.png\" alt=\"img_star.png\" width=\"400\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74687bf0-3ca7-4f2d-9ad3-b9c304582b23",
   "metadata": {},
   "outputs": [],
   "source": [
    "#catalog文件需要放到默认路径，即$LOCALPATH/csst_msc_sim/catalog/**\n",
    "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/catalog/Catalog_example_test2024.py\n",
    "setBG_outcell('black')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cad37712-a130-4846-b6ae-ee5827f97c60",
   "metadata": {},
   "outputs": [],
   "source": [
    "#修改配置文件后，运行仿真任务\n",
    "setBG_incell('honeydew')\n",
    "\n",
    "!srun -p debug -n 1 -c 4 python3 /public/home/chengliang/csstsimTest2024/csst_msc_sim/run_sim.py \\\n",
    "                                 --catalog Catalog_example_test2024 \\\n",
    "                                 --config_file config_overall_test2024.yaml \\\n",
    "                                 -c /public/home/chengliang/csstsimTest2024/data_test2024/run_test2024/config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dffd0a5b-597a-4dc0-9880-56292cbff80d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#检查仿真结果\n",
    "!tree $LOCALPATH/data_test2024/run_test2024/outputs/testRunX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8583552-904c-427c-9bb0-4839b03ce83c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#查看仿真星表\n",
    "!cat $LOCALPATH/data_test2024/run_test2024/outputs/testRunX/10109100100413/CSST_MSC_MS_SCI_20231022050242_20231022050512_10109100100413_08_L0_V01.cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00697fbb-dbd2-46ee-b79e-58cbff53a3a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#查看仿真图像(ds9)\n",
    "filename = LOCALPATH + \"/data_test2024/run_test2024/outputs/testRunX/10109100100413/CSST_MSC_MS_SCI_20231022050242_20231022050512_10109100100413_08_L0_V01\"\n",
    "hdu = fitsio.open(filename+\".fits\")\n",
    "\n",
    "fn = open(filename+\".cat\", 'r')\n",
    "header1 = fn.readline()\n",
    "ximg = []\n",
    "yimg = []\n",
    "for line in fn:\n",
    "    line   = line.strip()\n",
    "    columns= line.split()\n",
    "    ximg.append(float(columns[3]))\n",
    "    yimg.append(float(columns[4]))\n",
    "ximg = np.array(ximg)\n",
    "yimg = np.array(yimg)\n",
    "\n",
    "plt.figure(figsize=(12,12))\n",
    "plt.imshow(np.log10(hdu[1].data), origin='lower', vmin=np.log10(438), vmax=np.log10(550), cmap='grey')\n",
    "plt.plot(ximg, yimg,c='#90EE90', marker=\"o\", mfc=\"none\", ms = 20)\n",
    "\n",
    "plt.figure(figsize=(6,6))\n",
    "plt.imshow(np.log10(hdu[1].data)[int(yimg[0])-32:int(yimg[0])+32, int(ximg[0])-32:int(ximg[0])+32], origin='lower',\n",
    "           vmin=np.log10(438), vmax=np.log10(550), cmap='grey', extent=([int(ximg[0])-32, int(ximg[0])+32, int(yimg[0])-32, int(yimg[0])+32]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d917f8e5-c634-4797-9b3e-dee241f3659f",
   "metadata": {},
   "source": [
    "+ data_test2024/run_test2024/catalog/C9_Catalog.py\n",
    "  + 将star/galaxy/quasar加入仿真列表: self.objs.append(obj)\n",
    "  + 大星表推荐使用healpix天区索引\n",
    "  + 更新SED的读取函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "317941f2-8960-4e2f-ab81-a78af40c6952",
   "metadata": {},
   "outputs": [],
   "source": [
    "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/catalog/C9_Catalog.py\n",
    "setBG_outcell('black')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6db5d367-424c-4637-b996-c02beadf2704",
   "metadata": {},
   "source": [
    "  + 打开所有效应的仿真图像\n",
    "  <img src=\"pics/img_cti.png\" alt=\"img_cti.png\" width=\"1000\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6948ab17-3392-437b-9c24-24e573b2c38d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d04845d2-af7d-4b7b-a97b-775b2f1ef365",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "143546c0-8dfb-4b9d-81e9-18e7af5e2f8a",
   "metadata": {},
   "source": [
    "## 4. fits贴图"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34107217-f7ba-4d58-aac2-7d4d1a67cd21",
   "metadata": {},
   "source": [
    "+ steps:\n",
    "  1. 准备fits库(**图像**+**SED**): RA, DEC, z, mag_use_normal, image, pixScale\n",
    "  2. 建立星表索引文件(*.hdf5): tools/index_fits_hdf5.py\n",
    "  3. **修改catalog.py** (e.g. data_test2024/run_test2024/catalog/C9_Catalog_fits.py)\n",
    "     + 将fits数据加入仿真列表:self.objs.append(obj)\n",
    "     + 更新SED的读取函数\n",
    "  5. 修改配置文件: config.yaml\n",
    "  6. 提交仿真任务"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca23e1f3-b1bd-4e25-bdb3-ba01a6ebd5e6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!tree -L 1 $LOCALPATH/data_test2024/stampCats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44f8de98-a170-4c32-a889-c91fd3179b84",
   "metadata": {},
   "outputs": [],
   "source": [
    "#使用index_fits_hdf5.py构建索引文件\n",
    "\n",
    "setBG_incell(\"honeydew\")\n",
    "!cp $LOCALPATH/csst_msc_sim/tools/index_fits_hdf5.py $LOCALPATH/data_test2024\n",
    "#python3 index_fits_hdf5.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a2e5966-d6ca-41be-9c75-4e745fe0c46b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/catalog/C9_Catalog_fits.py\n",
    "setBG_outcell('black')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "193619be-9462-4be6-8201-9da571cb0055",
   "metadata": {},
   "source": [
    "  + 添加fits贴图的测试结果\n",
    "  <img src=\"pics/img_fits.png\" alt=\"img_fits.png\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd7c0489-0817-49da-9f97-db75c03b8b79",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3d9d9d3d-4ba0-45f2-8ee8-bdcbaed30ad8",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2360837e-4cd9-4d26-a61c-1e662db8a783",
   "metadata": {},
   "source": [
    "## 5.实用工具 (csst_msc_sim/tools)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "614ecfdf-a9e0-43c7-9aeb-02959fa4202d",
   "metadata": {},
   "source": [
    "### 5.1 [index_fits_hdf5.py](https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim/-/blob/develop/tools/index_fits_hdf5.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c18c3e1-6914-4d93-94b5-94595c56f0ed",
   "metadata": {},
   "source": [
    "  + 目的：把fits图像库封装成仿真可读取的fits星表格式\n",
    "  + 使用方法：\n",
    "    + 准备fits图像数据库\n",
    "    + 在index_fits_hdf5.py中，指定fits图像目录dir_cat\n",
    "  + 在终端运行：\n",
    "    + python index_fits_hdf5.py \n",
    "  + 输出：代码完成后将生成fits图像索引文件，数据格式为hdf5\n",
    "  + 说明：该程序可独立于仿真代码单独运行，即不需要预先安装仿真代码。如已安装仿真代码，该程序使用方法不受影响。产生的索引文件和对应fits图像库能够用作仿真软件的输入星表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b06f15cf-0399-4c02-9528-1e04a2efd951",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/tools/index_fits_hdf5.py\n",
    "setBG_outcell('black')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83c3c7ef-d53b-4e55-b233-15e838ce866b",
   "metadata": {},
   "source": [
    "### 5.2 [get_PSF.py](https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim/-/blob/develop/tools/get_PSF.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "831dd90d-04b4-4030-9d87-3c19b2e9639e",
   "metadata": {},
   "source": [
    "  + 目的：给定焦面位置，返回对应位置处的光学PSF图像\n",
    "  + 使用方法：\n",
    "      + 给定焦面采样位置x, y = imgPos[iobj, :]，在示例中通过读取仿真输出的.cat星表来设置imgPos\n",
    "  + 在终端运行：\n",
    "      + python get_PSF.py \n",
    "  + 输出：代码完成后将生成对应位置的PSF图像\n",
    "  + 说明：该程序可独立于仿真代码单独运行，即不需要预先安装仿真代码。如已安装仿真代码，该程序使用方法不受影响。产生的PSF图像暂不包含探测器部分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "263c7f7e-e96d-4616-8da1-ced7501c71c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/tools/get_PSF.py\n",
    "setBG_outcell('black')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24effd04-2110-41ff-83dd-879b72aaccfd",
   "metadata": {},
   "source": [
    "### 5.3 ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5df4f734-d89a-4984-acae-4f01973a81ad",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}