{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "0a723089-b919-4aac-b5a9-350e5ac9d29f", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import astropy.io.fits as fitsio\n", "\n", "from IPython.display import HTML, display\n", "\n", "def setBG_incell(color): \n", " script = (\n", " \"var cell = this.closest('.jp-CodeCell');\"\n", " \"var editor = cell.querySelector('.jp-Editor');\"\n", " \"editor.style.background='{}';\"\n", " \"this.parentNode.removeChild(this)\"\n", " ).format(color)\n", " \n", " display(HTML(''.format(script)))\n", "\n", "def setBG_outcell(color): \n", " script = (\n", " \"var cell = this.closest('.jp-CodeCell');\"\n", " \"var editor = cell.querySelector('.jp-OutputArea');\"\n", " \"editor.style.background='{}';\"\n", " \"this.parentNode.removeChild(this)\"\n", " ).format(color)\n", " \n", " display(HTML(''.format(script)))" ] }, { "cell_type": "markdown", "id": "87788753-5d20-4c2d-9633-f67d645b18d2", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "bfa294a2-fe05-4344-a715-aa45adb708c5", "metadata": {}, "source": [ "# CSST主巡天仿真\n", "+ 仿真软件git仓库: [https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim](https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim)\n", "\n", " \"csst_sim_git.png\"" ] }, { "cell_type": "markdown", "id": "bbccb71f-1932-4f91-8c01-c3f367399ae6", "metadata": {}, "source": [ "## 1. 安装" ] }, { "cell_type": "markdown", "id": "33813cc5-8343-445e-8968-ae681185e5da", "metadata": {}, "source": [ "## 1.1 [**CSST仿真v3.0.0 版本软件使用手册** ](https://www.kdocs.cn/l/cjyiU0SXGyn2)\n", " + 获取代码: **git clone** https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim.git\n", " + 配置环境: anaconda\n", " + **install.sh**, **requirements.txt**\n", " + python3.11\n", " + numpy, astropy, scipy, galsim, healpy, h5py, toml, lmfit, psutil, pyyaml, sep, numba, cython, cfitsio, mpi4py\n", " + 安装软件: **pip install -e .**\n", " + 测试数据: [**data_test2024**](https://pan.baidu.com/s/1WazLj3qCXZqgU7ngw0VKbw?pwd=kbsm)\n", " + PSF数据库\n", " + 多色成像: data_test2024/set1_dynamic\n", " + 无缝光谱: data_test2024/SLS_PSF_PCA_fp\n", " + 测试星表数据\n", " + 恒星星表: data_test2024/starcat_C9\n", " + 星系星表: data_test2024/galcat_C9/cat2CSSTSim_bundle-50sqDeg\n", " + 类星体星表: data_test2024/galcat_C9/qsosed\n", " + fits数据: data_test2024/stampCatsIndex.hdf5 (stampCats)\n", " + 曝光指向数据: data_test2024/pointing50_C9\n", "\n" ] }, { "cell_type": "markdown", "id": "1a443dbb-64c3-41c0-818a-e9fa4ef464cf", "metadata": {}, "source": [ "### 1.2 运行\n", "\n", "\n", "+ [mpirun -np N] **python3** LOCALPATH/csst_msc_sim/run_sim.py **--catalog** C9_Catalog_fits **--config_file** config_overall_test2024.yaml **-c** LOCALPATH/data_test2024/run_test2024/config" ] }, { "cell_type": "code", "execution_count": null, "id": "e0d31aa9-76ce-412f-8992-5588148a9b25", "metadata": {}, "outputs": [], "source": [ "#配置conda环境\n", "!conda info --envs" ] }, { "cell_type": "code", "execution_count": null, "id": "3fcceeca-8720-48aa-abdf-e3fdc90610ef", "metadata": {}, "outputs": [], "source": [ "#查看python版本\n", "!python --version" ] }, { "cell_type": "code", "execution_count": null, "id": "343e430e-9c51-4f88-95bb-f614ea57dfe6", "metadata": {}, "outputs": [], "source": [ "LOCALPATH=\"/public/home/chengliang/csstsimTest2024\"\n", "!ls --color $LOCALPATH" ] }, { "cell_type": "code", "execution_count": null, "id": "f3b06b32-4ac2-4b7d-bb29-9e2ae22d2402", "metadata": {}, "outputs": [], "source": [ "#准备测试数据\n", "!tree -C -L 2 $LOCALPATH/data_test2024/run_test2024" ] }, { "cell_type": "code", "execution_count": null, "id": "6210ffd7-389e-4c43-a3c0-b14a461d3c14", "metadata": {}, "outputs": [], "source": [ "#下载CSST仿真软件\n", "!tree -C -L 1 $LOCALPATH/csst_msc_sim" ] }, { "cell_type": "code", "execution_count": null, "id": "f31f465c-8bc8-4133-afaa-fede73b0af3a", "metadata": {}, "outputs": [], "source": [ "#进入CSST仿真软件目录并完成安装\n", "%cd $LOCALPATH/csst_msc_sim\n", "!pip install -e ." ] }, { "cell_type": "code", "execution_count": null, "id": "1d02e07d-8b2a-4f45-8e3d-cd4f79607c2f", "metadata": {}, "outputs": [], "source": [ "#检查CSST仿真软件模块\n", "import observation_sim\n", "observation_sim.__file__" ] }, { "cell_type": "code", "execution_count": null, "id": "97504b20-40b5-4f8e-a56b-69855ece6d2f", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "7f8285c4-90a1-43e9-bb5c-d268906963e1", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "9bfd2585-f7bb-4cd3-8b6e-91ebe7728aa2", "metadata": {}, "source": [ "## 2.配置" ] }, { "cell_type": "markdown", "id": "9d39722c-d24a-48e0-b070-e0e11176f5e6", "metadata": {}, "source": [ "+ data_test2024/run_test2024/config\n", " + 全局配置: config_overall_test2024.yaml，主要是路径信息\n", " + 观测配置: obs_config_SCI_WIDE_phot.yaml，主要是成像chip设置和各类效应开关\n", "\n", "\n", "\"fp.png\"\n", "\n", "\n", " + 为了便于检查，可以先按2*8格式输出图像: **format_output: NO**\n", " \n", "\"img_out2x8.png\" **$\\rightarrow$** \"img_out1x16.png\"\n" ] }, { "cell_type": "code", "execution_count": null, "id": "95b16f37-7a35-4a14-8807-9c405c0ba41f", "metadata": {}, "outputs": [], "source": [ "%cd $LOCALPATH/data_test2024" ] }, { "cell_type": "code", "execution_count": null, "id": "042f05fe-4252-489c-9bb8-5682a2ef7934", "metadata": {}, "outputs": [], "source": [ "!bat --theme=\"Monokai Extended Bright\" --plain run_test2024/config/config_overall_test2024.yaml \n", "setBG_outcell('black')" ] }, { "cell_type": "code", "execution_count": null, "id": "e7f6ec2a-abc5-4be1-84ad-e2c959cae26f", "metadata": { "scrolled": true }, "outputs": [], "source": [ "!bat --theme=\"Monokai Extended Bright\" --plain run_test2024/config/obs_config_SCI_WIDE_phot.yaml\n", "setBG_outcell('black')" ] }, { "cell_type": "code", "execution_count": null, "id": "7b68c16a-2fd9-4930-bc68-f6df9d6e9584", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "ccab997f-9566-4f27-8c9f-5e2531e8b72d", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "63e67d64-ef29-4f47-b9e2-7ae7fc2d0061", "metadata": {}, "source": [ "## 3.多色成像" ] }, { "cell_type": "markdown", "id": "47e51d31-5f88-4ffc-bcb0-695e4ebaa718", "metadata": {}, "source": [ "+ data_test2024/run_test2024/catalog/Catalog_example_test2024.py\n", " + **ra, dec, z**: [e.g., 27.87411627, -47.86911686, 0.]\n", " + **star**: [e.g., 0-galaxy, 1-star, 2-quasar, 3-stamp, 4-calib]\n", " + **mag_use_normal**: [e.g., 20.]\n", " + **sed**: [e.g., wave, flux]\n", "\n", "\"img_catalogExample.png\" \"img_star.png\"\n" ] }, { "cell_type": "code", "execution_count": null, "id": "74687bf0-3ca7-4f2d-9ad3-b9c304582b23", "metadata": {}, "outputs": [], "source": [ "#catalog文件需要放到默认路径，即$LOCALPATH/csst_msc_sim/catalog/**\n", "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/catalog/Catalog_example_test2024.py\n", "setBG_outcell('black')" ] }, { "cell_type": "code", "execution_count": null, "id": "cad37712-a130-4846-b6ae-ee5827f97c60", "metadata": {}, "outputs": [], "source": [ "#修改配置文件后，运行仿真任务\n", "setBG_incell('honeydew')\n", "\n", "!srun -p debug -n 1 -c 4 python3 /public/home/chengliang/csstsimTest2024/csst_msc_sim/run_sim.py \\\n", " --catalog Catalog_example_test2024 \\\n", " --config_file config_overall_test2024.yaml \\\n", " -c /public/home/chengliang/csstsimTest2024/data_test2024/run_test2024/config" ] }, { "cell_type": "code", "execution_count": null, "id": "dffd0a5b-597a-4dc0-9880-56292cbff80d", "metadata": {}, "outputs": [], "source": [ "#检查仿真结果\n", "!tree $LOCALPATH/data_test2024/run_test2024/outputs/testRunX" ] }, { "cell_type": "code", "execution_count": null, "id": "a8583552-904c-427c-9bb0-4839b03ce83c", "metadata": {}, "outputs": [], "source": [ "#查看仿真星表\n", "!cat $LOCALPATH/data_test2024/run_test2024/outputs/testRunX/10109100100413/CSST_MSC_MS_SCI_20231022050242_20231022050512_10109100100413_08_L0_V01.cat" ] }, { "cell_type": "code", "execution_count": null, "id": "00697fbb-dbd2-46ee-b79e-58cbff53a3a6", "metadata": {}, "outputs": [], "source": [ "#查看仿真图像(ds9)\n", "filename = LOCALPATH + \"/data_test2024/run_test2024/outputs/testRunX/10109100100413/CSST_MSC_MS_SCI_20231022050242_20231022050512_10109100100413_08_L0_V01\"\n", "hdu = fitsio.open(filename+\".fits\")\n", "\n", "fn = open(filename+\".cat\", 'r')\n", "header1 = fn.readline()\n", "ximg = []\n", "yimg = []\n", "for line in fn:\n", " line = line.strip()\n", " columns= line.split()\n", " ximg.append(float(columns[3]))\n", " yimg.append(float(columns[4]))\n", "ximg = np.array(ximg)\n", "yimg = np.array(yimg)\n", "\n", "plt.figure(figsize=(12,12))\n", "plt.imshow(np.log10(hdu[1].data), origin='lower', vmin=np.log10(438), vmax=np.log10(550), cmap='grey')\n", "plt.plot(ximg, yimg,c='#90EE90', marker=\"o\", mfc=\"none\", ms = 20)\n", "\n", "plt.figure(figsize=(6,6))\n", "plt.imshow(np.log10(hdu[1].data)[int(yimg[0])-32:int(yimg[0])+32, int(ximg[0])-32:int(ximg[0])+32], origin='lower',\n", " vmin=np.log10(438), vmax=np.log10(550), cmap='grey', extent=([int(ximg[0])-32, int(ximg[0])+32, int(yimg[0])-32, int(yimg[0])+32]))" ] }, { "cell_type": "markdown", "id": "d917f8e5-c634-4797-9b3e-dee241f3659f", "metadata": {}, "source": [ "+ data_test2024/run_test2024/catalog/C9_Catalog.py\n", " + 将star/galaxy/quasar加入仿真列表: self.objs.append(obj)\n", " + 大星表推荐使用healpix天区索引\n", " + 更新SED的读取函数" ] }, { "cell_type": "code", "execution_count": null, "id": "317941f2-8960-4e2f-ab81-a78af40c6952", "metadata": {}, "outputs": [], "source": [ "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/catalog/C9_Catalog.py\n", "setBG_outcell('black')" ] }, { "cell_type": "markdown", "id": "6db5d367-424c-4637-b996-c02beadf2704", "metadata": {}, "source": [ " + 打开所有效应的仿真图像\n", " \"img_cti.png\"\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6948ab17-3392-437b-9c24-24e573b2c38d", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "d04845d2-af7d-4b7b-a97b-775b2f1ef365", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "143546c0-8dfb-4b9d-81e9-18e7af5e2f8a", "metadata": {}, "source": [ "## 4. fits贴图" ] }, { "cell_type": "markdown", "id": "34107217-f7ba-4d58-aac2-7d4d1a67cd21", "metadata": {}, "source": [ "+ steps:\n", " 1. 准备fits库(**图像**+**SED**): RA, DEC, z, mag_use_normal, image, pixScale\n", " 2. 建立星表索引文件(*.hdf5): tools/index_fits_hdf5.py\n", " 3. **修改catalog.py** (e.g. data_test2024/run_test2024/catalog/C9_Catalog_fits.py)\n", " + 将fits数据加入仿真列表:self.objs.append(obj)\n", " + 更新SED的读取函数\n", " 5. 修改配置文件: config.yaml\n", " 6. 提交仿真任务" ] }, { "cell_type": "code", "execution_count": null, "id": "ca23e1f3-b1bd-4e25-bdb3-ba01a6ebd5e6", "metadata": { "scrolled": true }, "outputs": [], "source": [ "!tree -L 1 $LOCALPATH/data_test2024/stampCats" ] }, { "cell_type": "code", "execution_count": null, "id": "44f8de98-a170-4c32-a889-c91fd3179b84", "metadata": {}, "outputs": [], "source": [ "#使用index_fits_hdf5.py构建索引文件\n", "\n", "setBG_incell(\"honeydew\")\n", "!cp $LOCALPATH/csst_msc_sim/tools/index_fits_hdf5.py $LOCALPATH/data_test2024\n", "#python3 index_fits_hdf5.py" ] }, { "cell_type": "code", "execution_count": null, "id": "8a2e5966-d6ca-41be-9c75-4e745fe0c46b", "metadata": {}, "outputs": [], "source": [ "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/catalog/C9_Catalog_fits.py\n", "setBG_outcell('black')" ] }, { "cell_type": "markdown", "id": "193619be-9462-4be6-8201-9da571cb0055", "metadata": {}, "source": [ " + 添加fits贴图的测试结果\n", " \"img_fits.png\"" ] }, { "cell_type": "code", "execution_count": null, "id": "cd7c0489-0817-49da-9f97-db75c03b8b79", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "3d9d9d3d-4ba0-45f2-8ee8-bdcbaed30ad8", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "2360837e-4cd9-4d26-a61c-1e662db8a783", "metadata": {}, "source": [ "## 5.实用工具 (csst_msc_sim/tools)" ] }, { "cell_type": "markdown", "id": "614ecfdf-a9e0-43c7-9aeb-02959fa4202d", "metadata": {}, "source": [ "### 5.1 [index_fits_hdf5.py](https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim/-/blob/develop/tools/index_fits_hdf5.py)" ] }, { "cell_type": "markdown", "id": "5c18c3e1-6914-4d93-94b5-94595c56f0ed", "metadata": {}, "source": [ " + 目的：把fits图像库封装成仿真可读取的fits星表格式\n", " + 使用方法：\n", " + 准备fits图像数据库\n", " + 在index_fits_hdf5.py中，指定fits图像目录dir_cat\n", " + 在终端运行：\n", " + python index_fits_hdf5.py \n", " + 输出：代码完成后将生成fits图像索引文件，数据格式为hdf5\n", " + 说明：该程序可独立于仿真代码单独运行，即不需要预先安装仿真代码。如已安装仿真代码，该程序使用方法不受影响。产生的索引文件和对应fits图像库能够用作仿真软件的输入星表。" ] }, { "cell_type": "code", "execution_count": null, "id": "b06f15cf-0399-4c02-9528-1e04a2efd951", "metadata": { "scrolled": true }, "outputs": [], "source": [ "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/tools/index_fits_hdf5.py\n", "setBG_outcell('black')" ] }, { "cell_type": "markdown", "id": "83c3c7ef-d53b-4e55-b233-15e838ce866b", "metadata": {}, "source": [ "### 5.2 [get_PSF.py](https://csst-tb.bao.ac.cn/code/csst-sims/csst_msc_sim/-/blob/develop/tools/get_PSF.py)" ] }, { "cell_type": "markdown", "id": "831dd90d-04b4-4030-9d87-3c19b2e9639e", "metadata": {}, "source": [ " + 目的：给定焦面位置，返回对应位置处的光学PSF图像\n", " + 使用方法：\n", " + 给定焦面采样位置x, y = imgPos[iobj, :]，在示例中通过读取仿真输出的.cat星表来设置imgPos\n", " + 在终端运行：\n", " + python get_PSF.py \n", " + 输出：代码完成后将生成对应位置的PSF图像\n", " + 说明：该程序可独立于仿真代码单独运行，即不需要预先安装仿真代码。如已安装仿真代码，该程序使用方法不受影响。产生的PSF图像暂不包含探测器部分。" ] }, { "cell_type": "code", "execution_count": null, "id": "263c7f7e-e96d-4616-8da1-ced7501c71c5", "metadata": {}, "outputs": [], "source": [ "!bat --theme=\"Monokai Extended Bright\" --plain $LOCALPATH/csst_msc_sim/tools/get_PSF.py\n", "setBG_outcell('black')" ] }, { "cell_type": "markdown", "id": "24effd04-2110-41ff-83dd-879b72aaccfd", "metadata": {}, "source": [ "### 5.3 ..." ] }, { "cell_type": "code", "execution_count": null, "id": "5df4f734-d89a-4984-acae-4f01973a81ad", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 5 }