Using ``csst_common`` ===================== In this chapter, we introduce the basic usages of some IMPORTANT functions / classes in ``csst_common``. Package source code and installation guide can be found in the link below. - https://csst-tb.bao.ac.cn/code/csst-l1/csst_common ``csst_common.data_manager.CsstMsDataManager`` ---------------------------------------------- This class provides an interface to switch between local files and DFS. To initilize from local directory: .. code-block:: python from csst_common.data_manager import CsstMsDataManager # for full basic initialization dm = CsstMsDataManager.from_dir( ver_sim="C5.2", # version of simulation, "C5.2" is the latest datatype="mbi", # data type, "mbi", "sls" or "all" dir_l0=".", # the L0/input directory dir_l1=".", # the L1/output directory path_aux="", # aux file paths (master bias, dark, flat) dfs_mode=False, # if True, use DFS in pipeline modules dfs_node='kmust', # DFS server, chosen from {"kmust", "pml"} ) dm.target_detectors = None # use all available detectors In particular, if you are on ``dandelion`` or ``PML node``, you can use a ``quickstart`` method. .. code-block:: python # for a quick start () dm = CsstMsDataManager.quickstart( ver_sim='C5.2', # version of simulation, only "C5.2" is supported datatype='mbi', # data type, "mbi", "sls" or "all" dir_l1='.', # the L1/output directory exposure_id=100, # the exposure ID dfs_mode=False, # if True, use DFS in pipeline modules dfs_node='kmust', # DFS server, chosen from {"kmust", "pml"} ) dm You will get the outputs: .. code-block:: Data type = mbi Valid detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25] Available detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25] Target detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25] dir_l0 = /nfsdata/share/csst_simulation_data/Cycle-5-SimuData/multipleBandsImaging/NGP_AstrometryON_shearOFF/MSC_0000100/ dir_l1 = . dfs_mode = False dfs_node = kmust .. note:: This ``quickstart`` method is ONLY available on ``dandelion`` or the ``PML node``. In your local development environment, you have to use ``CsstMsDataManager.from_dir(...)`` to initialize it. Attributes ^^^^^^^^^^ Some attributes can be accessed as below. .. code-block:: python # access L0 directory dm.dir_l0 # access L1 directory dm.dir_l1 # access dir_pcref dm.dir_pcref # access path_aux dm.path_aux # access ver_sim dm.ver_sim # access target detectors dm.target_detectors # access available detectors dm.available_detectors # define an L1 file (detector-specified) dm.l1_detector(detector=6) # define an L1 file (non-detector-specified) dm.l1_file("flipped_image.fits") To get master files: .. code-block:: python # to get bias dm.get_bias() # to get dark dm.get_dark() # to get flat dm.get_flat() Query Reference Catalog ^^^^^^^^^^^^^^^^^^^^^^^ ``CsstMsDataManager`` provides a method ``query_rc`` to query reference catalog from ``DFS``. .. code-block:: python cat = dm.query_rc(ra=180, dec=0, radius=2, min_mag=0, max_mag=30) if cat is None: print("The query failed!") else: print("The query succeeded!") Access code to data products ---------------------------- ``Multi-Band Imaging`` ^^^^^^^^^^^^^^^^^^^^^^ ========================== ===== ========================================================== ================== Data source Level File name / access code Notes ========================== ===== ========================================================== ================== raw 0 ``dm.l0_detector(detector=detector)`` sci image raw 0 ``dm.l0_crs(detector=detector)`` cosmic ray image raw 0 ``dm.l0_cat(detector=detector)`` input catalog raw 0 ``dm.l0_log(detector=detector)`` log file ``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="img.fits")`` corrected image ``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="flg.fits")`` flag image ``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="wht.fits")`` weight image ``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="img.head")`` L0 fits header ``DFS`` -- ``dm.l1_file(name="gaia_dfs.fits")`` reference catalog ``csst_ms_mbi_distortion`` -- ``dm.l1_detector(detector=detector, post="wcs.head")`` fits head with WCS ``csst_ms_mbi_position`` -- ``dm.l1_detector(detector=detector, post="img.acat")`` photometry catalog ``csst_ms_mbi_position`` -- ``dm.l1_detector(detector=detector, post="img.rcat")`` reference catalog ``csst_ms_mbi_flux`` 1 ``dm.l1_detector(detector=detector, post="img_L1.fits")`` sci image ``csst_ms_mbi_flux`` 1 ``dm.l1_detector(detector=detector, post="flg_L1.fits")`` flag image ``csst_ms_mbi_flux`` 1 ``dm.l1_detector(detector=detector, post="wht_L1.fits")`` weight image ``csst_ms_mbi_photometry`` 1 ``dm.l1_detector(detector=detector, post="cat.fits")`` source catalog ``csst_ms_mbi_photometry`` 1 ``dm.l1_detector(detector=detector, post="psf.fits")`` source catalog ========================== ===== ========================================================== ================== ``Slit-Less Spectra 2D`` ^^^^^^^^^^^^^^^^^^^^^^^^ ========================== ===== =========================================================== ================== Data source Level File name / access code Notes ========================== ===== =========================================================== ================== raw 0 ``dm.l0_detector(detector=detector)`` sci image raw 0 ``dm.l0_crs(detector=detector)`` cosmic ray image raw 0 ``dm.l0_cat(detector=detector)`` input catalog raw 0 ``dm.l0_log(detector=detector)`` log file ``csst_ms_sls_instrument`` 1 ``dm.l1_detector(detector=detector, post="L1_1_flt.fits")`` corrected image ``csst_ms_sls_position`` -- ``dm.l1_detector(detector=detector, post="L1_1.fits")`` WCS determined ========================== ===== =========================================================== ================== ``Slit-Less Spectra 1D`` ^^^^^^^^^^^^^^^^^^^^^^^^ Note that for 1D pipeline, ``dm.target_detectors`` is a list containing only one element. =========================== ===== =============================================================== ================== Data source Level File name / access code Notes =========================== ===== =============================================================== ================== ``csst_ms_sls_position`` -- ``dm.l1_detector(detector=detector, post="L1_1.fits")`` L1 SLS image ``csst_common`` -- ``dm.target_detectors[0]`` detector number ``csst_ms_sls_mosaic`` -- ``dm.l1_detector(detector=detector, post="coadd.fits")`` coadded image ``csst_ms_sls_mosaic`` -- ``dm.l1_detector(detector=detector, post="coadd.weight.fits")`` coadded weight ``csst_ms_sls_directimage`` -- ``dm.l1_detector(detector=detector, post="dir_img.fits")`` cropped image ``csst_ms_sls_directimage`` -- ``dm.l1_detector(detector=detector, post="dir_wht.fits")`` cropped weight ``csst_ms_sls_objextract`` -- ``dm.l1_detector(detector=detector, post="comb.cat")`` combined catalog ``csst_ms_sls_objextract`` -- ``dm.l1_detector(detector=detector, post="segmentation.fits")`` segmentation file ``csst_ms_sls_sky`` -- ``dm.l1_detector(detector=detector, post="L1_1_sky.fits")`` sky background ``csst_ms_sls_axe`` -- ``dm.l1_detector(detector=detector, post="L1_1_axe.fits")`` aXe spectra ``csst_ms_sls_cde`` -- ``dm.l1_detector(detector=detector, post="L1_1_cde.fits")`` CDE spectra =========================== ===== =============================================================== ================== ``csst_common.file_recorder.FileRecorder`` ------------------------------------------ Get an empty ``FileRecorder``. This is initially a ``list``-like object. Use ``FileRecorder.add_record()`` to add file records. .. code-block:: :linenos: >>> from csst_common.file_recorder import FileRecorder >>> fr = FileRecorder() >>> for i in range(3): >>> fr.add_record(filepath="test{:03d}.txt".format(i), >>> db=True, >>> comment="Test file {:d}".format(i)) >>> fr.pprint_all() filepath db comment existence ----------- ---- ----------- --------- test000.txt True Test file 0 False test001.txt True Test file 1 False test002.txt True Test file 2 False ``FileRecorder`` inherits from ``list``. Therefore, if you are collecting records from subprocesses, here is the way to combine records: .. code-block:: python :linenos: >>> from csst_common.file_recorder import FileRecorder >>> fr1 = FileRecorder() >>> fr1.add_record(filepath="test1.txt", db=False, comment="test file 1") >>> fr2 = FileRecorder() >>> fr2.add_record(filepath="test2.txt", db=False, comment="test file 2") >>> fr_all = FileRecorder([*fr1, *fr2]) >>> fr_all.pprint_all() filepath db comment existence --------- ----- ----------- --------- test1.txt False test file 1 False test2.txt False test file 2 False >>> fr_all.extend(fr1) >>> fr_all.pprint_all() filepath db comment existence --------- ----- ----------- --------- test1.txt False test file 1 False test2.txt False test file 2 False test1.txt False test file 1 False ``csst_common.logger.get_logger()`` ----------------------------------- Get the default configured ``logging.Logger``. .. code-block:: python :linenos: from csst_common.logger import get_logger logger = get_logger() .. warning:: Developers should NOT use ``print`` function extensively in code. Generally ``print`` can be replaced with ``logger.debug("msg")``. For important information, use ``logger.info("msg")``. For warnings, use ``logger.warning("msg")``. For errors, use ``logger.error("msg")``. Conf https://docs.python.org/3/library/logging.html for more information about the usage of ``logging.Logger``. ``csst_common.status.CsstStatus`` --------------------------------- Developers should use ``csst_common.status.CsstStatus`` to return the status of their interfaces. .. code-block:: python :linenos: from csst_common.status import CsstStatus # presently 3 kinds of status are available, they are CsstStatus.PERFECT # all images are processed smoothly CsstStatus.WARNING # problematic CsstStatus.ERROR # unavailable, DO NOT use this An example interface -------------------- We recommend our developers to use the code structure used in the example interfaces, e.g., ``process_single_image`` and ``process_multiple_images``. The example code is shown below. Source code ^^^^^^^^^^^ .. literalinclude:: csst_common/example_interface.py :caption: ``example_interface.py`` :emphasize-lines: 7-11,25-29,75-88,103-116,173-175 :linenos: :language: python Rendered ``docstring`` ^^^^^^^^^^^^^^^^^^^^^^ .. automodule:: example_interface :members: ``csst_common.params`` ---------------------- See https://csst-tb.bao.ac.cn/code/csst-l1/csst_common/-/blob/main/csst_common/data/csst_params.yml. To use the parameters, use .. code-block:: python from csst_common.params import CSST_PARAMS as CP Module Identifier ----------------- to be updated DFS interfaces -------------- ==================================== ====================== ========================================== Method Function Original DFS interface ==================================== ====================== ========================================== ``CsstMsDataManager.dfs_query_rc()`` query catalog from DFS ``csst_dfs_api.common.catalog.CatalogApi`` ``CsstMsDataManager.dfs_query_l0()`` query L0 data from DFS -- ``CsstMsDataManager.dfs_query_l1()`` query L1 data from DFS -- ``CsstMsDataManager.dfs_push_l1()`` push L1 data to DFS ``csst_dfs_api.msc.level1.Level1DataApi`` ==================================== ====================== ==========================================