Newer
Older
In this chapter, we introduce the basic usages of some IMPORTANT functions / classes in ``csst_common``.
Package source code and installation guide can be found in the link below.
- https://csst-tb.bao.ac.cn/code/csst-l1/csst_common
``csst_common.data_manager.CsstMsDataManager``
----------------------------------------------
This class provides an interface to switch between local files and DFS.
To initilize from local directory:
datatype="mbi", # data type, "mbi", "sls" or "all"
dir_l0=".", # the L0/input directory
dir_l1=".", # the L1/output directory
dfs_mode=False, # if True, use DFS in pipeline modules
dfs_node='kmust', # DFS server, chosen from {"kmust", "pml"}
dm.target_detectors = None # use all available detectors
In particular, if you are on ``dandelion`` or ``PML node``, you can use a ``quickstart`` method.
dm = CsstMsDataManager.quickstart(
ver_sim='C5.2', # version of simulation, only "C5.2" is supported
datatype='mbi', # data type, "mbi", "sls" or "all"
dir_l1='.', # the L1/output directory
exposure_id=100, # the exposure ID
dfs_mode=False, # if True, use DFS in pipeline modules
dfs_node='kmust', # DFS server, chosen from {"kmust", "pml"}
dm
You will get the outputs:
.. code-block::
<CsstMsDataManager>
Data type = mbi
Valid detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25]
Available detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25]
Target detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25]
dir_l0 = /nfsdata/share/csst_simulation_data/Cycle-5-SimuData/multipleBandsImaging/NGP_AstrometryON_shearOFF/MSC_0000100/
dir_l1 = .
dfs_mode = False
dfs_node = kmust
``CsstMsDataManager.from_dir(...)`` to initialize it.
dm.get_flat()
Query Reference Catalog
^^^^^^^^^^^^^^^^^^^^^^^
``CsstMsDataManager`` provides a method ``query_rc`` to query reference catalog from ``DFS``.
.. code-block:: python
cat = dm.query_rc(ra=180, dec=0, radius=2, min_mag=0, max_mag=30)
if cat is None:
print("The query failed!")
else:
print("The query succeeded!")
Access code to data products
----------------------------
``Multi-Band Imaging``
^^^^^^^^^^^^^^^^^^^^^^
========================== ===== ========================================================== ==================
Data source Level File name / access code Notes
========================== ===== ========================================================== ==================
raw 0 ``dm.l0_detector(detector=detector)`` sci image
raw 0 ``dm.l0_crs(detector=detector)`` cosmic ray image
raw 0 ``dm.l0_cat(detector=detector)`` input catalog
raw 0 ``dm.l0_log(detector=detector)`` log file
``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="img.fits")`` corrected image
``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="flg.fits")`` flag image
``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="wht.fits")`` weight image
``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="img.head")`` L0 fits header
``DFS`` -- ``dm.l1_file(name="gaia_dfs.fits")`` reference catalog
``csst_ms_mbi_distortion`` -- ``dm.l1_detector(detector=detector, post="wcs.head")`` fits head with WCS
``csst_ms_mbi_position`` -- ``dm.l1_detector(detector=detector, post="img.acat")`` photometry catalog
``csst_ms_mbi_position`` -- ``dm.l1_detector(detector=detector, post="img.rcat")`` reference catalog
``csst_ms_mbi_flux`` 1 ``dm.l1_detector(detector=detector, post="img_L1.fits")`` sci image
``csst_ms_mbi_flux`` 1 ``dm.l1_detector(detector=detector, post="flg_L1.fits")`` flag image
``csst_ms_mbi_flux`` 1 ``dm.l1_detector(detector=detector, post="wht_L1.fits")`` weight image
``csst_ms_mbi_photometry`` 1 ``dm.l1_detector(detector=detector, post="cat.fits")`` source catalog
``csst_ms_mbi_photometry`` 1 ``dm.l1_detector(detector=detector, post="psf.fits")`` source catalog
========================== ===== ========================================================== ==================
``Slit-Less Spectra 2D``
^^^^^^^^^^^^^^^^^^^^^^^^
========================== ===== =========================================================== ==================
Data source Level File name / access code Notes
========================== ===== =========================================================== ==================
raw 0 ``dm.l0_detector(detector=detector)`` sci image
raw 0 ``dm.l0_crs(detector=detector)`` cosmic ray image
raw 0 ``dm.l0_cat(detector=detector)`` input catalog
raw 0 ``dm.l0_log(detector=detector)`` log file
``csst_ms_sls_instrument`` 1 ``dm.l1_detector(detector=detector, post="L1_1_flt.fits")`` corrected image
``csst_ms_sls_position`` -- ``dm.l1_detector(detector=detector, post="L1_1.fits")`` WCS determined
========================== ===== =========================================================== ==================
``Slit-Less Spectra 1D``
^^^^^^^^^^^^^^^^^^^^^^^^
Note that for 1D pipeline, ``dm.target_detectors`` is a list containing only one element.
=========================== ===== =============================================================== ==================
Data source Level File name / access code Notes
=========================== ===== =============================================================== ==================
``csst_ms_sls_position`` -- ``dm.l1_detector(detector=detector, post="L1_1.fits")`` L1 SLS image
``csst_common`` -- ``dm.target_detectors[0]`` detector number
``csst_ms_sls_mosaic`` -- ``dm.l1_detector(detector=detector, post="coadd.fits")`` coadded image
``csst_ms_sls_mosaic`` -- ``dm.l1_detector(detector=detector, post="coadd.weight.fits")`` coadded weight
``csst_ms_sls_directimage`` -- ``dm.l1_detector(detector=detector, post="dir_img.fits")`` cropped image
``csst_ms_sls_directimage`` -- ``dm.l1_detector(detector=detector, post="dir_wht.fits")`` cropped weight
``csst_ms_sls_objextract`` -- ``dm.l1_detector(detector=detector, post="comb.cat")`` combined catalog
``csst_ms_sls_objextract`` -- ``dm.l1_detector(detector=detector, post="segmentation.fits")`` segmentation file
``csst_ms_sls_sky`` -- ``dm.l1_detector(detector=detector, post="L1_1_sky.fits")`` sky background
``csst_ms_sls_axe`` -- ``dm.l1_detector(detector=detector, post="L1_1_axe.fits")`` aXe spectra
``csst_ms_sls_cde`` -- ``dm.l1_detector(detector=detector, post="L1_1_cde.fits")`` CDE spectra
=========================== ===== =============================================================== ==================
``csst_common.file_recorder.FileRecorder``
------------------------------------------
Get an empty ``FileRecorder``.
This is initially a ``list``-like object.
Use ``FileRecorder.add_record()`` to add file records.
.. code-block::
:linenos:
>>> from csst_common.file_recorder import FileRecorder
>>> fr = FileRecorder()
>>> for i in range(3):
>>> fr.add_record(filepath="test{:03d}.txt".format(i),
>>> db=True,
>>> comment="Test file {:d}".format(i))
>>> fr.pprint_all()
<FileRecorder length=3>
filepath db comment existence
----------- ---- ----------- ---------
test000.txt True Test file 0 False
test001.txt True Test file 1 False
test002.txt True Test file 2 False
Therefore, if you are collecting records from subprocesses,
here is the way to combine records:
.. code-block:: python
:linenos:
>>> from csst_common.file_recorder import FileRecorder
>>> fr1 = FileRecorder()
>>> fr1.add_record(filepath="test1.txt", db=False, comment="test file 1")
>>> fr2 = FileRecorder()
>>> fr2.add_record(filepath="test2.txt", db=False, comment="test file 2")
>>> fr_all = FileRecorder([*fr1, *fr2])
>>> fr_all.pprint_all()
<FileRecorder length=2>
filepath db comment existence
--------- ----- ----------- ---------
test1.txt False test file 1 False
test2.txt False test file 2 False
>>> fr_all.extend(fr1)
>>> fr_all.pprint_all()
<FileRecorder length=3>
filepath db comment existence
--------- ----- ----------- ---------
test1.txt False test file 1 False
test2.txt False test file 2 False
test1.txt False test file 1 False
``csst_common.logger.get_logger()``
-----------------------------------
Get the default configured ``logging.Logger``.
.. code-block:: python
from csst_common.logger import get_logger
logger = get_logger()
Developers should NOT use ``print`` function extensively in code.
Generally ``print`` can be replaced with ``logger.debug("msg")``.
For important information, use ``logger.info("msg")``.
For warnings, use ``logger.warning("msg")``.
For errors, use ``logger.error("msg")``.
Conf https://docs.python.org/3/library/logging.html for more information
about the usage of ``logging.Logger``.
``csst_common.status.CsstStatus``
---------------------------------
Developers should use ``csst_common.status.CsstStatus`` to return the
status of their interfaces.
.. code-block:: python
from csst_common.status import CsstStatus
# presently 3 kinds of status are available, they are
CsstStatus.PERFECT # all images are processed smoothly
CsstStatus.WARNING # problematic
CsstStatus.ERROR # unavailable, DO NOT use this
We recommend our developers to use the code structure used in the example interfaces,
e.g., ``process_single_image`` and ``process_multiple_images``.
The example code is shown below.
:emphasize-lines: 7-11,25-29,75-86,101-112,169-171
Rendered ``docstring``
^^^^^^^^^^^^^^^^^^^^^^
.. automodule:: example_interface
:members:
See https://csst-tb.bao.ac.cn/code/csst-l1/csst_common/-/blob/main/csst_common/data/csst_params.yml.
To use the parameters, use
.. code-block:: python
from csst_common.params import CSST_PARAMS as CP
DFS interfaces
--------------
==================================== ====================== ==========================================
Method Function Original DFS interface
==================================== ====================== ==========================================
``CsstMsDataManager.dfs_query_rc()`` query catalog from DFS ``csst_dfs_api.common.catalog.CatalogApi``
``CsstMsDataManager.dfs_query_l0()`` query L0 data from DFS --
``CsstMsDataManager.dfs_query_l1()`` query L1 data from DFS --
``CsstMsDataManager.dfs_push_l1()`` push L1 data to DFS ``csst_dfs_api.msc.level1.Level1DataApi``
==================================== ====================== ==========================================