ch08_csst_common.rst

Using ``csst_common``
=====================

In this chapter, we introduce the basic usages of some IMPORTANT functions / classes in ``csst_common``.
Package source code and installation guide can be found in the link below.

- https://csst-tb.bao.ac.cn/code/csst-l1/csst_common

``csst_common.data_manager.CsstMsDataManager``
----------------------------------------------

This class provides an interface to switch between local files and DFS.

To initilize from local directory:

.. code-block:: python

    from csst_common.data_manager import CsstMsDataManager
    # for full basic initialization
    dm = CsstMsDataManager.from_dir(
        ver_sim="C5.2",                  # version of simulation, "C5.2" is the latest
        datatype="mbi",                  # data type, "mbi", "sls" or "all"
        dir_l0=".",                      # the L0/input directory
        dir_l1=".",                      # the L1/output directory
        path_aux="",                     # aux file paths (master bias, dark, flat)
        dfs_mode=False,                  # if True, use DFS in pipeline modules
        dfs_node='kmust',                # DFS server, chosen from {"kmust", "pml"}
    )
    dm.target_detectors = None           # use all available detectors


In particular, if you are on ``dandelion`` or ``PML node``, you can use a ``quickstart`` method.

.. code-block:: python

    # for a quick start ()
    dm = CsstMsDataManager.quickstart(
        ver_sim='C5.2',                  # version of simulation, only "C5.2" is supported
        datatype='mbi',                  # data type, "mbi", "sls" or "all"
        dir_l1='.',                      # the L1/output directory
        exposure_id=100,                 # the exposure ID
        dfs_mode=False,                  # if True, use DFS in pipeline modules
        dfs_node='kmust',                # DFS server, chosen from {"kmust", "pml"}
    )
    dm

You will get the outputs:

.. code-block::

    <CsstMsDataManager>
    Data type = mbi
    Valid detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25]
    Available detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25]
    Target detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25]
    dir_l0 = /nfsdata/share/csst_simulation_data/Cycle-5-SimuData/multipleBandsImaging/NGP_AstrometryON_shearOFF/MSC_0000100/
    dir_l1 = .
    dfs_mode = False
    dfs_node = kmust


.. note::
    This ``quickstart`` method is ONLY available on
    ``dandelion`` or the ``PML node``.
    In your local development environment, you have to use
    ``CsstMsDataManager.from_dir(...)`` to initialize it.

Attributes
^^^^^^^^^^

Some attributes can be accessed as below.

.. code-block:: python

    # access L0 directory
    dm.dir_l0
    # access L1 directory
    dm.dir_l1
    # access dir_pcref
    dm.dir_pcref
    # access path_aux
    dm.path_aux
    # access ver_sim
    dm.ver_sim
    # access target detectors
    dm.target_detectors
    # access available detectors
    dm.available_detectors
    # define an L1 file (detector-specified)
    dm.l1_detector(detector=6)
    # define an L1 file (non-detector-specified)
    dm.l1_file("flipped_image.fits")

To get master files:

.. code-block:: python

    # to get bias
    dm.get_bias()
    # to get dark
    dm.get_dark()
    # to get flat
    dm.get_flat()

Query Reference Catalog
^^^^^^^^^^^^^^^^^^^^^^^

``CsstMsDataManager`` provides a method ``query_rc`` to query reference catalog from ``DFS``.

.. code-block:: python

    cat = dm.query_rc(ra=180, dec=0, radius=2, min_mag=0, max_mag=30)
    if cat is None:
        print("The query failed!")
    else:
        print("The query succeeded!")


Access code to data products
----------------------------

``Multi-Band Imaging``
^^^^^^^^^^^^^^^^^^^^^^

========================== ===== ========================================================== ==================
Data source                Level File name / access code                                     Notes
========================== ===== ========================================================== ==================
raw                        0     ``dm.l0_detector(detector=detector)``                      sci image
raw                        0     ``dm.l0_crs(detector=detector)``                           cosmic ray image
raw                        0     ``dm.l0_cat(detector=detector)``                           input catalog
raw                        0     ``dm.l0_log(detector=detector)``                           log file
``csst_ms_mbi_instrument`` --    ``dm.l1_detector(detector=detector, post="img.fits")``     corrected image
``csst_ms_mbi_instrument`` --    ``dm.l1_detector(detector=detector, post="flg.fits")``     flag image
``csst_ms_mbi_instrument`` --    ``dm.l1_detector(detector=detector, post="wht.fits")``     weight image
``csst_ms_mbi_instrument`` --    ``dm.l1_detector(detector=detector, post="img.head")``     L0 fits header
``DFS``                    --    ``dm.l1_file(name="gaia_dfs.fits")``                       reference catalog
``csst_ms_mbi_distortion`` --    ``dm.l1_detector(detector=detector, post="wcs.head")``     fits head with WCS
``csst_ms_mbi_position``   --    ``dm.l1_detector(detector=detector, post="img.acat")``     photometry catalog
``csst_ms_mbi_position``   --    ``dm.l1_detector(detector=detector, post="img.rcat")``     reference catalog
``csst_ms_mbi_flux``       1     ``dm.l1_detector(detector=detector, post="img_L1.fits")``  sci image
``csst_ms_mbi_flux``       1     ``dm.l1_detector(detector=detector, post="flg_L1.fits")``  flag image
``csst_ms_mbi_flux``       1     ``dm.l1_detector(detector=detector, post="wht_L1.fits")``  weight image
``csst_ms_mbi_photometry`` 1     ``dm.l1_detector(detector=detector, post="cat.fits")``     source catalog
``csst_ms_mbi_photometry`` 1     ``dm.l1_detector(detector=detector, post="psf.fits")``     source catalog
========================== ===== ========================================================== ==================


``Slit-Less Spectra 2D``
^^^^^^^^^^^^^^^^^^^^^^^^
========================== ===== =========================================================== ==================
Data source                Level File name / access code                                     Notes
========================== ===== =========================================================== ==================
raw                        0     ``dm.l0_detector(detector=detector)``                       sci image
raw                        0     ``dm.l0_crs(detector=detector)``                            cosmic ray image
raw                        0     ``dm.l0_cat(detector=detector)``                            input catalog
raw                        0     ``dm.l0_log(detector=detector)``                            log file
``csst_ms_sls_instrument`` 1     ``dm.l1_detector(detector=detector, post="L1_1_flt.fits")`` corrected image
``csst_ms_sls_position``   --    ``dm.l1_detector(detector=detector, post="L1_1.fits")``     WCS determined
========================== ===== =========================================================== ==================


``Slit-Less Spectra 1D``
^^^^^^^^^^^^^^^^^^^^^^^^

Note that for 1D pipeline, ``dm.target_detectors`` is a list containing only one element.

=========================== ===== =============================================================== ==================
Data source                 Level File name / access code                                         Notes
=========================== ===== =============================================================== ==================
``csst_ms_sls_position``    --    ``dm.l1_detector(detector=detector, post="L1_1.fits")``         L1 SLS image
``csst_common``             --    ``dm.target_detectors[0]``                                      detector number
``csst_ms_sls_mosaic``      --    ``dm.l1_detector(detector=detector, post="coadd.fits")``        coadded image
``csst_ms_sls_mosaic``      --    ``dm.l1_detector(detector=detector, post="coadd.weight.fits")`` coadded weight
``csst_ms_sls_directimage`` --    ``dm.l1_detector(detector=detector, post="dir_img.fits")``      cropped image
``csst_ms_sls_directimage`` --    ``dm.l1_detector(detector=detector, post="dir_wht.fits")``      cropped weight
``csst_ms_sls_objextract``  --    ``dm.l1_detector(detector=detector, post="comb.cat")``          combined catalog
``csst_ms_sls_objextract``  --    ``dm.l1_detector(detector=detector, post="segmentation.fits")`` segmentation file
``csst_ms_sls_sky``         --    ``dm.l1_detector(detector=detector, post="L1_1_sky.fits")``     sky background
``csst_ms_sls_axe``         --    ``dm.l1_detector(detector=detector, post="L1_1_axe.fits")``     aXe spectra
``csst_ms_sls_cde``         --    ``dm.l1_detector(detector=detector, post="L1_1_cde.fits")``     CDE spectra
=========================== ===== =============================================================== ==================


``csst_common.file_recorder.FileRecorder``
------------------------------------------

Get an empty ``FileRecorder``.
This is initially a ``list``-like object.
Use ``FileRecorder.add_record()`` to add file records.

.. code-block::
    :linenos:

    >>> from csst_common.file_recorder import FileRecorder
    >>> fr = FileRecorder()
    >>> for i in range(3):
    >>>     fr.add_record(filepath="test{:03d}.txt".format(i),
    >>>                   db=True,
    >>>                   comment="Test file {:d}".format(i))
    >>> fr.pprint_all()

    <FileRecorder length=3>
      filepath   db    comment   existence
    ----------- ---- ----------- ---------
    test000.txt True Test file 0     False
    test001.txt True Test file 1     False
    test002.txt True Test file 2     False

``FileRecorder`` inherits from ``list``.
Therefore, if you are collecting records from subprocesses,
here is the way to combine records:

.. code-block:: python
    :linenos:

    >>> from csst_common.file_recorder import FileRecorder
    >>> fr1 = FileRecorder()
    >>> fr1.add_record(filepath="test1.txt", db=False, comment="test file 1")
    >>> fr2 = FileRecorder()
    >>> fr2.add_record(filepath="test2.txt", db=False, comment="test file 2")
    >>> fr_all = FileRecorder([*fr1, *fr2])
    >>> fr_all.pprint_all()
    <FileRecorder length=2>
     filepath   db    comment   existence
    --------- ----- ----------- ---------
    test1.txt False test file 1     False
    test2.txt False test file 2     False

    >>> fr_all.extend(fr1)
    >>> fr_all.pprint_all()
    <FileRecorder length=3>
     filepath   db    comment   existence
    --------- ----- ----------- ---------
    test1.txt False test file 1     False
    test2.txt False test file 2     False
    test1.txt False test file 1     False


``csst_common.logger.get_logger()``
-----------------------------------

Get the default configured ``logging.Logger``.


.. code-block:: python
    :linenos:

    from csst_common.logger import get_logger
    logger = get_logger()

.. warning::
    Developers should NOT use ``print`` function extensively in code.
    Generally ``print`` can be replaced with ``logger.debug("msg")``.
    For important information, use ``logger.info("msg")``.
    For warnings, use ``logger.warning("msg")``.
    For errors, use ``logger.error("msg")``.
    Conf https://docs.python.org/3/library/logging.html for more information
    about the usage of ``logging.Logger``.


``csst_common.status.CsstStatus``
---------------------------------

Developers should use ``csst_common.status.CsstStatus`` to return the
status of their interfaces.

.. code-block:: python
    :linenos:

    from csst_common.status import CsstStatus
    # presently 3 kinds of status are available, they are
    CsstStatus.PERFECT
    CsstStatus.WARNING
    CsstStatus.ERROR


An example interface
--------------------

We recommend our developers to use the code structure used in the example interfaces,
e.g., ``process_single_image`` and ``process_multiple_images``.
The example code is shown below.

Source code
^^^^^^^^^^^

.. literalinclude:: csst_common/example_interface.py
    :caption: ``example_interface.py``
    :emphasize-lines: 7-11,25-29,75-86,101-112,169-171
    :linenos:
    :language: python


Rendered ``docstring``
^^^^^^^^^^^^^^^^^^^^^^

.. automodule:: example_interface
    :members:


``csst_common.params``
----------------------

See https://csst-tb.bao.ac.cn/code/csst-l1/csst_common/-/blob/main/csst_common/data/csst_params.yml.
To use the parameters, use

.. code-block:: python

    from csst_common.params import CSST_PARAMS as CP



Module Identifier
-----------------

to be updated

DFS interfaces
--------------

==================================== ====================== ==========================================
Method                               Function               Original DFS interface
==================================== ====================== ==========================================
``CsstMsDataManager.dfs_query_rc()`` query catalog from DFS ``csst_dfs_api.common.catalog.CatalogApi``
``CsstMsDataManager.dfs_query_l0()`` query L0 data from DFS --
``CsstMsDataManager.dfs_query_l1()`` query L1 data from DFS --
``CsstMsDataManager.dfs_push_l1()``  push L1 data to DFS    ``csst_dfs_api.msc.level1.Level1DataApi``
==================================== ====================== ==========================================