ch03_packaging.rst

Python Packaging
================
Python projects should be packaged in the standard way as shown in the
official `Python Packaging User Guide`_, and more specifically `here`_.

.. _here: https://packaging.python.org/en/latest/tutorials/packaging-projects/
.. _Python Packaging User Guide: https://packaging.python.org/en/latest/


Project structure
-----------------
An example code structure is shown below.
Files/directories with asterisks (``*``) marks are optional.

.. code-block::

    csst_proto                                # the repository name
    ├── csst_proto                            # the package name
    │   ├── data                              # package associated data directory
    │   ├── __init__.py                       # necessary file for a Python package
    │   ├── demo.py                           # Python modules
    │   ├── flip_image.py
    │   ├── scratch.py
    │   └── api.py                            # the top level interface module
    ├── doc                                   # *sphinx-based documentation directory
    │   ├── build
    │   ├── source
    │   ├── Makefile
    │   ├── apidoc.sh
    │   ├── contents.md
    │   ├── make.bat
    │   └── preview.sh
    ├── examples                              # *example scripts
    │   ├── how_this_code_will_be_used.py
    │   └── how_to_write_docstring.py
    ├── tests                                 # unit tests
    │   ├── test_flip_image.py
    │   └── test_other_functions.py
    ├── LICENSE                               # license file
    ├── README.md                             # README file (Markdown recommended)
    ├── install.sh                            # a single-line installation script
    ├── install_local.sh                      # local installation script
    ├── readthedocs.yml                       # *configuration file for readthedocs
    ├── requirements.txt                      # package dependencies
    └── setup.py                              # setup file

.. note::
    The ``__init__.py`` files is important, without which
    the directory will not be seen as a ``Python package``.

Python version
--------------

Developers should use ``Python 3.9.X`` to implement algorithms.
The base docker image will be ``continuumio/anaconda3``, which uses
``Python 3.9.12``.

Relative import
---------------
When import a class / function within the same package, a ``relative import``
should be used instead of ``absolute import``.
An example is below (``top_level_interface.py``):

.. literalinclude:: ../../csst_proto/top_level_interface.py
   :linenos:
   :language: python

.. note::
    Note the ``.`` means it's a relative import.

README.md
---------

A Markdown format text file to describe your package.

.. literalinclude:: ../../README.md

LICENSE
-------

Among the many choices, we encourage our developers to use ``MIT`` LICENSE.
An example is below.

.. literalinclude:: ../../LICENSE


``requirements.txt``
--------------------
``requirements.txt`` is a text file, within which a line represents a package and its version.

.. literalinclude:: ../../requirements.txt
   :linenos:

.. note::
    For each package, developers must specify a version.
    DO NOT use ``>``, ``>=``, ``<``, ``<=`` or ``~=``, use ``==`` ONLY!


``setup.py``
------------
We use a classic format of ``setup.py`` file as shown below.
We refer readers to https://setuptools.pypa.io/en/latest/ for more information
and usages about the package `setuptools`_.

.. _setuptools: https://setuptools.pypa.io/en/latest/

.. literalinclude:: ../../setup.py
   :linenos:
   :language: python

.. note::

   The ``install_requires`` keyword argument is commented because it takes no effect
   when installed with ``pip install <your_package> --no-deps``.
   Please DO NOT use it.

Include data
------------

Data files can be packaged together with code.
To include ``csst_proto/data/test_image.txt`` and ``csst_proto/data/table_data.vsb``,
set the keyword ``package_data`` in ``setup.py`` as below:

.. code-block:: python

    ...
    package_data={"": ["LICENSE", "README.md"],
                  "csst_proto": ["data/test_image.txt",
                                 "data/table_data.csv"
                                 ]},
    ...

.. note::
    The ``""`` key represents the data files located directly in the project directory.
    For files associated with package ``csst_proto``, link the data files to ``csst_proto`` key.

.. note::
    DO NOT include any test data (e.g., CSST images) in the package.
    We only encourage developers to pacakge configuration files inside.
    Please keep your package *clean* and *tight*.

To access the data, developers should use ``relative path``.
In the example below, we show the function ``read_test_image``,
an example of how to access data under ``csst_proto/data/``.
In ``__init__.py``:

.. literalinclude:: ../../csst_proto/__init__.py
    :linenos:
    :language: python

and in ``flip_image.py``:

.. literalinclude:: ../../csst_proto/flip_image.py
    :linenos:
    :language: python


Including C code
----------------

To include C code, developers should follow the guide provided by ``setuptools``:
https://setuptools.pypa.io/en/latest/userguide/ext_modules.html.
And also consider using https://docs.astropy.org/en/latest/development/ccython.html#building-c-or-cython-extensions.


``top_level_interface`` module
------------------------------
This is a special requirement for all of CSST DAS packages.
This module is used to store the interfaces (classes / functions) that
will be called by users.

.. tip::
    Remember to set the ``__all__`` variable.
    This should be a list containing all interfaces.

.. note::
    All the interfaces in this module should use a complete ``Numpy``-style docstring
    -- They will go through docstring format validations in Jenkins!


How to install package
----------------------
We recommend the following way to install your package.

.. code-block:: bash

    # install requirements
    pip install -r requirements
    # build extensions in place
    python setup.py build_ext --inplace
    # build source code
    python setup.py sdist
    # install from the source code
    pip install dist/*.tar.gz --force-reinstall --no-deps


Sphinx-based documentation
--------------------------
This is currently regarded as an optional section.
But we still recommend developers to read a bit of the ``sphinx`` documentation.

The sphinx homepage:

- https://www.sphinx-doc.org/en/master/index.html

A tutorial on how to write restructured text (.rst) files:

- https://docutils.sourceforge.io/rst.html