Skip to content
GitLab
Commit 5590255b authored by BO ZHANG's avatar BO ZHANG 🏀
Browse files

delete unnecessary files

parent 55e4b081
Pipeline #1464 passed with stage
in 0 seconds
Hide whitespace changes
Inline Side-by-side
.gitlab-ci.yml_ deleted 100644 → 0
View file @ 55e4b081
# This file is a template, and might need editing before it works on your project.
# This is a sample GitLab CI/CD configuration file that should run without any modifications.
# It demonstrates a basic 3 stage CI/CD pipeline. Instead of real tests or scripts,
# it uses echo commands to simulate the pipeline execution.
#
# A pipeline is composed of independent jobs that run scripts, grouped into stages.
# Stages run in sequential order, but jobs within stages run in parallel.
#
# For more information, see: https://docs.gitlab.com/ee/ci/yaml/index.html#stages
#
# You can copy and paste this template into a new `.gitlab-ci.yml` file.
# You should not add this template to an existing `.gitlab-ci.yml` file by using the `include:` keyword.
#
# To contribute improvements to CI/CD templates, please follow the Development guide at:
# https://docs.gitlab.com/ee/development/cicd/templates.html
# This specific template is located at:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Getting-Started.gitlab-ci.yml
stages: # List of stages for jobs, and their order of execution
- build
- test
build-job: # This job runs in the build stage, which runs first.
stage: build
script:
- which ssh
- echo "Compiling the code..."
- echo "Compile complete."
unit-test-job: # This job runs in the test stage.
stage: test # It only starts when the job in the build stage completes successfully.
script:
- echo "Running unit tests... "
- whoami
- uname -a
- pwd
- ls -a
- ls /nfsdata/share/csst_simulation_data/Cycle-5-SimuData/multipleBandsImaging/NGP_AstrometryON_shearOFF
- which python
- which pip
- python --version
- source /home/csstpipeline/anaconda3/bin/activate
- which python
- which pip
- python --version
- pip install -r requirements.txt
- sh install_local.sh
- pip show csst_proto
- pytest
docs/Makefile deleted 100644 → 0
View file @ 55e4b081
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = source
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
docs/apidoc.sh deleted 100644 → 0
View file @ 55e4b081
rm -rf source/api/*
sphinx-apidoc -o source/api ../csst_proto --ext-viewcode --ext-githubpages
docs/latexpdf.sh deleted 100644 → 0
View file @ 55e4b081
make latexpdf LATEXMKOPTS="-silent"
cd build/latex || exit
latexmk -pdf -dvi- -ps- -silent 'aguideforcsstdasdevelopers.tex'
cd ../..
open ./build/latex/aguideforcsstdasdevelopers.pdf
\ No newline at end of file
docs/make.bat deleted 100644 → 0
View file @ 55e4b081
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.https://www.sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
docs/preview.sh deleted 100755 → 0
View file @ 55e4b081
make clean html
open ./build/html/index.html
\ No newline at end of file
docs/requirements.txt deleted 100644 → 0
View file @ 55e4b081
numpy==1.23.3
scipy==1.9.1
joblib==1.2.0
setuptools==58.0.4
astropy==5.1
pycodestyle==2.9.1
nbsphinx
#sphinx_copybutton==0.5.0 # this does not exclude linenos
git+https://github.com/executablebooks/sphinx-copybutton.git
git+https://csst-tb.bao.ac.cn/code/csst-l1/csst_common.git
git+https://csst-tb.bao.ac.cn/code/csst-dfs/csst-dfs-commons.git
git+https://csst-tb.bao.ac.cn/code/csst-dfs/csst-dfs-proto-py.git
git+https://csst-tb.bao.ac.cn/code/csst-dfs/csst-dfs-api-local.git
git+https://csst-tb.bao.ac.cn/code/csst-dfs/csst-dfs-api-cluster.git
git+https://csst-tb.bao.ac.cn/code/csst-dfs/csst-dfs-api.git
docs/source/_static/custom.css deleted 100644 → 0
View file @ 55e4b081
.wy-nav-content {
max-width: 80% !important;
}
\ No newline at end of file
docs/source/_static/logo-wordmark-light.svg deleted 100644 → 0
View file @ 55e4b081
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
width="2000"
height="400"
id="svg5241"
version="1.1"
inkscape:version="0.48.5 r10040"
sodipodi:docname="logo-wordmark-light.svg"
inkscape:export-filename="/Users/anthony/dev/readthedocs/guidelines/assets/logo-wordmark-light.png"
inkscape:export-xdpi="90"
inkscape:export-ydpi="90">
<defs
id="defs5243" />
<sodipodi:namedview
id="base"
pagecolor="#32322a"
bordercolor="#666666"
borderopacity="1.0"
inkscape:pageopacity="1"
inkscape:pageshadow="2"
inkscape:zoom="0.70710678"
inkscape:cx="774.9861"
inkscape:cy="86.360425"
inkscape:document-units="px"
inkscape:current-layer="text5298"
showgrid="false"
inkscape:window-width="1366"
inkscape:window-height="723"
inkscape:window-x="0"
inkscape:window-y="26"
inkscape:window-maximized="0" />
<metadata
id="metadata5246">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(0,-652.3622)">
<g
id="g5310"
transform="translate(-7.2877533,-6.3546821e-5)"
style="fill:#fafafa;fill-opacity:1">
<g
inkscape:export-ydpi="90"
inkscape:export-xdpi="90"
inkscape:export-filename="/home/anthony/secure/docs/rtd/assets/rtd-logo-2.png"
transform="matrix(0.55753644,0,0,0.55753644,62.308135,1038.8762)"
id="g3990"
style="fill:#fafafa;fill-opacity:1" />
<g
id="g3878"
transform="matrix(8.4462802,0,0,8.4462802,-93.469267,-7827.1593)"
style="fill:#fafafa;fill-opacity:1">
<g
transform="matrix(0.55753644,0,0,0.55753644,68.308135,1050.1262)"
id="g3857"
style="fill:#fafafa;fill-opacity:1">
<path
id="path3929-8"
style="font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-indent:0;text-align:start;text-decoration:none;line-height:normal;letter-spacing:normal;word-spacing:normal;text-transform:none;direction:ltr;block-progression:tb;writing-mode:lr-tb;text-anchor:start;baseline-shift:baseline;color:#000000;fill:#fafafa;fill-opacity:1;stroke:none;stroke-width:3.22848844999999995;marker:none;visibility:visible;display:inline;overflow:visible;enable-background:accumulate;font-family:Sans;-inkscape-font-specification:Sans"
d="M -67.28125,-41.59375 A 1.6144056,1.6144056 0 0 0 -67.5,-38.4375 c 0,0 3.931695,1.301049 10.625,1.84375 5.374689,0.435786 11.46875,-0.375 11.46875,-0.375 a 1.6144056,1.6144056 0 1 0 -0.40625,-3.1875 c 0,0 -5.961605,0.737066 -10.8125,0.34375 C -63.013171,-40.330461 -66.5,-41.5 -66.5,-41.5 a 1.6144056,1.6144056 0 0 0 -0.78125,-0.09375 z m 0,-8 A 1.6144056,1.6144056 0 0 0 -67.5,-46.4375 c 0,0 3.931695,1.301049 10.625,1.84375 5.374689,0.435786 11.46875,-0.375 11.46875,-0.375 a 1.6144056,1.6144056 0 1 0 -0.40625,-3.1875 c 0,0 -5.961605,0.737066 -10.8125,0.34375 C -63.013171,-48.330461 -66.5,-49.5 -66.5,-49.5 a 1.6144056,1.6144056 0 0 0 -0.78125,-0.09375 z m 0,-8 A 1.6144056,1.6144056 0 0 0 -67.5,-54.4375 c 0,0 3.931695,1.301049 10.625,1.84375 5.374689,0.435786 11.46875,-0.375 11.46875,-0.375 a 1.6144056,1.6144056 0 1 0 -0.40625,-3.1875 c 0,0 -5.961605,0.737066 -10.8125,0.34375 C -63.013171,-56.330461 -66.5,-57.5 -66.5,-57.5 a 1.6144056,1.6144056 0 0 0 -0.78125,-0.09375 z m 0,-8 A 1.6144056,1.6144056 0 0 0 -67.5,-62.4375 c 0,0 3.931695,1.301049 10.625,1.84375 5.374689,0.435786 11.46875,-0.375 11.46875,-0.375 a 1.6144056,1.6144056 0 1 0 -0.40625,-3.1875 c 0,0 -5.961605,0.737066 -10.8125,0.34375 C -63.013171,-64.330461 -66.5,-65.5 -66.5,-65.5 a 1.6144056,1.6144056 0 0 0 -0.78125,-0.09375 z m -11.207892,-8.437005 c -8.407221,0.05606 -11.539425,2.645057 -11.539425,2.645057 l 0,62.7837755 c 0,0 3.05858,-2.6415165 12.905554,-2.2381255 9.846974,0.403391 11.878255,3.8552765 23.979914,4.0983855 12.101659,0.243109 15.143679,-1.86026 15.143679,-1.86026 l 0.174399,-64.0045705 c 0,0 -5.446133,1.541392 -16.044742,1.627727 -10.598609,0.08634 -13.146074,-2.696144 -22.875385,-3.022922 -0.608082,-0.02042 -1.183512,-0.0328 -1.743994,-0.02907 z m 7.034109,4.098386 c 0,0 5.094376,1.68402 14.504214,2.150925 7.953019,0.39462 15.928477,-0.784797 15.928477,-0.784797 l 0,56.883263 c 0,0 -4.036665,2.1158549 -14.12635,1.395195 -7.819331,-0.558499 -16.422608,-3.517054 -16.422608,-3.517054 l 0.116267,-56.127532 z m -4.912249,1.482394 a 1.6277275,1.6277275 0 0 1 0,3.255455 c 0,0 -2.634985,0.01353 -4.243719,0.1744 -2.701025,0.270103 -4.534383,1.249862 -4.534383,1.249862 a 1.6251955,1.6251955 0 1 1 -1.511462,-2.87759 c 0,0 2.391605,-1.26521 5.726113,-1.598661 1.926801,-0.192699 4.563451,-0.203466 4.563451,-0.203466 z m -1.569595,8.022372 c 0.899775,-0.02279 1.569595,0 1.569595,0 a 1.625,1.625 0 0 1 0,3.226388 c 0,0 -2.634985,0.01352 -4.243719,0.174399 -2.701025,0.270104 -4.534383,1.249863 -4.534383,1.249863 a 1.6251955,1.6251955 0 0 1 -1.511462,-2.87759 c 0,0 2.391605,-1.26521 5.726113,-1.598661 0.963401,-0.09635 2.094081,-0.151612 2.993856,-0.174399 z m 1.569595,7.993304 a 1.6277275,1.6277275 0 0 1 0,3.255455 c 0,0 -2.634985,-0.01554 -4.243719,0.145333 -2.701025,0.270103 -4.534383,1.249862 -4.534383,1.249862 a 1.6251952,1.6251952 0 0 1 -1.511462,-2.877589 c 0,0 2.391605,-1.265211 5.726113,-1.598661 1.926801,-0.1927 4.563451,-0.1744 4.563451,-0.1744 z"
inkscape:connector-curvature="0" />
</g>
<g
style="font-size:32px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;line-height:125%;letter-spacing:-1.99000001px;word-spacing:0px;fill:#fafafa;stroke:none;font-family:Bitter;-inkscape-font-specification:Bitter"
id="text5298">
<path
d="m 63.380232,1038.8522 0,-2.592 1.056,-0.096 c 0.618664,-0.064 0.927997,-0.3627 0.928,-0.896 l 0,-14.88 -1.824,-0.096 0,-2.72 8.736,0 c 2.474654,0 4.373319,0.4587 5.696,1.376 1.322649,0.9173 1.983982,2.4213 1.984,4.512 -1.8e-5,1.4507 -0.384018,2.6453 -1.152,3.584 -0.746683,0.9387 -1.642682,1.6213 -2.688,2.048 0.767985,0.2773 1.375985,0.928 1.824,1.952 l 2.304,5.024 1.824,0.064 0,2.72 -7.904,0 0,-2.592 0.928,-0.096 c 0.490653,-0.064 0.735986,-0.256 0.736,-0.576 -1.4e-5,-0.128 -0.04268,-0.2774 -0.128,-0.448 l -1.504,-3.2 c -0.234679,-0.5334 -0.501345,-0.9067 -0.8,-1.12 -0.277344,-0.2347 -0.682677,-0.352 -1.216,-0.352 l -2.912,0 0,5.568 2.08,0.096 0,2.72 -7.968,0 m 5.888,-11.424 2.784,0 c 2.623987,0 3.935986,-1.1733 3.936,-3.52 -1.4e-5,-1.344 -0.352014,-2.2187 -1.056,-2.624 -0.682679,-0.4053 -1.792011,-0.608 -3.328,-0.608 l -2.336,0 0,6.752"
style="fill:#fafafa;font-family:Bree Serif;-inkscape-font-specification:Bree Serif Bold"
id="path3068" />
<path
d="m 90.310094,1022.2762 c 1.599989,0 2.869321,0.3947 3.808,1.184 0.938652,0.768 1.407985,1.856 1.408,3.264 -1.5e-5,0.9387 -0.202681,1.7706 -0.608,2.496 -0.405347,0.704 -0.90668,1.2586 -1.504,1.664 -0.597345,0.4053 -1.322678,0.736 -2.176,0.992 -1.429342,0.4266 -3.040007,0.64 -4.832,0.64 0.06399,1.1306 0.415994,2.048 1.056,2.752 0.639993,0.6826 1.621325,1.024 2.944,1.024 1.322656,0 2.645321,-0.4694 3.968,-1.408 l 1.216,2.592 c -0.426681,0.384 -1.152014,0.7786 -2.176,1.184 -1.002678,0.4053 -2.154677,0.608 -3.456,0.608 -2.602673,0 -4.512005,-0.7147 -5.728,-2.144 -1.216002,-1.4507 -1.824002,-3.4347 -1.824,-5.952 -2e-6,-2.5174 0.693331,-4.6293 2.08,-6.336 1.386662,-1.7067 3.327993,-2.56 5.824,-2.56 m -1.472,7.328 c 0.789325,-0.1494 1.514657,-0.4587 2.176,-0.928 0.661322,-0.4907 0.991989,-1.0667 0.992,-1.728 -1.1e-5,-1.3013 -0.640011,-1.952 -1.92,-1.952 -1.194675,0 -2.112007,0.48 -2.752,1.44 -0.640006,0.9387 -0.992006,2.0693 -1.056,3.392 0.93866,-0.021 1.791993,-0.096 2.56,-0.224"
style="fill:#fafafa;font-family:Bree Serif;-inkscape-font-specification:Bree Serif Bold"
id="path3070" />
<path
d="m 110.8632,1023.3962 0,11.904 c -1e-5,0.3413 0.0533,0.576 0.16,0.704 0.12799,0.128 0.34132,0.2026 0.64,0.224 l 1.024,0.064 0,2.56 -5.088,0 0,-1.856 -0.096,-0.032 c -1.06668,1.536 -2.52801,2.304 -4.384,2.304 -2.176,0 -3.786669,-0.6934 -4.831999,-2.08 -1.045335,-1.3867 -1.568001,-3.2854 -1.568,-5.696 -10e-7,-2.9014 0.703998,-5.1627 2.112,-6.784 1.407999,-1.6213 3.519989,-2.432 6.335999,-2.432 1.81332,0 3.71199,0.3733 5.696,1.12 m -3.712,10.368 0,-8.288 c -0.59734,-0.2773 -1.41868,-0.416 -2.464,-0.416 -1.42934,0 -2.46401,0.576 -3.104,1.728 -0.64,1.152 -0.96,2.6773 -0.96,4.576 0,3.456 1.10933,5.184 3.328,5.184 0.93866,0 1.70666,-0.2774 2.304,-0.832 0.59732,-0.576 0.89599,-1.2267 0.896,-1.952"
style="fill:#fafafa;font-family:Bree Serif;-inkscape-font-specification:Bree Serif Bold"
id="path3072" />
<path
d="m 121.12069,1022.2762 c 0.91733,0 1.79199,0.128 2.624,0.384 l 0,-3.232 c -1e-5,-0.4693 -0.27734,-0.7253 -0.832,-0.768 l -1.376,-0.096 0,-2.528 5.952,0 0,19.456 c 0.0213,0.4906 0.28799,0.736 0.8,0.736 l 1.12,0.064 0,2.56 -5.184,0 0,-1.888 -0.096,-0.032 c -0.96001,1.5573 -2.41067,2.336 -4.352,2.336 -2.432,0 -4.13867,-0.8107 -5.12,-2.432 -0.896,-1.472 -1.344,-3.2747 -1.344,-5.408 0,-2.7734 0.68267,-4.992 2.048,-6.656 1.38666,-1.664 3.30666,-2.496 5.76,-2.496 m 2.624,11.584 0,-8.288 c -0.76801,-0.3413 -1.57867,-0.512 -2.432,-0.512 -1.40801,0 -2.44267,0.5653 -3.104,1.696 -0.64,1.1307 -0.96,2.5706 -0.96,4.32 0,3.584 1.152,5.376 3.456,5.376 0.87466,0 1.59999,-0.2454 2.176,-0.736 0.57599,-0.512 0.86399,-1.1307 0.864,-1.856"
style="fill:#fafafa;font-family:Bree Serif;-inkscape-font-specification:Bree Serif Bold"
id="path3074" />
<path
d="m 141.89219,1036.3322 c 0,0 -1.484,0.56 -2.268,0.56 -0.784,0 -1.092,-0.392 -1.092,-1.4 0,-0.448 0.056,-1.036 0.168,-1.764 l 1.204,-7.476 3.864,0 0.336,-2.1 -3.864,0 0.672,-4.088 -2.772,0.56 -0.56,3.528 -2.8,0.28 -0.308,1.82 2.772,0 -1.232,7.756 c -0.112,0.644 -0.168,1.26 -0.168,1.792 0,2.212 0.924,3.332 2.828,3.332 1.568,0 3.668,-1.288 3.668,-1.288 l -0.448,-1.512"
style="font-size:28px;font-style:italic;font-weight:normal;fill:#fafafa;font-family:Bitter;-inkscape-font-specification:Bitter Italic"
id="path3076" />
<path
d="m 150.57225,1017.2922 -5.124,0.14 -0.252,1.54 2.352,0.56 -3.08,19.32 2.66,0 0.924,-5.04 c 0,0 2.212,-7.7 5.852,-7.7 1.11999,0 1.456,0.812 1.456,1.848 0,0.392 -0.056,0.812 -0.112,1.232 l -1.596,9.66 5.124,-0.28 0.252,-1.54 -2.352,-0.42 1.26,-7.84 c 0.084,-0.588 0.14,-1.148 0.14,-1.652 0,-2.016 -0.81201,-3.388 -3.052,-3.388 -4.256,0 -6.496,5.404 -6.608,5.712 l 2.156,-12.152"
style="font-size:28px;font-style:italic;font-weight:normal;fill:#fafafa;font-family:Bitter;-inkscape-font-specification:Bitter Italic"
id="path3078" />
<path
d="m 170.69755,1035.6322 c 0,0 -2.548,1.26 -4.592,1.26 -2.1,0 -3.08,-0.924 -3.08,-2.912 0,-0.364 0.028,-0.784 0.084,-1.204 5.796,0 9.828,-2.184 9.828,-5.404 0,-2.212 -1.792,-3.64 -4.62,-3.64 -4.45199,0 -8.092,4.564 -8.092,10.36 0,3.08 1.96,5.04 5.04,5.04 3.304,0 6.272,-2.072 6.272,-2.072 l -0.84,-1.428 m -7.336,-4.816 c 0.728,-2.94 2.772,-4.984 4.816,-4.984 1.428,0 2.1,0.588 2.1,1.82 0,1.848 -2.94,3.164 -6.916,3.164"
style="font-size:28px;font-style:italic;font-weight:normal;fill:#fafafa;font-family:Bitter;-inkscape-font-specification:Bitter Italic"
id="path3080" />
<path
d="m 177.64305,1038.8522 0,-2.592 1.056,-0.096 c 0.61867,-0.064 0.928,-0.3627 0.928,-0.896 l 0,-14.88 -1.824,-0.096 0,-2.72 8.768,0 c 3.13599,0 5.57865,0.832 7.328,2.496 1.77065,1.664 2.65598,4.128 2.656,7.392 -2e-5,2.0266 -0.26668,3.7973 -0.8,5.312 -0.53335,1.4933 -1.25868,2.6666 -2.176,3.52 -1.83468,1.7066 -4.11734,2.56 -6.848,2.56 l -9.088,0 m 5.888,-18.176 0,15.104 3.264,0 c 1.83466,0 3.26399,-0.6614 4.288,-1.984 1.02399,-1.3227 1.53599,-3.2427 1.536,-5.76 -10e-6,-4.9067 -2.08001,-7.36 -6.24,-7.36 l -2.848,0"
style="fill:#fafafa;font-family:Bree Serif;-inkscape-font-specification:Bree Serif Bold"
id="path3082" />
<path
d="m 205.62659,1036.5162 c 2.32532,0 3.48798,-1.8774 3.488,-5.632 -2e-5,-1.8987 -0.26668,-3.3387 -0.8,-4.32 -0.51201,-0.9813 -1.38668,-1.472 -2.624,-1.472 -1.21601,0 -2.11201,0.4693 -2.688,1.408 -0.57601,0.9387 -0.86401,2.2186 -0.864,3.84 -1e-5,3.008 0.55466,4.896 1.664,5.664 0.49066,0.3413 1.09866,0.512 1.824,0.512 m -7.36,-5.728 c 0,-1.5787 0.23466,-2.944 0.704,-4.096 0.46933,-1.1733 1.09866,-2.0693 1.888,-2.688 1.51466,-1.152 3.18932,-1.728 5.024,-1.728 1.27999,0 2.35732,0.2133 3.232,0.64 0.89598,0.4053 1.58932,0.8853 2.08,1.44 0.51198,0.5333 0.93865,1.3227 1.28,2.368 0.36265,1.024 0.54398,2.24 0.544,3.648 -2e-5,2.944 -0.71469,5.1733 -2.144,6.688 -1.42935,1.5146 -3.26401,2.272 -5.504,2.272 -2.21868,0 -3.95734,-0.7147 -5.216,-2.144 -1.25867,-1.4507 -1.888,-3.584 -1.888,-6.4"
style="fill:#fafafa;font-family:Bree Serif;-inkscape-font-specification:Bree Serif Bold"
id="path3084" />
<path
d="m 218.42846,1030.6282 c -1e-5,1.8133 0.34133,3.2213 1.024,4.224 0.68266,1.0026 1.66399,1.504 2.944,1.504 1.30132,0 2.58132,-0.4587 3.84,-1.376 l 1.376,2.464 c -1.51468,1.2373 -3.41335,1.856 -5.696,1.856 -2.28268,0 -4.08534,-0.7147 -5.408,-2.144 -1.30134,-1.4507 -1.952,-3.584 -1.952,-6.4 0,-2.816 0.74666,-4.928 2.24,-6.336 1.51466,-1.4293 3.21066,-2.144 5.088,-2.144 1.89865,0 3.65865,0.4373 5.28,1.312 l 0,4.16 -2.944,0.224 0,-1.536 c -1e-5,-0.576 -0.21335,-0.928 -0.64,-1.056 -0.40535,-0.1493 -0.83201,-0.224 -1.28,-0.224 -2.58134,0 -3.87201,1.824 -3.872,5.472"
style="fill:#fafafa;font-family:Bree Serif;-inkscape-font-specification:Bree Serif Bold"
id="path3086" />
<path
d="m 236.74521,1025.1882 c -0.51201,-0.192 -1.13068,-0.288 -1.856,-0.288 -0.72534,0 -1.31201,0.1707 -1.76,0.512 -0.42668,0.32 -0.64001,0.7253 -0.64,1.216 -1e-5,0.4693 0.0747,0.8427 0.224,1.12 0.17066,0.256 0.42666,0.48 0.768,0.672 0.53332,0.2773 1.17332,0.5226 1.92,0.736 0.74665,0.192 1.30132,0.352 1.664,0.48 0.36265,0.1066 0.81065,0.2986 1.344,0.576 0.55465,0.2773 0.97065,0.576 1.248,0.896 0.74665,0.7893 1.11998,1.8026 1.12,3.04 -2e-5,1.6 -0.58668,2.8586 -1.76,3.776 -1.15201,0.896 -2.62401,1.344 -4.416,1.344 -2.60268,0 -4.56534,-0.3307 -5.888,-0.992 l 0,-4.448 2.88,-0.224 0,1.536 c -1e-5,0.9386 0.89599,1.408 2.688,1.408 1.79199,0 2.68799,-0.6507 2.688,-1.952 -1e-5,-0.4694 -0.16001,-0.8534 -0.48,-1.152 -0.29868,-0.2987 -0.59735,-0.5014 -0.896,-0.608 -0.29868,-0.1067 -0.66134,-0.2134 -1.088,-0.32 -0.40534,-0.1067 -0.81068,-0.2134 -1.216,-0.32 -0.38401,-0.1067 -0.81068,-0.2454 -1.28,-0.416 -0.44801,-0.192 -0.94934,-0.4587 -1.504,-0.8 -1.08801,-0.704 -1.63201,-1.8774 -1.632,-3.52 -1e-5,-1.664 0.58666,-2.944 1.76,-3.84 1.17333,-0.896 2.64532,-1.344 4.416,-1.344 1.79199,0 3.56265,0.4267 5.312,1.28 l 0,3.84 -2.88,0.224 0,-1.344 c -1e-5,-0.5333 -0.24535,-0.896 -0.736,-1.088"
style="fill:#fafafa;font-family:Bree Serif;-inkscape-font-specification:Bree Serif Bold"
id="path3088" />
</g>
</g>
</g>
</g>
</svg>
docs/source/api/csst_proto.rst deleted 100644 → 0
View file @ 55e4b081
API
===
In this chapter, we show what the rendered Numpy-style docstring looks like
in sphinx documentation.
.. automodule:: csst_proto.api
:noindex:
:members:
:undoc-members:
:show-inheritance:
docs/source/c8_changes.rst deleted 100644 → 0
View file @ 55e4b081
C8主巡天流水线接口
=========================
Change list
- 仓库名修改：
- ``csst_ms*`` 修改为 ``csst_msc_*``, 注意修改文件夹路径和 ``setup.py`` 中的名称
- 接口文件修改：
``top_level_interface.py`` 文件改名为 ``api.py`` , 这个文件中用于导入每个 python package 与流水线的接口函数
- 接口名称修改：
每个开发模块需要把接口函数命名为 ``base_`` 开头的函数，例如 ``base_phot``
- ``DataManager`` ``FileRecorder`` 不再使用, 底层接口都尽可能使用文件路径作为参数
- 使用 ``csst_common.status.CsstResult`` 作为返回类型, 包含三个部分：
- 运行状态 ``status`` 必须是 ``CsstStatus`` 的三种类型之一
- 输出文件列表 ``file_list``, 是一个 ``list``, 需要包含所有输出文件，包括临时文件
- 额外输出 （一般不需要，除非有其他额外输出）
- 程序中禁止使用当前文件夹 ``.`` 或 ``./``，所有路径应使用绝对路径
- 尽可能避免创建临时文件夹等操作
- 单元测试
- 覆盖率合格线可以设置稍低
- 由于代码接口定义更新，原则上单元测试应该会更加容易，
- 单元测试服务器： csstunittest@10.3.10.10 外网 -p 2000 159.226.170.52
- 单元测试需要一个大表来统计所有模块的每个案例(文件、类、函数名)和其简单解释(~10个字)
- Jenkins上需要有一个随时更新的单元测试统计结果 包括覆盖率，测试案例列表等
- 接口定义修改：
``csst_msc_mbi_distortion`` 样例代码如下
.. code-block:: python
import os
import numpy as np
import numpy.typing as npt
from astropy import table
from astropy.io import fits
from csst_common.status import CsstStatus, CsstResult
def read_data(file_path: str = "/path/to/data") -> np.ndarray:
return fits.getdata(file_path)
def _do_step_one(data: npt.NDArray) -> npt.NDArray:
reduced_data = np.fliplr(data)
return reduced_data
def _do_step_two(data: npt.NDArray) -> npt.NDArray:
reduced_data = np.flipud(data)
return reduced_data
def _do_step_three(data: npt.NDArray, rc: str = "/path/to/catalog") -> npt.NDArray:
trc = table.Table.read(rc)
return data + np.mean(trc["ra"]) + np.mean(trc["dec"])
def base_msc_l1_mbi_image_distortion(
input_file: str,
output_file: str,
rc: str = "/path/to/gaia_dr3.fits",
**kwargs
) -> CsstResult:
"""your docstring here"""
# read data
data = read_data(input_file)
# do your algorithm
data = _do_step_one(data)
data = _do_step_two(data)
data = _do_step_three(data, rc=rc)
# write results
hl = fits.HDUList(fits.PrimaryHDU(data=data))
hl.writeto(output_file, overwrite=True)
assert os.path.exists(output_file)
# construct CsstResult
result = CsstResult(
status=CsstStatus.PERFECT,
file_list=[
output_file,
"/path/to/result1.fits",
"/path/to/result2.fits",
],
)
return result
导入模块
---------
.. code-block:: python
# import modules
import numpy as np
import matplotlib.pyplot as plt
from astropy.io import fits
from astropy import table
# import modules
from scipy import signal
from astropy import units as u
from astropy import constants as const
# import classes / functions
from astropy.coordinates import SkyCoord
from scipy.interpolate import least_squares
# DO NOT use 'import *'
.. 主巡天模块-接口列表
.. ----------------------
.. .. code-block:: python
.. # csst-l1/mbi/csst_msc_mbi_instrument
.. def base_instcorr(input_file, img_file, wht_file, flg_file) -> CsstResult
.. # csst-l1/mbi/csst_msc_mbi_distortion
.. def base_distortion(input_file, img_file, wht_file, flg_file): -> CsstResult
.. # csst-l1/mbi/csst_msc_mbi_position
.. def base_position_single(input_file, img_file, wht_file, flg_file): -> CsstResult
.. def base_position_multiple(input_file, img_file, wht_file, flg_file): -> CsstResult
.. # csst-l1/mbi/csst_msc_mbi_flux
.. def base_flux(input_file, img_file, wht_file, flg_file): -> CsstResult
.. # csst-l1/mbi/csst_msc_mbi_photometry
.. def base_phot(input_file, img_file, wht_file, flg_file): -> CsstResult
.. # csst-l1/sls/csst_msc_sls_instrument
.. def base_instcorr(input_file, output_file): -> CsstResult
.. # csst-l1/sls/csst_msc_sls_position
.. def base_position(input_file, output_file): -> CsstResult
.. # csst-l1/sls/csst_msc_sls_mosaic
.. def base_mosaic(input_files, output_file): -> CsstResult
.. # csst-l1/sls/csst_msc_sls_directimage
.. def base_dimg(input_file, output_file): -> CsstResult
.. # csst-l1/sls/csst_msc_sls_sky
.. def base_skybkg(input_file, output_file): -> CsstResult
.. # csst-l1/sls/csst_msc_sls_objextraction
.. def base_objext(input_file, output_file): -> CsstResult
.. # csst-l1/sls/csst_msc_sls_axe
.. def base_axe(input_file, output_file, dir_slsconf): -> CsstResult
.. # csst-l1/sls/csst_msc_sls_cde
.. def base_cde(input_file, output_file, dir_slsconf): -> CsstResult
.. # csst-l1/qc/csst_msc_qc0
.. def base_qc0(input_file, output_file): -> CsstResult
.. # csst-l1/qc/csst_msc_sls_qc1
.. def base_qc1(input_file, output_file): -> CsstResult
docs/source/ch01_team.rst deleted 100644 → 0
View file @ 55e4b081
CSST DAS
========
The `CSST`_ (Chinese Space Station Telescope) is a 2-meter space-based telescope
which is scheduled to be launched at December 2023.
The **CSST DAS** (Data Analysis System), led by Chao Liu, is the official team at
NAOC (National Astronomical Observatories, CAS) to build the data
reduction pipelines.
.. _CSST: https://en.wikipedia.org/wiki/Xuntian
Facility resources
------------------
Here should be a link to the CSST website
Timeline
--------
To fit the schedule of CSST, a software suite is underway, including
- survey schedule
- simulation
- data pipeline
The timeline is splitted into cycles, and each cycle spans 6 months.
- C6 (2022.7-2022.12)
- integrate L1 pipelines
- C5 (2022.1-2022.6)
Weekly meeting
--------------
- 2022-08-26 (Fri)
- 2022-09-02 (Fri)
- 2022-09-09 (Fri)
- 2022-09-16 (Fri)
- 2022-09-23 (Fri)
- 2022-09-30 (Fri)
- 2022-10-14 (Fri)
- 2022-10-21 (Fri)
docs/source/ch02_vcs.rst deleted 100644 → 0
View file @ 55e4b081
Version Control
===============
A functional quality software needs a version control system,
which provides several advantages.
1. Record versions
so that you can rollback the software to any history version.
2. Collaboration between developers
so that multiple developers can work simultaneously to enhance efficiency.
3. Keep multiple braches
so that software can be developed in a separate branch
and it does not affect the stable (released) version.
4. Code safety
the approve-merge mechanism, etc.
``git``
-------
``git`` is probably the most widely used version control system presently.
- official website: https://git-scm.com/
- **Pro Git** (2nd Ed.): https://git-scm.com/book/en/v2
Basic usages
------------
To initialize a repository:
.. code-block:: bash
git init
To add a remote source
.. code-block:: bash
git remote add origin https://csst-tb.bao.ac.cn/code/csst-l1/csst_proto.git
To show a remote source `origin`
.. code-block:: bash
git remote show origin
To clone a remote repository
.. code-block:: bash
git clone https://csst-tb.bao.ac.cn/code/csst-l1/csst_proto.git
To pull remote code to local
.. code-block:: bash
git pull origin main
To view the git log
.. code-block:: bash
git log
To view the current status
.. code-block:: bash
git status
To add your revisions to version control
.. code-block:: bash
git add ./test.py
To make a commit
.. code-block:: bash
git commit -m "make a test commit"
To push to remote repository
.. code-block:: bash
git push origin main
To reset repository to a specific commit
.. code-block:: bash
git reset 82cc809698b3b52327a0360eea10b88aacbadec6 --hard
To configure your git
.. code-block:: bash
git config --global user.name "Your Name"
git config --global user.email "your_email@address"
You can also store your credentials on disk via
.. code-block:: bash
git config --global credential.helper store
.. note::
Please DO NOT use command ``git add .`` which makes the Version Control system
track all files in the repository -- It may include many useless files such as
``*.py~`` (temporary files generated by text editor).
Work flow with ``git``
----------------------
In CSST DAS pipeline development, each developer registers an account at the
`CSST DAS GitLab`_, which is a self self-managed `GitLab`_ instance
to host data processing pipelines at NAOC.
More specifically, the L1 Pipelines are developed in the `csst-l1`_ group.
Developers are required to upload their code to the repository.
The repository should have three branches:
- ``main`` branch: for a stable version
- ``dev`` branch: for development
- ``release`` branch: for code release
After the pipeline is integrated (probably the end of C6),
a code reviewer is required to merge approved code revisions into ``main`` branch.
.. _CSST DAS GitLab: https://csst-tb.bao.ac.cn/code
.. _GitLab: https://about.gitlab.com/
.. _csst-l1: https://csst-tb.bao.ac.cn/code/csst-l1
Code review mechanism
---------------------
Starting from C7, developers of all packages should NOT directly push any code
to *main* branch which is protected.
A reviewer is needed to approve the committed modifications before
merging into the *main* branch.
Developers should switch to a ``dev`` branch before doing anything.
Then commit code and push them to ``dev`` branch.
Workflow with ``dev`` branch
----------------------------
.. code-block::
:caption: As a developer:
(base) cham@MBP16 csst_proto % git branch -a # show all branch
dev
* main
remotes/origin/main
(base) cham@MBP16 csst_proto % git checkout dev # check out dev -> then you are in dev branch
M doc/source/conf.py
M doc/source/index.rst
M doc/source/integration.rst
M doc/source/vcs.rst
Switched to branch 'dev'
(base) cham@MBP16 csst_proto % git status # view code changes
On branch dev
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: doc/source/conf.py
modified: doc/source/index.rst
modified: doc/source/integration.rst
modified: doc/source/vcs.rst
Untracked files:
(use "git add <file>..." to include in what will be committed)
doc/contents.md
no changes added to commit (use "git add" and/or "git commit -a")
(base) cham@MBP16 csst_proto % git add doc/source/*.rst doc/source/conf.py # add your code changes
(base) cham@MBP16 csst_proto % git commit -m "updated doc: author info and docker info" # commit your changes
[dev bbed3fa] updated doc: author info and docker info
4 files changed, 73 insertions(+), 4 deletions(-)
(base) cham@MBP16 csst_proto % git push origin dev # as a developer you are not allowed to push to main branch
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 10 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 1.67 KiB | 1.67 MiB/s, done.
Total 8 (delta 6), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for dev, visit:
remote: http://10.3.10.28/code/csst-l1/csst_proto/-/merge_requests/new?merge_request%5Bsource_branch%5D=dev
remote:
To https://csst-tb.bao.ac.cn/code/csst-l1/csst_proto.git
* [new branch] dev -> dev
.. code-block::
:caption: As a code reviewer:
(base) cham@MBP16 csst_proto % git checkout main # check out main branch
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
(base) cham@MBP16 csst_proto % git merge dev # merge dev branch into main AFTER view changes
Updating d9f64d3..bbed3fa
Fast-forward
doc/source/conf.py | 2 +-
doc/source/index.rst | 11 +++++++++++
doc/source/integration.rst | 50 +++++++++++++++++++++++++++++++++++++++++++++++---
doc/source/vcs.rst | 14 ++++++++++++++
4 files changed, 73 insertions(+), 4 deletions(-)
(base) cham@MBP16 csst_proto % git push origin main # push to remote main branch
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://csst-tb.bao.ac.cn/code/csst-l1/csst_proto.git
d9f64d3..bbed3fa main -> main
As a developer, you can also create a merge requrest to ask code reviewers to merge your changes on GitLab.
First, click ``Merge requests`` tag, click ``
.. image:: vcs/merge_request.png
Second, select source branch and target branch, create a new merge request.
.. image:: vcs/pushed_to_dev.png
The code reviewers should then view the changes, and decide whether the changes should be approved.
.. image:: vcs/create_merge_request.png
If the changes are accepted, approve and merge the changes into the `main` branch.
.. image:: vcs/merge_approve.png
After the changes have been merged, code reviewers should leave comment on the changes.
.. image:: vcs/merged.png
\ No newline at end of file
docs/source/ch03_packaging.rst deleted 100644 → 0
View file @ 55e4b081
Python Packaging
================
Python projects should be packaged in the standard way as shown in the
official `Python Packaging User Guide`_, and more specifically `here`_.
.. _here: https://packaging.python.org/en/latest/tutorials/packaging-projects/
.. _Python Packaging User Guide: https://packaging.python.org/en/latest/
Project structure
-----------------
An example code structure is shown below.
Files/directories with asterisks (``*``) marks are optional.
.. code-block::
csst_proto # the repository name
├── csst_proto # the package name
│ ├── data # package associated data directory
│ ├── __init__.py # necessary file for a Python package
│ ├── demo.py # Python modules
│ ├── flip_image.py
│ ├── scratch.py
│ └── api.py # the top level interface module
├── doc # *sphinx-based documentation directory
│ ├── build
│ ├── source
│ ├── Makefile
│ ├── apidoc.sh
│ ├── contents.md
│ ├── make.bat
│ └── preview.sh
├── examples # *example scripts
│ ├── how_this_code_will_be_used.py
│ └── how_to_write_docstring.py
├── tests # unit tests
│ ├── test_flip_image.py
│ └── test_other_functions.py
├── LICENSE # license file
├── README.md # README file (Markdown recommended)
├── install.sh # a single-line installation script
├── install_local.sh # local installation script
├── readthedocs.yml # *configuration file for readthedocs
├── requirements.txt # package dependencies
└── setup.py # setup file
.. note::
The ``__init__.py`` files is important, without which
the directory will not be seen as a ``Python package``.
Python version
--------------
Developers should use ``Python 3.9.X`` to implement algorithms.
The base docker image will be ``continuumio/anaconda3``, which uses
``Python 3.9.12``.
Relative import
---------------
When import a class / function within the same package, a ``relative import``
should be used instead of ``absolute import``.
An example is below (``api.py``):
.. code-block:: python
from .flip_image import flip_image, read_test_image
from .demo import a_demo_function, ADemoClass
__all__ = ["flip_image", "read_test_image", "a_demo_function", "ADemoClass"]
.. note::
Note the ``.`` means it's a relative import.
README.md
---------
A Markdown format text file to describe your package.
.. literalinclude:: ../../README.md
LICENSE
-------
Among the many choices, we encourage our developers to use ``MIT`` LICENSE.
An example is below.
.. literalinclude:: ../../LICENSE
``requirements.txt``
--------------------
``requirements.txt`` is a text file, within which a line represents a package and its version.
.. literalinclude:: ../../requirements.txt
:linenos:
.. note::
For each package, developers must specify a version.
DO NOT use ``>``, ``>=``, ``<``, ``<=`` or ``~=``, use ``==`` ONLY!
``setup.py``
------------
We use a classic format of ``setup.py`` file as shown below.
We refer readers to https://setuptools.pypa.io/en/latest/ for more information
and usages about the package `setuptools`_.
.. _setuptools: https://setuptools.pypa.io/en/latest/
.. literalinclude:: ../../setup.py
:linenos:
:language: python
.. note::
The ``install_requires`` keyword argument is commented because it takes no effect
when installed with ``pip install <your_package> --no-deps``.
Please DO NOT use it.
Include data
------------
Data files can be packaged together with code.
To include ``csst_proto/data/test_image.txt`` and ``csst_proto/data/table_data.vsb``,
set the keyword ``package_data`` in ``setup.py`` as below:
.. code-block:: python
...
package_data={"": ["LICENSE", "README.md"],
"csst_proto": ["data/test_image.txt",
"data/table_data.csv"
]},
...
.. note::
The ``""`` key represents the data files located directly in the project directory.
For files associated with package ``csst_proto``, link the data files to ``csst_proto`` key.
.. note::
DO NOT include any test data (e.g., CSST images) in the package.
We only encourage developers to pacakge configuration files inside.
Please keep your package *clean* and *tight*.
To access the data, developers should use ``relative path``.
In the example below, we show the function ``read_test_image``,
an example of how to access data under ``csst_proto/data/``.
In ``__init__.py``:
.. literalinclude:: ../../csst_proto/__init__.py
:linenos:
:language: python
and in ``flip_image.py``:
.. literalinclude:: ../../csst_proto/flip_image.py
:linenos:
:language: python
Including C code
----------------
To include C code, developers should follow the guide provided by ``setuptools``:
https://setuptools.pypa.io/en/latest/userguide/ext_modules.html.
And also consider using https://docs.astropy.org/en/latest/development/ccython.html#building-c-or-cython-extensions.
``top_level_interface`` module
------------------------------
This is a special requirement for all of CSST DAS packages.
This module is used to store the interfaces (classes / functions) that
will be called by users.
.. tip::
Remember to set the ``__all__`` variable.
This should be a list containing all interfaces.
.. note::
All the interfaces in this module should use a complete ``Numpy``-style docstring
-- They will go through docstring format validations in Jenkins!
How to install package
----------------------
We recommend the following way to install your package.
.. code-block:: bash
# install requirements
pip install -r requirements
# build extensions in place
python setup.py build_ext --inplace
# build source code
python setup.py sdist
# install from the source code
pip install dist/*.tar.gz --force-reinstall --no-deps
Sphinx-based documentation
--------------------------
This is currently regarded as an optional section.
But we still recommend developers to read a bit of the ``sphinx`` documentation.
The sphinx homepage:
- https://www.sphinx-doc.org/en/master/index.html
A tutorial on how to write restructured text (.rst) files:
- https://docutils.sourceforge.io/rst.html
docs/source/ch04_codestyle.rst deleted 100644 → 0
View file @ 55e4b081
Code Style
==========
PEP 8 : the base Python style
-----------------------------
Python code should follow **PEP 8**
- https://peps.python.org/pep-0008/
Whereas C/C++ code should follow **PEP 7**
- https://peps.python.org/pep-0007/
We have a few requirements for our developers in addition to the original PEP8.
These are grabbed from the `Coding Guidelines for astropy-affiliated packages`_.
.. _Coding Guidelines for astropy-affiliated packages: https://docs.astropy.org/en/latest/development/codeguide.html#coding-style-conventions
- Only use 4-space indent
- ``import *`` is not allowed
- The use of automatic code formatters (e.g., Black) is strongly discouraged
Exceptions to PEP 8
-------------------
The following table summarizes all PEP 8 guidelines that are **not followed**
by the this Code Style Guide.
E501
Allow lines to have > 80 characters, but <=120
E722
Allow use of bare ``try ... except ...``
E121, E123, E126, E133, E226, E241, E242, E704, W503, W504 and W505
They are ignored because they are not rules unanimously accepted,
and PEP 8 does not enforce them.
.. tip::
Click `here`_ for error code explanations and conventions
.. _here: https://pycodestyle.pycqa.org/en/latest/intro.html#error-codes
Naming conventions
------------------
All CSST Python source code is consistent with PEP 8 naming in the following ways:
- class names are ``CamelCase`` with leading uppercase
- function names should be ``lower_case_with_underscore``
- module variables used as module global constants are ``UPPERCASE_WITH_UNDERSCORES``
Code style check
----------------
Many tools can be used to check code style, such as
- PyLint: https://pylint.pycqa.org/en/latest/index.html
- Flake8: https://flake8.pycqa.org/en/latest/index.html
- pycodestyle: https://pycodestyle.pycqa.org/en/latest/index.html
All of them are from the PyCQA organization.
In our case, we recommend our developers to use the following command
(with pycodestyle) to check code style before committing code.
.. code-block:: bash
pycodestyle ./**/*.py --ignore=E121,E123,E126,E226,E24,E704,W503,W504,E501,E722
Recommended IDEs
----------------
For writing Python packages, two powerful IDEs are worth to recommend:
- PyCharm: https://www.jetbrains.com/pycharm/
- VSCode: https://code.visualstudio.com/
Both of them have plugins or built-in tools to check code style.
Numpy-style docstring
---------------------
There are several popular docstring styles, namely Google, reStructuredText, and Numpy.
The CSST DAS adopts Numpydoc-style among the three.
We refer our developers to the Numpydoc official style guide for instructions on
how to write docstrings.
- https://numpydoc.readthedocs.io/en/latest/format.html
In particular, a complete example is available here.
- https://numpydoc.readthedocs.io/en/latest/example.html
In our case, we recommend the following method to check docstring
.. code-block:: bash
python -m numpydoc --validate {YOUR_PACKAGE}.api.{YOUR_FUNCTION/CLASS}
For example,
.. code-block:: bash
python -m numpydoc --validate csst_proto.top_level_interface.flip_image
An example with typical Numpy-style docstring is shown below
.. literalinclude:: ../../csst_proto/demo.py
:linenos:
:language: python
.. note::
Numpy-style docstring uses a two-section summary for functions / classes,
including ``a short summary`` (usually a single line) right after the quote
and a following detailed ``extended summary``.
.. note::
The ``See Also`` section is not required for our development.
.. note::
Presently, Numpy style docstring check is only performed on the functions / classes
listed in the ``__all__`` variable in ``top_level_interface`` module.
Markup languages
----------------
Markdown and reStructrured Text are two of popular markup languages.
reStructrured Text is used in Numpydoc docstrings as well as sphinx-based documentation.
Markdown is much easier and usually used to write `README` for a package.
- Markdown
- https://www.markdownguide.org/
- reStructured Text
- https://www.writethedocs.org/guide/writing/reStructuredText/
docs/source/ch05_preference.rst deleted 100644 → 0
View file @ 55e4b081
Code Preference
===============
Initially we want our developers to following the
`coding guidelines for astropy-affiliated packages <https://docs.astropy.org/en/latest/development/codeguide.html>`_
as much as possible.
A few important conventions and special cases should be outlined here.
Package preference
------------------
Several packages are favored over others if they can be used to solve the problem under study.
Developers should use them as much as possible.
Standard libraries
Python standard libraries have the highest priorities, e.g., ``os``, ``re``, etc.
``numpy``, ``scipy``, ``matplotlib``
The ``BIG 3`` for Python scientific computing.
``astropy`` and its ``astropy``-affiliated packages
For example, ``astropy.io.fits`` is favored over ``pyfits``.
Parallel computing
------------------
The two packages are preferred for implementing `embarrassingly` parallel computing (without inter-communication).
- ``multiprocessing``: https://docs.python.org/3/library/multiprocessing.html
- ``joblib``: https://joblib.readthedocs.io/en/latest/
.. literalinclude:: preference/example_multiprocessing.py
:linenos:
:language: python
:caption: an example of using ``multiprocessing`` for parallel computing
The output is
.. code-block::
Total time cost: 5.095193147659302 sec!
.. literalinclude:: preference/example_joblib.py
:linenos:
:language: python
:caption: an example of using ``joblib`` for parallel computing
The output is
.. code-block::
[Parallel(n_jobs=5)]: Using backend LokyBackend with 5 concurrent workers.
[Parallel(n_jobs=5)]: Done 1 tasks | elapsed: 5.2s
[Parallel(n_jobs=5)]: Done 2 out of 5 | elapsed: 5.2s remaining: 7.8s
[Parallel(n_jobs=5)]: Done 3 out of 5 | elapsed: 5.2s remaining: 3.5s
[Parallel(n_jobs=5)]: Done 5 out of 5 | elapsed: 5.2s remaining: 0.0s
[Parallel(n_jobs=5)]: Done 5 out of 5 | elapsed: 5.2s finished
Total time cost: 5.1958301067352295 sec!
.. tip::
``joblib`` is recommended for its highly concise syntax and verbose info -- do every thing within one statement.
``n_jobs`` can be set to ``-1`` to use almost all CPUs, ``backend`` can be set to ``multiprocessing``
to use the backend built by standard library ``multiprocessing``, or ``loky`` for alleged high robustness.
Visit https://joblib.readthedocs.io/en/latest/ for more info and usages of ``joblib``,
such as the ``batch_size`` and ``verbose`` parameters.
For parallel computing with inter-communication or distributed computing,
we recommend developers to consider using ``mpi4py``: https://github.com/mpi4py/mpi4py.
Global variables
----------------
Usage of ``global`` should be prohibited.
In most cases, variables should be kept in their default scopes.
Using ``subprocess``
--------------------
``subprocess.run()`` is favored over ``subprocess.Popen()`` and ``os.system()`` as suggested in Python documentation:
- The ``subprocess`` module allows you to spawn new processes, connect to their input/output/error pipes,
and obtain their return codes. This module intends to replace several older modules and functions:
.. code-block:: python
os.system
os.spawn*
- The recommended approach to invoking ``subprocesses`` is to use the ``run()`` function for all use cases
it can handle. For more advanced use cases, the underlying ``Popen`` interface can be used directly.
Numpy multithreading
--------------------
Numpy sometime automatically uses multithreading. To see if you are actually using OpenBLAS or MKL, use
.. code-block:: python
numpy.__config__.show()
To set shut down this feature, use
.. code-block:: python
export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export OMP_NUM_THREADS=1
export VECLIB_MAXIMUM_THREADS=1
.. note::
In most cases, this automatic multithreading does not enhance the performance in practice.
Therefore, the above setting will be used in CSST pipeline globally.
docs/source/ch06_unittest.rst deleted 100644 → 0
View file @ 55e4b081
Unit Tests
==========
Unit tests should be based on `unittest`_,
while `doctest`_ is not recommended for limited usage.
In the end, `pytest`_ is used to perform the unittests.
And `coverage`_ is used to assess the unit test coverage.
Introduction to ``unittest``
----------------------------
Here we show an example unit test on ``csst_proto.top_level_interface.flip_image``.
.. literalinclude:: ../../tests/test_flip_image.py
:linenos:
:language: python
Run unit tests
--------------
Unit tests should be based on the standard library ``unittest``,
while ``doctest`` is not recommended and should not be used.
The import mode we use here is ``importlib``.
Several packages should be installed before you run unit tests.
They are
- ``pytest``: a tool to collect unit tests and run them
- ``coverage``: coverage statistics
- ``pytest-cov``: a plugin to provide ``--cov`` parameter for ``pytest``
To run unit tests collected from directory ``.``, and calculate coverage for source code in directory ``csst_proto``,
.. code-block:: bash
coverage run -m pytest --cov=csst_proto . --import-mode=importlib --cov-report=html --cov-report=term-missing
You will see messages like below
.. code-block:: none
============================ test session starts ============================
platform darwin -- Python 3.9.7, pytest-7.1.3, pluggy-0.13.1
rootdir: /Users/cham/CsstProjects/csst_proto
plugins: anyio-2.2.0, asdf-2.13.0, cov-4.0.0
collected 4 items
tests/test_demos.py . [ 25%]
tests/test_flip_image.py ... [100%]
---------- coverage: platform darwin, python 3.9.7-final-0 -----------
Name Stmts Miss Cover Missing
-----------------------------------------------------------------
csst_proto/__init__.py 3 0 100%
csst_proto/demo.py 8 3 62% 56-57, 60
csst_proto/flip_image.py 21 0 100%
csst_proto/top_level_interface.py 3 0 100%
-----------------------------------------------------------------
TOTAL 35 3 91%
Coverage HTML written to dir htmlcov
============================= 4 passed in 0.73s =============================
The coverage info is shown, and a coverage report in html format will be generated in directory ``htmlcov``.
.. _unittest: https://docs.python.org/3.9/library/unittest.html
.. _doctest: https://docs.python.org/3.9/library/doctest.html
.. _pytest: https://docs.pytest.org/en/7.1.x
.. _coverage: https://coverage.readthedocs.io/en/6.5.0/
Exclude files
-------------
To exclude specific files from coverage report, write a `.coveragerc` file.
.. code-block::
[run]
omit =
# omit anything in a .local directory anywhere
*/.local/*
# omit everything in /usr
/usr/*
# omit this single file
utils/tirefire.py
An example `.coveragerc` for `csst_proto` is
.. code-block::
[run]
omit =
tests/*
setup.py
csst_proto/top_level_interface.py
test_codestyle.py
Conventions
-----------
Before write his own unit test scripts, a developer should have a glance at
the conventions on how `pytest`_ discovers unit tests (e.g, `test_*.py` or `*_test.py` files).
For more information, click `here`_.
.. _here: https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#test-discovery
Data access
-----------
For the Main Survey, i.e., the multi-band imaging (MBI) and the slit-less spectra (SLS)
developers might use two kinds of data to run their unit tests.
Auxiliary data (data used for your unit test ONLY)
It should be put in ``/nfsdata/users/csstpipeline/L1Pipeline/unittests/<your_project_name>/``
You can obtain this path from the environment in your unit test scirpts,
.. code-block:: python
import os
os.getenv("AUX_DIR")
Simulation data (officially released simulation data)
It should be accessed with the ``csst_common.data_manager.CsstMsDataManager``.
The ``CsstMsDataManager.quickstart`` provides an interface to initialize
an instance. See the next chapter for more details.
Here we show an example which initializes the ``CsstMsDataManager`` with the 100th exposure of ``C5.2``
simulation to perform a unit test for the project ``csst_ms_sls_instrument``.
.. code-block:: python
import os
import unittest
from csst_common.data_manager import CsstMsDataManager
from csst_ms_sls_instrument.top_level_interface import sls_instrument_pipeline
aux_dir = os.getenv("AUX_DIR")
out_dir = os.path.join(aux_dir, "csst_ms_sls_instrument")
class TestCsstMsSlsInstrument(unittest.TestCase):
def setUp(self) -> None:
self.dm = CsstMsDataManager.quickstart(ver_sim="C5.2", datatype="sls", dir_l1=out_dir, exposure_id=100)
def test_instrument(self):
test_detector = 1
# for l0 image path:
self.dm.l0_detector(test_detector)
# for l1 image path:
self.dm.l1_detector(detector=test_detector, post="flt.fits")
self.dm.l1_file(name="merged.fits", comment="This is a merged image.")
self.assertTrue(os.path.exists(self.dm.l0_detector(test_detector)))
For developers on other instruments (CPIC, THZ, MCI and IFS),
unit tests might be performed in their local environments.
docs/source/ch07_simulation.rst deleted 100644 → 0
View file @ 55e4b081
Using Simulation Data
=====================
Versions
--------
The current available CSST simulation data products are not versioned.
Therefore, in pipeline system, we use ``ver_sim`` keyword in code (e.g., ``CsstMsDataManager``).
The ``ver_sim`` uses a ``C{X}.{Y}`` format string, where
``X`` denotes the Cycle during which the simulation is released,
and ``Y`` denotes the serial number.
For example, ``C5.2`` represents for the second released version of simulation in Cycle 5.
Server details
--------------
Server
``dandelion``.
The ``IP``, ``Username`` and ``Password`` are available via request to ``csst_das@nao.cas.cn``.
Simulation data path
- ``C5.2``: ``/nfsdata/share/csst_simulation_data/Cycle-5-SimuData``
- ``C3``: ``/nfsdata/share/csst_simulation_data/Cycle-3-SimuData``
.. note::
Presently, the ``CsstMsDataManager`` only supports ``C5.2`` simulation.
Naming conventions
------------------
.. code-block:: python
>>> import re
>>> pattern = re.compile(
>>> r"CSST_"
>>> r"(?P<facility>[A-Z]+)_"
>>> r"(?P<project>[A-Z]+)_"
>>> r"(?P<data_type>[A-Z]+)_"
>>> r"(?P<t_start>[0-9]{14})_"
>>> r"(?P<t_stop>[0-9]{14})_"
>>> r"(?P<obs_id>[0-9]{9})_"
>>> r"(?P<detector>[0-9]{2})_"
>>> r"L(?P<level>[0-9]+)_"
>>> r"(?P<version>[A-Z0-9]+).fits"
>>> )
>>> mo = re.fullmatch(pattern, r"CSST_MSC_MS_SCI_20270626203558_20270626203828_100000066_01_L0_1.fits")
>>> print(mo.groupdict())
{'facility': 'MSC', 'project': 'MS', 'data_type': 'SCI', 't_start': '20260612100759', 't_stop': '20260612101029', 'obs_id': '100000036', 'detector': '16', 'level': '0', 'version': '1'}
docs/source/ch08_csst_common.rst deleted 100644 → 0
View file @ 55e4b081
Using ``csst_common``
=====================
In this chapter, we introduce the basic usages of some IMPORTANT functions / classes in ``csst_common``.
Package source code and installation guide can be found in the link below.
- https://csst-tb.bao.ac.cn/code/csst-l1/csst_common
``csst_common.data_manager.CsstMsDataManager``
----------------------------------------------
This class provides an interface to switch between local files and DFS.
To initilize from local directory:
.. code-block:: python
from csst_common.data_manager import CsstMsDataManager
# for full basic initialization
dm = CsstMsDataManager.from_dir(
ver_sim="C5.2", # version of simulation, "C5.2" is the latest
datatype="mbi", # data type, "mbi", "sls" or "all"
dir_l0=".", # the L0/input directory
dir_l1=".", # the L1/output directory
path_aux="", # aux file paths (master bias, dark, flat)
dfs_mode=False, # if True, use DFS in pipeline modules
dfs_node='kmust', # DFS server, chosen from {"kmust", "pml"}
)
dm.target_detectors = None # use all available detectors
In particular, if you are on ``dandelion`` or ``PML node``, you can use a ``quickstart`` method.
.. code-block:: python
# for a quick start ()
dm = CsstMsDataManager.quickstart(
ver_sim='C5.2', # version of simulation, only "C5.2" is supported
datatype='mbi', # data type, "mbi", "sls" or "all"
dir_l1='.', # the L1/output directory
exposure_id=100, # the exposure ID
dfs_mode=False, # if True, use DFS in pipeline modules
dfs_node='kmust', # DFS server, chosen from {"kmust", "pml"}
)
dm
You will get the outputs:
.. code-block::
<CsstMsDataManager>
Data type = mbi
Valid detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25]
Available detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25]
Target detectors = [6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 25]
dir_l0 = /nfsdata/share/csst_simulation_data/Cycle-5-SimuData/multipleBandsImaging/NGP_AstrometryON_shearOFF/MSC_0000100/
dir_l1 = .
dfs_mode = False
dfs_node = kmust
.. note::
This ``quickstart`` method is ONLY available on
``dandelion`` or the ``PML node``.
In your local development environment, you have to use
``CsstMsDataManager.from_dir(...)`` to initialize it.
Attributes
^^^^^^^^^^
Some attributes can be accessed as below.
.. code-block:: python
# access L0 directory
dm.dir_l0
# access L1 directory
dm.dir_l1
# access dir_pcref
dm.dir_pcref
# access path_aux
dm.path_aux
# access ver_sim
dm.ver_sim
# access target detectors
dm.target_detectors
# access available detectors
dm.available_detectors
# define an L1 file (detector-specified)
dm.l1_detector(detector=6)
# define an L1 file (non-detector-specified)
dm.l1_file("flipped_image.fits")
To get master files:
.. code-block:: python
# to get bias
dm.get_bias()
# to get dark
dm.get_dark()
# to get flat
dm.get_flat()
Query Reference Catalog
^^^^^^^^^^^^^^^^^^^^^^^
``CsstMsDataManager`` provides a method ``query_rc`` to query reference catalog from ``DFS``.
.. code-block:: python
cat = dm.query_rc(ra=180, dec=0, radius=2, min_mag=0, max_mag=30)
if cat is None:
print("The query failed!")
else:
print("The query succeeded!")
Access code to data products
----------------------------
``Multi-Band Imaging``
^^^^^^^^^^^^^^^^^^^^^^
========================== ===== ========================================================== ==================
Data source Level File name / access code Notes
========================== ===== ========================================================== ==================
raw 0 ``dm.l0_detector(detector=detector)`` sci image
raw 0 ``dm.l0_crs(detector=detector)`` cosmic ray image
raw 0 ``dm.l0_cat(detector=detector)`` input catalog
raw 0 ``dm.l0_log(detector=detector)`` log file
``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="img.fits")`` corrected image
``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="flg.fits")`` flag image
``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="wht.fits")`` weight image
``csst_ms_mbi_instrument`` -- ``dm.l1_detector(detector=detector, post="img.head")`` L0 fits header
``DFS`` -- ``dm.l1_file(name="gaia_dfs.fits")`` reference catalog
``csst_ms_mbi_distortion`` -- ``dm.l1_detector(detector=detector, post="wcs.head")`` fits head with WCS
``csst_ms_mbi_position`` -- ``dm.l1_detector(detector=detector, post="img.acat")`` photometry catalog
``csst_ms_mbi_position`` -- ``dm.l1_detector(detector=detector, post="img.rcat")`` reference catalog
``csst_ms_mbi_flux`` 1 ``dm.l1_detector(detector=detector, post="img_L1.fits")`` sci image
``csst_ms_mbi_flux`` 1 ``dm.l1_detector(detector=detector, post="flg_L1.fits")`` flag image
``csst_ms_mbi_flux`` 1 ``dm.l1_detector(detector=detector, post="wht_L1.fits")`` weight image
``csst_ms_mbi_photometry`` 1 ``dm.l1_detector(detector=detector, post="cat.fits")`` source catalog
``csst_ms_mbi_photometry`` 1 ``dm.l1_detector(detector=detector, post="psf.fits")`` source catalog
========================== ===== ========================================================== ==================
``Slit-Less Spectra 2D``
^^^^^^^^^^^^^^^^^^^^^^^^
========================== ===== =========================================================== ==================
Data source Level File name / access code Notes
========================== ===== =========================================================== ==================
raw 0 ``dm.l0_detector(detector=detector)`` sci image
raw 0 ``dm.l0_crs(detector=detector)`` cosmic ray image
raw 0 ``dm.l0_cat(detector=detector)`` input catalog
raw 0 ``dm.l0_log(detector=detector)`` log file
``csst_ms_sls_instrument`` 1 ``dm.l1_detector(detector=detector, post="L1_1_flt.fits")`` corrected image
``csst_ms_sls_position`` -- ``dm.l1_detector(detector=detector, post="L1_1.fits")`` WCS determined
========================== ===== =========================================================== ==================
``Slit-Less Spectra 1D``
^^^^^^^^^^^^^^^^^^^^^^^^
Note that for 1D pipeline, ``dm.target_detectors`` is a list containing only one element.
=========================== ===== =============================================================== ==================
Data source Level File name / access code Notes
=========================== ===== =============================================================== ==================
``csst_ms_sls_position`` -- ``dm.l1_detector(detector=detector, post="L1_1.fits")`` L1 SLS image
``csst_common`` -- ``dm.target_detectors[0]`` detector number
``csst_ms_sls_mosaic`` -- ``dm.l1_detector(detector=detector, post="coadd.fits")`` coadded image
``csst_ms_sls_mosaic`` -- ``dm.l1_detector(detector=detector, post="coadd.weight.fits")`` coadded weight
``csst_ms_sls_directimage`` -- ``dm.l1_detector(detector=detector, post="dir_img.fits")`` cropped image
``csst_ms_sls_directimage`` -- ``dm.l1_detector(detector=detector, post="dir_wht.fits")`` cropped weight
``csst_ms_sls_objextract`` -- ``dm.l1_detector(detector=detector, post="comb.cat")`` combined catalog
``csst_ms_sls_objextract`` -- ``dm.l1_detector(detector=detector, post="segmentation.fits")`` segmentation file
``csst_ms_sls_sky`` -- ``dm.l1_detector(detector=detector, post="L1_1_sky.fits")`` sky background
``csst_ms_sls_axe`` -- ``dm.l1_detector(detector=detector, post="L1_1_axe.fits")`` aXe spectra
``csst_ms_sls_cde`` -- ``dm.l1_detector(detector=detector, post="L1_1_cde.fits")`` CDE spectra
=========================== ===== =============================================================== ==================
``csst_common.file_recorder.FileRecorder``
------------------------------------------
Get an empty ``FileRecorder``.
This is initially a ``list``-like object.
Use ``FileRecorder.add_record()`` to add file records.
.. code-block::
:linenos:
>>> from csst_common.file_recorder import FileRecorder
>>> fr = FileRecorder()
>>> for i in range(3):
>>> fr.add_record(filepath="test{:03d}.txt".format(i),
>>> db=True,
>>> comment="Test file {:d}".format(i))
>>> fr.pprint_all()
<FileRecorder length=3>
filepath db comment existence
----------- ---- ----------- ---------
test000.txt True Test file 0 False
test001.txt True Test file 1 False
test002.txt True Test file 2 False
``FileRecorder`` inherits from ``list``.
Therefore, if you are collecting records from subprocesses,
here is the way to combine records:
.. code-block:: python
:linenos:
>>> from csst_common.file_recorder import FileRecorder
>>> fr1 = FileRecorder()
>>> fr1.add_record(filepath="test1.txt", db=False, comment="test file 1")
>>> fr2 = FileRecorder()
>>> fr2.add_record(filepath="test2.txt", db=False, comment="test file 2")
>>> fr_all = FileRecorder([*fr1, *fr2])
>>> fr_all.pprint_all()
<FileRecorder length=2>
filepath db comment existence
--------- ----- ----------- ---------
test1.txt False test file 1 False
test2.txt False test file 2 False
>>> fr_all.extend(fr1)
>>> fr_all.pprint_all()
<FileRecorder length=3>
filepath db comment existence
--------- ----- ----------- ---------
test1.txt False test file 1 False
test2.txt False test file 2 False
test1.txt False test file 1 False
``csst_common.logger.get_logger()``
-----------------------------------
Get the default configured ``logging.Logger``.
.. code-block:: python
:linenos:
from csst_common.logger import get_logger
logger = get_logger()
.. warning::
Developers should NOT use ``print`` function extensively in code.
Generally ``print`` can be replaced with ``logger.debug("msg")``.
For important information, use ``logger.info("msg")``.
For warnings, use ``logger.warning("msg")``.
For errors, use ``logger.error("msg")``.
Conf https://docs.python.org/3/library/logging.html for more information
about the usage of ``logging.Logger``.
``csst_common.status.CsstStatus``
---------------------------------
Developers should use ``csst_common.status.CsstStatus`` to return the
status of their interfaces.
.. code-block:: python
:linenos:
from csst_common.status import CsstStatus
# presently 3 kinds of status are available, they are
CsstStatus.PERFECT # all images are processed smoothly
CsstStatus.WARNING # problematic
CsstStatus.ERROR # unavailable, DO NOT use this
An example interface
--------------------
We recommend our developers to use the code structure used in the example interfaces,
e.g., ``process_single_image`` and ``process_multiple_images``.
The example code is shown below.
Source code
^^^^^^^^^^^
.. literalinclude:: csst_common/example_interface.py
:caption: ``example_interface.py``
:emphasize-lines: 7-11,25-29,75-88,103-116,173-175
:linenos:
:language: python
Rendered ``docstring``
^^^^^^^^^^^^^^^^^^^^^^
.. automodule:: example_interface
:members:
``csst_common.params``
----------------------
See https://csst-tb.bao.ac.cn/code/csst-l1/csst_common/-/blob/main/csst_common/data/csst_params.yml.
To use the parameters, use
.. code-block:: python
from csst_common.params import CSST_PARAMS as CP
Module Identifier
-----------------
to be updated
DFS interfaces
--------------
==================================== ====================== ==========================================
Method Function Original DFS interface
==================================== ====================== ==========================================
``CsstMsDataManager.dfs_query_rc()`` query catalog from DFS ``csst_dfs_api.common.catalog.CatalogApi``
``CsstMsDataManager.dfs_query_l0()`` query L0 data from DFS --
``CsstMsDataManager.dfs_query_l1()`` query L1 data from DFS --
``CsstMsDataManager.dfs_push_l1()`` push L1 data to DFS ``csst_dfs_api.msc.level1.Level1DataApi``
==================================== ====================== ==========================================
\ No newline at end of file
docs/source/ch09_build.rst deleted 100644 → 0
View file @ 55e4b081
Build and Test
==============
A push to git repository will trigger an automatic build and test on the Jenkins platform.
Then a notification email from ``csst_das@nao.cas.cn`` is composed,
attached with a ``build log`` and a ``coverage report``,
is sent to relavant developers and code reviewers.
``build status`` explained
--------------------------
The ``build status`` is in the subject of the notification email.
Here are the explanations of them.
Successful
Congratulations! Your code passes all tests.
Fixed
Although last build fails, this build is successful.
Failure
Current build fails in at least one test.
Still Failing
Current build fails in at least one test. And last build fails.
``build log`` explained
-----------------------
The build log includes 6 sections.
[1/6] Check on versions of requirements
The status will be non-zero if any of your requirements is behind other developers.
[2/6] Check on installation of requirements
The status will be non-zero if ``pip install -r requirements.txt`` exit with non-zero status.
[3/6] Check on installation of package
The status will be non-zero if the following code are unsuccessfully executed
.. code-block:: bash
rm -rf dist # remove dist directory if exists
python setup.py build_ext --inplace # build extension
python setup.py sdist # build source code
pip install dist/*.tar.gz --force-reinstall --no-deps # install package without dependencies
[4/6] Check on import of interfaces (``top_level_interface``)
The status will be non-zero if the following code are unsuccessfully executed
.. code-block:: python
from <your_package>.top_level_interface import *
[5/6] Run unit tests and coverage
The status will be non-zero if the following code are unsuccessfully executed
.. code-block:: bash
coverage run -m pytest --import-mode=importlib
[6/6] ``Numpydoc`` validation of interfaces
The status will be non-zero if the following code are unsuccessfully executed
.. code-block:: bash
python -m numpydoc --validate {YOUR_PACKAGE}.top_level_interface.{YOUR_FUNCTION/CLASS}
An example of test summary is at the end of build log (the attached ``build.zip``).
If all checks/tests/validations passed, you will see a section as below.
.. code-block:: none
===================== TEST SUMMARY =====================
Test passed!
Check requirements: 0
Install requirements: 0
Install package: 0
Importability status: 0
Unit test status: 0
Numpydoc status: 0
Unit test coverage report
-------------------------
The ``build log`` also contains a section of unit test coverage report.
An example is below.
.. code-block:: none
---------- coverage: platform darwin, python 3.9.7-final-0 -----------
Name Stmts Miss Cover Missing
-----------------------------------------------------------------
csst_proto/__init__.py 3 0 100%
csst_proto/demo.py 8 3 62% 56-57, 60
csst_proto/flip_image.py 21 0 100%
csst_proto/top_level_interface.py 3 0 100%
-----------------------------------------------------------------
TOTAL 35 3 91%
This means the overall unit test coverage is 91%.
The missing terms are also shown in the last column.
You can also find a detailed coverage report in ``html`` format in the attached ``htmlcov.zip``.
Caveats
-------
The Numpy style docstring check is only for the packages listed below.
- csst_proto
- csst_cicd
- csst_ms_mbi_instrument
- csst_ms_mbi_distortion
- csst_ms_mbi_position
- csst_ms_mbi_flux
- csst_ms_mbi_photometry
We welcome other packages participate in.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment