Merge pull request #2 from cmarmo/master

Weighting documentation section

.gitignore

@@ -16,6 +16,8 @@
 /src/Makefile.in
 /src/fits/Makefile.in
 /src/wcs/Makefile.in
+/src/levmar/Makefile.in
+/tests/Makefile.in
 
 # configure generated files
 /Makefile
@@ -30,20 +32,26 @@
 /man/Makefile
 /sextractor.spec
 /src/.deps/
+/src/levmar/.deps/
 /src/Makefile
 /src/fits/.deps/
 /src/fits/Makefile
 /src/wcs/.deps/
 /src/wcs/Makefile
+/src/levmar/Makefile
+/tests/Makefile
 /stamp-h1
 /man/sex.1
 
 # build objects and targets
 *.o
 /src/sex
+/src/ldactoasc
 /src/f
 /src/fits/libfits.a
 /src/wcs/libwcs_c.a
+/src/levmar/liblevmar.a
+/doc/build/
 
 # generated by icc with -prof-gen=srcpos option
 *.dyn

doc/src/Using.rst

 .. File Using.rst
 
+.. _using-sextractor:
+
 Using SExtractor
 ================
 

doc/src/Weighting.rst

+.. File Weighting.rst
+
+Weighting
+=========
+
+The noise level in astronomical images is often fairly constant, that is, constant values for the gain,
+the background noise and the detection thresholds can be used over the whole frame.
+Unfortunately in some cases, like strongly vignetted or composited images, this approximation
+is no longer good enough.
+This leads to detecting clusters of detected noise peaks in the noisiest parts of the image,
+or missing obvious objects in the most sensitive ones.
+|SExtractor| is able to handle images with variable noise.
+It does it through *weight maps*, which are frames having the same size as the images where objects are detected or
+measured, and which describe the noise intensity at each pixel.
+These maps are internally stored in units of *absolute variance* (in :math:`ADU^2`)
+We employ the generic term *weight map* because these maps can also be interpreted as quality index maps:
+infinite variance (:math:`\ge 10^{30}` by definition in |SExtractor|) means that the related pixel in the science frame is totally unreliable and should be ignored.
+The variance format was adopted as it linearizes most of the operations done over weight maps (see below).
+
+This means that the noise covariances between pixels are ignored.
+Although raw CCD images have essentially white noise, this is not the case for warped images, for which resampling may
+induce a strong correlation between neighbouring pixels.
+In theory, all non-zero covariances within the geometrical limits of the analysed patterns should be taken into account to derive
+thresholds or error estimates.
+Fortunately, the correlation length of the noise is often smaller than the patterns to be detected or measured,
+and constant over the image. In that case one can apply a simple *fudge factor* to the estimated variance to account for
+correlations on small scales. This proves to be a good approximation in general, although it certainly leads to
+underestimations for the smallest patterns.
+
+.. _weigth-map-format:
+
+Weight-map formats
+------------------
+
+|SExtractor| accepts in input, and converts to its internal variance format, several types of weight-maps.
+This is controlled through the ``WEIGHT_TYPE`` configuration keyword.
+Those weight-maps can either be read from a FITS file, whose name is specified by the ``WEIGHT_IMAGE`` keyword,
+or computed internally.
+Valid ``WEIGHT_TYPE`` values are:
+
+* ``NONE`` :
+    No weighting is applied. The related ``WEIGHT_IMAGE`` and ``WEIGHT_THRESH`` (see below) parameters are ignored.
+
+
+* ``BACKGROUND`` :
+    the science image itself is used to compute internally a variance map (the related ``WEIGHT_IMAGE``
+    parameter is ignored). Robust (:math:`3\sigma`-clipped) variance estimates are first computed within the same
+    background meshes as those described in ?? [#f1].
+    The resulting low-resolution variance map is then bicubic-spline-interpolated on the fly to produce
+    the actual full-size variance map.
+    A check-image with ``CHECKIMAGE_TYPE`` ``MINIBACK_RMS`` can be requested to examine the low-resolution variance map.
+
+
+* ``MAP_RMS`` :
+    the FITS image specified by the ``WEIGHT_IMAGE`` file name must contain a weight-map in units of
+    absolute standard deviations (in ADUs per pixel).
+
+
+* ``MAP_VAR`` :
+    the FITS image specified by the ``WEIGHT_IMAGE`` file name must contain a weight-map in units of
+    relative variance. A robust scaling to the appropriate absolute level is then performed by comparing this
+    variance map to an internal, low-resolution, absolute variance map built from the science image itself.
+
+
+* ``MAP_WEIGHT`` :
+    the FITS image specified by the ``WEIGHT_IMAGE`` file name must contain a weight-map in units of
+    relative weights.
+    The data are converted to variance units (by definition variance :math:`\propto\frac{1}{weight}`),
+    and scaled as for ``MAP_VAR``.
+    ``MAP_WEIGHT`` is the most commonly used type of weight-map: a flat-field, for example, is generally
+    a good approximation to a perfect weight-map.
+
+Effect of weighting
+-------------------
+
+Weight-maps modify the working of |SExtractor| in the following respects:
+
+#. Bad pixels are discarded from the background statistics.
+   If more than :math:`50\%` of the pixels in a background mesh are bad, the local background value and
+   its standard deviation are replaced by interpolation of the nearest valid meshes.
+
+#. The detection threshold *t* above the local sky background is adjusted for each pixel *i*
+   with variance :math:`\sigma^2_i` : :math:`t_i=DETECT\_THRESH\times\sqrt{\sigma^2_i}`, where
+   ``DETECT_THRESH`` is expressed in units of standard deviations of the background noise.
+   Pixels with variance above the threshold set with the ``WEIGHT_THRESH`` parameter are therefore simply not detected.
+   This may result in splitting objects crossed by a group of bad pixels.
+   Interpolation (see ???) should be used to avoid this problem.
+   If convolution filtering is applied for detection, the variance map is convolved too.
+   This yields optimum scaling of the detection threshold in the case where noise is uncorrelated from pixel to pixel.
+   Non-linear filtering operations (like those offered by artificial retinae) are not affected.
+
+#. The cleaning process (??) takes into account the exact individual thresholds assigned to each pixel for
+   deciding about the fate of faint detections.
+
+#. Error estimates like ``FLUXISO_ERR`` , ``ERRA_IMAGE`` , etc., make use of individual variances too.
+   Local background-noise standard deviation is simply set to :math:`\sqrt{\sigma^2_i}`.
+   In addition, if the ``WEIGHT_GAIN`` parameter is set to ``Y`` (which is the default), it is assumed that the
+   local pixel gain (i.e., the conversion factor from photo-electrons to ADUs) is inversely proportional to
+   :math:`\sigma^2_i`, its median value over the image being set by the ``GAIN`` configuration parameter.
+   In other words, it is then supposed that the changes in noise intensities seen over the images are due to gain changes.
+   This is the most common case: correction for vignetting, or coverage depth.
+   When this is not the case, for instance when changes are purely dominated by those of the read-out noise,
+   ``WEIGHT_GAIN`` shall be set to ``N``.
+
+#. Finally, pixels with weights beyond ``WEIGHT_THRESH`` are treated just like pixels discarded by the masking process (??).
+
+Combining weight maps
+---------------------
+
+All the weighting options listed in :ref:`weigth-map-format` can be applied separately to detection and measurement images
+(:ref:`using-sextractor`), even if some combinations may not always make sense.
+For instance, the following set of configuration lines:
+
+``WEIGHT_IMAGE`` *rms.fits*, *weight.fits*
+
+``WEIGHT_TYPE`` ``MAP_RMS``, ``MAP_WEIGHT``
+
+will load the FITS file *rms.fits* and use it as an RMS map for adjusting the detection threshold and cleaning,
+while the *weight.fits* weight map will only be used for scaling the error estimates on measurements.
+This can be done in single- as well as in dual-image mode (:ref:`using-sextractor`).
+``WEIGHT_IMAGE`` can be ignored for ``BACKGROUND`` ``WEIGHT_TYPE``.
+It is of course possible to use weight-maps for detection or for measurement only.
+The following configuration:
+
+``WEIGHT_IMAGE`` *weight.fits*
+
+``WEIGHT_TYPE`` ``NONE``, ``MAP_WEIGHT``
+
+will apply weighting only for measurements; detection and cleaning operations will remain unaffected.
+
+.. rubric:: Footnotes
+
+.. [#f1] The mesh-filtering procedures act on the variance map, too.
+
+.. include:: keys.rst
+

doc/src/conf.py

@@ -81,7 +81,7 @@ language = None
 # There are two options for replacing |today|: either, you set today to some
 # non-false value, then it is used:
 #
-today = 'Thu Oct 26 2017'
+today = 'Mon Nov 06 2017'
 #
 # Else, today_fmt is used as the format for a strftime call.
 #

doc/src/index.rst

@@ -20,6 +20,7 @@ Contents
    Using
    Config
    Param
+   Weighting
    Measurements
    references
 

man/sex.1

-.TH SEXTRACTOR "1" "October 2017" "SExtractor 2.24.2" "User Commands"
-.SH NAME
-sex \- extract a source catalogue from an astronomical FITS image
-.SH SYNOPSIS
-.B sex \fIimage\fR [\fI-c configuration-file\fR]
-.RS
-[\fI-parameter1 value1 -parameter2 value2 ...\fR]
-.RE
-.TP
-.B sex \fIimage\_detect\fR \fIimage_measure\fR [\fI-c configuration-file\fR]
-.RS
-[\fI-parameter1 value1 -parameter2 value2 ...\fR]
-.RE
-.TP
-.B sex \fI-d\fR
-.SH DESCRIPTION
-SExtractor is a program that builds a catalogue of objects from an astronomical
-image. Although it is particularly oriented towards reduction of large scale
-galaxy-survey data, it performs rather well on moderately crowded star fields.
-.RE
-See http://astromatic.net/software/sextractor for more details.
-.SS "Operation modes:"
-.TP
-\fB\-h\fR, \fB\-\-help\fR
-print this help, then exit
-.TP
-\fB\-V\fR, \fB\-\-version\fR
-print version number, then exit
-.TP
-\fB\-d\fR, \fB\-\-dump\fR
-dump a default configuration file
-.SH MANUAL
-The full documentation for
-.B SExtractor
-is maintained as a PDF manual available at
-.B http://astromatic.net/software/sextractor
-.SH "REPORTING BUGS"
-Please report bugs to the SExtractor software forum at
-http://astromatic.net/forum.
-.SH "SEE ALSO"
-.BR
-eye(1), missfits(1), psfex(1), scamp(1), sky(1), stiff(1), stuff(1), swarp (1),
-ww(1)
-.SH AUTHORS
-Written by Emmanuel Bertin (Institut d'Astrophysique de Paris)
-.PP
-.SH LICENSE
-SExtractor is licensed under the terms of the General Public License (GPL),
-version 3.
-.SH COPYRIGHT
-.PP
-This is free software; see the source for copying conditions.  There is NO
-warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-