Doc: added Flagging section and flag description tables in Windowed...

Doc: added Flagging section and flag description tables in Windowed measurement and Weighting sections.

doc/src/Flagging.rst

+.. File Flagging.rst
+
+.. include:: global.rst
+
+.. _flagging:
+
+Flagging
+========
+
+Both *internal* and *external* flags are available for each detection as catalog parameters:
+
+* Several sets of internal flags are produced by the various processes within |SExtractor|:
+
+  * :ref:`Extraction flags <flags_def>` (:param:`FLAGS`)
+  * :ref:`Weighting flags <flags_weight_def>` (:param:`FLAGS_WEIGHT`)
+  * :ref:`Windowed measurement flags <flags_win_def>` (:param:`FLAGS_WIN`)
+  * :ref:`Model-fitting flags <flags_model_def>` (:param:`FLAGS_MODEL`);
+
+* External flags are derived from *flag maps*. Flag maps are images in integer format, and have the same size as the science images, where pixel values can be used to flag some pixels (for instance, "bad" or noisy pixels). Different combinations of flags can be applied within the isophotal area that defines each object, to produce a unique value that will be written to the catalog.
+
+Each catalog parameter comprises several flag bits as a sum of powers of 2 coded in decimal. For example, a saturated detection close to an image boundary will have :param:`FLAGS` = 4+8 = 12 (see below).
+
+.. _flags_def:
+
+Extraction flags: :param:`FLAGS`
+--------------------------------
+
+:param:`FLAGS` contains 8 flag bits with basic warnings about the source extraction process, in order of increasing concern.
+
+.. _flags_table:
+
+.. csv-table:: :param:`FLAGS` description
+  :header: "Value", "Meaning"
+  :widths: 3 60
+
+  1, ":ref:`aperture photometry <photometry>` is likely to be biased by neighboring sources or by more than 10% of bad pixels in any aperture"
+  2, "the object has been deblended"
+  4, "at least one object pixel is saturated"
+  8, "the isophotal footprint of the detected object is truncated (too close to an image boundary)"
+  16, "at least one photometric aperture is incomplete or corrupted (hitting buffer or memory limits)"
+  32, "the isophotal footprint is incomplete or corrupted (hitting buffer or memory limits)"
+  64, "a memory overflow occurred during deblending"
+  128, "a memory overflow occurred during extraction"
+
+
+External flags: :param:`IMAFLAGS_ISO`
+-------------------------------------
+
+The processing of external flags is triggered whenever :param:`IMAFLAGS_ISO` or :param:`NIMAFLAGS_ISO` are present in the catalog parameter file.
+The file names of the images containing flag maps in |FITS| format are set with the ``FLAG_IMAGE`` configuration keyword.
+Flag maps must be stored as 2-dimensional arrays of 8, 16 or 32 bits integers.
+The integers may be signed or unsigned, although for Boolean operations the sign bit will be interpreted as bit 7, 15 or 31, depending on integer size.
+Obviously all flag maps must have the same dimensions as the image used for detection. 
+Flag maps can be created using, e.g., the |WeightWatcher|_ software :cite:`2008ASPC_394_619M`.
+
+The values of flag map pixels that overlap the isophotal area of a given detected object are combined and stored in the catalog as the long integer :param:`IMAFLAGS_ISO`.
+The ``FLAG_TYPE`` configuration keyword sets the type of combination applied individually to each flag map.
+5 types of combination are currently supported:
+
+- ``OR``: arithmetic (bit-to-bit) **OR** of flag map pixels.
+- ``AND``: arithmetic (bit-to-bit) **AND** of non-zero flag map pixels.
+- ``MIN``: minimum of the (signed) flag map pixel values.
+- ``MAX``: maximum of the (signed) flag map pixel values.
+- ``MOST``: most frequent non-zero flag map pixel value.
+
+The :param:`NIMAFLAGS_ISO` catalog parameter contains the number(s) of relevant flag map pixels: the number of non-zero flag map pixels if ``FLAG_TYPE`` is set to ``OR`` or ``AND``, or the number of pixels with value :param:`IMAFLAGS_ISO` if ``FLAG_TYPE`` is set to ``MIN`` , ``MAX``, or ``MOST``.

doc/src/Model.rst

@@ -133,10 +133,14 @@ Minimization
 Minimization of the loss function :math:`\lambda(\boldsymbol{q})` is carried out using the `Levenberg-Marquardt algorithm <http://en.wikipedia.org/wiki/Levenberg%E2%80%93Marquardt_algorithm>`_, and more specifically the |LevMar|_ implementation :cite:`lourakis04LM`.
 The library approximates the Jacobian matrix of the model from finite differences using Broyden's :cite:`Broyden1965ACo` rank one updates.
 The fit is done inside a disk which diameter is scaled to include the isophotal footprint of the object, plus the FWHM of the PSF, plus a 20 % margin.
+
 The number of iterations is returned in the :param:`NITER_MODEL` measurement parameter.
 It is generally a few tens.
 The final value of the modified chi square term in :eq:`loss_func`, divided by the number of degrees of freedom, is returned in :param:`CHI2_MODEL`.
-The :param:`FLAGS_MODEL` parameter flags various issues which may happen during the fitting process (see the flags section for details on how flags are managed in |SExtractor|):
+
+.. _flags_model_def:
+
+The :param:`FLAGS_MODEL` catalog parameter flags various issues which may happen during the fitting process (see the :ref:`flagging` section for details on how flags are managed in |SExtractor|):
 
 .. _flags_model_table:
 

doc/src/Photom.rst

@@ -2,6 +2,8 @@
 
 .. include:: global.rst
 
+.. _photometry:
+
 Photometry
 ==========
 

doc/src/PositionWin.rst

@@ -5,8 +5,8 @@
 Windowed positional parameters
 ==============================
 
-Measurements performed through a *window* function (an *envelope*) do not have many of the drawbacks of isophotal measurements. |SExtractor| implements “windowed” versions for most
-of the measurements described in the :ref:`previous section<pos_iso_def>`:
+Measurements performed through a *window* function (an *envelope*) do not have many of the drawbacks of isophotal measurements.
+|SExtractor| implements “windowed” versions for most of the measurements described in the :ref:`previous section<pos_iso_def>`:
 
 .. note::
   Unless otherwise noted, all parameter names given below are only prefixes. They must be followed by _IMAGE if the results shall be expressed in pixel coordinates or :param:`_WORLD`, :param:`_SKY`, :param:`_J2000` or :param:`_B1950` for |WCS|_ coordinates (see :ref:`coord_suffix`).
@@ -25,8 +25,7 @@ of the measurements described in the :ref:`previous section<pos_iso_def>`:
 | :param:`CXX`, :param:`CYY`, :param:`CXY`        | :param:`CXXWIN`, :param:`CYYWIN`, :param:`CXYWIN`        |
 +-------------------------------------------------+----------------------------------------------------------+
 
-The computations involved are roughly the same except that the pixel values are integrated within a circular Gaussian window as opposed to
-the object’s isophotal footprint.
+The computations involved are roughly the same except that the pixel values are integrated within a circular Gaussian window as opposed to the object’s isophotal footprint.
 The Gaussian window is scaled to each object; the Gaussian FWHM is the diameter of the disk that contains half of the object flux (:math:`d_{50}`).
 Note that in double-image mode (§[chap:using]) the window is scaled based on the *measurement* image.
 
@@ -212,7 +211,22 @@ The error ellipse parameters are:
    {\rm cov}^2(\overline{x_{\tt WIN}},\overline{y_{\tt WIN}})}.
    \end{aligned}
 
+.. _flags_win_def:
 
+Windowed measurement flags: :param:`FLAGS_WIN`
+----------------------------------------------
+
+The :param:`FLAGS_WIN` catalog parameter flags various issues which may happen with windowed measurements (see the :ref:`flagging` section for details on how flags are managed in |SExtractor|):
+
+.. _flags_win_table:
+
+.. csv-table:: :param:`FLAGS_WIN` description
+  :header: "Value", "Meaning"
+  :widths: 3 60
+
+  1, ":ref:`Windowed second-order moments <moments_win_def>` are inconsistent (:math:`\overline{x_{\tt WIN}^2}\overline{y_{\tt WIN}^2} \le \overline{x_{\tt WIN}y_{\tt WIN}}^2`)"
+  2, ":ref:`Windowed second-order moments <moments_win_def>` are less than or equal to 0"
+  4, "Windowed flux is less than or equal to 0"
 
 .. [#win_accuracy] See http://www.astromatic.net/forum/showthread.php?tid=581 .
 

doc/src/Weighting.rst

@@ -20,7 +20,7 @@ Fortunately, the correlation length of the noise is often smaller than the patte
 In that case one can apply a simple *fudge factor* to the estimated variance to account for correlations on small scales.
 This proves to be a good approximation in general, although it certainly leads to underestimations for the smallest patterns.
 
-.. _weigth-map-format:
+.. _weight-map_format:
 
 Weight-map formats
 ------------------
@@ -66,7 +66,7 @@ Weight-maps modify the working of |SExtractor| in the following respects:
 #. Bad pixels are discarded from the background statistics.
    If more than :math:`50\%` of the pixels in a background mesh are bad, the local background value and the background standard deviation are replaced by interpolation of the nearest valid meshes.
 
-#. The detection threshold *t* above the local sky background is adjusted for each pixel *i* with variance :math:`\sigma^2_i` : :math:`t_i=DETECT\_THRESH\times\sqrt{\sigma^2_i}`, where ``DETECT_THRESH`` is expressed in units of standard deviations of the background noise.
+#. The detection threshold *t* above the local sky background is adjusted for each pixel *i* with variance :math:`\sigma^2_i`: :math:`t_i=`\ ``DETECT_THRESH``\ :math:`\times\sqrt{\sigma^2_i}`, where ``DETECT_THRESH`` is expressed in units of standard deviations of the background noise.
    Pixels with variance above the threshold set with the ``WEIGHT_THRESH`` parameter are therefore simply not detected.
    This may result in splitting objects crossed by a group of bad pixels.
    Interpolation (see ???) should be used to avoid this problem.
@@ -88,7 +88,7 @@ Weight-maps modify the working of |SExtractor| in the following respects:
 Combining weight maps
 ---------------------
 
-All the weighting options listed in :ref:`weigth-map-format` can be applied separately to detection and measurement images (:ref:`using-sextractor`), even if some combinations may not always make sense.
+All the weighting options listed in :ref:`weight-map_format` can be applied separately to detection and measurement images (:ref:`using-sextractor`), even if some combinations may not always make sense.
 For instance, the following set of configuration lines:
 
 ``WEIGHT_IMAGE`` *rms.fits*, *weight.fits*
@@ -96,7 +96,7 @@ For instance, the following set of configuration lines:
 ``WEIGHT_TYPE`` ``MAP_RMS``, ``MAP_WEIGHT``
 
 will load the FITS file *rms.fits* and use it as an RMS map for adjusting the detection threshold and cleaning, while the *weight.fits* weight map will only be used for scaling the error estimates on measurements.
-This can be done in single- as well as in dual-image mode (:ref:`using-sextractor`).
+This can be done in single- as well as in dual-image mode (see :ref:`using-sextractor`).
 ``WEIGHT_IMAGE`` can be ignored for ``BACKGROUND`` ``WEIGHT_TYPE``.
 It is of course possible to use weight-maps for detection or for measurement only.
 The following configuration:
@@ -107,6 +107,22 @@ The following configuration:
 
 will apply weighting only for measurements; detection and cleaning operations will remain unaffected.
 
+.. _flags_weight_def:
+
+Weight-map flags: :param:`FLAGS_WEIGHT`
+---------------------------------------
+
+The :param:`FLAGS_WEIGHT` catalog parameter flags various issues which may happen during the weighting process (see the :ref:`flagging` section for details on how flags are managed in |SExtractor|):
+
+.. _flags_weight_table:
+
+.. csv-table:: :param:`FLAGS_WEIGHT` description
+  :header: "Value", "Meaning"
+  :widths: 3 60
+
+  1, "at least one measurement-image weight with a value below ``WEIGHT_THRESH`` *overlaps* the isophotal footprint of the detected object"
+  2, "at least one measurement-image weight with a value below ``WEIGHT_THRESH`` *touches* the isophotal footprint of the detected object"
+
 .. rubric:: Footnotes
 
 .. [#f1] The mesh-filtering procedures act on the variance map, too.

doc/src/index.rst

@@ -21,6 +21,7 @@ Contents
    Config
    Param
    Weighting
+   Flagging
    Measurements
    references
 

doc/src/keys.rst

@@ -85,6 +85,9 @@
 .. |WCSLIB| replace:: :program:`WCSLIB`
 .. _WCSLIB: http://www.atnf.csiro.au/people/mcalabre/WCS/wcslib
 
+.. |WeightWatcher| replace:: :program:`WeightWatcher`
+.. _WeightWatcher: http://astromatic.net/software/weightwatcher
+
 .. |Windows| replace:: Microsoft Windows\ :sup:`®`\ 
 .. _Windows: http://www.microsoft.com/windows
 

doc/src/references.bib

@@ -334,6 +334,20 @@ booktitle = {Astronomical Data Analysis Software and Systems (ADASS) XIII},
   adsnote = {Provided by the SAO/NASA Astrophysics Data System}
 }
 
+@INPROCEEDINGS{2008ASPC_394_619M,
+   author = {{Marmo}, C. and {Bertin}, E.},
+    title = "{MissFITS and WeightWatcher: Two Optimised Tools for Managing FITS Data.}",
+booktitle = {Astronomical Data Analysis Software and Systems XVII},
+     year = 2008,
+   series = {Astronomical Society of the Pacific Conference Series},
+   volume = 394,
+   editor = {{Argyle}, R.~W. and {Bunclark}, P.~S. and {Lewis}, J.~R.},
+    month = aug,
+    pages = {619},
+   adsurl = {http://adsabs.harvard.edu/abs/2008ASPC..394..619M},
+  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
+}
+
 @INPROCEEDINGS{2012SPIE_8451E_0DM,
    author = {{Mohr}, J.~J. and {Armstrong}, R. and {Bertin}, E. and {Daues}, G. and {Desai}, S. and {Gower}, M. and {Gruendl}, R. and {Hanlon}, W. and {Kuropatkin}, N. and {Lin}, H. and {Marriner}, J. and {Petravic}, D. and {Sevilla}, I. and {Swanson}, M. and {Tomashek}, T. and {Tucker}, D. and {Yanny}, B.},
     title = "{The Dark Energy Survey data processing and calibration system}",