3.5 Quality assessment and validation

Author(s): Lennart Lindegren

Assessing the quality of the astrometric data is challenging, given the scarcity of independent data sets that have sufficient quality for a meaningful comparison. The quality assessment and validation must therefore to a large extent rely on various internal checks on the integrity and consistency of the data, and of the adopted models, algorithms, and software. Quality assurance for algorithms and software make use of simulations, test cases, and standard software engineering methods not further discussed here.

Broadly speaking, the quality assessment and validation methods applicable to the astrometric processing can be divided into the following categories.

1. 1.

   Basic data checks: These include checks on the amount of input and output data (e.g., what percentage of the elementary observations are actually used, and what fraction of the time do they cover) and range checks on all output quantities.

2. 2.

   Internal consistency: The astrometric processing is essentially a weighted least-squares solution and statistical tests of the residuals (including graphical output such as histograms, sky maps, and scatter plots) can be powerful method to monitor the quality and progress of the data processing and detect potential problems. Ideally, the (weighted) residuals should be unbiased, uncorrelated, Gaussian, independent of other variables, and consistent with calculated uncertainties.

3. 3.

   Cross-validation: The high redundancy of elementary observations per source makes it possible to partition the data into sets that are to a large extent statistically independent. Processing the complementary data sets separately, and comparing the astrometric results, may give a very good indication of the data quality.

4. 4.

   Model robustness and diagnostic parameters: While the astrometric model of the sources is unique and immutable, other parts of the modelling involving the nuisance parameters (for the instrument and attitude models) are to some extent arbitrary and the astrometric results should be insensitive to trivial changes e.g. of the attitude knot sequence or break times for the calibration model. Additionally, the astrometric solution may include diagnostic parameters that are expected to be zero if the basic modelling is correct. Examples are colour- and magnitude-dependent centroid displacements. Non-zero values of the diagnostic parameters indicate that the models are inadequate or insufficiently calibrated.

5. 5.

   Comparison with independent measurements: The astrometric parameters are compared with independent data sets, e.g. from other space missions (Hipparcos) or techniques (VLBI observations). Even if a strict validation of the Gaia results ideally requires external data of similar or better quality, it is in some cases possible to obtain a statistically significant assessment also when the comparison data are less precise.

6. 6.

   Astrophysical validation: This method relies on astrophysical models of certain kinds of objects. Examples are the parallaxes and proper motions of quasars, parallaxes of distant standard candles (Cepheids, RR Lyrae variables, Miras, etc.), the internal kinematics of stellar clusters, and statistical tests based on the assumed non-negativity of true parallaxes.

The following sections describe the outcome of tests made as part of the validation activities at CU3 level. In many cases the tests use data that are not part of the released data, including alternative solutions and data sets to which the final filtering was not yet applied. A number of the tests are reported in the Gaia DR1 astrometry paper (Lindegren et al. 2016), to which the reader is referred for additional details.

Author(s): Lennart Lindegren

The astrometric data in Gaia DR1 consist of two distinct data sets, with very different properties:

1. 1.

   The primary data set, or TGAS solution, comprises 2 057 050 sources for which all five astrometric parameters (position at the reference epoch J2015.0, parallaxes, and proper motion components) are provided, along with their standard uncertainties, correlation coefficients, and other statistics. This data set is the result of the TGAS solution using positions around epoch J1991.25 from Tycho-2 or, when available, from the Hipparcos Catalogue (van Leeuwen 2007) in order to improve the solution and in particular the proper motions. These sources are typically brighter than $G\simeq 11.5$. Basic statistics are given in Table 3.4 and Table 3.5.

2. 2.

   The secondary data set comprises 1 140 622 719 sources down to the Gaia limiting magnitude ($G\simeq 20.7$) for which only approximate positions at J2015.0 are given. This data set was obtained in the secondary solution (Section 4.3) using a mild prior on the parallaxes and proper motions in order to constrain these to reasonable values. Basic statistics are given in Table 3.6.

Additionally, all sources have $G$ magnitudes and associated statistics.

Author(s): Lennart Lindegren

Author(s): David Hobbs

Attitude telemetry from Gaia arrives daily over the mission lifetime. The main processing of Gaia data proceeds over six month cycles. In each cycle new data in blocks of roughly one day must be added to the previous data. At first sight this may look straightforward, however, complications arise as there will be overlaps and gaps between existing blocks of attitude data. Additionally, care may have to be taken when attitude data with different accuracies is combined.

The attitude processing consists of a number of different attitudes with increasing degrees of accuracy as discussed in Section 2.4.5.

• •

  The commanded (requested) attitude.

• •

  The on-board star tracker measured raw attitude contained in telemetry.

• •

  IOGA – The Initial on-ground attitude which is a spline fit to the available raw attitude.

• •

  OGA1 – A Kalman filter designed to smooth the attitude (IOGA) and to improve its accuracy to the order of 50 milli-arcseconds (mas) (see Section 2.4.5).

• •

  OGA2 – A First Look (FL) process does a direct astrometric solution on a single days worth of data resulting in B-splines and quaternions, ODAS (see Section 2.4.5).

• •

  OGA3 – AGIS is the final step in the attitude improvement where all the available observations for primary stars are used together with the available attitude and calibration parameters to iteratively arrive at the final attitude solution with a targeted accuracy of $\sim$10 of $\mu$as (see Section 3.3.4).

Generally, OGA1 and OGA2 data arrive in batches of 1 day but may contain data gaps and may or may not overlap with existing data. Initially, it was proposed to use OGA1 (supplemented by IOGA or OGA2 if OGA1 was not available) as the starting values for the AGIS attitude improvement. However, after some experimentation and due to the patched nature of OGA1-IOGA-OGA2 it was decided to start AGIS from the crude commanded attitude. The convergence of AGIS proved to be very good provided data gaps (see Section 3.3.4) were handled correctly and only a small number of iterations were needed to improve the solution to be better than OGA1.

This also made the verification of the AGIS attitude very simple by comparing the AGIS attitude with commanded attitude in angles and rates about the SRS axes. Checking for range out of bounds (extreme values) and checking for attitude spikes (manual inspection). RSE values of the differences between the starting values and the current values are plotted in the AGIS monitor and the updates in the attitude are also plotted. Sampling of about 1/5 of a knot interval is needed. Additionally, manual checks of the attitude around dead times and manual checks on extra knots could be performed.

Author(s): Alex Bombrun

The geometric calibration model is described in Section 3.3.5. The first verification consists in the analysis of the fitted values for the geometric calibration model. In particular one can compute the basic angle offset contained in this model (See Equation 19 in Lindegren et al. 2012). The BA changes are consistent with the payload activities described in Section 1.3.3. The four major payload events (the two de-contaminations and the two refocusing, Table 1.3) correspond to the 4 major jump of the basic angle offset visible on (See Figure (A.1) in Lindegren et al. 2016).

Basic angle jumps have not been included in the basic angle correction based on harmonic analysis of BAM data. However, 44 calibration break points have been set at the time of largest basic angle jumps identified in BAM data, see Section 3.2.5. Consequently the amplitude of these jumps was fitted in the geometric calibration model. (See zoom of Figure (A.1) in Lindegren et al. 2016) and Figure 3.10, both show that there is good agreement between the fitted amplitudes (AGIS) and the measured ones (BAM).

The AGIS residuals are analysed to investigate the limitation of the current model. In particular there are large correlations within the residuals due to some colour dependencies (See Figure (D.2) in Lindegren et al. 2016). A validation run with colour-magnitude effects added to the nominal calibration model was computed (See Appendix (E.1) in Lindegren et al. 2016, for a detailed analysis). Figure 3.11 and Figure 3.12 show the evolution of the colour calibration in the validation run over the focal plane. The effect is larger in AF1 than in AF9 and in row 7 than in row 1. Additionally the effect is time dependent, none the less, it seems to be stabilised after the last decontamination.

Author(s): Lennart Lindegren

See Section 3.5.3 and Section 3.5.3.

Author(s): Lennart Lindegren

Correlations among the astrometric data exist on several different levels, e.g.:

• •

  within-source correlations (between the different astrometric parameters of the same source);

• •

  between-source correlations (between the same astrometric parameter, e.g. parallax, for different sources);

• •

  general correlations (between arbitrary astrometric parameters of different sources).

Within-source correlations are always provided, when relevant, in the Gaia Archive. They are estimated from the $5\times 5$ normal matrix of the individual sources. Because they are computed by neglecting the between-source correlations, they are only approximations of the actual within-source correlations, but probably sufficiently good for all astrophysical applications. They are principally needed when transforming the five astrometric parameters to other representations (e.g. calculation of galactic coordinates, tangential velocity, or epoch transformation).

Between-source correlations are important when calculating quantities that depend on several sources, such as the mean parallax or internal kinematics of a stellar cluster. In the final Gaia data such correlations are expected to be important mainly on small angular scales (less than a few degrees), but in Gaia DR1 they could exist on much larger angular scales (tens of degrees).

Between-source correlations and general correlations are much harder to estimate than the within-source correlations, principally because their rigorous calculation would involve the inversion of extremely large matrices. Approximate methods to estimate general correlations exist (e.g. Holl and Lindegren 2012; Holl et al. 2012) but have not been implemented in the Gaia data processing. Empirically, the between-source correlations can be estimated by analysing the spatial correlations of the astrometric residuals, or from statistical analysis of the parallaxes and proper motions for distinct groups of sources, such as in stellar clusters and quasars. For Gaia DR1 the between-source correlations have not been extensively studied but they are believed to be very significant and arising mainly from modelling errors in the attitude or instrument. A limited correlation study of the astrometric residuals in Gaia DR1 is reported in Appendix D.3 of Lindegren et al. (2016).

Author(s): Lennart Lindegren

See Section 3.5.3.

Author(s): Uli Bastian

The results of the astrometric solution were compared with ground-based astrometric data. As expected, this essentially showed the precision and the systematic errors of the ground-based data, rather than contributing to the validation of the Gaia astrometry. The sole exception are the VLBI positions of quasars, which both gave a confirmation of the quality of the Gaia solution, as well as a first mas-level information on the typical offset between the optical and the radio centres of light for quasars.

Results from the comparisons are given in the Gaia DR1 astrometry paper, Lindegren et al. (2016) and in the Gaia DR1 validation paper Arenou et al. (2017). See also 3.5.3.

Author(s): Uli Bastian

In addition to the above-mentioned direct comparison with ground-based astrometry, other prior consensus knowledge on distances and motions of stars was used to verify/validate the Gaia DR1 astrometry. Among the source classes and criteria used for this purpose are:

• •

  LMC & SMC members as a (regional) check on the parallax zeropoint

• •

  quasars as a (global) check on the parallax zeropoint

• •

  distant Cepheids and RR Lyrae stars as a (global) check on the parallax zeropoint

• •

  quasar positions as check against the ICRF (as mentioned above already)

• •

  quasars as a check on proper-motion precision at the faint end

• •

  stellar aggregates (LMC, SMC, star clusters) as a check on proper-motion precision at the bright end

• •

  stellar aggregates as check on colour- and magnitude-dependent biases

• •

  negative-parallaxes tail as check on parallax precision

Results from the checks are given in the Gaia DR1 astrometry paper, Lindegren et al. (2016) and in the Gaia DR1 validation paper Arenou et al. (2017). See also Section 3.5.3.

Author(s): Lennart Lindegren

The various quality assessment and validation procedures give confidence in the overall correctness of TGAS and the secondary solution, but also indicate many weaknesses and problems that have to be addressed and removed in subsequent releases. Some of the important caveats concern the completeness of the data (e.g. missing bright stars, high proper motions stars, and faint stars in high-density regions) and the presence of systematic errors. The TGAS parallaxes, for example, are expected to have position- and colour-dependent systematics on the level of $\pm 0.3$ mas. These systematics are strongly correlated for stars within areas of several degrees, which means that they cannot be reduced by averaging e.g. in a cluster. It is probable that the position and proper motion errors have similar systematics, although this is more difficult to determine. For individual sources the stated standard uncertainties of the astrometric parameters appear to be fairly reliable in the primary data set (TGAS), while the positional uncertainties in the secondary data set appear to be underestimated.

A more independent catalogue consolidation and validation of the science results for Gaia DR1 was also performed and are documented in Chapter 7 and will be published in Arenou et al. (2017).