Featured Product
This Week in Quality Digest Live
CMSC Features
Atul Minocha
It’s all about ROI
Ryan E. Day
September Speaker Series in review
Ryan E. Day
Realigning a cornerstone of industry
David H. Parker
Practical implications for electronic distance measurement
Belinda Jones
Users of CMMs, industrial scanners, and portable metrology systems are urged to participate

More Features

Voting for 2022–2023 term will open at CMSC 2022 starting July 25
New facility in Toronto area will showcase multiple Hexagon product lines
API division named ‘Top External Provider 2018’
Exact Metrology selected for project
Faster and more powerful than ever before
Accurate measurement out of the box
Engineering and design teachers will benefit from enhanced 3D scanning performance
Partnering with FARO Technologies

More News

Joseph Hayden, Manal Khreishi, Theodore Hadjimichael, and Raymond Ohl

Joseph Hayden, Manal Khreishi, Theodore Hadjimichael, and Raymond Ohl’s default image


Monte Carlo Method for Uncertainty Propagation in JWST Metrology Databases

Published: Thursday, May 21, 2015 - 15:41

Large-volume metrology measurements are conducted at every major phase of alignment and integration of the Integrated Science Instrument Module (ISIM), which forms the main payload of the James Webb Space Telescope (JWST) at NASA Goddard Space Flight Center (GSFC). Both ambient and cryogenic metrology measurements are used to verify that ISIM and ground support equipment (GSE) conform to the predicted position and orientation at various integration and alignment states.

The fusing of multiple measurements and their uncertainties from a network of different metrology instruments is done using SpatialAnalyzer software. The resulting measurements and their uncertainties constitute a metrology calibration database for a particular test configuration. A custom Monte Carlo (MC) uncertainty propagation routine, written in MATLAB, was developed as a means to robustly propagate, analyze, and account for the uncertainty associated with spatially transforming targets and unit vectors between databases with common targets. The MC error propagation routine generates N-different geometric best-fit transformations between two separate databases with at least three corresponding targets. Each of the geometric transformations involves simulating target measurements from the sum of the nominal targets and their randomly drawn uncertainties. The randomly drawn uncertainties are defined by Student's t-probability density functions (PDF) for each of the targets in the databases. The 2 sigma standard deviation is calculated for all targets in the N-simulated MC transformations. The MC error propagation method has been shown to be beneficial for propagating uncertainty through multiple JWST databases and for estimating transformation uncertainty for line-of-sight modeling scenarios where one or more targets may be omitted from a best-fit transformation.


A metrology calibration database is created for each new test configuration of ISIM for both ambient and cryogenic environments. Each configuration of ISIM requires new metrology measurements and a corresponding metrology database. The database is comprised of a list of target names, target types (points or unit vectors), number of measurement samples, average XYZ values, and corresponding Student's t 95-percent confidence interval (CI) uncertainties. The initial database provides a foundation for the reference coordinate system to which all subsequent databases may be related.

Best-fit transformations are routinely used in metrology to adjust the position and orientation of points, objects, coordinate systems, and measurement instruments. A best-fit spatial transformation, however, transforms nominal targets without regard to the uncertainty in those targets. In principle, it is possible to apply hundreds of transformations back-and forth between common targets in two different databases with only negligible round-off error when using double precision operations. In practice, however, there is an accuracy penalty associated with applying a transformation to common measured targets that is dependent on the uncertainty in those targets. The Monte Carlo Transformation Uncertainty (MCTU) code was developed as a means to robustly propagate, analyze, and account for the uncertainty associated with spatially transforming point targets and unit vectors originating from a variety of different metrology sources.

The MCTU is primarily used as a supplement to best-fit transformations and is not a substitute for a network bundle adjustment. The simultaneous registration of all instruments and their measurements is most accurately addressed using the network bundle adjustment capability1-5 within most commercial metrology software, such as the Universal Spatial Metrology Network (USMN)6,7 in SpatialAnalyzer (SA) by New River Kinematics. The bundle adjustment is able to optimize the network using the instrument uncertainty and unique point-cloud uncertainty fields associated with each target. Uncertainty fields of measurements exported from SA can only be approximated by their constituent axial 95-percent CI Student's t-uncertainties. This approximation will be accurate to the extent that the composite uncertainty field within SA is ellipsoidal with major and minor axes aligned to the coordinate system. Most target uncertainty fields for JWST metrology measurements are generally spherical as a result of the symmetric placement of multiple laser trackers and laser radar stations.

Often, however, metrology measurements and their uncertainty statements originate from multiple independent measurement sources. Measurements may have been provided by a vendor or a calibration facility, or they may have originated from independent measurement instruments such as theodolites, coordinate measuring machines (CMM), or photogrammetry (PG) cameras. The MCTU method provides a relatively fast and structured approach to propagating uncertainty in databases with common targets that reside outside of a metrology software environment.

Metrology databases

A metrology database is comprised of average point targets and average unit vectors along with their corresponding uncertainty statements for a specific test configuration. A test configuration may constitute different test environments, different load conditions, different orientations, or could be identical in all respects to a previous test configuration but conducted at a later date. The database is an official record of the metrology survey for a particular test configuration and is usually the final step in the metrology analysis process.

A typical metrology survey consists of multiple laser tracker stations, laser radar stations, and theodolites. The USMN capability in SA software is used to bundle measurements from all metrology instruments. At the completion of a USMN, all instruments and their measurements are transformed to a reference coordinate system. All measurements in the network, along with all associated planes and unit vectors, are then exported from SA to an ASCII file. This file is read into MATLAB and the average and 95-percent Student's t-confidence intervals8 are calculated for all common point targets and unit vectors. Outlier targets deactivated during the USMN process are ignored when calculating the average and Student's t-uncertainties in MATLAB. Finally, each of the target names, target types, number of samples, position or direction, and uncertainties are exported to the Excel formatted spreadsheet.

Database targets and uncertainties for this work use a "P", "Q", "Pt", and "QPt" nomenclature, where P-targets are always best-fit transformed to the Q targets in the reference coordinate system to determine the Pt targets and their uncertainties. The term "involved target" refers to a point target that is used directly in the MC best-fit transformations. The term "uninvolved target" refers to targets that are not used in the best-fit transformation, either because there are no common targets between P and Q, or because the user has intentionally uninvolved that target. Unit vectors are always treated as uninvolved targets and are only rotated by the best-fit rotation matrix. Sample P-, Q-, and Pt-database files containing both points and direction cosines are shown in table 1, with a corresponding graphical representation in figure 1. At the completion of the MC transformations the resulting database can be saved using either Q priority, Pt priority, or lowest uncertainty priority.

The Q-priority database method gives Q targets and their uncertainties precedence in the final QPt database over the corresponding transformed Pt targets and their uncertainties. If a Pt target does not have a corresponding Q target then that Pt target is used in the database. The Pt priority gives Pt targets precedence in the final QPt database over corresponding Q targets. The lowest uncertainty priority database method gives the Q or Pt target with the lower uncertainty magnitude precedence in the final QPt database.
The Q priority is commonly used to tie-in a more comprehensive set of P targets to more recently measured Q targets, but does not permit Pt targets to overwrite corresponding Q targets. The Pt priority is used to transform more recently measured P targets back to a reference coordinate system established by the Q targets and does permit overwriting of Q targets by corresponding Pt targets. The minimum uncertainty priority is used to tie-in P targets to more recently measured Q targets and permits overwriting of Q targets where corresponding Pt target uncertainties are lower.

Table 1. Sample metrology database files for (a) P targets, (b) Q targets, and (c) Pt targets, with average and Student's t-95 percent confidence interval uncertainties. The resulting database with nominal values and Monte Carlo uncertainties is formed from the Q and Pt targets using Q priority, Pt priority, or lowest uncertainty priority database export methods. Click here for larger image.


Figure 1. Graphical display of P targets, Q targets, and Pt targets (top), and three types of database export types: Q priority, Pt priority, and minimum uncertainty priority (bottom). Click here for larger image.

Monte Carlo uncertainty propagation

Monte Carlo simulations are routinely used in the aerospace industry to explore the sensitivity in the output of complex systems by iteratively varying the input parameters within statistical limits. These complex systems can be mathematical or physical in nature and usually contain multiple coupled degrees of freedom where closed-form solutions are difficult or impossible to obtain.9,10 A Monte Carlo approach is well suited to characterizing the sensitivity of transformed 3D points because of the nonlinearity associated with a composite rotation matrix.

The MCTU software, written in MATLAB, was developed as a means of propagating uncertainty when transforming one database to another using common targets and their uncertainties. The following least-squares problem

is used to determine the rotation matrix Ri and the translation vector di that maps Pi + ∆Pi targets to the corresponding Qi + ∆Qi targets for each of N-Monte Carlo transformation iterations. The best-fit transformation routine used for the nominal and Monte Carlo transformations employs the Singular Value Decomposition (SVD) optimization method to solve the Orthogonal Procrustes problem.11-14 The SVD method is used because it offers a mature, fast, closed-form solution to least-squares fitting to solve for absolute orientation.

The first step in the MCTU software flow, as seen in figure 2, requires the user to select a P and a Q database file. The software then identifies all point targets common to the two files that could be used in the MC transformations. An initial best-fit transformation is done between the nominal P and Q targets to calculate the residuals to identify common target names that will be involved in the MC transformations. The user has the option of uninvolving common outlier targets identified in the residuals prior to running the MC transformations. The user specifies the number of MC transformations to be used, where N = 1,000 has been shown to produce stable results in the uncertainty of the transformed targets. Then N-random 95-percent confidence interval (CI) uncertainties are calculated for each target using the MATLAB random number generator that draws from a Student's t-PDF, scaled by the X, Y, and Z 95-percent CI uncertainties specified in the database for each target. The Student's t-distribution is a function of the degrees of freedom (DOF) and probability, where the DOF for each target are also obtained from the database. A Student's t-distribution is used, rather than a normal distribution, since the use of random numbers from a normal PDF results in an underestimation of the 95-percent CI when limited samples are involved. The Student's t-uncertainty is calculated as t σ/√N - 1, where t is the Student's t "t-value", σ is the sample standard deviation, and N is the number of samples. The t-value is calculated from the tinv function in MATLAB as tinv(P, v, ), where P=1-.05/2 for the 95-percent CI of the 2-tailed T-distribution, and the degrees of freedom, v, are defined as v = N – 1. The sample standard deviation, σ, is defined as

where xi is the i-th sample and is the average of the samples.

At the completion of N-Monte Carlo transformations, the average 6 DOF rigid body transformation and 2s standard deviation of all Pt targets are calculated. The standard deviation may be used rather than Student's t-95-percent CI since the two distributions converge for large numbers of MC simulations.

Typical run times for the MCTU code range from 15 seconds for 10 targets to 2 minutes for several hundred targets on a 64-bit Windows 7 computer, using 1,000 MC transformations. The nominal transformed targets and average of all MC transformations typically agree to within 1 μm, so values from the nominal transformation are usually used in the database for consistency in transformed results.


Figure 2. Flowchart for MCTU propagation method. Click here for larger image.


The custom MCTU software written in MATLAB has been used extensively during integration and testing of ISIM to estimate and propagate the uncertainty in metrology databases. Multiple ambient and cryogenic databases have been constructed from metrology surveys on ground support equipment such as the BALL Image Analyzer (BIA) sensor head, alignment fixtures such as the Master Alignment Test Fixture (MATF) and Integrated Alignment Test Fixture (IATF), and flight hardware such as Integrated Science Instrument Module (ISIM).15,16 The MCTU software offers a robust approach to characterizing and propagating the uncertainty in transformed metrology database targets based on the uncertainty in a rigid body constellation of targets involved in the transformation.

It is important to note that although the MCTU method offers a more comprehensive approach to propagating uncertainty in metrology databases than simply root-sum-squaring error terms, it is not as comprehensive an approach to uncertainty propagation as a full network bundle adjustment. A network bundle adjustment, such as the USMN in SA, has access to composite uncertainty fields for every measurement. When possible, it is preferable to apply transformations within the framework of a network bundle adjustment, because the exact point-cloud information will be considered in the adjustment.


Thanks to Steve Seiler and Jeremy Winn from New River Kinematics for helpful comments on USMN point-cloud uncertainties.

1 Zhang, X. F., "Fusion Estimation of Point Sets from Multiple Stations of Spherical Coordinate Instruments Utilizing Uncertainty," Measurement Science Review, Vol. 12, No. 2, pp. 40–45.
2 Weckenmann, A., Jiang, X., Sommer, K.-D., Neuschaefer-Rube, U., Seewig, J., and Shaw, L., "Multisensor Data Fusion in Dimensional Metrology," CIRP Annals—Manufacturing Technology, No, 58, pp. 701–721.
3 Forbes, A. B., Hughes, B., and Sun, W., "Comparison of Measurements in Coordinate Metrology," Measurement, Vol. 42, No. 10, pp. 1473–1477, 2009.
4 Mitchell, J., Spence, A., Hoang, M., et al, "Sensor Fusion of Laser Trackers for Use in Large-Scale Precision Metrology", Proceedings of SPIE, Vol. 5263, pp. 57–65, 2004.
5 Predmore, C., "Bundle Adjustment of Multi-Position Measurements Using the Mahalanobis Distance," Precision Engineering, Vol. 34, No. 1, pp. 113–123.
6 Calkins, J., "Quantifying Coordinate Uncertainty Fields in Coupled Spatial Measurement Systems," Ph.D. Thesis, pp. 1–226, 2002.
7 New River Kinematics, Inc. Spatial Analyzer (SA) User Manual, pp. 580-594, V2013.12.09.
8 Bevington, P. R. and Robinson, D. K., Data Reduction and Error Analysis for the Physical Sciences, McGraw-Hill Science/Engineering/Math, Third Edition, 2002.
9 Kosarevsky, S. V. and Latypov, V. N., "Practical Procedure for Position Tolerance Uncertainty Determination via Monte Carlo Error Propagation," Measurement Science Review, Vol. 12, No. 1, 2012.
10 Brujic, D. and Ristic, M., "Monte Carlo Simulation and Analysis of Free-Form Surface Registration," Proceedings of the Institution of Mechanical Engineers, Vol. 211, No. 8, pp. 605–617, 1997.
11 Soderkvist, I., "Perturbation Analysis of the Orthogonal Procrustes Problem," BIT, Vol. 33, No. 4, pp. 687–694, 1993.
12 Arun, K. S., Huang, T. S., and Blostein, S.D., "Least-Squares Fitting of Two 3D Point Sets," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 9, pp. 698–700, 1987.
13 Gander, W. and Hrebicek, J., Solving Problems in Scientific Computing using MATLAB and Maple, Springer Verlag, 1993.
14 Hanson, R. J. and Norris, M. J., "Analysis of Measurements Based on the Singular Value Decomposition", SIAM Journal on Scientific and Statistical Computing, Vol. 2, pp. 363–373, 1981.
15 Hadjimichael, T., et al, "Cryogenic Metrology for the James Webb Space Telescope Integrated Science Instrument Module Alignment Target Fixtures Using Laser Radar Through a Chamber Window", Proceedings of SPIE, Vol. 7793, 2010.
16 Ohl, R., et al, "Metrology Trades for Ambient Testing of the James Webb Space Telescope," The Journal of CMSC, Autumn 2009.


About The Author

Joseph Hayden, Manal Khreishi, Theodore Hadjimichael, and Raymond Ohl’s default image

Joseph Hayden, Manal Khreishi, Theodore Hadjimichael, and Raymond Ohl

Joseph Hayden is the senior optical engineer at Sigma Space Corp.

Manal Khreishi, Theodore Hadjimichael, and Raymond Ohl are on the optics metrology team at NASA Goddard Space Flight Center