Measurement uncertainty is one of the root causes of waste due to variation in industrial manufacturing. This article establishes the impact of certain factors on measurement uncertainty while using coordinate measurement machines as well as its reduction through the usage of recognized Gage R&R methodology to include ANOVA. Measurement uncertainty stemming from equipment and appraiser variation is identified and ranked according to its degree of impact. A comparative analysis is conducted showing how different CMMs of similar design can generate differing amounts of measurement uncertainty. The approach set forth in this paper not only proves effective with CMMs but can also be applied to other complex multivariate measurement systems.
Keywords: coordinate measuring machine, gage repeatability & reproducibility, equipment variation, appraiser variation, measurement system analysis, measurement uncertainty.
Measurement error is one of the root causes of variation, or waste, in any manufacturing process. As such, all measurement errors must be properly identi ed and understood in order to determine the quality of a manufacturing process. This task is accomplished by performing a Measurement System Analysis (MSA) on each measurement system in a given manufacturing process. Perhaps the most well known type of MSA performed is a Gage Repeatability & Reproducibility (R&R) Study.
Gage Repeatability is primarily the variation observed in the measurement gage itself and is often considered to be the equipment variation. Gage Reproducibility is variation introduced into the measurement system when a variable is changed, that is, a different operator using the same gage. The repeatability and reproducibility combined are what determines the total “measurement error,” or “noise” in a given measurement system. This noise is the source of measurement uncertainty in the measurement data (Kappele, 2005).
Typically, a measurement system is considered capable if it has a low amount of uncertainty. Ideally, this is determined by less than 1% noise and a total Gage R&R of less than 10%. The total percentage of Gage R&R is a combination of both the measurement uncertainty as well as part-to-part variation. Systems having more than 10% noise or a combined Gage R&R of more than 30% are considered unacceptable, and every effort should be made to improve the measurement system (MSA Workgroup, 2010).
The three most recognized methods of Gage R&R as set forth in the Measurement Systems Analysis Reference Manual are the Range Method, the Average & Range Method, and the Analysis of Variance (ANOVA) Method. The Measurement Systems Analysis Reference Manual is a publication put together by the Automotive Industry Action Group (AIAG) and serves as a reference for Gage R&R methodology that has been sanctioned by the Chrysler Group LLC, Ford Motor Company, and General Motors Corporation Supplier Quality Requirements Task Force (MSA Workgroup, 2010).
Coordinate measuring machines are a common measurement device used to secure measurements of high accuracy across various industries. They range in type from table-top bridge-type machines with touch type probes to much more advanced technology using hand-held devices and optical laser type probes. Rather than technically measuring component parts in the traditional sense, CMMs obtain discrete points or hits on a three-dimensional Cartesian plane and generate measurement data through algorithmic mathematical computation.
A number of researchers have made significant progress toward developing methodologies aimed at estimating measurement uncertainty that results in coordinate measurements, particularly using contact-mode probes. Yet, considerable research remains to be performed to fully account for measurement uncertainty and to improve their estimation. For example, techniques to model and estimate task specific uncertainty for contact-probe coordinate measuring machines were developed by Wilhelm et al., (2001), who reported that for any task specific uncertainty method to gain universal acceptance, standardized inputs would be highly desirable, if not a requirement. In their investigation of measurement uncertainty estimation of CMMs in accordance with the Guide to the Expression of Uncertainty in Measurement (GUM), Fang and Sung (2005) noted that measurement uncertainties mainly come from the calibration of the CMM and temperature. For a measurement range of 0mm to 400 mm, they estimated an expanded uncertainty of 3.4 µm with a coverage factor of 1.98 at a 95% confidence interval. Their further analysis showed that the measurement uncertainty can be reduced by using a high precision instrument, such as laser interferometer.
The principal factors that impact measurement uncertainty have been studied extensively. Barini et al. (2010) investigated the source and effects of differing uncertainty contributors by point-by-point sampling of complex surface measurements using tactile CMMs. By carrying out a four-factor (machine, probe, operator, and procedure), two-level randomized factorial experiment and choosing adequate process parameter settings, a subsequent decrease of the measurement uncertainty from 34 µm to 8 µm was observed.
Other researchers have used other approaches. In their work, Phillips et al. (2010) utilized a computer simulation software approach to investigate the validation of CMM measurement uncertainty. All the measurement errors found in the physical measurements were well inside their corresponding uncertainty intervals. From their investigation, Phillips et al. suggested a well-documented list of reference value tests as a useful tool to employ before starting the more expensive aspect of real physical measurements of calibrated parts.
Data for this study was collected using two separate CMMs; the first is located in the CMM Laboratory of the School of Engineering and Information Systems, Morehead State University and the second from a CMM Lab located in a Tier 1 Original Equipment Manufacturer (OEM) facility. Both machines were similar in design, utilizing a bridge type table design with Direct Computer Control (DCC) capability.
The primary differences of the two machines were that they are manufactured by different companies and each functions with a different operating software. The machine used in the university laboratory was manufactured by Brown & Sharpe and is operated by PC•DMIS 2014 software; while the second machine was manufactured by the Zeiss company and operated by Calypso 4.8 software.
As many controls as possible were maintained during the study to ensure a quality comparative analysis between the two machines. For example, the same participants were used to collect measurement data on each of the CMMs. Additionally, all data were collected by measuring the same three sample parts with each machine. The DCC mode was utilized on each machine in lieu of a third operator to provide baseline data.
The overall methodology of this research is based on the American Automotive Industry standard requirement for Gauge R&R studies. One of the challenges faced by quality professionals who supply products to customers in the American Automotive Industry (AAI) is not only complying with an extensive list of customer specific requirements, but also complying with those requirements in the specific manner prescribed by the customer as well. An especially good example of this challenge is encountered when attempting to comply with customer requirements pertaining to MSAs and the documentation of the measurement variation present with each gage used to release product to the customer.
While the core tools reference manuals contain good practices and methods this is not the same as being “best” method across the board in every instance. As the name implies these manuals were originally created as “reference” guides, but over the years the AAI requirements have evolved to the point that these guides have changed from references to requirements for the entire automotive supply chain. This phenomenon in and of itself poses its own set of unique obstacles and challenges to those in the field of quality due to the fact that “not all measurements systems are created alike.” The result is a tendency to analyze a measurement system through the lens of the required method for the purpose of compliance to the requirement, rather than analyzing a measurement system with the intent of truly understanding the capability and uncertainty of the system itself.
The three separate dimensional measurement types selected are referred to in the CMM operator’s manual as ‘geometric features’. They were selected due to the way a CMM generates three-dimensional measurements differently than two-dimensional measurements. When three-dimensional objects are measured, probe radius compensation is made perpendicular to the surface of the object as opposed to the active work plane used in two-dimensional objects.
The decision between a manually operated CMM versus a DCC type is typically determined by the type of operation in a given organization. Most production-oriented environments choose a DCC type, while companies specializing in prototype development and reverse engineering are more likely to prefer a manual CMM (Meredith, 1999). However this does not mean to imply that a CMM with DCC capability is always being utilized in DCC mode. DCC CMMs can still be operated in manual mode.
The two different CMM setup types that are under investigation are those of manual setup and CAD setup. Both setup types are directly linked to a DCC operation type. The integration of advanced CAD inspection programs has provided yet another layer of part inspection versatility to the realm of metrology. Through the usage of CAD enhanced CMM software, it is now possible to graphically test and debug inspection routines before executing a new part program with the CMM (PC•DMIS, 2014).
With nearly all types of measurement system analysis, the operator(s) involved tend to contribute significantly towards the overall measurement variation in the system.
An Analysis of Variance (ANOVA) was performed to determine the significance of impact of the four selected factors on overall measurement uncertainty associated with a CMM. This test was conducted with a 95% confidence level. When a statistical significance was discovered while comparing a set of three or more means, a post hoc Tukey Test was performed to determine which means were significantly different.
A series of Gage R&R studies were performed using the equations and methodology set forth by the MSA Reference Manual. The results of the Gage R&R studies determine the percentages of variance that each of the four categorical factors contribute towards the overall measurement uncertainty in this particular study. These equations are as follows
Equation 1:
K_{1} = a compensation constant based on the number of trials used.
Equation 1 was used to calculate Repeatability / Equipment Variation (EV)
Equation 2:
K_{2} is a compensation constant based on the number of appraisers used.
n = parts r = trials
X_{diff} is the difference between the greatest and least X_{bar} of all trials and all parts for each appraiser
Equation 2 was used to calculate Reproducibility / Appraiser Variation (AV).
Equation 3:
Equation 3 was used to calculate GRR (combined Gage R & R).
Equation 4:
K3 is a compensation constant based on the number of parts used
Rp is the range of part averages
Equation 4 was used to calculate the PV (Part Variation).
Equation 5:
Equation {5} was used to calculate TV (Total Variation)
The following equations were used to calculate the percentages of AV, EV, & GRR.
(The percentage of PV is of no consequence in this research since the part to part variation is not under investigation.)
Equation 6:
Equation 7:
Equation 8:
A statistical comparative analysis was performed using data collected from two separate CMMs.
Data was collected at the university lab over the course of several days. The same participants were involved throughout the entire data collection process. The temperature in the lab was carefully monitored and measurements were only taken when the lab temperature was maintained within 20°C +/- 2°C in accordance to the universally accepted standards established by the NIST (Doiron, 2007).
The Gage R&R data were collected while operating the CMMs in different configurations. While collecting data using the CAD capability, the software was completely closed out and reopened in identical sequence for each iteration.
While measuring the conic sections, it was necessary to calculate angular measurements due to the limited capabilities of one of the CMMs.
Equation 9 was used to convert from linear into angular for the outer base angle of the cone.
Equation 9:
θ represents the outer base angle of the Cone.
Diameter_{1} & Diameter_{2} are the diameters of the upper and lower planes of the conic section.
Since all raw data were in the form of different units of measure it was first necessary to normalize data before performing an ANOVA. Data were normalized according to each distinct dimension type. Data normalization was performed using Equation 10.
Equation 10:
After analyzing the data, the following findings were discovered, and the four factors being analyzed were then ranked according to their impact on the overall MSA.
Residuals plots were analyzed in connection with each factor (Figures 2-4) each of which demonstrate no indication of bias in any of the ANOVAs. The histogram and normality plots show no skew or outliers, and the residuals appear normally distributed. The Residuals vs. Fits graphs all indicate that the residuals have a constant variance and the Residuals vs. Order plots show no apparent correlations between any of the data.
In addition to each ANOVA, a Tukey Pairwise Comparison of Means test, also known as Tukey’s Honestly Significant Difference test, was conducted when P values of the ANOVA were discovered below the risk value. A Tukey Test was used to reveal which means in a given data set differed significantly from the rest (Olleveant, 1999).
An analysis of the variation related to 3D measurement type when operating the CMM in manual mode revealed a notable amount of both EV and AV variation when compared to the variation encountered from the other factors. The cylinder consistently yielded the highest amount of both EV and AV variation. The cone ranked second with the sphere displaying the least amount of variation.
The first ANOVA was conducted on data relating to 3D measurement type. Because the P-value for the appraisers was determined to be below the alpha value of 0.05 (although only slightly at 0.048. (Table 1), it was determined that the difference in means between the appraisers in this particular study was statistically significant. All data was first analyzed and found to be normally distributed.
Figure 1 shows that the data collected by Appraisers 1 and 2 were significantly different from each other while measuring the different three dimensional shapes.
Table 1: ANOVA of Measurement Type
Source |
DF |
Seq SS |
Adj SS |
Adj MS |
F |
P |
Appraiser |
2 |
0.8262 |
0.8262 |
0.4131 |
3.12 |
0.048 |
Dimension Type |
2 |
0.3626 |
0..3626 |
0.1813 |
1.37 |
0.258 |
Error |
130 |
17.2324 |
17.2324 |
0.1326 |
||
Total |
134 |
18.4212 |
Figure 1. Tukey Test of Appraisers for Measurement Type
The second factor investigated was Machine Operation Type. The Gage R & R data reveals the presence of both EV and AV variation while in Manual Mode but not while in DCC mode.
This finding was also supported by the ANOVA which demonstrated statistical significance for the variation between the two modes of operation as shown in Table 2.
Figure 2. Four in One Residulas Plot for Measurement Type
Table 2: ANOVA of Operation Type
Source |
DF |
Seq SS |
Adj SS |
Adj MS |
F |
P |
Appraiser |
2 |
0.1006 |
0.1006 |
0.0503 |
0.38 |
0.682 |
Dimension Type |
1 |
1.4624 |
1.4624 |
1.4624 |
11.17 |
0.001 |
Error |
176 |
23.0415 |
23.0415 |
0.1309 |
||
Total |
179 |
24.6045 |
Table 3: ANOVA of Setup Type
Source |
DF |
Seq SS |
Adj SS |
Adj MS |
F |
P |
Appraiser |
2 |
0.0339 |
0.0339 |
0.0169 |
0.12 |
0.887 |
Dimension Type |
1 |
0.2576 |
0.2576 |
0.2576 |
1.83 |
0.18 |
Error |
86 |
12.1049 |
12.1049 |
0.1408 |
||
Total |
89 |
12.3964 |
The third factor, Set-up Type, proved to bethe least significant of each of the four factors analyzed. Very little if any variation was observed in both AV and EV for either set-up type. This produced a negligible GRR% overall. These results were supported by the ANOVA, as seen in Table 3. The data reveal no presence of measurement noise when utilizing CAD capable software in both DCC and manual modes.
Figure 3. Four in One Residulas Plot for Operation Type
Figure 4. Four in One Residulas Plot for Setup Type
The fourth factor investigated was variation attributable to machine operators. In order to determine the impact of this variation it was first necessary to examine the AV in connection with each of the other independent factors. As the data in Table 4 shows, Appraiser Variation is present in some configurations but not in all. As seen in Table 4, the two configurations with the least amount of appraiser variation were a) when measuring spherical objects, and b) when operating the CMM in DCC mode.
Table 4: Summary of Gage R&R Results
Factor |
EV µ |
% EV |
AV µ |
% AV |
GRR µ |
% GRR |
1a) 3D Measurement / Sphere |
0.015 |
0.21% |
0.002 |
0.03% |
0.015 |
0.21% |
1b) 3D Measurement / Cylinder |
0.195 |
1.11% |
0.131 |
0.75% |
0.235 |
1.34% |
1c) 3D Measurement / Cone |
0.058 |
0.50% |
0.064 |
0.55% |
0.086 |
0.74% |
2a) Operation Type / Manual |
0.105 |
0.83% |
0.067 |
0.53% |
0.124 |
0.99% |
2b) Operation Type / DCC |
0.000 |
0.00% |
0.004 |
0.04% |
0.004 |
0.04% |
3a) Setup Type / Manual |
0.006 |
0.21% |
0.006 |
0.22% |
0.009 |
0.31% |
3b) Setup Type / CAD |
0.005 |
0.16% |
0.004 |
0.14% |
0.006 |
0.21% |
Table 5: Ranking of Individual Factors
Factors by Ranking |
Combined Average of Individual Factors |
1) Measurement Type |
(0.15 + 0.195 + 0.058) ÷ 3 = 0.134 |
2) Operation Type |
(0.105 + 0.00) ÷ 2 = 0.0525 |
3) Operator |
(0.002 + 0.131 + 0.064 + 0.067 + 0.004 + 0.006 + 0.004) ÷ 7 = 0.040 |
4) Setup Type |
(0.006 + 0.005) ÷ 2 = 0.006 |
(Result 2 – Ranking the factors)
After extracting the EV for each factor using the series of Gage R&R methodology, the grand mean of each individual component of EV was calculated. This data was then sorted in Table 5 providing a ranking of the impact of each individual factor on the overall measurement uncertainty of the system.
Figures 5 and 6 are images taken of both individual CMMs used for the comparative analysis. The individual parts measured in the study can also be seen clearly on the table surface of the CMM in Figure 5.
Figure 5. Image of CMM Located in University Laboratory
Figure 6. Image of CMM Located in Tier 1 Industry Laboratory
For the purpose of conducting the comparative analysis, a few individual components were selected and then firmly secured to the table top surface of each CMM prior to securing measurements. The statistical data for comparative analysis are displayed in Tables 6 and 7 below.
Tables 6 and 7 clearly demonstrate differing amounts of EV and AV data are present when identical parts are measured by the same operators on different CMMs of similar design. Additionally, both the GRR results as well as the statistical summary indicate a greater presence of measurement variation when using the CMM located in the university lab.
Table 6: Case Study Gage R & R Summary
Different CMMs |
EV µ |
% EV |
AV µ |
% AV |
GRR µ |
% GRR |
1) Machine 1- MSU Laboratory |
0.150 |
0.51% |
0.096 |
0.33% |
0.178 |
0.61% |
2) Machine 2 - Industry Laboratory |
0.094 |
0.32% |
0.066 |
0.23% |
0.115 |
0.39% |
Table 7: Case Study Statistical Summary
Statistic |
Tier 1 OEM CMM |
University CMM |
1) Mean Standard Deviation |
0.104 µ |
0.360 µ |
2) Mean Range |
0.386 µ |
1.036 µ |
This study has revealed the presence of varying degrees of measurement uncertainty while operating CMMs in different configurations. The extent of impact of this uncertainty for CMMs must be determined by the associated user. It is noteworthy that this work revealed that differing amounts of equipment variation (EV) and appraiser variation (AV) are present when identical parts are measured by the same operators on different CMMs of similar design.
The impact of each of the four individual factors on the overall measurement uncertainty was revealed in rank order from highest to lowest as: Measurement Type, Operator Type, Operator, and Set-up Type. Furthermore, a Comparative Analysis between the industry CMM and university CMM, based on both the Gage RR and ANOVA results, indicated a greater presence of measurement variation when using the university CMM.
While the methodology set forth in this study may not necessarily encompass all cases, it has proven effective to adequately perform an MSA on a CMM. The same result could be expected for future studies of similar complex multivariate measurement devices.
Jayson Minix works as a Quality Engineer for the Parker Hannifin Corporation in the O-ring Division of the Seals Group.
Dr. Hans Chapman is an Associate Professor of Design and Manufacturing at the Department of Engineering and Technology Management, School of Engineering and Information Systems at Morehead State University (MSU), Morehead, Kentucky. He is a member of the Gamma Mu chapter of Epsilon Pi Tau.
Dr. Nilesh Joshi is an Associate Professor at the Department of Engineering and Technology Management, School of Engineering and Information Systems at Morehead State University, Morehead, Kentucky.
Dr. Ahmad Zargari is Associate Dean of School of Engineering and Information Systems, and Professor of Engineering and Technology Management at Morehead State University. He is a member of Alpha Gamma Chapter of Epsilon Pi Tau.
Barini, E. M., Tosello G., and De Chiffre L. (2010). Uncertainty analysis of point-by-point sampling complex surfaces using touch probe CMMs. DOE for complex surfaces verification with CMM, Journal of Precision Engineering, Elsevier, 34, 16 – 21.
Doiron, T. (2007). 20°C – A short history of the standard teference temperature for industrial dimensional measurements. Journal of Research of the National Institute of Standards and Technology, vol. 112, No. 1. http://nvlpubs.nist.gov/nistpubs/jres/112/1/V112.N01.A01.pdf
Fang C. Y., & Sung C. K. (2005). Measurement uncertainty analysis of CMM with ISO GUM, Available: aspe.net/publications/Annual.
Kappele, W.D., & Raffaldi, J. (2005). Quality 101: An introduction to Gage R&R. Quality Magazine. Retrieved from: http://www.qualitymag.com/articles/83529
Martin, D. (2008). A spreadsheet tool for learning the multiple regression F-test, t-tests, and multicollinearity. Journal of Statistical Education, 3( 3). http://www.amstat.org/publications/jse/v16n3/martin.html
Meredith, S. (1999). Choosing a CMM: Selecting the system that’s best for you. Quality Digest. Retrieved from: http://www.qualitydigest.com/jan99/html/choosecmm.html
MINITAB (2010). Gage Studies for Continuous Data.
Montgomery, D.C. (2009). Introduction to statistical quality control. (6th ed.) New York, NY: John Wiley & Sons.
Montgomery, D.C. (2014). Design and analysis of experiments international student version. New York, NY: John Wiley & Sons.
MSA Workgroup. (2010). Measurement systems analysis MSA (4th ed.). Automotive Industry Action Group (AAIG).
Olleveant, N.A. (1999). Tukey multiple comparison test. Journal of Clinical Nursing. Blackwell Science. Retrieved from: http://www.blackwellpublishing.com/specialarticles/jcn_8_304.pdf
PC•DMIS (2014). Integrating CAD and Metrology. PC•DMIS. Hexagon Metrology. Retrieved from: http://www.pcdmis.com/products/pc-dmis-cmm/pc-dmis-cad
Phillips S. D., Borchardt B., Abackerli A. J., Shakarji C., Sawyer D., Murray P., Rasnick B., Summerhays K. D., Baldwin J. M., Henke R. P., & Henke M. P. (2010). The validation of CMM task specific measurement uncertainty software. National Institute of Science and Technology (NIST). Available: nist.gov/manuscript-publication.
Wilhelm R. G., Hocken R., & Schwenke H. (2001). Task specific uncertainty in coordinate measurement. Journal of Cirp Annals of Manufacturing Technology, 50(2), 553-563.