Usability Engineering: When the Surface Becomes the Touchscreen

Published: Wednesday, August 14, 2013 - 15:21

This article is about how augmented reality (AR) techniques can change the way we use measurement equipment. Three applications are presented to demonstrate how much easier, faster, and more intuitive an inspection task gets through gesture control (GC) and presenting information on the inspected surface.

We demonstrate these principles using a fastener-flushness measurement system for the aerospace industry as an example. A gauge repeatability and reproducibility (gauge R&R) test proves that results are repeatable and reproducible, meaning there is very little operator variation. This has been and still is a big problem when using manual dial gauges for the same purpose.

From usability engineering we know that there are three important limits to device response times: 0.1 second for immediate response, 1 second for uninterrupted user workflow, and 10 seconds for keeping a user’s attention. We will explain what these limits mean when working with measurement equipment, and why we think it’s important to meet these criteria for an effective workflow that results in better user experience and, ultimately, better inspection results.

The examples are based on a very fast, structured-light scanner that integrates application-specific inspection software to address chronic and well-defined application problems. In our example, these are inspection flushness of fasteners, gap and flush of assembled parts, and small dents in smooth surfaces. The common approach of all three examples involves inspection based on extracting 2D and 3D features and rapidly presenting the inspection results as a colored projection overlay directly on the inspected part. The whole process of scan, analysis, and display of results is completed within a couple seconds.

For an operator accustomed to long evaluation and interpretation cycles, such AR techniques enable instant analysis. Also there is no need to transfer results from a computer screen to the matching areas of a part because the results are displayed right next to where the measured features are.

Although AR renders obsolete the traditional computer monitor display, GC techniques and touchscreens can do the same for keyboards and mice. This allows for a more intuitive user experience when initializing and triggering the measurement process. As people get more familiar with touchscreens, we think GC will mark the next natural evolution of how we interact with devices. We will describe how this idea is implemented on an AR-enabled, dent-inspection application to produce information about surface deformation right at your fingertips.

To our knowledge the techniques use, for the first time, completely self-contained white-light scanners with integrated AR and GC. With seamlessly integrated battery and on-board CPU, these systems are ready to use out of the box in seconds.

White light scanning

During the past decade, white light scanning has become a well-established tool for many different applications, including 3D digitizing (i.e., turning real-world objects into 3D models) and 3D inspection and measuring (comparing scan data to CAD models). Fields of use are just as versatile, including automotive, aerospace, archaeology, cultural heritage, life-science and medical applications. This section outlines the basic principle of operation, resulting output, and associated limitations of white-light scanning.

Principle of operation
A projection unit, similar to a slide projector, is used to project a series of patterns onto the object to be digitized (see figure 1).

Figure 1: A series of typically four to 12 different line patterns is projected onto the object. Images are captured from an angle that is different from the angle of projection.

Usually one or two digital cameras observe this scene from different perspectives (i.e., viewing angles). From the point of view of these cameras, the shape of the projected lines appears to be changed by the shape of the object. The series of patterns are imaged, stored, and processed on a computer.

System setup and use
Most of today’s commonly available systems are designed so that they have a broad range of applications. Sometimes they come with different lens sets so that the operator can adjust the field of view by switching lens objectives. All the systems are operated through dedicated software that runs on a powerful standalone computer. The scanner itself is connected through a cable to that computer.

Although this setup is widely used, it has some drawbacks:
• Systems are relatively bulky and while hooked up to the computer cannot be easily moved on the shop floor or in difficult-to-access areas where measurement is required.
• To solve a measurement problem to the point where an actionable result is achieved takes much more time than simply scanning; it requires additional software and scripting, operator
skills, and engineering interpretation of complex point clouds.
• Results are delivered on a standalone computer monitor that sometimes can be far from the place of measurement and requires walking back and forth
• Systems are relatively expensive due to their versatile design and setup

As a result of these properties, today’s commonly available measurement systems are rarely used for the daily routine of quality control jobs on the shop floor. Instead, these systems are relegated to use by engineering departments and trained operators.

Below we suggest some changes to the design, setup, and use to this model that overcome the shortcomings mentioned above.

Usability engineering

At this point we want to introduce a few points that are important when assessing how “usable” a measurement system is. Much attention is paid to the accuracy and repeatability of results delivered by a measurement system. It is generally agreed that this factor is of ultimate importance. However, in giving it such overarching priority, basic usability aspects are sorely overlooked.

Here are two examples. The first addresses ease of use when operating a system. The second highlights the importance of producing results that are meaningful, readily understood, and immediately actionable in the form of a pass/fail or go/no-go decision. By overlooking such usability factors, traditional systems deliver a negative operator experience and yield very poor inspection efficiency.

Speed: Make it fast
From usability engineering, we know that there are three important limits to device response times:
• 0.1 seconds for immediate response. If a response is 0.1 second or faster, the user doesn’t perceive any delay and experiences immediate response. Common examples of this can be seen when adjusting the volume on a radio or changing a channel on TV.
• 1 second for uninterrupted user workflow. For more involved operations, a user typically will allow up to 1 second of time, before shifting their thoughts away from the task at hand. In such cases, it is not necessary to inform the user about progress.
• 10 seconds for keeping users attention. When response times approach 10 seconds, it becomes important that the user receive feedback on
the progress of the process in order to retain his attention. Displaying a progress bar serves this purpose adequately.

Response time delays greater than 10 seconds should be avoided if the goal is to deliver a continuous workflow experience. Exceeding this time increases the risk that the user will be distracted with other things that would affect the quality of his work.

It’s important to keep these limits in mind when designing and using a measurement system, as well as in any other computerized system. When designing our new systems, we emphasized maintaining immediate (0.1 second) or uninterrupted (1 second) user workflow, while still delivering high-measurement quality.

Simplicity: Keep it simple
Everybody recognizes simple and easy systems quickly, but making a system simple can be a complex process. It involves finding the right balance between functionality and ease of use. Many engineers like to squeeze in too many features that eventually end up making the system complex, and thereby limiting adoption.

A few factors that help drive simpler design include:
• Fewer mouse clicks
• Fewer parameter settings
• No operation modes
• No complex menus
• Hard-wired buttons instead of software keys
• Immediate response, little latency

Control: Put the user in the driver’s seat
For best results, it’s important that the user is in control all the time. To achieve this, processes must be short, and feedback of results needs to be quick. If this can’t be accomplished because of complex and time-consuming calculations, then the user should be kept informed about what is going on and how long the calculations are going to take. Progress bars are a typical way to accomplish this. But if the user decides she wants to terminate this time-consuming process and start over again, this should be possible at all times. Thus, to keep the user in control:
• Keep processes short
• Keep the user informed about the progress, if processes are long
• Always allow the user to interrupt the process

Augmented reality

Augmented reality (AR), as defined by Wikipedia is “a live, direct or indirect view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics, or GPS data.”

Although we might not be familiar with the term “augmented reality,” we are all familiar with how AR is used in today’s sports telecasting (see figure 2). One example is the yellow “first down” line seen in television broadcasts of U.S. football games. This is the line the offensive team must cross to receive a first down. AR is also used in association with many other sporting events to show commercial advertisements overlaid onto the view of the playing area.

Figure 2: AR overlay of commercials and the yellow line in sports broadcasting

We think AR is a great concept for many applications, but to our knowledge one that is rarely used for any measurement device.

In our application for fastener-flushness measurement, we use AR principles to project measurement results right next to the features that have been measured. These are numerical results of fastener depth and angle as well as simple color codes that mark features relative to tolerance settings. In figure 3, fasteners within tolerance are green, and fasteners out of tolerance are red or blue, depending on their direction. In addition, we use the overlay image to provide status information that gives the operator real-time feedback. Examples of this include information to help the operator maintain the appropriate working distance and brightness settings.

The obvious benefit in this application is that there is no doubt about which measurement result belongs to which feature. Traditional solutions that present the results on a separate monitor not only require additional bulky hardware but also introduce uncertainty about identifying the right fastener on the screen, which can lead to another source of error.

Figure 3: Augmented overlay for fastener-flushness measurement

Limitations for this method of projecting results onto the measured part may arise in the following cases, where visibility of a projection can be limited:
• Very shiny parts
• Very dark and matte parts
•  Parts that don’t have a flat-enough surface to display something
• Transparent surfaces

Hand-gesture recognition

Once we successfully removed the need for a computer monitor to operate the system, our goal was to make a keyboard and computer mouse also obsolete. We wanted to trigger the measurement and change other necessary settings by using hand gestures only.

Figure 4: Examples of hand gestures

Although this user-experience is still under development, we can showcase some early results of how our system can be operated with natural and easy-to-learn hand gestures. Advantages include:
• Intuitive operation
• Easy to learn
• No additional hardware (e.g., remotes, keyboards, batteries) necessary

Putting it all together

Putting together all of the above ideas and features into a fast white-light scanner that is completely self-contained and battery-powered results in an interesting new category of measuring devices.

The fastCHECK system
Based on a scalable hardware platform, 8tree has developed a range of products that incorporate all of the features described here. The fastCHECK system for checking fastener flushness on aerospace parts will be used to demonstrate the capabilities of such a system as well as to prove repeatability and reproducibility of the measurement results (see figure 5).

Figure 5: The fastCHECK system checks flushness of aerospace fasteners and projects the results back to the surface.

The application for this system is checking flushness and angularity of installed fasteners (especially rivets and Hi-Loks) on aerospace parts. The standard method uses manual dial gauges; this is time-consuming, and the results depend on the operator’s experience level (see figures 6 and 7). The goal is to improve this process by increasing the throughput, and improving the measurement quality by reducing the operator influence.

Figure 6: A manual dial gauge is used to measure fastener flushness.

Figure 7: Typical problems that can occur when installing fasteners

The system uses a structured light approach and projects a sequence of different patterns within 0.1 second. The light color can be adjusted to match the color of the measured surface. Then, the system identifies the fasteners from the 2D image and builds a reference plane around each fastener, making use of the 3D coordinates available at every camera pixel (see figure 8).

Figure 8: How structured light scanning picks up depth and height of fasteners

Another plane is then built on the head of each fastener. These two planes can now be compared in terms of distance (i.e., from the middle of each fastener) and in terms of angles if the fastener is not installed exactly perpendicular. These measurement results are saved and optionally projected next to each fastener head. A color code is projected on top of each fastener to inform the operator about the measurement result in a go/no-go fashion (see figure 9).

Figure 9: A color code marks each fastener for easy interpretation of measurement results.

The following color code is used to make results easy to understand:
• Green: fastener is within tolerance
• Red: fastener is above plus tolerance
• Blue: fastener is below minus tolerance
• Compass needle: angular tolerance is exceeded
• Size of dot: The size of the projected dot is proportional to the tolerance, e.g., the larger the deviation, the bigger the projected dot

Together with Airbus in the United Kingdom, we performed several gauge R&R studies with the system in order to verify performance for angular and depth measurement on different surfaces, lighting conditions, and different size fasteners. We share some of the results here with special attention to reproducibility or operator variation. This is a chronic problem in the industry when using manual dial gauges, which can be pretty accurate but can be strongly influenced and dependent on the person who uses it.

Gauge R&R test

A commonly used method to analyze the performance of a measurement system is to perform a gauge R&R study. In this study, a gauge or measurement system is used to obtain repeated measurements on selected parts by several operators. The two main elements generated during the study are repeatability and reproducibility. Repeatability represents the variability when the system is used to measure the same part by the same operator. Reproducibility refers to the variability from different operators measuring the same part. The main purpose of a gage study is to determine how much variation in the data is specifically due to the measurement system, and whether the system meets the required performance. A gauge study helps isolate and remove other factors such as operator variation and part variation.

Several methods are available to support a gauge R&R study. For example using a simple EXCEL spreadsheet that models the guidelines from the Automotive Industry Action Group  (AIAG), or using software packages like Minitab that performs all relevant calculations.

Typically, a gauge R&R study is performed using 10 parts, two or three operators, and two or three replications (per AIAG guidelines).

During the gauge R&R study the following estimates are determined and expressed as percentage values:
• Appraiser variation—repeatability
• Equipment variation—reproducibility
• Combined R&R
• Part variation
• Total variation

To establish the repeatability and reproducibility of a system, the variation of the measurement is compared with the variation in the test samples set. The relations between these measures are explained in figure 10:

Figure 10: Combination of AV and EV to R&R in form of right-angle triangle

The contribution of equipment and appraiser to the total variation can be expressed as the statistical variance or standard deviation (or multiple thereof). Additionally to the overall estimates, distribution and variation of the measurement data are analyzed. For this two charts are created, the Xbar chart and the Rbar chart.

Xbar chart
The chart plots the averages of multiple readings by each operator on each part, and the control limits are calculated using the repeatability variation. The repeatability variation from an acceptable measurement system should be much less than the part-to-part variation, which is reflected by the variation of the plotted points on the chart. Therefore, with an acceptable gauge, most plotted points should fall outside the control limits.

Rbar chart
In gauge R&R studies, an R chart is used to check the reproducibility variation.

Specifically, you can answer the following questions with an R chart:
• Does each operator measure all the parts consistently? If not, which part is more difficult to measure consistently?
• Do all operators have similar measurement variation? Do specific operators measure with significantly greater variation?

If no points are outside the limits, this indicates that all parts were measured with similar consistency.

Example gauge R&R study with fastCHECK
• Test sample: 10 fasteners on a test panel
• Operators: Three operators with different skills level All received a half hour introduction to the system operation
• Sequence: All operators measured the 10 fasteners three times. The fasteners were placed such that the operators needed to perform at least two scans to capture them all.

Results, Xbar chart

Figure 11: The plots show that the chosen samples represent a good variety. Click here for larger image.

Results, Rbar chart

Figure 12: The plots show the variation of measurements is within the calculated limits. Click here for larger image.

The operator’s influence
The purpose of this study is to analyze the influence of the operator in the overall measurement variation. For this we use a calculation following the Minitab analysis. The results in figure 13 show the detail components of the total variation:

Figure 13: Results table of the gauge R&R study

In this example, 1.07 percent of the total variation can be contributed to the appraiser or reproducibility.

The study variance for the value, based on 6 x standard deviation, is 4.7 percent. This shows that, thanks to the usability engineering, the operator’s influence can be kept small even for a complex metrology instrument.

Further applications: dentCHECK and gapCHECK
Currently, additional products are being developed by 8tree to address other measurement tasks in a similarly intuitive and effective manner. Among these new products, two examples are pictured below: dentCHECK for checking size and depth of dents in automotive or aerospace parts, and gapCHECK for measuring gap and flush (see figures 14 and 15).

Figure 14: dentCHECK measures dents and projects color deviation plots on the part.

Figure 15: gapCHECK is able to measure gap and flush deviation.

A&M Precision Measuring Services is the North America reseller for 8tree products.  A&M also has an established history as a specialist in tooling, machining, and metrology services addressing the needs of many different industries.

About The Authors

Erik Klaas

Erik Klaas is the CTO and co-founder of 8tree. He co-founded 8tree to create very easy-to-use, application-specific, white-light-scanning systems that revolutionize the user experience with augmented reality and gesture-based techniques. Previously, during a 20-year career at Breuckmann GmbH, he held a variety of engineering and project management roles, and served as CTO of the company during the last 10 years. Klaas obtained his engineering degree in optical 3D metrology and color topometry from the Cologne University of Applied Science.

Arun Chhabra

Pia Böttcher

Pia Böttcher is on the operations and business development team at 8tree. She joined 8tree with a shared belief in the vision of easier-to-use 3D-scanners. She brings a rigorous, process-based approach to managing customer engagements, defining strategy, and improving the overall customer experience. During the past 20 years, she has held a variety of project management, sales/marketing, and business management roles in the machine-vision and life-sciences sectors. Böttcher has an MBA from Warwick Business School and a graduate-level degree in computer science from the University of Applied Sciences Konstanz.