No, this isn’t about promises you made on New Year’s Eve and broke the next day. It’s about measurement resolution—the number of decimal places to which a measured value is calculated and presented. The measurement can be of any parameter—voltage, distance, weight, temperature, whatever. The principle of measured and displayed resolution applies in all cases.

You may think that lots of digits after the decimal point means high-resolution measurement. If a device shows more digits to the right of the decimal, does that mean it has a higher resolution than one with fewer digits? Maybe; maybe not. Those extra digits to the far right of a display that are beyond the actual resolution of the measuring device are defined as empty resolution. They’re more for show than for performance.

What’s the right resolution?
A number of factors come into play when choosing the right resolution for a measurement. A common one is the 10:1 rule, which states that the measurement resolution should be 10 times finer than (or 1/10th) of the specification being measured. In other words, if the specification is 2.3 units, the measuring device should resolve to X.XX units, or one more decimal place than the specification. You might think that if one more digit is good, then two, three or more than that would be even better. Maybe not.

What’s measured vs. what’s displayed
In a perfect world, the number of digits in a display readout would represent the capabilities of the measuring device displaying those values. If, let’s say, the device can resolve to four decimal places, the display would have four digits after the decimal point. However, some devices fib and display more decimal places than they can actually measure. Did you ever notice a digital display with lots of decimal places where the last (least significant) first, second, or third digits are constantly flickering between different values? Is that because the parameter being measured is varying or is it noise in the measuring device itself? This question leads to a useful rule of thumb: Do not assume that a device with a display of x digits after the decimal point can actually resolve measurements to x digits. In other words, just because a display has half a dozen digits to the right of the decimal point doesn’t mean the device can measure to that resolution. Those last few digits may be empty resolution.

Here’s an example of why this matters: Let’s say you’re measuring a voltage with a specification of 5.00 +/- 0.05 volts. You grab a voltmeter with five decimal places to the right of zero and get a reading of 5.04992 V, but the last three digits are flickering between values (in such a case, the least significant digit would probably vary so much as to be unintelligible). This brings up a few questions. Since the specification is only two decimal places, should you even care about the last two or three digits? Is the display flickering because the measured voltage is changing? Is it noise in the voltmeter? Whether it’s the voltage or the display (or both) that are varying in those least significant digits, does the reading show the voltage to be in or out of specification? If the reading were stable and repeated at 5.04, it wouldn’t matter what the last three digits were doing—that reading is within the specification. But it would be good to know how close the reading is to the 5.05 limit; so noting that third digit is very meaningful. In this example, the least significant digits are of no consequence.

Specifying the proper resolution is relatively easy if the measurements aren’t near the limits of what the measurement device can resolve. However, if you’re measuring extremely small parameters, such as microvolts, nanometers, or pico-seconds, you may be near or past the limits of your measuring device, regardless of how many digits are in its display. This is why it’s important to know how well the measuring device measures at the extremes of its display resolution. And remember that simply adding more decimal places doesn’t mean that you’re extending the measurement capability to those additional digits—they might be empty resolution.

The right resolution
Some people fall into the trap of setting their specifications and tolerances to the display resolution of their measuring devices. “Since I can measure to a ten-thousandth of an inch, why not specify all my parts to that resolution?” There are a number of ways to respond to this question, the most important of which is with another question: “What resolution is necessary?” For example, just because you have a stopwatch that can measure to a millisecond, it doesn’t mean you have to measure everything to a millisecond. The nearest second, minute or hour may be close enough. As with everything else that’s measured, the resolution depends on the particular specification, not simply on the performance of the measuring device.

What you resolve is not accuracy
What does all this have to do with accuracy? Some people believe that if their measuring device can measure to four decimal places, they’re accurate at four decimal places. Wrong. No matter what a measuring device can resolve, it has little to do with how accurately it measures. Remember that accuracy is how well a measurement matches a known value. Of course, it helps if your measuring device has the necessary resolution to verify the accuracy. Although this sounds like circular logic, keep one thing in mind: Don’t trust your accuracy to empty resolution.

Until next time: Yes, measurement matters.