{domain:"www.qualitydigest.com",server:"169.47.211.87"} Skip to main content

User account menu
Main navigation
  • Topics
    • Customer Care
    • FDA Compliance
    • Healthcare
    • Innovation
    • Lean
    • Management
    • Metrology
    • Operations
    • Risk Management
    • Six Sigma
    • Standards
    • Statistics
    • Supply Chain
    • Sustainability
    • Training
  • Videos/Webinars
    • All videos
    • Product Demos
    • Webinars
  • Advertise
    • Advertise
    • Submit B2B Press Release
    • Write for us
  • Metrology Hub
  • Training
  • Subscribe
  • Log in
Mobile Menu
  • Home
  • Topics
    • 3D Metrology-CMSC
    • Customer Care
    • FDA Compliance
    • Healthcare
    • Innovation
    • Lean
    • Management
    • Metrology
    • Operations
    • Risk Management
    • Six Sigma
    • Standards
    • Statistics
    • Supply Chain
    • Sustainability
    • Training
  • Login / Subscribe
  • More...
    • All Features
    • All News
    • All Videos
    • Contact
    • Training

Do You Suffer from Empty Resolution?

Meaningless numbers

Fred Mason
Wed, 09/13/2006 - 22:00
  • Comment
  • RSS

Social Sharing block

  • Print
Body

No, this isn’t about promises you made on New Year’s Eve and broke the next day. It’s about measurement resolution—the number of decimal places to which a measured value is calculated and presented. The measurement can be of any parameter—voltage, distance, weight, temperature, whatever. The principle of measured and displayed resolution applies in all cases.

You may think that lots of digits after the decimal point means high-resolution measurement. If a device shows more digits to the right of the decimal, does that mean it has a higher resolution than one with fewer digits? Maybe; maybe not. Those extra digits to the far right of a display that are beyond the actual resolution of the measuring device are defined as empty resolution. They’re more for show than for performance.

What’s the right resolution?
A number of factors come into play when choosing the right resolution for a measurement. A common one is the 10:1 rule, which states that the measurement resolution should be 10 times finer than (or 1/10th) of the specification being measured. In other words, if the specification is 2.3 units, the measuring device should resolve to X.XX units, or one more decimal place than the specification. You might think that if one more digit is good, then two, three or more than that would be even better. Maybe not.

 …

Want to continue?
Log in or create a FREE account.
Enter your username or email address
Enter the password that accompanies your username.
By logging in you agree to receive communication from Quality Digest. Privacy Policy.
Create a FREE account
Forgot My Password

Add new comment

Image CAPTCHA
Enter the characters shown in the image.
Please login to comment.
      

© 2025 Quality Digest. Copyright on content held by Quality Digest or by individual authors. Contact Quality Digest for reprint information.
“Quality Digest" is a trademark owned by Quality Circle Institute Inc.

footer
  • Home
  • Print QD: 1995-2008
  • Print QD: 2008-2009
  • Videos
  • Privacy Policy
  • Write for us
footer second menu
  • Subscribe to Quality Digest
  • About Us
  • Contact Us