{domain:"www.qualitydigest.com",server:"169.47.211.87"} Skip to main content

User account menu
Main navigation
  • Topics
    • Customer Care
    • FDA Compliance
    • Healthcare
    • Innovation
    • Lean
    • Management
    • Metrology
    • Operations
    • Risk Management
    • Six Sigma
    • Standards
    • Statistics
    • Supply Chain
    • Sustainability
    • Training
  • Videos/Webinars
    • All videos
    • Product Demos
    • Webinars
  • Advertise
    • Advertise
    • Submit B2B Press Release
    • Write for us
  • Metrology Hub
  • Training
  • Subscribe
  • Log in
Mobile Menu
  • Home
  • Topics
    • 3D Metrology-CMSC
    • Customer Care
    • FDA Compliance
    • Healthcare
    • Innovation
    • Lean
    • Management
    • Metrology
    • Operations
    • Risk Management
    • Six Sigma
    • Standards
    • Statistics
    • Supply Chain
    • Sustainability
    • Training
  • Login / Subscribe
  • More...
    • All Features
    • All News
    • All Videos
    • Contact
    • Training

How Many Decimal Places Are Enough?

Determining what’s reasonable for equipment resolution

Photo by Sable Flow on Unsplash

Henry A. Zumbrun
Mon, 05/13/2024 - 12:03
  • Comment
  • RSS

Social Sharing block

  • Print
Body

We’ve been asked by customers how many decimal places are enough. It’s either because they want to maximize the resolution of their purchase or they expect far more resolution from the equipment than is reasonable.

ADVERTISEMENT

We’ve seen a lot, even a meter being set to read 100,000.001. That’s 100 million counts, though this instrument (i.e., readout) wasn’t even stable to 1 count. The extra 0.001 meant nothing.

Then there’s the other extreme, where some say that rounding is affected. They might be right, depending on what they’re trying to accomplish. Many meters will round based on an invisible digit to the right of the decimal place. That means if a meter can read 10,000 counts by a resolution of ±1 count, the meter is likely rounding based on 10,000.X, where X equals 0.1 – 0.9. Thus, 10,000.6 becomes 10,001, and 10,000.3 becomes 10,000.

This article isn’t about formulas or rounding. But it is about determining how many decimal places are enough.

Repeatability and resolution

Let’s start with a situation where we can set the appropriate resolution to display some degree of repeatability between measurements.

 …

Want to continue?
Log in or create a FREE account.
Enter your username or email address
Enter the password that accompanies your username.
By logging in you agree to receive communication from Quality Digest. Privacy Policy.
Create a FREE account
Forgot My Password

Add new comment

Image CAPTCHA
Enter the characters shown in the image.
Please login to comment.
      

© 2025 Quality Digest. Copyright on content held by Quality Digest or by individual authors. Contact Quality Digest for reprint information.
“Quality Digest" is a trademark owned by Quality Circle Institute Inc.

footer
  • Home
  • Print QD: 1995-2008
  • Print QD: 2008-2009
  • Videos
  • Privacy Policy
  • Write for us
footer second menu
  • Subscribe to Quality Digest
  • About Us
  • Contact Us