Building gold-standard AI systems requires gold-standard AI measurement science—the scientific study of methods used to assess AI systems’ properties and effects. The National Institute of Standards and Technology (NIST) works to improve measurements of AI performance, reliability, and security that American companies and consumers rely on to develop, adopt, and benefit from AI technologies.

Among other groups at NIST, the Center for AI Standards and Innovation (CAISI) works with the larger community of AI practitioners to identify and make progress on open questions that are key to maturing the field of AI measurement science and advancing AI innovation. This article highlights an initial selection of such questions that CAISI has identified through its initiatives to date.

The need for improved AI measurement science

Many modern evaluations of AI systems don’t precisely articulate what’s been measured, much less whether the measurements are valid. Metrology, the field of measurement science, provides methods for rigorous and trustworthy use of measured values, qualified with assessments of uncertainty.

 …