Quality Digest      
  HomeSearchSubscribeGuestbookAdvertise June 13, 2024
This Month
Need Help?
ISO 9000 Database
Web Links
Back Issues
Contact Us

by Kennedy Smith

Software bugs cost the U.S. economy an estimated $59.5 billion each year, and more than half of that cost is shouldered by the end-user, according to a 2002 study from the National Institute of Standards and Technology.

The cost of failed projects for the U.S. information technology industry in 2000 was estimated at $84 billion, reports The Boston Globe.


Case in Point: SAS

Founded in 1976 by James H. Goodnight, SAS (www.sas.com) develops business intelligence software that serves more than 40,000 businesses, government agencies and educational organizations in 118 countries. Among other awards, SAS was chosen as one of FORTUNE magazine's "100 Best Companies to Work for in America" in 2003.

In the following interview, Cynthia Morris, SAS's research and development director in charge of software quality, discusses how the company integrates quality throughout the product life cycle. Morris, who has been with the company for 12 years, leads three groups at SAS: Tools and Infrastructure, Testing Integration, and Process and Research.

QD: How does quality come into play in the process of developing your software?

Morris: It starts with our employees; they're qualified and loyal. We have a very low turnover rate because of our environment; satisfied employees create satisfied customers.

Infrastructures are in place to support quality--like R&D trainers to educate new hires in our internal processes and technologies. We organize so that the testers and developers are sitting side by side, working together to create a quality product. Also, we try to get the technical support team and the marketing strategists located nearby or at least promote interaction to ensure that the right products are delivered. We have internal tools to help the developers and testers work seamlessly along with automated internal reports to track the release progress. We use quality management systems to document our processes and create a model, which helps us develop and sustain repeatable processes.

QD: How has the software quality process changed in the decade that you've been with SAS?

Morris: The awareness that quality is the responsibility of everyone working on the software project now permeates the organization. We're using complete data to inform the team and management about the quality status. It's on our internal Web so every employee has access to it. And we're open to new models, such as prototyping and peer programming. During the past several years, we've been using more tools to help us automate reports and tests.

QD: How involved is top management in the quality process?

Morris: Top management sees it as a priority. Jim Goodnight, our CEO, is a programmer himself. He understands the need to have a quality system. Like Goodnight, our CTO Keith Collins understands the value of a quality system. He wants to be proud--we want to be proud--of what we deliver.

Just this week [Goodnight] hosted an internal Webcast, and he was talking about how he really enjoys programming and wishes he had more time to do it. He's a man who understands what programmers have to go through, and he's also focused on quality. So, quality does come from the top down as well as the bottom up.

Because we lease our software on a yearly basis, we have to continuously prove quality to our customers. That focuses the minds of the developers and sales force to make sure we're always meeting customer needs and delivering quality goods because the customer can decide, "No, thank you. We want to go somewhere else."

QD: How are defects handled at SAS?

Morris: When customers report a defect, we mark it as such: a customer-reported problem. We have reports that highlight the customer-reported problems to ensure that we address them. The SAS User Group International conference provides a forum where developers can talk to customers directly. Ballots are sent to the customers so we can find out what they'd like us to improve upon--what their top issues and feature requests are. We address something from that ballot every year.

We evaluate our key quality milestones for every release and determine whether we've met our objectives. If we haven't, we make hard decisions about cutting certain features, delaying the schedule, or a combination of both to ensure a quality product.

QD: What quality improvement suggestions would you give to software companies that aren't implementing a quality initiative?

Morris: Metrics are critical to increase understanding of how their software compares with what the customer wants and needs and to really find out where their quality stands. They should hire people that have quality experience to implement quality processes. It's a big task. There are lots of measurements, lots of good practices available, but it does take time to incorporate them into the fabric of any software organization.

Communication is key. You have to have good communication with your customers as well as good communication within R&D, marketing and throughout the company.

QD: What is your opinion of the state of quality in software?

Morris: The pressure of having superior quality while getting to the marketplace in a timely manner continues to increase. Software product complexity and operating system interdependence continue to grow. At the same time, these trends provide us with more data and better processes and tools to help us with those challenges. Software quality is growing in maturity, and it will continue to get better and better. Customers demand it.

IEEE Software Magazine reports that the best commercial software companies remove about 95 percent of all known defects before releasing a product to the customer. However, the industry average is less than 85 percent.

Watts Humphrey of Carnegie Mellon University's Software Engineering Institute estimates that for every 1,000 lines of code that software professionals create, there are about 100 defects.

The Standish Group's Software Chaos report states that the software performance success rate is about 30 percent, and runaway and failed projects are common.

If these statistics regarding the quality of software don't make you cringe, you must have never used a computer. Surely everyone who has can relate a story about how software bugs have affected his or her work. Your computer locked up before you had a chance to save that important report; your Web page was down and you couldn't figure out why; you called tech support to report a problem only to have them tell you, "We're already aware of that defect, and you owe us $35 for the phone call." But can the state of quality in software really be as bad as the statistics suggest? And if so, what can software development companies do about it? This article will attempt to answer these questions.

The state of software quality

What is the state of software quality today? It really depends on whom you ask. Various reports surface every year or so that attempt to explain where the United States stands in terms of software quality, but there seem to be as many opinions as there are experts on the subject.

"There's been some improvement in software quality over the years," explains Herb Krasner, founder of the Software Quality Institute at the University of Texas. "About five years ago, The Standish Group was reporting that the software project performance success rate was about 25 percent, meaning that 75 percent of all software projects that were undertaken were ultimately unsuccessful. Today, it's up to 30 percent; so yes, there is slight improvement."

Carolyn Fairbank, CEO of the Quality Assurance Institute, believes that software quality will remain below par until organizations restructure their thinking to be based not upon deadlines, but upon good processes. "We're far too focused on product delivery, not process capability," she argues. "We're too busy trying to get the product out the door. Granted, this is a market-driven phenomenon, but we'll have to change that deadline-driven attitude to one of good processes. If you get the process right, the product will have a far better chance at success. Unfortunately, many IT professionals still don't quite understand the concept of process management."

"In the beginning of a new technology's life cycle, there's a time when it's really exciting but crappy in terms of quality," explains Mark Minasi, technology writer and author of several books on software, including The Software Conspiracy: Why Software Companies Put Out Faulty Products, How They Can Hurt You, and What You Can Do About It (McGraw-Hill Trade, 1999). "We've been in the software business around 55 years, and by now we should be in the part of the life cycle where quality becomes important. I like to ask people, 'How would you feel if Interstate 95 was completely computerized and a Pentium 4 was running it?' They all say, 'I'd never get on that.' We don't trust software, and it's a terrible shame."

Why don't some software companies invest in quality?

Experts cite several reasons software companies excuse themselves from investing more heavily in quality practices:

"Our customers aren't complaining; therefore they're satisfied." That's wrong, says Fairbank. On the contrary, customers feel that they have no other choice but to put up with defective software, so they don't bother complaining. In other words, the public has become used to poor-quality software. However, Fairbank warns companies against this "like it or leave it" mentality. "In the beginning, if you wanted a desktop you either went with Apple or Microsoft," she explains. "But, during the last several years, new operating systems like Linux have emerged, which is giving some of the larger companies a run for their money. Customers are slowly starting to realize that they really do have alternatives and, more important, they have legal recourse."

"We've tried quality and we didn't see any results." Like all quality initiatives, results aren't always immediately noticeable--it's what Fairbank refers to as the software industry's tendency to be instant gratification-oriented.

"The improvement I see with companies that invest in software quality is 10 times, easily," adds Krasner. "There's a lag time between the investment and the payoff, and it varies by organizational size. So, once you get them to invest in improving the quality, management has to be patient enough to wait to see the return on the bottom line."

"Our customers want new features, not better quality." This is another misconception, say the experts. "Marketing is constantly saying, 'It's got to have new features; otherwise we can't compete,' " says Minasi. "We need to convince software vendors that it's perfectly OK to spend less time on features and more time on quality. Who really needs Mongolian character sets? If you want to sell me your newest version, make it crash-proof."

"In order to get the product out in a timely manner, we must sacrifice some quality." Software development is a time-to-market industry, which may cause some vendors to assume this attitude. "Software vendors would like to provide a quality product, but they've got to get it out to market before someone down the street does," explains Fairbank. "We've accelerated our time-to-market by leaps and bounds, but that doesn't mean quality has improved. In fact, the companies that don't have quality processes in place are the ones that have trouble getting a product out to market in the speed at which it needs to be delivered."

Minasi describes it as the "iron triangle": You can have it fast; you can have it cheap; you can have it good. Sounds nice, but the customer is only allowed to pick two. "Software vendors think that customers will choose fast and cheap," he says.

"Top management doesn't care about quality." Yes, they do. However, those who pitch quality to top management must explain it in terms they understand: What is the cost of poor-quality software?

"It's impossible to create low-defect software; by nature, software contains bugs." The experts disagree. Software development companies that don't use quality processes have high chances of creating bug-ridden software simply because they're starting from scratch every time they develop something new, explains Minasi. As an example, he compares software companies' processes to The Boeing Co.'s. A significant portion of new product development at Boeing lies within the development and testing phases. Not one part is built before it's been tested in theory and on paper. This way, Boeing can be almost 100-percent sure that the product will work even before they've built it.

In contrast, many software makers create a new program and then test it for defects, leaving them prone to mistakes.

Another problem experts cite is that software companies are often immune to the consequences of their poor-quality product. "With some exceptions, software companies have been shielded from litigation for years," explains Fairbank. "The customer will one day say, 'We're fed up with this, so we're not buying your product anymore.' When companies take the fall from either litigation or financial disaster, we'll start to see an accelerated trend toward software quality."

The tricky part, however, is convincing software companies that it's in their best interest to invest in quality now. This involves showing top management the bottom-line results and making a culture change toward quality throughout the entire organization. It also means teaching software development companies about the various methods of attaining that goal.

The cost of software quality

There are several approaches a software developer can take when implementing quality practices. Krasner has developed an approach called the "cost of software quality," which outlines the economic benefits of delivering good quality software. The cost of quality approach has long been used in the manufacturing industry, and Krasner has expanded on that idea to include software development.

CoSQ defines key dimensions of software quality, including the level of customer satisfaction; the degree to which software has value for its stakeholders; key attributes such as reliability, usability and maintainability; the degree to which a software product is defective and the implementation of quality processes. (See page 34.)

In his report, "Using the Cost of Quality Approach for Software," Krasner explains that companies need to answer three questions: How much does poor software quality cost? How much does good software quality cost? How good is our software quality? Once these questions are answered, the software providers can compare quality costs to overall software production costs and software profits, compare quality costs to benchmarks and norms, better analyze product quality to improve their competitive situation, measure improvement actions and bottom-line effect of quality programs, visibly see previously hidden costs related to poor quality, and more clearly see the economic tradeoffs involved with software quality.

"By putting the cost of quality model into action in an organization, what immediately becomes obvious to management and the executives is the amount of money they're losing on fixing bugs and delivering poor-quality products," comments Krasner. "They were never able to quantify that before, so they could only guess about how it was affecting their bottom line. The cost of quality model gives them results in a form that they understand: dollars."

The Capability Maturity Model

Another approach to software quality improvement is the Capability Maturity Model. The Carnegie Mellon Software Engineering Institute, which developed the CMM, describes it as the outline of the principles of software process maturity that assists software organizations in improving their software processes in terms of product evolution. It comprises five maturity levels (This information can be found at the SEI Web site: www.sei.cmu.edu):

Initial. The software is characterized as specialized (ad hoc) and, occasionally, even chaotic. Few processes are defined and success depends on individual effort.

Repeatable. Basic project management processes are established to track cost, schedule and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications.

Defined. The software process for both management and engineering activities is documented, standardized and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software.

Managed. Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled.

Optimizing. Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.

The CMM contends that as an organization moves up these five levels, the results are predictability, effectiveness and control of the organization's software.

Software quality standards

Other ways to systematically improve software quality are through established standards. The Institute of Electrical and Electronics Engineers produces more than 30 percent of the world's published literature in electrical engineering, computers and control technology. This includes more than 40 standards related to engineering terminology, process documentation, quality tools, project management, safety, dependability and other software quality issues. To learn more about specific IEEE standards, visit http:/href="http://qualitydigest.com/standards/index.lasso".ieee.org/software.

The International Organization for Standardization also develops specific standards related to information technology and software engineering, including software testing requirements, software product evaluation, software process assessment and others. A complete list of software-related ISO standards is available at www.iso.org.

The future of software quality

Although they differ about the state of quality in software today, experts agree that it will continue to improve. Minasi believes that as software developers realize that there are only so many special features they can add, the focus will move toward quality. "Once quality becomes a perceived possibility, once it becomes something you can buy, people will upgrade to that system," he suggests.

Fairbank says that software companies can improve quality by adopting an enterprisewide quality culture. "There are small things an organization can do to make an immediate bottom-line improvement," she states. "But, instilling the philosophies of quality into an organization takes a culture change that doesn't happen overnight. The ones that stay in it for the long haul have seen tremendous results via improved customer satisfaction, outstanding products and services, heightened employee morale, and substantial ROI."

Krasner sees a slow but steady trend toward quality. "In 10 years, instead of a 30-percent success rate on our software projects, I hope we'll have a 75-percent success rate," he says. "My hope is that instead of losing about $60 billion a year because of software quality problems, we'll only be losing $30 billion."

He also agrees with Fairbank that a culture change is needed. "The catalyst for change is re-educating people who are in decision-making positions to understand that quality methods are good for business. We've educated the developers and the project managers, and they understand why quality is important. Now it's time to prove it to the people above them."

About the Author

Kennedy Smith is Quality Digest's associate editor. Letters to the editor regarding this article can be sent to letters@qualitydigest.com.