Featured Product
This Week in Quality Digest Live
Operations Features
Taran March @ Quality Digest
If at first you don’t succeed, make it a quality problem
Richard Ruiz
Seven ways automation can focus layered process audits on quality improvement rather than administrative workload
Ryan E. Day
Dimensional Engineering uses FARO QuantumS ScanArm for a complex reverse engineering project in petrochemical industry
Bruce Hamilton
Why sharing and teaching are the best ways to learn
Janelle Farkas
Data and analysis don’t have to be complicated to yield bottom-line benefits

More Features

Operations News
Help drive team productivity with customizable preprinted templates
A guide for practitioners and managers
Examine nine ways to help understand context, diffuse politics, heal relationships, and find the right work at the right time
Provides eight operating modes and five alarms
Making lean Six Sigma easier and adaptable to current workplaces
Deep Reality Viewer creates stereo, high-definition 3D images without using a monitor
April 25, 2019 workshop focused on hoshin kanri and critical leadership skills related to strategy deployment and A3 thinking
Makes it faster and easier to find and return tools to their proper places
Process concerns technology feasibility, commercial potential, and transition to marketplace

More News

Michael Brundage

Operations

When a Manufacturer Asks ‘How Do We Get Smart?’

A working explanation of natural language processing for manufacturing operations

Published: Tuesday, October 22, 2019 - 11:03

How do we get smart? I was first asked this question while sitting in on a call with a small manufacturer who supplied parts for automotive manufacturers. With all the buzz around “smart manufacturing,” this manufacturer wanted to join the movement. The problem was that its leaders didn’t know where to start.

They bought sensors without any real understanding of what to do with them. When I asked why, they said they didn’t have any useful data and they needed data to be “smart.” I asked them, “How do you not have any useful data?” They said they only had documents capturing the history of each maintenance event in the facility. These “textual maintenance work orders” were hardly useful due to their unstructured, jargon-filled nature. At the time they were right; these data in its natural form seemed useless.

However, when my colleagues and I in the Knowledge Extraction and Application for Manufacturing Operations project manually analyzed their data, we discovered a treasure trove of hidden information. Hydraulic leaks had occurred more than 40 times in three months, but they were going unnoticed because technicians wrote down “hyd leak” or “hydraulics were leaking,” or “hydaurlic [sic] burst and leak.” They were not noticing that a maintenance technician only worked on cleaning the base of a machine because they wrote down “clean base,” “base clean,” or “cleaned base.”

After we presented this information to the company's leaders, they realized they had the data to become “smart,” but they still could not perform the analysis. After we spoke with other manufacturers, both big and small, and with other experts across various industries from mining to heating, ventilation, and air conditioning (HVAC), we realized this problem of obtaining value from maintenance work orders (MWOs) was ubiquitous.

Enter NIST

First, we needed to find out why there were so many people describing the same thing so differently. We performed an experiment: We had multiple NIST manufacturing experts describe what happened in this highly viewed clip of a machine-shop mishap.

Credit: Gussepe

For the purposes of this discussion, check out run time 1:25 to 1:45. If you want to see more of what makes machinists cringe, check out the whole video.

How would you describe what is happening here? Does your version match the NIST experts?

Expert 1: The cutting tool snapped off. Need to replace tool and inspect spindle for damage. Looks like they were cutting too deep in one pass for the strength of the tool.

Expert 2: The depth of cut is too large and the feed too high for the slot, such that the forces increase until tool breakage as the tool approaches the vise. It probably wasn’t smart, either, to machine toward the vise as they have anyway. A typical approach to avoid this problem is to ramp into the slot.

Expert 3: All-around operator error. Looks to be too high a depth of cut at too high a feed-rate. Also looks like the move at the end put too high a stress on the tool. Operator should have retracted the tool before making that move if he/she wanted to keep that depth of cut.

Expert 4: Too large of an engagement at too high of a feed.

What do you notice here? Four very different descriptions! Some of their diagnoses were longer, some shorter, some with abbreviations, some with jargon, some with shorthand, others with full sentences, but still some commonalities between them. Now imagine describing this problem two hours, four hours, or eight hours later at the end of your shift as a technician. The descriptions might be even more different. This is the problem that maintenance technicians face on a day-to-day basis.

So how do we make it easier on the technicians while still finding the patterns between the descriptions? Let’s start by looking at the commonalities between the descriptions of the video.

Most said that the “tool is broken”; others said that the “depth of cut is too large,” or that the “feed rate is too high”; some describe the “bad process plan” and the overall “operator error.” Knowing that these commonalities exist, can we just force the technician to use a prepopulated problem code (for example, a drop-down menu)? Well, let’s try that. Which of these potential problem codes make the most sense for this video?
• Broken CNC*
• Poor planning
• CNC is broken
• Broken tool
• High depth of cut
• Error by operator
• Miscellaneous
• Too high of feed rate
• Operator error
• Broken machine tool
• Poor process planning
• Other
• Broken end mill
• Technician error
• M2 is broken
• Broken drill
• Mike!!!
• Milling machine down
• Problem code 083AM
• Power supply problem
• Broken drill tip
• Part planning is bad
• Design error
• N/A

It’s really hard to find the correct one, if there even is one. Now imagine you have to run to another machine failure and need to do this quickly. Most technicians just select “other” and write down what’s wrong because it is quicker, and they are getting paid to fix problems, not write down good data. This leads to the problems we described above. So instead of using a drop-down menu, what if we parse through the descriptions written down by the technicians using natural language processing (NLP)? Here, a computer interprets and organizes natural language, like all the various errors described above, and turns it into useful, actionable data.

Possible solutions

What would some solutions look like? Well, we could go through manually, work order by work order, and determine the cause, effect, and solution for each problem, as shown here.

Work Order Data Cause Effect Solution
HP coolant pressure at 75 psi. Bad gauge/low-pressure lines cleaned ou [sic] Broken gauge Low coolant pressure Cleaned out low-pressure lines

Although this certainly brings more structure to the data, it is still time-consuming and can be inconsistent. If the same person sees “bad gauge” on the thousandth work order, will he remember to put “broken gauge”? Adding to this, it takes a lot of time and cognitive effort to think of which portion of the original data is the cause and which portion is the effect. (Was “broken gauge” the cause or effect in this scenario?) In fact, when one person tried to annotate the data, annotating 800 work orders took 12 hours!

We could also potentially use this manually annotated “cause, effect, and solution” as training data for a machine learning model. Although this certainly is possible, it would require much more than 800 annotated work orders, which would take more annotators, more time, and might be impossible depending on your problem and budget.

Now, one of the problems in the above method is the amount of effort it takes to determine the cause, effect, and solution. In the example above, is the “bad gauge” the cause or effect? Figuring this out is difficult for an annotator, so what if we changed what we wanted the clean data to look like? What if instead of identifying the cause, effect, and solution, the annotator simply identifies words as “items,” “problems” and “solutions”? This “tagging” paradigm is simpler for an annotator because most words in the MWOs are mutually independent, e.g. “replace” is almost always a “solution,” whereas “gear” is almost always an “item.”

So, what does this look like?

Work Order Data Items Problems Solutions
HP coolant pressure at 75 psi. Bad gauge/low-pressure lines cleaned ou (sic) High-pressure coolant, gauge, low-pressure lines Broken, low pressure Cleaned out

Using this process, one person manually annotated 1,200 MWOs in 12 hours: a marginal improvement.

Because manually annotating one work order at a time takes an extraordinary amount of time and effort, what if instead we leveraged natural language processing to analyze the entire set of MWOs at once? NLP methods can extract and rank all the words, which we call “tokens,” from the raw text throughout all of the MWOs, and then present them to a human annotator to classify into items, problems, and solutions (we also include “irrelevant” and “unknown” classifications). This approach takes advantage of both the strengths of NLP and human expertise.

So, what does this look like? There are six steps to the process.

1. Extract tokens from maintenance work orders (MWOs)
First, we start with the raw data—in this example, the data have a description and resolution field: 

2. Rank tokens
Next, we look at the tokens across all the work orders. These tokens are automatically ranked in order of importance. As you can see, the tokens include misspellings, abbreviations, and jargon. This is because the tokens are pulled from the raw text.

3. Select similar words
Once these words are given to a user in order, the user is presented with similar words from other inputted work orders. The user can then analyze words of similar spelling. This step takes advantage of the computer’s ability to quickly identify potential similar words, but also uses a human’s ability to understand and determine what words are actually the same.

In this example, “rep” might mean “repaired” or “replace”; thus, the user selects only “repaired = reapirs = repair = repir = reparis = repaird.”

4. Create preferred alias
After determining similar words, the user can now determine the proper “alias” for the term, in this example, it’s “repaired”—and the “classification” of that word.

5. Determine word type
For the classification, we use “items,” “problem actions,” and “solution actions,” as well as “irrelevant” (terms that are not important for analysis, e.g., “this,” “that”) and “ambiguous” (terms that are unclear with only one word, e.g. “hot” could mean “too hot,” which is a problem, or “hot water,” which is an item). This allows the user to classify a token as one of those five classes. In this example, “repaired” is a “solution” (indicated by the green “S” in the figure).

The user can then perform this same process for more tokens. As opposed to a drop-down menu with a limited set of terms, the advantage of this system is the ability of the user to perform this process on as many tokens as she wants. Since the tokens are ranked in terms of importance, the user can annotate a lot of work orders quickly because words like “repaired” occur much more frequently than words like “asset145.” This allows for a large return on investment: A small amount of time classifying tokens leads to a large amount of annotated work orders.

6. Automatically annotate MWOs
Once this process is completed, each individual MWO is automatically tagged with the words that appear in each work order, according to their classification and alias.

Now the work orders will have the “items,” “problem actions,” and “solution actions,” that occur in each work order. The “irrelevant” terms and “unknown” terms will be recorded as well.

Using this methodology on the same data set as described above, a user spent 45 minutes merging similar words, classifying those words, and creating aliases. Using the same dataset in which it took 12 hours to fully manually tag 1,200 MWOs, this system allowed for 3,300 MWOs to be fully tagged. Out of the remaining 2,200, 2,000 had some tags associated with them. A huge improvement from the other methods described!

To help manufacturers utilize this method, the NIST team developed a free, open-source application called Nestor. We have been working with manufacturers, both large and small, to pilot this method, perform case studies, and create reproducible decision workflows.

We also recently held a workshop where we had more than 25 participants from industry, government, and academia, to discuss the current trends, successes, challenges, and needs with respect to natural language document analysis for decision support in manufacturing, and to further the work of a critical manufacturing standards community to produce publicly available guidelines for industry. We are actively working on a report documenting the results, which will be out in 2020.

Lastly, we produced a state-of-the-art paper for both the ASME Manufacturing Science and Engineering Conference (MSEC) and ASME Journal of Manufacturing Science and Engineering (JMSE) to tackle the problem of where to start a smart manufacturing journey. The paper, titled “Where do we start? Guidance for technology implementation in maintenance management for manufacturing” is available on the NIST website.

Next steps

Further pilots with industry to create reproducible case studies are needed to allow manufacturers to implement this method in their decision workflows for analysis of their work orders.

Reference datasets need to be produced to allow researchers to test and validate their work on a common dataset. Some work has been done by the University of Western Australia with mining excavator data, but more is needed. Following the success of the workshop, there is a lot of buzz about creating a community of interest around this work to further the field of NLP in the maintenance community. This community of interest will be started in FY20.  During the interim, some of this work is being investigated by the ASME Standards Subcommittee on Advanced Monitoring, Diagnostics, and Prognostics for Manufacturing Operations.

We hope that this work and the work of other researchers can help manufacturers and other maintainers improve their decision-making process to further help them get “smart.” If you have interest in any of these activities, please leave a comment!

First published Oct. 4, 2019, on NIST’s Taking Measure blog.

Discuss

About The Author

Michael Brundage’s picture

Michael Brundage

Michael Brundage is an industrial engineer in the Information Modeling and Testing Group. He serves as the project leader for the Knowledge Extraction and Application for Manufacturing Operations project in the Model-Based Enterprise Program. Michael also has a personal interest in augmented reality and even proposed to his now fiancé using an augmented reality application that he created.