3rd European

3rd European-American Workshop - Meetings & Breakout Session results

Reliability Working Group meeting September 9, 2002

Reliability committee working group meeting notes:

Meeting opened at 10am

Attendees:

Damir Marcocic– Croatia

Background:

Head of NDE lab

R&R focus within NDE

Working group V – European Joint Research – Focus mine detection (40 members ~15 active) human factors focus

Transfer standardization lessons learned to Mine detection

Christina Mueller – Head of NDE Reliability lab BAM, Focus on NDE and detection reliability since 1986

Ripi Singh – Program manager, Pratt Whitney

– Inspection reliability for USAF, 15 years reasearch. Structural integrity and reliability. Focus on maintainability and inspection program.

Lloyd Schaefer

Honeywell – Principal Engineer Global Repair Development Engineering. Focus on NDE systems development and reliability analysis in support of the life prediction effort for repairs of turbines

Sharon Vukelich – Head of efforts at USAF engine structural integrity program, related to NDE. 20 years experience, with 8 years GE engines. Author MIL-STD 1823 for reliability.

Peter Wilrich

Prof. Emer Frei Univ Berlin – Statistician – EE PhD – Tech Univ Hamburg

Measurement uncertainty. Chair ISO standards for interlab testing (esp precision) 5725 measurement reliability, also quantitative reliability. Looking at both demining and NDE. Capability 11843 ISO detection capability.

Perspectives on resources and actions within industrial and professional societies for statistical analysis of NDE and related processes:

Perspectives on use of statistical methods within industries and professional societies in Europe and America

Peter provided some perspectives on his experiences with the use and support of statistical methods within German and American industries:

ASQ stronger than Euro Quality society, with only ASQ possessing a statistical division.

6 Sigma just taking hold in EuroQuality and some industries, eg. Siemens

Peter shared his experiences with development/management of ISO interlab measurement reliability standards. A look ahead at an appropriate capability/detection standard targeted to demining activities, eg ISO 11843

Possible sources of support for creation of standards were discussed

DIN dropped interest in a measurement standard and the perception is that DIN will only get involved where there is strong industry backing.

The group discussed the need to determine currently what involvement/support is possible, and to establish a "pull" oriented feedback group, it was decided to create an action where AE reliability group members at this meeting would contact risk, lifing, and other reliability customer data groups to conduct a Voice of the Customer survey. This feedback would then be reflected on the web to motivate producers of NDE and detection reliability data, and also top-down industrial and government support for the efforts.

ASNT & DGZFP Reliability Committee status

Schaefer inquired as to the current level of effort and prognosis for the ASNT reliability committee effort – Sharon Vukelich and Christina Mueller provided status. There is a recognition of need to address it. 10 years ago, it was rejected as a "Standards" activity, and had perhaps objectionable implications for the membership and existing certification programs. The initial response to this posture of 10 years ago was that ASTM created a performance demonstration section related to its specific needs. Ripi Singh suggested that to avoid this problem that the primary influence be retained on testing system reliability, and muting the focus on individual inspector certification. While inherently problematic, such as in penetrant inspection, qualifying the inspector group capability as input to the design or lifing customer is a feasible work around.

All agreed that the focus should be on testing the "process" vs the individual.

Within ASNT board, both past president Doggert, and also Mike Turnbow supported the concept of a committee for NDE reliability.

To date ASNT committee is active, but have held only one meeting

With support of a neutrally located and focused NDE reliability web, knowledge of specimen and POD assessment resources and results will help germinate essentials which will lead towards understanding of global reliability by process, and identification of Critical to Quality elements which influence the results.

DGZfP had terminated its reliability committee, however, current president Jorge Volker will re-form this committee and champion the importance of acquiring and delivering this knowledge to customers.

Creation of white paper to capture Voice of the Customer for Reliability and Detection products

The reliability committee agreed that it should proceed to create a voice of the customer white paper which will guide further activities. Members will solicit input from

ASIP
ENSIP – Vince Spinnel – Structures, but with lower NDE focus. Spanel (vince)
Croatian register of shipping – (damir markovich),
Lloyd’s of London and additional insurance, risk management companies.
Mine action centers
Reliability doorway for demining through CEN-BT126 (working group) CWA7 (mine detectors)
- Accepted modified performance demonstration requirement

ECNDT –

The white paper categories – VOC, end users, will focus on:
Petrochem
Nuclear
Commercial AF
Military AF
Commercial Engines
Military engines
Insurance/risk management
NATO – Ward Rummel, Floyd Spencer, CNRC – David Forsyth

Potential for global assessment standards

The potential for a standard, globally, for reliability assessment design was discussed. Sharon Vukelich related past attempts which ended (AGARD) due to lack of agreement on experimental design. The PIE approach recommended by Ripi was then adopted as the best near term approach to allow practitioners to input the elements of their studies, and for the committee to evaluate what distills out of this process. There is clear value in measuring and understanding the variability of NDE process elements, and it is fundamentally valuable for reliability confidence when detection variables are in control, despite imperfect knowledge of absolute POD.

Structure of Partner Information Exchange:

The structure of the PIE will be:

PIE- Platform for information exchange

Publish results and their basis

Criteria and comments clearing section for each input

Additional emphasis on collecting and providing miss and false call distributions

Ultimately an internal standard will emerge from the compiled results

Adoption of the optimal procedure will distill on its own

Airport security reliability

To expand the mine detection portion of the committee area of interest, the issues related to weapons and explosives detection on persons and luggage/cargo will be examined. Lloyd Schaefer took an action to determine existing quality/reliability procedures for these processes and report back to the committee.

Development of failure and flaw distribution knowledge:

To understand better what specific targets of detection distributions should be, it is agreed that knowledge of especially failure related undetected, or detected and accepted flaws consist of. Sharon Vukelich indicated she will publish and distribute a database of all Aero rotor failure incidents through 2002 by the end of the year. Breakdown of the information will be per operator and engine/model.

Need for blind trials:

High human factors NDE and detection tasks have stronger needs for blind trials than majority automated systems. Examples, mine detection, airport security and FPI/MPI NDE tests. How to account for the known test factor, and how to effect blind trials and deliver the inspection planning information is of interest. To date, the use of backprojection techniques, such as the SIRS database enables the generation of "miss" distributions, Lloyd Schaefer reported. It was pointed out that when we have such information, it may not be longer necessary to conduct specific NDE reliability demonstrations. Lloyd Schaefer will write a report to the group on the nature and measurement of variables in the SIRS approach. It was agreed that blind trials are more readily possible for the mine and weapon detection reliability efforts, and the adequacy of these procedures will be evaluated when the existing quality procedures are obtained. Christina Mueller will provide procedure inputs for the demining procedures.

Actions and meeting closure:

Identified actions by the committee members will be carried out, and a status report compiled in 60 days. Meeting was adjourned at 2pm.

Session 1 - Creating and applying POD, ROC to risk and life management needs - Wednesday September 11th

Actual problems in creating and applying POD, ROC to risk and life management needs.

a) Test sample design

b) Finding the right POD/CL for the application

c) Optimizing assessment design

Main conclusion:

We need a catalogue of "risk based" POD curves

Session 2 - The specifics of mine searching and the possibility to adopt NDE and NDE reliability methods

3rd European-American Workshop on NDE Reliability

Breakout Session 2

Sept 12. 2002, Berlin

The specifics of mine searching and the possibility to adopt NDE and NDE reliability

EXECUTIVE SUMMARY

The goal of the Workshop was to:

Accelerate the collaboration and information exchange within the humanitarian demining community (HDC), and across HD and NDT communities.
Advance the state of the knowledge by recognizing and tackling the highest priority issues.
Agree on the action plan / next steps to address the issues.

Add 1.

The WS included presentations by experts from Europe and North America addressing issues ranging from demining strategy and performance evaluation of modeling and field testing to actual field results of reliable mine clearance approaches.

The presenters shared their findings in field testing, demonstrated promising new methodologies, and provided a preview of new technology that will advance the reliability and speed of APMD and clearance and lower the cost per cleared area.

Add 2.

Additionally, participants had an opportunity to identify the highest priority issues in the demining activities area, and discuss the issues in a breakout session.

The two issues that surfaced during the session were:

Lack of a clear and globally accepted definition of quality requirements, success measure that would enable / drive common quality standards for:
- testing of equipment and methods
- modeling
- acquiring relevant data
- performing field operations

The above issues result in a lowered ability to compare testing, modeling and field results and therefore xxxxxx the ability to predict the success for various new methods and technologies.

Inadequate training and education of needed cadre across the spectrum of demining activities ranging from project management, organization, logistics, R&D, reliability testing, quality control to field activities.

The issue results in the inability to xxxxxxxx done successful models, average existing knowledge and provide enough capacity, capability to meet the demining goals set by the international community.

Add 3.

In the breakout session the participants agreed on several act...... items / immediate next steps that would ......... in an environment to successfully tackle.

Both issues:

establish communication network channel to continue discussing on quality requirements and communicate the findings to the wider demining community to accelerate advancement and avoid redundant work
expend the network to include other relevant stakeholders and test / confirm the identified categories:
- reliability, speed, safety and cost per cleared area as quality measures
follow through on the Croatian team proposal leverage on their success and internationally available body of knowledge / state-of-the-art , to establish a training and education center Mine Action Academy – MAA – and help meet the set capability and capacity needs.
ITEP supports for the MAA proposal beyond the WS participants, in particular invite Prof. V.Krstelj to next meeting in Geneva, "Standing Committee on mine clearance, mine awareness and mine action technologies", and other relevant stakeholders to initiate and accelerate the formation of MAA.

Conclusions:

Main conclusion:

We should apply the NDE reliability formula for a deep understanding and improvement of the demining process: better educations (MAA) and performance demonstration for systems in the field.

Thursday, Sept. 12 Friday, Sept. 13

Session 3 - The Modular Model and Reliability Formula for NDE and Demining

Breakout Session Summary

The Modular Model and Reliability Formula for NDE and Demining

Chairmen:

Philippe Benoist

Jay Fisher

The modular model as defined at a previous workshop is

where

and

At the beginning of the session, Christina Mueller summarized the modular model for the attendees.

There was a general consensus that the modular model has not been used mathematically. It was pointed out that the model was designed more as a way to understand factors that contribute to reliability, rather than to provide a mathematical approach to calculate inspection reliability.

Some participants thought that some quantitative use of the model might be possible. In particular, some claimed that IC and AP could be separated; others that HF could be separated from the other factors.

It was pointed out that different industries have developed different approaches to ensuring sufficient inspection reliability. In particular, the U.S. aerospace industry has embraced development of Probability of Detection (POD) curves, which essentially provide an overall measurement of inspection reliability. However, the U.S. nuclear industry has developed a performance demonstration that requires vendors to demonstrate a minimum level of detection and characterization on blind specimens, and only secondarily determines a quantitative measurement of inspection reliability. The European nuclear industry also has a qualification approach, but qualifies techniques and personnel separately, and uses demonstrations rather than blind tests. It was not clear that lessons from these industries could be easily applied to demining, which has a larger cost to less than 100% reliability and a large problem with false positives.

It was noted that existing published POD data are difficult to use: there is often not enough information to be able to judge how to apply existing data to a new situation. Christina Mueller suggested that a web site be created to collect data and standards on reliability. There was some discussion that it would be useful to develop common ways to describe terms that are used in describing reliability, so that reliability data could be easily understood and compared by people who did not generate the data.

Session 4 - How shall we combine calibration, reproducibility, and repeatability to POD/ROC investigations and the Modular Model

3rd European-American Workshop on NDE Reliability

Breakout Session 4

Sept 13. 2002, Berlin

How to combine calibration / R&R to POD/ROC investigations
and Modular model

EXECUTIVE SUMMARY

Mike Anderson,

rely on performance demonstration.

what happens with calibration ? important is to detect.

until operator finds a crack and size it – no problem with calibration !

instruments are digitally controlled, self-calibration... algorithm

separate between manual and automatic calibration procedures

calibration ?!? (Bob)

traceability to reference standard
i.e. amplitude and time domain

Bob,

standardization of procedure

POD can be used as a tool for process control

U. Ewert,

in RRT one have to describe how you collect data, formulae, put data, without list.

it is noticed that even in that case people do not obey instructions – because there was no computer program which could make multiple same calculations !

we need agreement on the tools

K. Osterloh,

one cannot compare crack with the mine :)

L. Schaefer,

standardization

bookkeeping of former results to trace information about calibrated thresholds

H. Stephens,

platform for comparison of results, design of standards with "typical" flaws

short term need:

couple of choices how to collect data

D. Markucic,

with each number extraction or even a single curve (POD) we loose quality of data

calibration "instruction" is a part of data description

Ripi Singh,

data are not portable, case stories !

information of knowledge

H. Stephens,

Lack of education (knowledge) for people outside NDE community with which we work/communicate. Need to move beyond POD = badness (Lloyd). Articulate the message to our NDT societies about this topics.

Main conclusion:

Traceability of results; Standardization of procedure; POD for process control (6 sigma)

Session 5 - Summary & Consensus

Summary & Consensus.

Breakout Session 5

(R.Singh, M.Anderson, C.Mueller)

presentations handle risk and reliability information
we have heard success stories
quantitative estimate did not come out from the sessions
output information of reliability have to be feedback for the risk
closer relationship with NDE and (probabilistic) fracture mechanics
challenge to asses risk in demining
risk in demining is when you activate mine (due to the most accidents of the pyrotechnicians), and in NDE when you miss a crack.

(A.Dean, K.Osterloh, J.O.Robertz, C.Mueller)

Breakout session put accent on the Mine Action Academy (MAA).
It is a good marriage (demining and NDE fields).
In general we do not come at WS with enough prepared problems for discussion.
To many acronyms which confuse during following the presentations.
deminers are living with false calls – metal detector is metal detector, not a mine detector
two issues:
detection, test and evaluation (3 phases)
clearance which is followed with risk assessment (combination of performance of two different systems)

(J.Fisher)

reliability formulae is not a mathematical expression >>> a guideline
three approaches for POD development (aerospace, US and European nuclear and power plants)

Main conclusion:

We need to quantify the risk in NDE and demining !

CLICK BELOW FOR BREAKOUT SESSIONS CONCLUSIONS

[ fuselage_ec_results.htm ] [ resume.htm ] [ LASchaeferCV.htm ] [ Birdmail.htm ]