Recommendation E.880 FIELD DATA COLLECTION AND EVALUATION ON THE PERFORMANCE OF EQUIPMENT, NETWORKS AND SERVICES 1 Introduction This Recommendation provides guidelines for the collection of field data relating to dependability. It covers general aspects with an overview of sources, measures and information that may be involved when collecting field data. It is anticipated that specific practical needs of operation, maintenance and planning staff, in applying these guidelines, will be dealt with in a handbook under preparation. The Recommendation emphasizes that meaningful data must include the data on successes (operation without failures) as well as data on failures and faults. In other words, this Recommendation is not intended to be only a failure reporting guideline. It is applicable, without any restriction, to different items ranging from components to systems and networks (including hardware, software and people). Terms and definitions used are in line with Recommendation E.800. 2 Scope It is the intention of this Recommendation to provide guidelines for setting up data collection and reporting schemes which can be applied either during monitoring of samples of items or on a more widespread basis on almost all items (of the same type) by large operational and maintenance organizations. It is considered that, if such guidelines are followed, accuracy and completeness of reporting are enforced and the quality of the monitored items and their parts can be improved on a medium- to long-term basis. Moreover, such an effort will facilitate the interchange of information between user and providers. No recommendations are made on how to organize maintenance support. It is nevertheless understood that some items are repaired on site, others only replaced on site and possibly repaired at centralized facilities. Field data may be obtained at each of those stages. In order to obtain maximum efficiency from the collection of data, it is suggested that the programmes of reporting, analysis and dissemination of results be closely co-ordinated. The items considered may either have been designed, manufactured, or installed and may be operated by the same organization or by different organizations. This Recommendation applies to all possible cases of provider-user relations. 3 The need for data collection Any data collection scheme must aim to provide the information required to enable the correct decisions to be taken in order to reach specified objectives; these objectives should be well defined and documented at the outset. Fascicle II.3 - Rec. E.880 PAGE1 The specific objectives of the field data collection and presentation are as follows: a) to provide for a survey of the actual performance level of the items monitored for information to management, operation and planning, maintenance support, training of personnel, etc.; b) to indicate a possible need for the improvement of: - items already installed and in operation, or - further items to be delivered; c) to compare the specified or predicted characteristics of the item(s) with the actual field performance; d) to improve future designs; e) to improve predictions (data bases and procedures); f) to inform the provider about the performance of items on a regular or on a single occasion basis; g) to have a common reporting basis. 4 Sources and means of data collection In the following, the various information sources are described and the methods for systematically collecting information are outlined. 4.1 Sources of data The following sources of data may generally be available: - maintenance activities; - repair activities (on site, repair and/or complaint centre); - performance observation activities (e.g. anomaly reports, traffic measurements); - existing information (e.g. stocklist, installation list, modifications, a regularly updated data base for configuration control purposes). 4.2 Means of collecting data It is not intended to recommend any particular format for the recording medium (e.g. paper based or computer data base), however it should be recognized that early consideration of the format is necessary and important in setting up an effective data collection scheme and also aids subsequent successful processing. Frequently the recording of data will be by manual means but automated and interactive data collection systems may be also considered. The advantages to be gained from holding data in a form suitable for processing by an electronic data processing system include easy and accurate updating of information and the possibility of performing new extended analyses. Data may be collected by one or several of the following reporting means. 4.2.1 Operation reporting Data reporting should be supported by information on the use of the items. Where systems are in operation for the reporting of all failures, it is necessary to collect data on the use of the whole population of items (the total number of similar items under observation). 4.2.2 Failure reporting At any level, failure reporting is dependent on the fault coverage test resources used at the considered level: cases such as "fault not found" or "right when tested" should be clearly mentioned. Failure reporting should cover all failures that have been observed. They should also contain sufficient information to identify failures. Failures considered to be attributable to any maintenance action should be so noted. The failure reporting should be sufficiently comprehensive to cover the requirements of detailed investigation of an individual failure and the resulting fault. Where economic reasons or lack of resources make it undesirable to collect all of the failure data indicated, it may be desirable to agree upon a shortened form of report which can be used to collect limited data on all relevant failures, with an option to call for the full report in specific cases. 4.2.3 Maintenance reporting The maintenance report should contain all information relevant to the manual or automatic action taken to restore the item. When there is need to distinguish between corrective maintenance and preventive maintenance reporting, if no replacements or repairs are made, the action can be classified as a preventive maintenance report. If a preventive maintenance action results in a replacement or repair, the report may be treated as a corrective maintenance report even though the item has in fact not failed in operation. PAGE4 Fascicle II.3 - Rec. E.880 4.3 Storage, updating and checking procedures Independently of the structure chosen for the data storage, data should be checked at the time of input so as to ensure validity. It is evident that every data bank needs an in-depth study appropriate to its specific requirements, in order to define the most suitable method of data checking, error correction, and updating. 5 List of dependability measures The selection of the data to be collected is very dependent on the kind of performance measures to be evaluated/estimated. Field data reporting may have to be limited by economic necessity to the minimum necessary to meet the requirement, whilst recognizing that collection systems should be capable of future expansion. It is likely that certain data may be needed for more than one purpose, and careful consideration can therefore result in the most cost-effective data collection scheme. The dependability measures that might be taken into consideration are listed as follows. 5.1 Reliability performance Failure rate Failure intensity Replacement intensity Mean operating time between failures [.]1) Up time. 5.2 Maintainability performance 5.2.1 Time related performance [.]1) Down time [.]1) Technical delay [.]1) Fault localization time [.]1) Fault correction time [.]1) Restart time [.]1) Checkout time [.]1) Repair time [.]1) Active corrective maintenance time. 5.2.2 Probabilities Probability of fault coverage Probability of false alarm Probability of fault nondetection Probability of alarm detection Probability of a failure being localized within a given number of replaceable units. 5.3 Maintenance support performance 5.3.1 Time related performance [.]2) Logistic time [.]2) Administrative delay. 5.3.2 Probabilities Spare parts shortage probability Test resource shortage probability Human resource shortage probability. 1) [.] indicates according to specific applications a mean value or a fractile. 2) [.] indicates according to specific applications a mean value or a fractile. Fascicle II.3 - Rec. E.880 PAGE1 5.4 Availability performance Steady state availability [.]2) Accumulated down time. 6 Data required Consideration of the foregoing objectives defines the need for a system which provides for the collection of documented data covering: a) the identity of items or population of items under observation; b) operational conditions; c) maintenance support conditions; d) performance monitoring. For each individual item, sufficient information has to be recorded to clearly identify the item itself and its operating environment. Depending on the item under consideration (e.g. equipment, printed circuit board, component, personnel), and on the depth and kind of analysis of collected data, the necessary item identification data shall be used, on a case by case basis. The item identification should also allow the analysis of the relationships between the items for which data is collected. In relation to the particular analysis to be done, some items may be considered as equivalent, therefore separate small items need not to be collected in such cases. The following information may be needed and could be collected or will be available from existing sources: - type of item - manufacture/provider - item configuration - individual No. or serial No. - date of manufacture - supplier - delivery date - installer (company) - installation date - customer (name) - site (geographic) - system. Consideration should be given to possible limitations due to non-completeness of collected data or possible difficulties in data collection or particular assumptions made for the collection itself. The choice of the kinds of data to be collected and the design of the related collection procedure depend on many factors, some of which are: - the required end-result; - the diversity of components or systems; - the duration of the data collection project; - the data handling method (manual or computer based); - a sufficient knowledge of the capability to collect the required quantity of information and the accessibility to data to be gathered. 6.1 Number of items to be considered The number of items to be considered depends mainly on the characteristics to be dealt with, the statistical aspect of the evaluation to be made and the cost involved. 6.2 Information on items being considered 6.2.1 Operating conditions 6.2.1.1 Environment classes a) Fixed (outdoors, indoors, underground, undersea, off-shore, etc.); PAGE4 Fascicle II.3 - Rec. E.880 b) Portable (item specially built for easy transportation by one man only); c) Mobile (in motor vehicle, in ship, in aircraft); d) Other (specify). 6.2.1.2 Specific environment data a) Climatic conditions - weather-protected, - not weather protected, - air temperature, - air pressure, - humidity; b) Electrical environment (EMC); c) Mechanical conditions (vibration, shocks, bumps); d) Mechanically active substances (sand, dust, etc.); e) Chemically active substances; f) Biological conditions. 6.2.1.3 Mode of operation a) Continuous; b) Intermittent (give cycle); c) Stand-by; d) Single operation (e.g. one shot devices); e) Storage. 6.2.1.4 Load conditions a) Overload; b) Other (specified). 6.3 Failure and fault description - Fault recognition: symptoms and indications, fault detected, fault not detected, false alarm. - Item fault mode (identification of functions affected). - Failure causes: a) Inherent to item under observation; b) Misuse failure; c) Induced by maintenance or administrative action; d) External to item under observation; e) Secondary (caused by related item); Fascicle II.3 - Rec. E.880 PAGE1 f) Other. In cases where the failure immediately follows a period of transport, storage or stand-by, the relevant conditions shall be stated. - Fault consequences - List (identification) and physical location of faulty replaced parts: a) quantity of suspected replaceable items; b) quantity of replaced items. - Fault evidence and documentation (printouts, photograph, etc.). - Action taken: Replacement, repair, adjustment, modification, lubrication, etc. - Active maintenance time (diagnostic + repair + tests + . . .). - Downtime, including, where applicable: - undetected fault time, - fault localization time, - reconfiguration time3), - technical delay, - logistic delay, - administrative delay, - fault correction time, - checkout time, - restart time. 6.4 Maintenance support data: - spare resources shortage, - test resources shortage, - resources shortage. 7 Data presentation for evaluation When collected data is offered for subsequent evaluation by using approximate statistical methods, all conditions for their correct use and understanding should be clearly stated. These conditions should encompass the purpose of the data gathering especially with respect to type and variation of the data chosen. Information on the circumstances should also be provided such as when (e.g. busy hour conditions), where (e.g. geographic considerations) and for how long the collection took place. Specific situations, which may limit the data application and use, should be indicated, e.g. difficulties encountered, particular assumptions made, non-completeness. Considerations should also be given to the form of presentation: where appropriate, a condensed form (e.g. diagrams, histograms) may prevail over a detailed raw data presentation. 8 Statistical methods for data treatment In most cases the need for data treatment appears in connection with one of the following activities: - estimation, - compliance evaluation, - monitoring of performances, - comparison of performances. For each performance of interest, estimations, hypothesis tests, control charts and comparison techniques are used for evaluating. The application of a given statistical procedure usually requires the fulfilment of some general conditions and assumptions which have to be carefully investigated. Some of these preliminary investigations relate directly to the properties and the characteristics of the (stochastic) process generating the collected data, some other relate to the distribution underlying the collected data. Both preliminary investigations and data treatment may require statistical procedures not dealt with in this Recommendation. International organizations other than CCITT, e.g. IEC, have produced valuable material in this field [1]. Reference 3) Time required for automatic reconfiguration (if manual operations are needed, they are integrated into technical delay). PAGE4 Fascicle II.3 - Rec. E.880 [1] International Electrotechnical Commission - Catalogue of Publications, Ed. 1987. Fascicle II.3 - Rec. E.880 PAGE1