Stable Implementation Agreements for Open Systems Interconnection Protocols: Part 24 - Conformance Testing Output from the December 1993 Open Systems Environment Implementors' Workshop (OIW) SIG Chair: Eva Kuiper, Hewlett Packard Workshop Editor: Brenda Gray, NIST PART 24 - Conformance Testing December 1993 (Stable) Foreword This part of the Stable Implementation Agreements was prepared by the Conformance Testing Special Interest Group (CTSIG) of the Open Systems Environment Implementors' Workshop (OIW). Text in this part has been approved by the Plenary of the above- mentioned Workshop. Future changes and additions to this version of these Implementor Agreements will be published as a new part. Deleted and replaced text will be shown as struck. New and replacement text will be shown as shaded. ii PART 24 - Conformance Testing December 1993 (Stable) Table of Contents Part 24 - Conformance Testing . . . . . . . . . . . . . . . . 1 0 Introduction . . . . . . . . . . . . . . . . . . . . . . 1 1 Scope . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 Normative References . . . . . . . . . . . . . . . . . . 1 3 Status . . . . . . . . . . . . . . . . . . . . . . . . . 1 4 Errata . . . . . . . . . . . . . . . . . . . . . . . . . 1 5 Guidelines on Interpretation of Disputed Test Cases . . . 1 6 Guidelines on the Choice of PICS . . . . . . . . . . . . 1 7 CT SIG Resolution for FTAM . . . . . . . . . . . . . . . 2 8 Guidelines for PCTR Test Campaign Summary . . . . . . . 2 iii PART 24 - Conformance Testing December 1993 (Stable) List of Tables Table 1 - Guidelines for the use of the "Selected" and "Not Run" columns in a PCTR . . . . . . . . . . . . . . . . . 3 iv Part 24 - Conformance Testing 0 Introduction (Refer to Working Implementation Agreements Document) 1 Scope (Refer to Working Implementation Agreements Document) 2 Normative References (Refer to Working Implementation Agreements Document) 3 Status This material is current as of December 18, 1992. 4 Errata Errata will be reflected in replacement pages of Version 6, Stable Document. 5 Guidelines on Interpretation of Disputed Test Cases (Refer to Working Implementation Agreements Document) 6 Guidelines on the Choice of PICS (Refer to Working Implementation Agreements Document) 1 PART 24 - Conformance Testing December 1993 (Stable) 7 CT SIG Resolution for FTAM (Refer to Working Implementation Agreements Document) 8 Guidelines for PCTR Test Campaign Summary The following table provides guidelines for handling the "selected", "not run" and "observations" columns in a Protocol Conformance Test Report. In the following table, the "criterion" column in taken with the "test case status" column to give the expected contents of the three PCTR columns. Note that this table does not contain all possible permutations of PICS support answer, IUT behavior, and test case status. The primary focus of this table is to provide PCTR guidelines for the permutations of "test case status". 2 PART 24 - Conformance Testing December 1993 (Stable) Table 1 - Guidelines for the use of the "Selected" and "Not Run" columns in a PCTR Criterion PCTR PCTR PCTR selected not run observat column column ion column I. TEST DESELECTED 1. feature not deselect implemented (f) 2. feature not deselect applicable (f) II. TEST SELECTED; TEST PURPOSE NOT ACHIEVED 1. ATS defect; no select not run ATS error (g) error in standard (a,c) 2. ATS defect; error select not run ATS or (g) error ambiguity in (a,c) standard 3. ETS defect select not run ETS (g) error (a,b,c) 4. No defect; select run manual Inconclusive verdict analysis (c,d) III. TEST SELECTED; TEST PURPOSE ACHIEVED 1. Test is defect- select run free (e) 2. ETS/ATS defect; select run manual workaround analysis available alternat e verdict (b,c,d) 3 PART 24 - Conformance Testing December 1993 (Stable) See Notes listed below: a) This criteria includes any test where the test purpose cannot be achieved by an IUT exhibiting valid behavior. b) This criteria includes any test where the ETC attempts to accomplish the test purpose in an overly restrictive way, resulting in an Inconclusive test case verdict, even though the IUT exhibits valid behavior. Manual analysis instructions are provided by the MOT supplier for such tests. These instructions are used by the test lab to determine whether the test purpose can be achieved. If the analysis shows that the test purpose cannot be achieved, the PCTR indicates this test as "Not Run". If the analysis shows that the test purpose can be achieved, the PCTR indicates this test as "Run". c) The Observation must include the MOT supplier's Defect Report number or a reference to official MOT documentation (such as the release notes or test specifications) or corrective action. If the MOT supplier does not agree that the test case is defective, then the observation must include a standard and/or profile justification. If the defect is IUT specific, the observation must include a test3-specific justification, or a reference to one. The observation must fully and adequately explain why the IUT's behavior is valid. d) The Observation must include a test-specific justification of verdict. Analysis of IUT behavior reveals no evidence of non-conformance and no know impact on interworking. Manual analysis also reveals no evidence of ATC/ETC defect. e) An observation is needed only if the test case verdict in not Pass. f) It is recommended that the observation column contain a reference to the test case selection expression used to de-select the test case. If the observation column is empty for a particular item, which has been de-selected, the default meaning is that the item was de-selected based on the PICS. If the test is deselected for PIXIT reasons, the reference is mandatory. g) The term "not run" as used here is not the normal English usage of the term. See ISO/IEC 9646-5 for the meaning of the "run" column of the PCTR. When "not run" appears in the run column of the PCTR it indicates a situation in which there is an ATS or ETS error. 4 PART 24 - Conformance Testing December 1993 (Stable) 5