INTERNATIONAL TELECOMMUNICATION UNION CCITT X.294 THE INTERNATIONAL TELEGRAPH AND TELEPHONE CONSULTATIVE COMMITTEE DATA COMMUNICATION NETWORKS OPEN SYSTEMS INTERCONNECTION (OSI) PROTOCOL SPECIFICATIONS, CONFORMANCE TESTING OSI CONFORMANCE TESTING METHODOLOGY AND FRAMEWORK FOR PROTOCOL RECOMMENDATIONS FOR CCITT APPLICATIONS – REQUIREMENTS ON TEST LABORATORIES AND CLIENTS FOR THE CONFORMANCE ASSESSMENT PROCESS Recommendation X.294 Geneva, 1992 Printed in Switzerland FOREWORD The CCITT (the International Telegraph and Telephone Consultative Committee) is a permanent organ of the International Telecommunication Union (ITU). CCITT is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The Plenary Assembly of CCITT which meets every four years, establishes the topics for study and approves Recommendations prepared by its Study Groups. The approval of Recommendations by the members of CCITT between Plenary Assemblies is covered by the procedure laid down in CCITT Resolution No. 2 (Melbourne, 1988). Recommendation X.294 was prepared by Study Group VII and was approved under the Resolution No. 2 procedure on the 17th of January 1992. ___________________ CCITT NOTE In this Recommendation, the expression “Administration” is used for conciseness to indicate both a telecommunication Administration and a recognized private operating agency. γ ITU 1992 All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, without permission in writing from the ITU. PAGE BLANCHE Recommendation X.294 Recommendation X.294 OSI CONFORMANCE TESTING METHODOLOGY AND FRAMEWORK FOR PROTOCOL RECOMMENDATIONS FOR CCITT APPLICATIONS – REQUIREMENTS ON TEST LABORATORIES AND CLIENTS FOR THE CONFORMANCE ASSESSMENT PROCESS1) The CCITT, considering (a) that the Recommendation X.200 defines the Reference Model of Open Systems Interconnection (OSI) for CCITT Applications; (b) that the objective of OSI will not be completely achieved until systems can be tested to determine whether they conform to the relevant OSI protocol Recommendations; (c) that standardized test suites should be developed for each OSI protocol Recommendation as a means to: – obtain wide acceptance and confidence in conformance test results produced by different testers, – provide confidence in the interoperability of equipments which passed the standardized conformance tests; (d) the need for standardizing the conformance testing process to achieve an acceptable and useful degree of comparability of results of conformance assessments of similar products, unanimously declares the view that the roles of the test laboratory and the client during the conformance assessment process should be in accordance with this Recommendation. CONTENTS 0 Introduction 1 Scope 2 References 3 Definitions 4 Abbreviations 4.1 Abbreviations defined in Recommendation X.290 4.2 Other abbreviations 5 Overview of the Conformance Assessment Process 5.1 Introduction 5.2 Preparation for testing 5.3 Test operations 5.4 Test report production 6 Preparation for testing 6.1 Introduction 6.2 Requirements for SUT testability 6.3 Communication between the test laboratory and the client 6.4 Documentation for conformance assessment 7 Test operations 7.1 Introduction 7.2 Static Conformance Review 7.3 Test selection 7.4 Test parameterization 7.5 Test Coordination Procedure verification 7.6 Test Campaign 7.7 Negociated Exits during the Test Campaign 8 Test report production 8.1 Conformance test reports 8.2 System Conformance Test Report (SCTR) 8.3 Protocol Conformance Test Report (PCTR) 9 Compliance 9.1 Test laboratory role 9.2 Client role Annex A – System Conformance Test Report (SCTR) Annex B – Protocol Conformance Test Report (PCTR) Annex C – Skeleton PIXIT Proforma Annex D – Guidance for a PIXIT Annex E – Summary of Conformance Testing documents 0 Introduction Conformance testing requires mutual understanding and agreement between the test laboratory and the client. This Recommendation addresses the roles of both the test laboratory and the client during the conformance assessment process, the need to reach mutual agreements between them, and the requirements on each of them. The conformance assessment process is the most visible process of conformance testing, where the results of test suite standardization are put to real use. This is also the stage at which there is potentially the most scope for variations to occur. As this Recommendation is concerned with the relatively formal process of testing implementations, it is important that the number and nature of such variations is very limited. One of the major objectives of standardizing the conformance testing process is to achieve an acceptable and useful degree of comparability of results of conformance assessments of similar implementations. If this is to be achieved, not only should the same source of tests be used (i.e. as specified in standards), but also the methods of selecting and parameterizing these tests, and presenting their results, should be, to a large extent, the same. This Recommendation addresses the issues which should be taken into account, by both the test laboratory and the client, if the necessary consistency of conformance assessment is to be achieved. The target audiences for this Recommendation are the test laboratories and their clients. The test laboratory is responsible for conducting the conformance assessment of an OSI implementation at the request of a client. Typically, test laboratories are: a)organizations developing or supplying OSI implementations (first-party test laboratories); b)organizations willing to verify OSI implementations themselves before using them (second-party test laboratories); c)organizations, independent of suppliers or users of OSI implementations, whose business is the testing of such implementations (third-party test laboratories). Clients may be implementors or suppliers of real open systems or other OSI systems, who are applying for their own implementations to be tested. Alternatively, they may be procurers of those implementations, or any other interested party. The applicability of this Recommendation is independent of the relationship between the client and the implementation. During the conformance assessment process, the client is responsible for the conformance statements accompanying the System Under Test (SUT) and for the configuration of the SUT. Secondary, but related, audiences to whom this Recommendation could also be of interest are: a)test realizers; b)organizations responsible for the accreditation of first , second- or third-party test laboratories; c)organizations responsible for the issue of test certificates which are based upon the conformance test reports issued by test laboratories; d)readers of conformance test reports. Within this Recommendation, the conformance assessment process relating to both the test laboratory and the client is divided into three phases: a)preparation for testing; b)test operation; c)production of test reports. An overview of these three phases is given in § 5. Sections 6 to 9 state requirements on how to conduct these three phases. For the purposes of this Recommendation, it is assumed that a test laboratory is available and is already organized to provide a conformance assessment service. The test laboratory is assumed to have acquired from a test realizer (whether or not the latter belongs to the same organization), Means of Testing Implementations under test (IUTs), for one or more OSI protocols and according to one or more Abstract Test Methods (ATM)s. This Recommendation specifies requirements on the test laboratory with respect to the conduct of the conformance assessment process for a particular client. Similarly, it is assumed that a client is ready to apply for conformance assessment of an OSI implementation. The client is assumed to be familiar with the appropriate standards, with the concepts of conformance testing and Abstract Test Methods, and to be ready to cooperate with the test laboratory. This Recommendation specifies requirements on the client with respect to both the testability of the proposed SUT and the conduct of the conformance assessment process. It is recommended that this Recommendation be read in conjunction with Recommendation X.290. This Recommendation is also to be published as ISO/IEC 9646- 5:1991. 1 Scope This Recommendation specifies requirements on both the test laboratory and the client, for the conduct of the conformance assessment process. The requirements are those necessary to achieve comparability of results of tests on similar implementations performed by different test laboratories. This Recommendation also provides some guidance on the conformance assessment process. The requirements include: a)requirements for the testability of the implementation with respect to ATMs; b)general requirements on the test laboratory and the client applicable to any conformance testing process; c)exchange of technical and administrative information, including a System Conformance Statement (SCS), a Protocol Implementation Conformance Statement (PICS) for each relevant OSI protocol standard, and Protocol Implementation extra Information for Testing (PIXIT) for each Abstract Test Suite (ATS) to be used for testing; d)cooperation between the test laboratory and the client to reach an agreement on the definition of the Implementation Under Test (IUT), on the ATMs and ATSs to be used and on the conditions under which testing will be performed; e)requirements for the structure and content of the conformance test reports that document the results of conformance assessment. This Recommendation is applicable equally to those test laboratories which are affiliated to suppliers or procurers, and those which are independent. This Recommendation is applicable to conformance assessment of implementations of OSI and ISDN two-party protocol standards that comply with the relevant requirements for testability in Recommendation X.291, based on conformance testing standards specified in compliance with Recommendation X.291, and using Means of Testing (MOT) in compliance with Recommendation X.293. The following are outside the scope of this Recommendation: a)the production of diagnostic trace information, additional to that in the conformance log, for the results of testing performed by the test laboratory, and its supply to the client; b)aspects of test laboratory operations which are not specific to testing implementations of OSI protocols; c)accreditation of test laboratories; d)certification of implementations of OSI protocols. 2 References Rec. X.200 (1988) – Reference Model of Open Systems Interconnection for CCITT Applications. (See also ISO 7498.) Rec. X.210 (1988)– Open Systems Interconnection Layer Service Definition Conventions. (See also ISO/TR 8509.) Rec. X.290 (1992)– OSI Conformance Testing Methodology and Framework for Protocol Recommendations for CCITT Applications – General concepts. (See also ISO/IEC 9646- 1.) Rec. X.291 (1992)– OSI Conformance Testing Methodology and Framework for Protocol Recommendations for CCITT Applications – Abstract Test Suite Specification. (See also ISO/IEC 9646-2.) Rec. X.292 – OSI Conformance Testing Methodology and Framework for Protocol Recommendations for CCITT Applications – Tree and Tabular Combined Notation (TTCN). (See also ISO/IEC 9646-3.)2) Rec. X.293 (1992)– OSI Conformance Testing Methodology and Framework for Protocol Recommendations for CCITT Applications – Test Realization. (See also ISO/IEC 9646- 4.) 3 Definitions For the purposes of this Recommendation, all the definitions given in Recommendation X.290 apply. The definitions in this clause also apply to this Recommendation. 3.1 client test manager The person identified by the client organization as being responsible for all matters relating to the conformance testing of the IUT. 3.2 test laboratory manager The person identified by the test laboratory as being responsible for all matters relating to test laboratory operations. 3.3 test operator The person or persons designated by the test laboratory as being responsible for running conformance tests against the IUT. 3.4 SUT operator The person or persons designated by the client organization as being responsible for operation of the SUT during conformance testing. 3.5 negotiated exit [NE] (from the conformance assessment process) A point in time at which the test laboratory and the client can mutually decide to terminate the conformance assessment process. 3.6 test laboratory checklist A record of test-related information supplied to the client by the test laboratory during the test preparation phase of the conformance assessment process. 3.7 client checklist A record of test-related information supplied to the test laboratory by the client during the test preparation phase of the conformance assessment process. 4 Abbreviations 4.1 Abbreviations defined in Recommendation X.290 For the purposes of this Recommendation, the following abbreviations defined in Recommendation X.290, § 4 apply: ASP Abstract service primitive ATM Abstract test method ATS Abstract test suite BIT Basic interconnection test IUT Implementation under test MOT Means of testing OSI Open systems interconnection PCO Point of control and observation PCTR Protocol conformance test report PETS Parameterized executable test suite PICS Protocol implementation conformance statement PIXIT Protocol implementation extra information for testing SAP service access point SATS Selected abstract test suite SCS System conformance statement SCTR System conformance test report SUT System under test TMP Test management protocol 4.2 Other abbreviations For the purposes of this Recommendation, the following abbreviation also applies: NE Negotiated exit 5 Overview of the Conformance Assessment Process 5.1 Introduction Figure 1/X.294 illustrates the conformance assessment process. Recommendation X.290, § 6.3 provides an overview of the conformance assessment process. An overview of the three phases (preparation for testing, test operations and test report production) is given as guidance in this section. 5.2 Preparation for testing The preparatory phase includes: a)general administrative steps, such as the application by the client, the provision of documents by the test laboratory describing the general policy, terms and conditions to be followed during test operations, and the provision of information about the SUT by the client; b)checking the completeness of the documents provided by the test laboratory (PIXIT proforma), of those provided by the client (PICS, PIXIT, SCS), and also of the checklists exchanged between the test laboratory and the client; c)analysis of the SUT configuration and choice of a conformance testing standard(s) for the OSI protocol(s) to be tested, either agreeing that the SUT and the test laboratory's MOT are both capable of supporting the ATM, or using a negotiated exit if an agreement cannot be reached; d)preparation of the SUT and the MOT for the testing configuration that results from the choice of an ATM. Requirements for test preparation, on both the test laboratory and the client, are given in § 6. 5.3 Test operations During the second phase, the test operations are carried out. These include: a)the static conformance review, during which detailed analysis of the PICS and PIXIT takes place; b)test selection and parameterization, applied to the executable (or abstract) test suite; this determines the Parameterized Executable Test Suite (PETS) that will be executed; FIGURE 1/X.294 = 16.5 cm [D01] c)one or more test campaigns, running: 1)basic interconnection tests (optional); 2)capability tests; 3)behaviour tests. If difficulties are encountered during test operations, it is possible for the client and test laboratory to negotiate a repetition of a test campaign as a whole or in part. Alternatively, they can take a negotiated exit from the conformance assessment process. Note – The reasons for the negotiated exit are documented in an informal test report. Requirements for test operations, on both the test laboratory and the client, are given in § 7. 5.4 Test report production After test operations are complete, the third phase commences: an evaluation of the conformance of the OSI implementation is made. This evaluation is recorded in System and Protocol Conformance Test Reports, the proformas for which are specified in Annexes A and B respectively. Requirements for test report production are given in § 8. 6 Preparation for testing 6.1 Introduction Section 6 specifies requirements for test preparation on both the test laboratory and the client. Figure 2/X.294 illustrates the preparatory phase of the conformance assessment process. During this phase, both parties ensure that the required documentation (including SCS, PICS, PIXIT and the checklists) is completed to their mutual satisfaction: in particular, the characteristics of the SUT which determine its configurations and affect the choice of test method must have been precisely defined. It is assumed that the requirements for SUT testability have been met by the client before approaching the test laboratory. As a prerequisite to test operations, the client and the test laboratory agree on the ATM and the conditions for the test campaign. If an agreement is reached, the test laboratory will select the MOT for the chosen test method, and proceed to the test operations phase; otherwise, a negotiated exit may be taken. 6.2 Requirements for SUT testability 6.2.1Client role 6.2.1.1 General The client shall ensure that the SUT is testable using at least one Abstract Test Method (ATM) as defined in Recommendation X.291. Note – This Recommendation does not constrain the client to agree to any particular ATM, as long as one method is made possible by appropriately configuring the SUT. Each of the ATMs described in Recommendation X.291, § 12, imposes particular requirements on the SUT with respect to its testability. The requirements vary according to the test method. For the ATM by which the client claims the SUT can be tested, the client shall ensure that the SUT provides the necessary means of control and observation and that it can enable the appropriate test coordination procedures to be performed. The next four subsections state further requirements for SUT testability for each of the ATMs. 6.2.1.2 Local Test Method 6.2.1.2.1 For the Local ATM, the client shall ensure that the IUT upper interface is realized in hardware and can be connected to a test system. Note – The only requirements are that the interface is standardized and that there is a well-defined mapping between the relevant ASPs and the hardware interface. 6.2.1.2.2 There are no requirements on the client for the test coordination procedures. 6.2.1.3 Distributed Test Method 6.2.1.3.1 For the Distributed ATM, the client shall ensure that the SUT contains the means of generating, and indicating the receipt of, the ASPs for the relevant Points of Control and Observation (PCOs), that are appropriate to the envisaged IUT. Note 1 – The only requirements are that means be realized within the SUT for the control and observation of the effects of the relevant ASPs and that the upper service boundary of the IUT is either a human-user interface or a standardized programming language interface. FIGURE 2/X.294 = 16.5 cm [D02] Note 2 – Any realization will satisfy this requirement provided that the appropriate ASPs can be generated and detected unambiguously. Example realizations are: pushing buttons, observing and controlling activities of OSI protocols which use the service primitives concerned, observing activity on peripherals and so on. There may be a one-to-one, one-to-many, or many-to-one correspondence between the means within the SUT and ASPs. 6.2.1.3.2 The client shall ensure that notification of the occurrence of ASP events within the SUT can be communicated by the SUT Operator to the Test Operator (at the test laboratory) when required by the test coordination procedures. Note – The means for transferring notification of the occurrence of generated and observed ASP events to the test laboratory is outside the scope of the X.290-series Recommendations, but could be achieved by the use of a separate communication channel, e.g. voice, telephone call, data call, etc. The way in which this communication takes place constitutes part of the informal test coordination procedures, and is stated by the client in the Client Checklist when applying to the test laboratory for conformance assessment. 6.2.1.4 Coordinated Test Method 6.2.1.4.1 For the Coordinated ATM, the client shall ensure that the SUT can support at least one upper tester that is an implementation of a standardized Test Management Protocol (TMP) appropriate to the envisaged IUT. Note – This requirement does not imply the implementation within the SUT of a real OSI service boundary with real service primitives. 6.2.1.4.2 Once an ATS for the Coordinated ATM has been chosen during test preparation, the client shall ensure that the SUT supports an upper tester which realizes the TMP for this ATS. Note – The client may use the assistance of the test laboratory during test preparation in order to meet this requirement. Alternatively, the client may choose to implement the upper tester and TMP for a particular ATS within the envisaged IUT before approaching a test laboratory for conformance assessment. 6.2.1.5 Remote Test Method 6.2.1.5.1 For the Remote ATM, once an ATS is chosen during test preparation, the client shall document in the PIXIT the degree of control and observation that is possible within the SUT to achieve the test coordination procedures that are stated informally in the standardized ATS. Note – Of all the test methods, the Remote ATM imposes the fewest constraints on the SUT. The SUT, however, is expected to perform in accordance with the claims made in the PIXIT. These claims may imply some degree of control, for example, where the SUT is required to initiate some event. 6.2.1.5.2 For those test events for which a claim of degree of control or observation has been made in the PIXIT, the client shall ensure that notification of the occurrence of such test events within the SUT can be communicated by the SUT Operator to the Test Operator (at the test laboratory), when required by the test coordination procedures. Note – The means for transferring notification of the occurrence of generated and observed test events to the test laboratory is outside the scope of the X.290-series Recommendations, but could be achieved by the use of a separate communication channel, e.g. voice, telephone call, data call, etc. The way in which this communication takes place constitutes part of the informal test coordination procedures, and is stated by the client in the Client Checklist when applying to the test laboratory for conformance assessment. 6.3 Communication between the test laboratory and the client 6.3.1Test laboratory and client checklists 6.3.1.1 Introduction During test preparation, the test laboratory and client exchange test-related information in order that they can agree on the definition of the IUT and on the choice of ATM(s) and ATS(s) to be used during testing. This information, some of which may be assembled as a result of discussion, is specified in a Test Laboratory Checklist and a Client Checklist. Based on the exchange and review of the checklists by both parties, an agreement can be reached on whether or not to proceed with the test preparation phase. An agreement might not be reached if the test laboratory is unable to offer a testing service which is compatible with the client's proposed IUT. If both parties agree to continue, the documents required for conformance assessment are prepared (refer to § 6.4). 6.3.1.2 Test laboratory role During the test preparation phase of the conformance assessment process, the test laboratory shall provide the client with a Test Laboratory Checklist, containing as a minimum the following information: a)requirements placed on the client by the test laboratory, concerning the provision of the SCS, PICS(s) and PIXIT(s); b)statement of compliance with Recommendation X.294; c)ATM(s) supported for each OSI protocol for which a testing service is offered; d)statement of conformance with the conformance testing standards for which a testing service is offered; e)statement of whether or not the test laboratory offers a comprehensive conformance testing service as defined in each of the E1 applicable conformance testing standards and in Recommendation X.291; f)limitations, if any, of the lower tester(s), for the supported test methods that are relevant to the client's SUT; g)specifications of the upper tester, if applicable, and the test coordination procedures for the supported test methods that are relevant to the client's SUT; h)description of the test laboratory procedures which are related to running the tests during the test campaign and which are relevant to the client, particularly those to be performed by the SUT Operator; i)references to all documentation to be used and produced by the test laboratory during the conformance assessment process. Note – The test laboratory may provide the client with: a)information on the testing services offered in the range of OSI protocols of interest to the client; b)assistance in the implementation of an upper tester, if applicable for the chosen ATM; c)information on whom to contact to obtain PIXIT proformas and any other necessary information; d)estimates of the time required to complete the Test Operations and Test Report Production phases of the conformance assessment process; e) statement of test laboratory accreditation (if applicable). 6.3.1.3 Client role During the test preparation phase of the conformance assessment process, the client shall provide the test laboratory with a Client Checklist, containing as a minimum the following information: a)statement of compliance with Recommendation X.294; b)specification of what part of the SUT is proposed to be the IUT and which protocols are to be tested; c)a claim of SUT testability based on specific ATM(s) and/or conformance testing standards; d)a statement of the test coordination procedures that are suitable for use with this IUT, and which correspond to the proposed ATM(s). Note – The client may provide the test laboratory with: a)information on any physical requirements for the SUT (for example, space, air conditioning, etc.) if relevant, and any other practical information that may be needed during the conformance assessment process; b)information on whom to contact during the conformance assessment process. 6.3.2Agreement on test methods and selection of test suites 6.3.2.1 Test laboratory role The test laboratory shall review the Client Checklist and shall determine if the test laboratory offers a testing service which is applicable to the client's proposed IUT. The test laboratory shall accommodate the client's choice of ATM for each protocol in the proposed IUT (see § 6.3.2.2) and shall select the corresponding reference standardized ATS (including if relevant the TMP specification) to be used in the conformance assessment process. For each standardized ATS selected, the test laboratory shall identify and use a MOT which complies with that standardized ATS and with Recommendation X.293. Note – The conformance testing standard(s) used should have the highest available standardization status (see Recommendation X.291, § 5). 6.3.2.2 Client role The client shall review the Test Laboratory Checklist and shall make the choice of which ATM is to be used for each protocol in the proposed IUT in accordance with the claims for SUT testability and the testing service offered by the test laboratory. Note 1 – The client may wish to select ATM(s) which impose no additional requirements on the SUT other than those contained in the OSI standards to which the SUT claims to conform. In this case, the client should select a test laboratory which provides a comprehensive testing service. (See Recommendation X.291, § 11.7.2 and § 9.1 of this Recommendation.) Note 2 – An IUT can consist of a single protocol entity or multiple protocol entities. If the IUT is multi-protocol, then single-layer embedded test methods should be used incrementally. (Refer to Recommendation X.290, § 7.6.) Note 3 – When using embedded single-layer test methods, the ATS is targetted at a single OSI protocol. Note 4 – Several IUTs can be defined in the SUT if several conformance assessments are to be performed on the same SUT for different combinations of protocols and test methods. 6.3.2.3 Mutual role After both parties have reviewed the information provided in the checklists, in order that the conformance assessment process can continue, both parties shall agree on a)the accuracy and sufficiency of the information provided in the checklists; b)the definition of the IUT; c)the ATM(s) and the corresponding ATS(s) which will be used for the conformance assessment process. If an agreement cannot be reached, a negotiated exit should be taken in order to terminate the conformance assessment process. If an agreement is reached, what has been agreed shall be recorded in the System Conformance Test Report (SCTR) issued at the end of the conformance assessment process. 6.3.3Management of technical issues There are no general requirements concerning procedures for the resolution of technical issues between the client and test labo ratory that may arise during the conformance assessment process. However, should differences be discovered between the conformance testing standard and the protocol standard, the protocol standard shall have precedence in problem resolution. Note – Unresolved issues of a technical nature, related to the interpretation of relevant standards, can be referred to the appropriate OSI standard defining group. 6.4 Documentation for conformance assessment 6.4.1Overview After the test laboratory and client have agreed on the definition of the IUT and on the ATM(s) and ATS(s) to be used during the conformance assessment, they exchange detailed information about the SUT. This information resides in four documents related to test preparation: the PICS, PIXIT, SCS and TMP Implementation Statement. The next four subsections state requirements on both the client and test laboratory related to the production and exchange of these documents. 6.4.2Protocol Implementation Conformance Statement (PICS) 6.4.2.1 Content of the PICS Detailed information on the role and scope of the PICS is given in Recommendation X.290, § 5.4 and basic guidance on the design of the PICS proforma is given in Recommendation X.291, Annex C. A PICS proforma is contained in each OSI protocol standard and in each OSI transfer syntax standard which complies with the requirements of Recommendation X.291 for testability. 6.4.2.2 Test laboratory role There is no requirement on the test laboratory for the provision of PICS proformas for use by the client. However, the test laboratory may provide copies of the relevant PICS proformas if necessary. 6.4.2.3 Client role The client shall provide a PICS for each OSI protocol standard (and for each OSI transfer syntax standard if applicable) which is implemented in the IUT and for which conformance is to be tested. The client shall complete the relevant PICS proformas from the relevant OSI standards. The requirements for the provision of PICS information are stated in the relevant standards. 6.4.3Protocol Implementation extra Information for Testing (PIXIT) 6.4.3.1 Content of the PIXIT The role and scope of the PIXIT is given in Recommendation X.290, § 6.2. Requirements and further guidance on the structure and content of the PIXIT is given in Annexes C and D of this Recommendation. 6.4.3.2 Test laboratory role The test laboratory shall produce a PIXIT proforma for each standardized ATS for which testing is offered, in accordance with the skeleton proforma specified in Annex C. Each PIXIT proforma shall include the augmented partial PIXIT proforma accompanying the MOT and associated with the relevant reference standardized ATS. It shall also contain the questions necessary to elicit all the information concerning the SUT, the IUT, and the relevant ATS parameters, which is not contained in the relevant SCS or PICS, and which is needed by the test laboratory in order to run the executable version of the standardized ATS. The test laboratory shall provide a PIXIT proforma to the client for each ATS that is to be used during the conformance assessment process. Note – There is one standardized ATS per protocol for single- layer and single-layer embedded test methods. Thus, for these test methods, there is one PIXIT proforma per protocol. 6.4.3.3 Client role The client shall provide a PIXIT for each ATS to be used for testing, by completing the relevant PIXIT proforma provided by the test laboratory. Note – Wherever the information requested is extensive, the PIXIT may refer to other documents which contain the necessary information. 6.4.4System Conformance Statement (SCS) 6.4.4.1 Content of the SCS The role and scope of the SCS is mentioned briefly in Recommendation X.290, § 5.4. The format of the SCS is not standardized. However, the SCS contains the following information as a minimum: a)information related to both the SUT and the client: 1)administrative information to identify the client; 2)system information to identify the SUT, for example, name and version number; b)information related to the protocols in the SUT for which a PICS is provided: 1)the identification of the OSI protocol (and OSI transfer syntax) standards, including version numbers; 2)a reference to the related PICS. The SCS optionally contains an indication of whether an SCTR and a PCTR, received from a previous conformance assessment, are available for information. 6.4.4.2 Test laboratory role There is no requirement on the test laboratory for the provision of an SCS proforma. The SCS provides test-independent information about the SUT. 6.4.4.3 Client role The client shall provide to the test laboratory an SCS as described in § 6.4.4.1. 6.4.5TMP Implementation Statement 6.4.5.1 Content of the TMP Implementation Statement The role and scope of the TMP Implementation Statement is given in Recommendation X.291, § 14. A proforma for the TMP Implementation Statement is contained in the TMP part of each conformance testing standard which uses the Coordinated ATM. 6.4.5.2 Test laboratory role The test laboratory shall provide the client with the proforma for the TMP Implementation Statement for each ATS based on the Coordinated ATM which will be used during conformance assessment. 6.4.5.3 Client role The client shall provide a TMP Implementation Statement for each ATS based on the Coordinated ATM which is to be used for testing. This is done by completing the relevant proforma provided by the test laboratory. 7 Test operations 7.1 Introduction Section 7 specifies requirements for test operations. Figure 3/X.294 shows typical test operations. For the purpose of this section, it is assumed that a single-protocol IUT is being tested using a single standardized ATS, a single PICS and a single PIXIT. However, this section does not exclude a multi-protocol IUT being tested incrementally, using a series of single-protocol standardized ATSs. If a multi-protocol IUT is to be tested incrementally, then a standardized ATS will be used for each protocol to be tested, and so there will be a PICS for each protocol and a PIXIT for each corresponding standardized ATS. 7.2 Static Conformance Review 7.2.1Test laboratory role 7.2.1.1 During the static conformance review, the test laboratory shall analyse the PICS according to the following criteria: a)the PICS shall be self-consistent; b)the PICS shall be consistent with the static conformance requirements specified in the OSI protocol standard(s) to which the IUT is claimed to conform. 7.2.1.2 As a minimum, the following checks shall be made for consistency between the PICS and the static conformance require ments: a)for each item which is indicated as mandatory in the status column, check that the item is indicated as supported; b)for each item which is indicated as optional and from which a defined subset must be supported, check that the indications of support are consistent with the requirement; FIGURE 3/X.294 = 20.5cm [D03] c)for each item which is indicated as conditional in the status column, evaluate the condition in order to determine the effective status and to perform the check which is appropriate to that status; d)apply the additional static conformance review checks, if any, which are specified explicitly in the PICS proforma itself. 7.2.1.3 No additional checks are needed for items which are indicated in the status column as: a)as an unqualified option (Boolean: do or not do); b)not applicable. 7.2.1.4 The test laboratory shall also check the consistency of the information presented in the PIXIT and SCS documents provided by the client. 7.2.1.5 The test laboratory shall inform the client of the results of the static conformance review before continuing with the con formance assessment process. 7.2.2Client role The client shall review the results of the static conformance review. 7.2.3 Mutual role If the results of the static conformance review reveal that, in the view of either the test laboratory or the client, to proceed with testing would not be productive, then a negotiated exit may be taken. 7.3 Test selection 7.3.1 Test laboratory role The test laboratory shall select all those abstract test cases appropriate for the IUT, based on the information in the PICS and PIXIT, in accordance with the documentation of the MOT and with the requirements of its reference standardized ATS. This means that the test laboratory shall select the following test cases provided that they are testable according to the PIXIT: a)all capability test cases for mandatory capabilities; b)all capability test cases for optional or conditional capabilities that are present in the IUT according to the PICS; c)all behaviour test cases for mandatory capabilities; d)all behaviour test cases that are consistent with the optional or conditional capabilities that are present in the IUT according to the PICS. This selected set of abstract test cases is the Selected Abstract Test Suite (SATS) for this IUT. The SATS might not be tangible (refer to Recommendation X.293, § 5). If the standardized ATS defines a list of Basic Interconnection Tests (BITs) and the client wishes to perform basic interconnection testing, the test laboratory shall select the set of BITs listed in order to perform basic interconnection testing at the beginning of the test campaign. 7.3.2 Client role The client shall inform the test laboratory whether or not basic interconnection testing should be performed during the test campaign. Note – If the standardized ATS does not identify a list of BITs, the test laboratory will not be able to accommodate this request. 7.4 Test parameterization 7.4.1 Test laboratory role 7.4.1.1 After the set of all test cases retained in test selection is determined, the information provided in the PIXIT shall be used to determine the appropriate values for each parameter in those test cases, in accordance with the documentation of the Means of Testing and with the requirements of its reference standardized ATS. The resulting Parameterized Executable Test Suite (PETS) is then ready to be executed, or to be generated on request. Information from the PIXIT (and possibly from the PICS) is used in the parameterization process. Examples of types of parame terization are: a)values of network addresses; b)values of connection end point identifiers; c)values of counters; d)values of timers; e)encoding strategies. Note – This list is not exhaustive. 7.4.1.2 After parameterization, the test laboratory shall ensure that all the test cases in the SATS are present in the PETS. Note – The parameterization performed should be validated as far as possible before capability testing takes place, for example, during basic interconnection testing. 7.4.2Client role There are no requirements on the client during test parameterization. 7.5 Test Coordination Procedure verification 7.5.1Introduction Some level of test coordination procedures is specified or implied in every standardized ATS. Upon completion of the test parameterization process, it should be verified that the MOT and the SUT are able to use the required test coordination procedures in order to carry out the test campaign. 7.5.2Mutual role If the ATM to be used for the test campaign is other than the Local or the Coordinated method, the verification of the test coordination procedures shall be informally accomplished by the test laboratory and the client. If the Coordinated ATM is to be used, the test laboratory shall verify the implementation of the TMP in the SUT. This shall be achieved by selecting the applicable TMP test cases from the standardized ATS, according to the TMP Implementation Statement, and executing them against the upper tester. If the Local ATM is to be used, verification of the test coordination procedures will be done by the test laboratory as part of the procedures for verifying the MOT. Regardless of the ATM, if the results of verifying the test coordination procedures are unsatisfactory to either the test laboratory or the client, then the test campaign shall not be attempted. 7.6 Test Campaign 7.6.1Introduction A test campaign is the process of executing the PETS for a particular IUT and producing information required for the conformance log. 7.6.2Test laboratory role 7.6.2.1 General The test laboratory shall ensure that the MOT and Test Operator are available throughout the agreed test campaign period, shall invoke all of the test cases in the PETS, and shall produce the information required for the conformance log. 7.6.2.2 Basic Interconnection Testing If BITs have been selected, the test laboratory shall invoke those tests before running further capability and behaviour tests. The test laboratory shall inform the client of the results of the BITs before proceeding with further test cases. Note – The client may wish to take a negotiated exit after analysing the results of the BITs. 7.6.2.3 Analysis of verdict assignments During the test campaign, the test laboratory shall establish for each test case in the SATS, which one of the following results applies: a)pass verdict; b)fail verdict; c)inconclusive verdict; d)abstract test case error; e)executable test case error; f)abnormal test case termination. For each test case that has an abstract test case error, the test laboratory shall indicate in the PCTR that the test case was “Not Run” together with the reason why. Note – The test laboratory should progress the updating of the standardized ATS to fix the error via the relevant rapid amendment procedures. For each test case that produced either an executable test case error or an abnormal test case termination result, the test laboratory shall re-run the test case. If the same result is produced, the test laboratory shall indicate in the PCTR that the test case was “Not Run” together with the reason why. For each test case that produced an Inconclusive verdict, the test laboratory shall re-run the test case at least once. If a Pass or Fail verdict is produced during a subsequent execution, that verdict shall be recorded in the PCTR. If an Inconclusive verdict is produced during subsequent execution(s) of the test case and the test case behaviour is the same as in previous executions, the Inconclusive verdict shall be recorded in the PCTR. For each test case that produced a Fail verdict, the test laboratory shall assess whether the verdict was associated with an unidentified test event in the abstract test case. If this is not the case, the test laboratory shall record the Fail verdict for this test case in the PCTR. However, if this is the case, the test laboratory shall determine whether there is an abstract test case error, that is, whether the event which matched the unidentified test event was valid according to the protocol and should have been defined in the abstract test case. If so, the test laboratory shall indicate in the PCTR that the test case was “Not Run” together with the reason why. While analysing the conformance log associated with a particular test case, the test laboratory might observe values in the PDUs received by the MOT which were not explicitly checked in the abstract test case. If the ATC does not contain explicit checks for these values, then the test laboratory shall not alter the verdict assignment because of its observation. Note – Any observations made by the test laboratory may be recorded in section 7 of the PCTR (see Annex B) but shall be considered only as additional information for the client. Prior to entering the test report production phase, the test laboratory shall inform the client of any test cases for which it intends to record a Fail verdict in the PCTR. 7.6.2.4 Production of Conformance Log If the client requests a conformance log for the test campaign, the test laboratory shall produce one that documents the test out-come for every execution of each test case which was requested by the client, including any repeated executions. 7.6.3Client role 7.6.3.1 General The client shall ensure that the SUT and, if required, an SUT Operator are available throughout the agreed test campaign period. The client shall cooperate with the test laboratory to make any changes to the IUT or its environment which are required in order to enable execution of all the test cases (see § 7.6.4.1) and shall review the documentation of such changes. 7.6.3.2 Basic Interconnection Testing If basic interconnection testing is to be performed, the client shall review the results of that testing and inform the test laboratory if a negotiated exit is to be taken. The client should do this before the test laboratory proceeds with further testing. 7.6.3.3 Analysis of verdict assignments There are no requirements on the client concerning analysis of verdict assignments. However, during the test campaign, the client may request a re-run of any test case that produced a Fail verdict, if not satisfied that the test case correctly diagnosed an error in the IUT. 7.6.4Mutual role 7.6.4.1 Changes to the test environment Once a test campaign has started, any changes to the IUT or its environment, or to the lower tester and its environment, or to the upper tester, shall occur only if agreed by both the client and test laboratory. For the sake of consistency of the test results, such changes shall be agreed to only if they do not invalidate the results of the test cases previously executed in that test campaign. If the changes are required in order to run test cases in the executable version of the reference ATS, then the fact that the requirement comes from the reference ATS is sufficient reason to deem that those test cases do not invalidate the results of previously executed test cases. Any changes shall be documented by the test laboratory and reviewed by both the Test Laboratory Manager and the Client Test Manager. Based on this review, a decision shall be reached as to whether the test campaign is to be: a)restarted; b)continued, following the necessary revisions; c)discontinued by taking a negotiated exit. 7.6.4.2 Analysis of verdict assignments If the client requests a re-run of a test case that produced a Fail verdict, the test laboratory and client shall assess whether or not the result was caused by some error in the IUT. If they are unable to establish the existence of an error in the IUT, the test laboratory shall apply the repetition procedure for test cases yielding Inconclusive verdicts (see § 7.6.2.2). 7.7 Negotiated Exits during the Test Campaign 7.7.1Use of a Negotiated Exit During the test campaign, a negotiated exit is a point in time when the test laboratory and the client decide together that the test results up to that point do not justify continuing the conformance assessment process. The request for a negotiated exit can be made by either party. If a negotiated exit is taken as a consequence of a dispute over the result(s) of specific test case(s), then the negotiated exit takes place before the assignment of verdict(s) to the disputed test case(s). 7.7.2Test laboratory role If a negotiated exit is agreed to by both the client and test laboratory, the test laboratory shall make available to the client, on request, documentation containing all the information recorded during the conformance assessment process. This documentation shall, if requested by the client, include the conformance log for the test cases that were run, as defined in Recommendation X.293, § 6.4. In addition, the test laboratory shall provide an informal test report which does not assume the status of either an SCTR or PCTR. It shall be considered simply, as guidance to the client on the results of the testing undertaken. This informal test report shall indicate the reasons why a negotiated exit was taken. Note – When a test campaign is terminated by a negotiated exit, diagnostic testing may proceed and further diagnostic trace information, additional to that in the conformance log, may be provided to the client, but this is outside the scope of the X.290- Series of Recommendations. 7.7.3Client role If a negotiated exit is taken, there are no further requirements on the client. Note – The client may request documentation containing all the information recorded during the conformance assessment process, including the conformance log. 7.7.4Mutual role After a negotiated exit, conformance testing shall not be restarted except by starting a new test campaign or by initiating a new conformance assessment process. 8 Test report production 8.1 Conformance test reports 8.1.1Introduction The conformance assessment process culminates in the production by the test laboratory of two types of test report: a System Conformance Test Report (SCTR); and a Protocol Conformance Test Report (PCTR) for each protocol tested. The test laboratory may also produce detailed diagnostic trace information, additional to that provided in the conformance log, to accompany the test reports if requested; such information is considered to be supplemental to the test reports themselves, and there is no requirement in this Recommendation that a test laboratory shall provide it. 8.2 System Conformance Test Report (SCTR) 8.2.1Test laboratory role Provided that a negotiated exit was not taken during the test campaign, the test laboratory shall produce an SCTR that provides a summary of the results of the conformance testing performed on the client's SUT. The test laboratory shall use the SCTR proforma given in Annex A. The SCTR shall contain the list of reference standardized ATS(s) against which testing has been carried out, together with their dates of publication, and in the case of ISO/IEC standards, details of any amendments or addenda with which the IUT is claimed to conform. The SCTR shall explain briefly the nature of the OSI testing and in particular that there is no guarantee that an SUT that has passed all the tests will interwork with other real open systems. (The SCTR proforma contains a paragraph for this purpose.) The SCTR shall state clearly if non-conformance has been demonstrated in any of the test cases, or if any areas of concern have been observed. Such statements should be clear and unambiguous. Section 1.7 of the SCTR shall record the agreement between the test laboratory and the client on the definition of what part(s) of the SUT are considered to be the IUT during testing and on the ATM(s) and ATS(s) to be used. The SCTR shall be made available to the client by test laboratory at the end of the conformance assessment process. 8.2.2Client role There are no requirements on the client during the production of the SCTR. Note – The client should review the SCTR and, in the case of disagreement with the test laboratory over its content, should supply comments in Section 1.8 of the SCTR. 8.2.3Mutual role The test laboratory and client shall ensure that information is provided in Section 1.6 of the SCTR describing any restrictions that apply to the use by the client of the SCTR, or to its release by the test laboratory. 8.3 Protocol Conformance Test Report (PCTR) 8.3.1Test laboratory role At the client's request, the test laboratory shall provide an accompanying PCTR for each OSI protocol for which conformance testing has been carried out during this conformance assessment process. The test laboratory shall use the PCTR proforma given in Annex B. Each PCTR shall record the conformance status of the IUT with respect to the protocol that has been tested. In Sections 3 and 5 of the PCTR, the test laboratory shall record the results of the static conformance review. In Section 6, the test laboratory shall list all of the abstract test cases in the order specified in the standardized ATS, including any BITs that were run, together with the following information: a)abstract test cases which were selected for inclusion in the PETS; b)abstract test cases for which corresponding executable test cases were run to completion during the test campaign; Note – Test cases which are selected but reported as being “Not Run” include those which produced a test case error or an abnormal test case termination. c)the verdicts assigned to those test cases that were run to completion; d)observations (if any) made by the test laboratory during the test campaign. The test laboratory shall ensure that the correct set of test cases has been performed for the IUT. If the MOT has selected a test which is not appropriate for the IUT, then the test laboratory shall not document the result of that test case in the PCTR but shall indicate that the test was “Not Run” together with the reason why. If the MOT failed to select a test case which is appropriate for the IUT, then the test laboratory shall indicate in the PCTR that the test is selected but that it was “Not Run”, together with the reason why. Note – The test laboratory should try to correct the error in either the standardized ATS (by submitting a defect report) or in the MOT as appropriate. If no Fail verdicts are to be recorded in the PCTR, the test laboratory shall complete section 4 to indicate that “the test campaign did not reveal errors in the IUT”. If, in addition, the PICS for the IUT is consistent with the static conformance requirements, the test laboratory shall complete section 1 to indicate that “the IUT has not been shown by conformance assessment to be non-conforming to the specified protocol standard”. At the client's request, the test laboratory shall also provide to the client any or all of the appropriate conformance logs, together with guidance on how to interpret them. Note – As a minimum, the conformance logs are required to be available on paper, but it is recommended that they are also available in a machine-readable format suitable for the client. The test laboratory shall record the retention date for the conformance logs in the PCTR. Note – It is recommended that, if available, either the ordered list of actual test events which is specific to the test system, or a machine-readable version of the conformance log, is retained until the retention date. 8.3.2Client role The client shall inform the test laboratory whether or not PCTR(s) and the associated conformance logs are to be provided. Note – The client should review each PCTR and, in the case of disagreement with the test laboratory over its content, should supply comments in Section 1.5 of the PCTR. 8.3.3Mutual role The test laboratory and client shall ensure that information is provided in Section 1.4 of the PCTR describing any restrictions that apply to the use by the client of the PCTR, or to its release by the test laboratory. 9 Compliance 9.1 Test laboratory role A test laboratory which claims to comply with this Recommendation shall a)for each SUT for which it carries out conformance assessment, comply with the requirements stated under the subsections entitled “Test laboratory role” and “Mutual role” for: 1)the preparation for testing, as stated in § 6 and Annex C; 2) test operation, as stated in § 7; 3)test report production, as stated in § 8 and Annexes A and B. b)test OSI protocol implementations by means of conformance testing standards which comply with Recommendation X.291, using MOT which comply with Recommendation X.293. In addition, if a test laboratory claims to provide a Comprehensive Conformance Testing Service for a specific OSI protocol, then it shall comply with the requirements for a comprehensive service stated in the relevant conformance testing standard(s) and in Recommendation X.291. Note 1 – There is no requirement in the X.290-Series Recommendations that a test laboratory has to offer a comprehensive service for any specific protocol. Note 2 – There is a requirement on ATS specifiers, in Recommendation X.291, § 11.7.2, that they must state the minimum requirements which are to be met by test laboratories in order to claim a comprehensive conformance testing service for the relevant protocol(s). Note 3 – If there is more than one standardized ATS for the same protocol, then a test laboratory can be said to provide a comprehensive conformance testing service even if it meets the requirements for such a service in only one of them. This could arise for example if the ISO/IEC standard and CCITT Recommendation differ for the same ATM and the same protocol. A test laboratory which makes a claim of conformance to the conformance testing standard(s) used for the conformance assess- ment, shall (in addition to satisfying the requirements stated in the conformance testing standard(s)) comply with the requirements stated under the subsections entitled “Test laboratory role” and “Mutual role” for: a)test selection, as stated in § 7.3.1; b)test parameterization, as stated in § 7.4.1; c)the test campaign, as stated in § 7.5.2. 9.2 Client role A client which claims to comply with this Recommendation shall, for each SUT presented for conformance assessment, comply with the requirements stated under the subsections entitled "Client role" and "Mutual role" throughout §§ 6, 7 and 8. ANNEX A (This annex is an integral part of this Recommendation) System Conformance Test Report (SCTR)3) A.1 Introduction This annex provides an SCTR proforma which shall be used by the test laboratory to document the results of a conformance assessment process for a specific client. Text in italics is comment for guidance purposes only, and is not to be included in the actual SCTR. A.2 SCTR Proforma The SCTR shall use the following format: SYSTEM CONFORMANCE TEST REPORT FOR SUT Name 1. IDENTIFICATION SUMMARY — 1.1 SYSTEM CONFORMANCE TEST REPORT — SCTR Number: SCTR Date: Test Laboratory Manager: Name Signature: Signature 1.2 TEST LABORATORY — Identification 1.3 CLIENT — Identification 1.4 SUT — Name: Version: Supplier: Dates for Testing: SCS Identifier: 1.5 NATURE OF CONFORMANCE TESTING — The purpose of Conformance Testing is to increase the probability that different implementations can interwork. However, the com plexity of OSI protocols makes exhaustive testing impractical on both technical and economic grounds. Furthermore, there is no guarantee that an SUT which has passed all the relevant tests conforms to a specification. Neither is there any guarantee that such an SUT will interwork with other real open systems. Rather, the passing of the tests gives confidence that the SUT has the stated capabilities and that its behaviour conforms consistently in representative instances of communication. 1.6 LIMITS AND RESERVATIONS — Additional information relevant to the technical contents or further use of the test report, or to the rights and obligations of the test laboratory and the client, may be given here. Such information may include restrictions on the publication of the report. 1.7 RECORD OF AGREEMENT — A definition of what part(s) of the SUT were considered to be the IUT(s) during testing, and of the abstract test method(s) and abstract test suite(s) that were used. 1.8 COMMENTS — Additional comments may be given by either the client or test laboratory on any of the contents of the SCTR, for example, to note disagreement between the two parties. 2. SYSTEM REPORT SUMMARY — For each protocol layer tested, a summary of the testing and conformance status of the implementation of that layer is required, using clauses of the format shown below. 2.n PROTOCOL LAYER TESTING SUMMARY FOR Protocol Name — Implementation identifier: Name and version number IUT definition reference: Reference from Section 1.7 Protocol standard/recommendation: Reference PICS: Reference PIXIT: Reference PCTR Number: Reference PCTR Date: Date of PCTR ATS standard/recommendation: Reference Abstract Test Method: Reference Means of Testing identifier: Name and version number Conformance Status: Static Conformance errors?: Yes/No Dynamic Conformance errors?: Yes/No Test cases run: Number Passed: Number Failed: Number Inconclusive: Number Observations (optional): If the SUT is not statically and dynamically conforming for this protocol, an additional summary may be given on aspects of non- conformance. Any difficulties encountered may be reported here. ANNEX B (This annex is an integral part of this Recommendation) Protocol Conformance Test Report (PCTR)4) B.1 Introduction This annex provides a PCTR proforma which shall be used by the test laboratory to document the results of conformance testing against a specific protocol using a specific ATS standard, for a specific client. Text in italics is comment for guidance purposes only, and is not to be included in the actual PCTR. B.2 PCTR Proforma The PCTR shall use the following format: PROTOCOL CONFORMANCE TEST REPORT FOR Protocol Name 1. IDENTIFICATION SUMMARY — 1.1 PROTOCOL CONFORMANCE TEST REPORT — PCTR Number: PCTR Date: Corresponding SCTR Number: Corresponding SCTR Date: Test Laboratory Manager: Name Signature: Signature 1.2 IUT — Name: Version: Protocol Standard/Recommendation: Reference PICS: Copy or reference Previous PCTRs if any (optional): References 1.3 TESTING ENVIRONMENT — PIXIT: Copy or reference ATS Standard/Recommendation: Reference Abstract Test Method: Means of Testing identification: Protocol information (optional): Timers, parameters, etc. Dates of testing: Conformance Log reference(s): Information required to obtain conformance logs Retention Date for Log reference(s): 1.4 LIMITS AND RESERVATIONS — Additional information relevant to the technical contents or further use of the test report, or to the rights and obligations of the test laboratory and the client, may be given here. Such information may include restrictions on the publication of the report. 1.5 COMMENTS — Additional comments may be given by either the client or the test laboratory on any of the contents of the PCTR, for example, to note disagreement between the two parties. 2. IUT CONFORMANCE STATUS — This IUT has/has not been shown by conformance assessment to be non- conforming to the specified protocol standard/recommendation. Strike the appropriate words in this sentence; if the PICS for this IUT is consistent with the static conformance requirements (as specified in Section 3 of this report) and there are no “Fail” verdicts to be recorded (in Section 6) strike the word “has/”, otherwise strike the words “/has not”. 3. STATIC CONFORMANCE SUMMARY — The PICS for this IUT is/is not consistent with the static conformance requirements in the specified protocol standard/recommendation. Strike the appropriate words in this sentence. 4. DYNAMIC CONFORMANCE SUMMARY — The test campaign did/did not reveal errors in the IUT. Strike the appropriate words in this sentence; if there are no “Fail” verdicts to be recorded in Section 6 of this report, strike the word “did/”, otherwise strike the words “/did not”. In addition, a descriptive summary of the results of groups of tests may be given. The detailed results of testing are provided in the table of Section 6; this section allows the test laboratory to make observations on those results, for example, “All the tests concerned with segmented data transfer failed.” 5 STATIC CONFORMANCE REVIEW REPORT — If Section 3 indicates non-conformance, this section itemises the mismatches between the PICS and the static conformance requirements of the specified protocol standard/recommendation. 6 TEST CAMPAIGN REPORT — This section shall use the following table which indicates both the test case selection that was performed by the test laboratory, and the results of testing. The order in which the abstract tests shall appear in this table is defined in the ATS standard/recommendation. Notes on the information that the Test Laboratory shall complete in the columns are provided below, and referenced as n). ATS Selected? Run? Verdict Observations Reference a) b) c) d) e) a) Reference to the abstract test case from the ATS standard/recommendation. b)Indicate whether or not the test was selected according to the PICS and PIXIT. If it was not selected due to the PIXIT information, indicate why. c)If the test was selected, indicate whether or not the test was run to completion. If the status of the test was "Not Run", indicate why (as defined in Clause 7.6.2.3). d) Enter the verdict as assigned during the test campaign. e) Enter a reference to any observations made in Section 7 of this report.. 7 OBSERVATIONS — Additional information relevant to the technical content of the PCTR may be given here. ANNEX C (This annex is an integral part of this Recommendation) Skeleton PIXIT Proforma5) C.1 Introduction This annex provides a skeleton structure for a PIXIT proforma. The test laboratory shall use this annex to produce a comprehensive PIXIT proforma of the same structure related to a specific abstract test suite and to the relevant Means of Testing. The comprehensive PIXIT proforma shall be provided to a client for completion when the related abstract test suite is to be used for testing against that client’s IUT. Note – This skeleton PIXIT proforma is not oriented to any specific protocols, abstract test method or Means of Testing and therefore should be modified and expanded as required by individual circumstances. Text in italics is comment for guidance purposes only, and is not to be included in the actual proforma. The test laboratory shall provide the information requested in Sections 1, 2 and 3 before issuing the proforma to the client. The client shall provide the information requested in Sections 4 and 5 before returning the proforma. Both parties shall contribute the information requested in the remaining sections. C.2 PIXIT Proforma The PIXIT proforma shall use the following format: PIXIT PROFORMA FOR Abstract Test Suite Name 1. IDENTIFICATION SUMMARY — PIXIT Number: Test Laboratory Name: Date of Issue: Issued to: Name of client The test laboratory may include client or contract references in the identification summary. 2. ABSTRACT TEST SUITE SUMMARY — Protocol Standard/Recommendation: Protocol(s) to be tested ATS Standard/Recommendation: Abstract Test Method: 3. TEST LABORATORY — This section shall contain information for the identification of the test laboratory and Means of Testing. Typically, this should include: Test Laboratory Identification: Name and addressing details Test Laboratory Manager: Name Means of Testing: SAP Address(es): Lower Tester addresses for testing the SUT Instructions for Completion: The test laboratory shall include any special instructions necessary for the completion and return of the proforma by the client. 4. CLIENT — This section shall contain information for the identification of the client. Typically, this should include: Client Identification: Name and addressing details Client Test Manager: Name Test Facilities Required: The client shall record the particular facilities required for testing, if a range of facilities is provided by the test laboratory. 5. SUT — Name: Version: SCS Number: Machine Configuration: Machine on which the SUT is mounted. Operating System Identification: Upper Tester Identification: if any Upper Tester Validation Date: if appropriate IUT Identification: PICS Reference for IUT: Limitations of the SUT: The client may provide information explaining if any of the abstract tests cannot be executed, for example, non-realization of ASPs if the Remote Test Method is used. Environmental Conditions: 6. ANCILLARY PROTOCOLS — In the following table, the client shall identify relevant information concerning each ancillary protocol of the SUT: PIGS PIXIT PCTR Protocol Version No. Ref. Ref. Ref. Name (opt.) (opt.) (opt.) One clause is required for each ancillary protocol included in the SUT other than the IUT itself. The information required is depend ent on the Means of Testing and the SUT, and covers all the addressing, parameter values, timer values and facilities (relevant to CCITT recommendations) as defined by the PICS for each protocol. 7. PROTOCOL LAYER INFORMATION (for Protocol Name) — 7.1 PROTOCOL IDENTIFICATION — Name: Version: PICS Reference: 7.2 IUT INFORMATION — This clause shall include such items as addresses, parameter values and timer values required to test the IUT. It shall be a refinement of the information provided in the PICS for the relevant protocol, but not conflict with it. Furthermore, it shall include test suite parameters where they are identified in the Abstract or Executable Test Suite specifications. 7.2.1ADDRESSES — This clause shall identify the SAP addresses to be used: a) by the lower tester to access the IUT (provided by the client); b)by the SUT to access the lower tester (provided by the test laboratory). 7.2.2PARAMETER VALUES — Parameter Parameter PICS Parameter Parameter Name Type Clause Range Value The client provides the parameter range and the exact parameter value to be used is agreed between the client and the test lab oratory. 7.2.3TIMER VALUES — Timer Name PICS Timer Timer Type Clause Range Value 7.2.4PROCEDURAL INFORMATION — This clause shall identify requirements for testing, placed by the ATS standard, which may not be realizable by the SUT and which may result in abstract test cases which cannot be executed; for example, if the Remote test method is used, non-realization of the control of sending a particular PDU by the IUT, that is, as required by the TTCN implicit send statement. ANNEX D (This annex is not an integral part of this Recommendation) Guidance for a PIXIT As the PIXIT will normally be used in conjunction with the PICS, it should not unnecessarily duplicate information provided by the PICS, other than that required to identify the client, the SUT and the protocol(s) to be tested. Cross references should be provided in the PIXIT to sections of the PICS wherever appropriate. Where the client and the test laboratory agree that large amounts of information are necessary to enable testing to be performed, the PIXIT should reference the appropriate documentation, rather than reproducing it. The PIXIT information will relate mainly to the protocol layer(s) to be tested. However, additional information relating to the ancillary layers (both higher and lower) involved in testing the IUT will also be required. The scope of this ancillary information will be dependent on the protocol layer to be tested, the test method and the Means of Testing, but may include protocol and addressing information for each ancillary layer in the SUT (see § C.6). Although the PIXIT is intended primarily to provide information about the testing environment of the IUT to the test laboratory, it is also useful to incorporate into the PIXIT that information (or references to it) which the client requires in order to prepare the SUT. The test laboratory incorporates this information into the PIXIT proforma that is supplied to the client for completion. Examples include lower tester information (for example addresses, timer values and parameter values) and the configuration of the ancillary layers to be used for testing. ANNEX E (This annex is not an integral part of this Recommendation) Summary of Conformance Testing documents E.1 Abbreviations in this annex In this annex, the following abbreviations apply: a) a reference in the format “4; M” refers to Recommendation X.294, Annex M; b)“test lab.” means test laboratory. E.2 Use of Conformance Testing documents by the client and test laboratory PICS PICS PIXIT PIXIT SCS Proforma Proforma Provided by: protocol 4; C client standard Partly completed test by: lab. Completed by: client client client Held by: test test test lab. lab. lab. SCTR SCTR PCTR PCTR Conforma Proforma Proforma nce log Provided by: 4; A 4; B test lab. Partly completed by: Completed by: test test test lab. lab. lab. Held by: test test test lab. lab. lab. _______________________________ 1) Recommendation X.294 and ISO 9646-5, Information technology – Open Systems Interconnection – Conformance Testing Methodology and Framework – Part 5: Requirements on Test Laboratories and Clients for Conformance Assessment Process, are technically aligned. 2) To be published. 3) Copyright release for SCTR Proformas Users of this Recommendation may freely reproduce the SCTR proforma in this annex so that it can be used for its intended purpose and may further publish the completed SCTR. 4) Copyright release for PCTR proformas. Users of this Recommendation may freely reproduce the PCTR proforma in this annex so that it can be used for its intended purpose and may further publish the completed PCTR. 5) Copyright release for PIXIT proformas. Users of this Recommendation may freely reproduce the PIXIT proforma in this annex so that it can be used for its intended purpose and may further publish the completed PIXIT.