November 16, 1989 TO: X3T9.3 Fiber Channel Working Group Members FROM: Roger Cummings SUBJECT: FIBER CHANNEL WORKING GROUP MINUTES Please find attached drafts of: a) The minutes of the ANSI X3T9.3 Fiber Channel Working Group of October 19, 1989. Note that there are also six Attachments to the minutes that relate to presentations at the meeting (and that the version of the minutes reflects the minor amendments made at the November meeting). b) The minutes of the ANSI X3T9.3 Fiber Channel Working Group of November 2 and 3, 1989. Note that there are also seven Attachments to the minutes that relate to presentations at the meeting. Note that the minutes in a) above were produced by Wayne Roiger of Cray Research, who thus deserves the thanks of the Working Group. For your information, there are four methods of obtaining these minutes: 1) Attend at least alternate Working Group meetings, as a result of which you will receive the minutes and attachments package directly. (Note that this is presently true, but as it is dependent upon volunteers to perform the mailing it may cease in the future). 2) Subscribe to the regular bimonthly X3T9.3 mailings produced by CEBEMA. The cost for this mailing for 1989 was $50 in addition to the $250 fee to be an Observer on X3T9.3. The contact at CBEMA is Katrina Grey at (202) 626- 5741 and the 1990 subscription begins with the April mailing. 3) Obtain individual X3T9.3 mailings from Global Engineering at (800) 854-7179. The price of each mailing is unknown. For 1989 the contents of the mailings are as follows: X3T9.3 April: January FC Minutes X3T9.3 June: No FC Minutes X3T9.3 August: March and May FC Minutes X3T9.3 October: July and September FC Minutes X3T9.3 December: No FC Minutes The October and November FC Minutes will be in the February 1990 mailing. 4) Download the minutes only (i.e. no attachments) from Area 17 of the SCSI Bulletin Board provided by NCR at (316) 636-8700. Rates of 300, 1200 and 2400 Baud are supported, and the recommended line settings are 8 data bits and no parity. The next Fiber Channel Working Group will be held on the Wednesday of the December Plenary Week (December 6), which is being hosted by NCR at the Catamaran Resort Hotel in San Diego CA. All attenders are asked to bring to the meeting details of the configurations which they require the Fiber Channel to support. A meeting notice for the December plenary week is attached along with a schedule of X3T9.3 meetings (both plenaries and working groups) for 1990. I would like to express my appreciation to all the attenders of the November Fiber Channel meeting for the thanks that I received for acting as Secretary over the last year. It is most gratifying to know that the effort to produce the minutes and mailings is appreciated. I have also passed on your thanks to the people at StorageTek who have provided such great help in this matter. If there are any corrections required to, or omissions noted from, the minutes I can be reached as follows: Phone: (303) 673-6357 (Business) (303) 665-0761 (Home) Internet: 70106.307@compuserve.com Fax: (303) 673-5891 Regards Roger Cummings Senior Engineer Subsystems Controller Development MD4271 #ndm0/rc The December X3T9 plenary week is being hosted by Fred Meadows and NCR at the Catamaran resort Hotecl in San Diego, CA. Call (619) 488-1081 for reservations. The rate for the X3T9/NCR group is $99 single and $114 double. 1990 SCHEDULE OF MEETINGS DATES EVENT LOCATION 01/08 - 01/12 Working Groups San Jose, CA 02/19 - 02/21 Plenaries and Working Groups Austin, TX 03/05 - 03/09 Working Groups Irvine, CA 04/23 - 04/25 Plenaries and Working Groups St. Pete., FL 05/07 - 05/11 Working Groups Chicago, IL 06/18 - 06/20 Plenaries and Working Groups Wichita, KS 07/09 - 07/13 Working Groups Rochester, MN 08/20 - 08/22 Plenaries and Working Groups Seattle, WA 09/04 - 09/07 Working Groups Denver, CO 10/15 - 10/17 Plenaries and Working Groups Valley Forge, PA 10/29 - 11/02 Working Groups TBD 12/03 - 12/05 Plenaries and Working Groups San Diego, CA The format of the week for months containing only working groups is as follows: Monday SCSI Common Access Method Group Tuesday and Wednesday SCSI Working Group Thursday and Friday Fiber Channel Working Group The format of the week for the months containing both plenaries and working groups is as follows: Monday Separate X3T9.2 (SCSI) and X3T9.3 plenaries (Monday portion of X3T9.3 devoted to IPI and HSC) Tuesday Separate X3T9.2 (SCSI) and X3T9.3 plenaries (Tuesday portion of X3T9.3 devoted to Fiber Channel) Wednesday Fiber Channel Working Group. MINUTES OF THE TWELFTH FIBER CHANNEL WORKING GROUP MEETING The Twelfth meeting of the ANSI X3T9.3 Fiber Channel Working Group was hosted by Ed Cardinal of Amdahl/Key Computer at the Amdahl Education Center at 3120 Scott Boulevard, Santa Clara, CA on November 2 and 3, 1989. A total of 46 people attended, as follows: ADVANCED MICRO DEVICES Bin Guv Jim Kubinec AMDAHL Rich Taborek AMDAHL/KEY COMPUTER LABS. Ed Cardinal Owen L. Gehlert ANCOR COMMUNICATION Terry Anderson AT&T MICROELECTRONICS Phillip Fraley CANSTAR Kumar Malavalli CONTROL DATA Wayne Sanderson CORNING INC. Kevin Able Steven E. Swanson CRAY RESEARCH INC. Marvin Bausman John Renwick Wayne Roiger ENDL Dal Allan ESL INC. David F. Hepner FORD AEROSPACE Gary Waldeck FUJITSU AMERICA Bob Driscal Kohji Mohri GAZELLE MICROCIRCUITS David Macmillan GIGABIT LOGIC Carl Deierling IBM Henry Brandt K. C. Chennappan Robert Dugan Charlie Martin Joseph R. Mathis Jerry Radcliffe Leon Skarshinski Ron Soderstrom Horst L Truestedt Rick Warren Carl Zeitler IBM GENERAL PRODUCTS DIVISION Gary Stephens LAWRENCE LIVERMORE NATIONAL LABS. John Severyn LOS ALAMOS NATIONAL LAB Don Tolmie MAXTOR CORP. Larry Lamers NATIONAL SEMICONDUCTOR Sam Laymoun NCR John Lohmeyer ORTEL Alexander Liebovich PCO Jim Goell STORAGETEK Roger Cummings Floyd Paurus Steve Zanowick SUN MICROSYSTEMS Robert N Snively TANDEM COMPUTERS Jim Smith UNISYS Doug Morrissey The meeting was opened by the Chairman of the Working Group, Dal Allan of ENDL Consulting, by reviewing some administrative matters connected with the working group. Dal noted that he had been asked to set up group discount schemes for air travel with United and Northwest to the Working Groups, but that the United group priviledges were about be be withdrawn due to lack of use. He encouraged all of the attenders to investigate the considerable savings that these schemes allow. A number of people responded that they had tried to use the group fares, but had considerable difficulty if convincing their corporate travel departments to do so. For future reference, the numbers necessary to use the group fares are: United Phone: 800-521-4041 Account: ENDL #413TA NW Phone: 800-328-1111 Account: ENDL #15201 The next subject addressed was that of the minutes and mailings of the Working Group. Roger Cummings of Storagetek, who had missed the meeting in Raleigh, agreed to resume as Secretary - and received a round of applause from the meeting as thanks for past work in this position. Roger confirmed that the rule limiting direct mailing of the minutes to attenders of the either of the most recent two meetings was being strictly enforced, and noted that volunteers to perform future mailings would be welcome. Dal Allan emphasized that the minutes were being included in the bimonthly X3T9.3 mailings, and noted that these mailings were now also obtainable from Global Engineering on an individual basis. Roger also noted that the text of the minutes only (i.e. no attachments) for all past Fiber Channel Working Groups could be found in Area 17 of the SCSI BBS on 316-636-8700. The schedule for working group meetings for 1990 was then discussed. Dal Allan and John Lohmeyer of NCR (the Chairman of the X3T9.2 (SCSI) committee) have arranged for a full week of working groups to be held at a single location in the months that do not contain plenary meetings. The schedule for this week is that the SCSI Common Access Method (CAM) group meets on Monday, the SCSI Working Group on Tuesday and Wednesday, and the Fiber Channel Working Group on Thursday and Friday. Offers were made by IBM to host the July meetings in Rochester MN., and Storagetek to host the September meetings in Denver, CO - with both being subject to confirmation by their respective managements. A host for the November meetings was still being sought, with a number of people suggesting Honolulu as the preferred location! The administrative matters having been concluded, Dal Allan distributed an agenda for the meeting. A copy of Dal's agenda is Attachment 1. It was noted that Ultra Technologies had again not provided a speaker, and thus they were dropped from the agenda. The minutes of the Eleventh Fiber Channel Working Group meeting, that was held in Raleigh NC. on October 18, were then reviewed. The minutes had been produced by Wayne Roiger of Cray Research. A number of minor amendments to the minutes were made, and these are included in the mailing for that meeting. The minutes of the HSC and Fiber Channel portions of the October X3T9.3 plenary were then distributed by Ed Cardinal of Amdahl/Key Labs. A copy of these minutes is Attachment 2. The first presentation was then given by Don Tolmie of Los Alamos National Labs. on the subject of Simplex versus Duplex configurations for the Fiber Channel. A copy of Don's slides is Attachment 3. Don described the data flows associated with a number of different types of equipment, and concluded that in a number of the situations these flows were either simplex or half-duplex. Therefore he suggested that the Fiber Channel should also only support a simplex configuration on the grounds of economy. However Don's scenario was immediately questioned. Horst Truestedt of IBM noted that the data flow of a disk controller could appear to be full duplex by multiplexing the traffic from different peripherals. Bob Dugan of IBM asked if the return of an acknowledgment of the delivery of the simplex data flow constituted a full duplex link, and Don replied that it did not in his terms. Wayne Sanderson of Control Data noted that IPI and SCSI are clearly moving towards true full duplex operation. John Renwick of Cray Research stated that the basic capability of fiber allows a duplex link to be built with little cost penalty, and Gary Stephens of IBM noted that many interfaces have balanced loads with CERN and NASA being significant exceptions. The major point that was noted during the discussion on Don's presentataion was that configuring the channel as a single full duplex link, or two simplex links in opposite directions, has a major impact on the design and complexity of a switch. In the former case the forward and reverse paths have to be switched at the same time, and in the latter both the protocol and the switch have to deal with the situation in which one of the paths becomes busy due to a routing conflict etc. Wayne Sanderson expressed the view that the Fiber Channel should not define a standard switch architecture, but should be designed to support a number of different architectures e.g. a space switch and a "broadcast hub" system a l… Canstar. This lead to a discussion on required switching performance, and Gary Stephens described one possible configuration of a cpu using one Fiber Channel to multiplex between thousands of devices using 4096 byte "bursts". At 100 Megabytes/s a burst therefore takes 40 microseconds, and the time to establish a route through the switch therefore needs to be short in comparison with this burst transfer time. Terry Anderson of Ancor Communications noted that for a space switch the time to establish a path is dependent upon the size of the switch, but that it could be as low as 700 nanoseconds. Don was followed by Joe Mathis of IBM, who described an updated version of the FC-2 protocol that he had first introduced at the September meeting in Minneapolis. A copy of Joe's slides is Attachment 4. Joe also distributed a preliminary FC-2 specification, noting that there were still some major inconsistencies amongst the sections produced by the three authors. The specification is included as Attachment 6 for information. Joe began by noting the definitions of the terms Originator, Responder, and Exchange that had been agreed at the October meeting. The definitions are as follows: Originator An equipment that injects FC-2 frames into the Fiber Channel. Responder An equipment that receives FC-2 frames from the Fiber Channel. Exchange A self-contained transfer of information between an Originator and one (or more) Responder(s) over the Fiber Channel. Joe reviewed the link level frame format, and in response to a question from Don Tolmie stated that all of the field sizes shown were in bytes. Joe then defined four Frame Types, one of which was a Device Long frame which included a data field for which four sizes were defined - namely 256, 512, 1024 and 2048 bytes. Dal Allan stated that he had learned that the maximum allowable packet size in FDDI was defined as 4500 bytes, and his suggestion that a data field size of 4096 bytes also be supported was accepted. Joe then moved on to the subject of Link Level frames, and noted that two types of reply protocol were supported. The Ready type protocol requires explicit notification from the responder that it has a buffer available before frame transmission, and the Acknowledge protocol returns an explicit indication that the frame was received successfully. This initiated a long discussion on the merits of the various schemes especially with relation to various classes of topology. Rich Taborek of Amdahl noted the absence of a Negative Acknowledge (NACK), and was told that it would be simple to add it if required. John Renwick postulated that FC-2 should not retransmit FC-3 information, only frames associated with itself. Eventually three types of reply protocol were defined, namely Ship and Pray, Acknowledge frames only until an error is detected, and Acknowledge all correctly received frames - and a straw poll indicated that the last option was preferable. It was also identified that out of sequence delivery of frames should be regarded as an error condition. Joe then described the new concept of Link Management frames, which provide a method by which cascaded switches can communicate. This is the equivalent of out of band signalling in current switch architectures. An special character at the end of a frame is used to indicate that an 80 byte Link Management frame can be inserted following that frame, and Joe demonstrated that this concept would still allow 100 Megabytes of data to be transported on a 1062.5 Megabaud link. It was pointed out that a switch that had inserted a Link Management frame would have to remove the special character that gave it permission to prevent the frame being overwritten by subsequent switches. Joe agreed but noted that there are many other ways that this function could be implemented, including the use of special replaceable characters. John Renwick asked if Flow Control could be implemented using these Management frames, and Joe replied that this was being considered. Joe moved on to consider the attributes of the Link Level frames, and agreed that the Link Management frame should be renamed Switch Management, and that Abort Operation would become Abort Exchange. Roger Cummings asked where the Reply Frames were generated, and was told that an Accept is generated by a Responder, but that a Busy or a Reject could be generated by a Responder or a Switch. Henry Brandt of IBM clarified that Reject is an error condition, but Busy indicated that the Exchange should be retried. The subjects of Path and Connection Management were then considered. Joe described a system topology of multiple cascaded switches, and showed an example of a path definition that did not span the entire path between one network node and another but allowed Virtual Connections from multiple nodes to share that defined path. This also generated much comment, and a number of people noted that this concept is equivalent to the manner in which multiple peripherals, each performing one operation, are multiplexed on to a single physical highway in today's IPI and SCSI systems. Joe then moved on to consider the frame fields required to perform flow control, and Roger Cummings and Gary Stephens pointed out that when multiple virtual connections were supported by an interface the buffers and status registers available would have to be partitioned amongst those multiple connections. This concluded the business of the meeting for the first day, and David Macmillan of Gazelle Microcircuits issued an invitation to the attenders to visit their plant (which is adjacent to Amdahl). The main purpose of the visit was to see a demonstration of their TTL-compatible Hot Rod Gallium Arsenide circuit, which acts as a transmitter and receiver for a full duplex 1 Gigabaud serial interface and uses 4B/5B encoding. A very brief data sheet for the Hot Rod is Attachment 7, and further information can be obtained directly from David at (408)982-0900. Tuesday morning began with a summary by Joe Mathis of the discussion that resulted from his presentation on the previous day. Joe began by suggesting a nomenclature as follows: [A] Network address of Node A [A]:[B]:......:[Z] Path from A to Z {[a],[b],......,[y]} Address domain of a switch Rich Taborek asked for the definition of a node, and Dal Allan suggested that the node address could be regarded as a port address. Terry Anderson suggested that an explicit path be defined as one that is defined as above, and an implicit path be defined as one where the routing is determined by the switching fabric. The other points of Joe's summary were:  No NACK frames will be used.  The ACK protocol will be as follows:  ACK all good frames;  FC-3 defines recovery policy (this requires use use of control flags at the beginning of an operation).  Frames within exchanges do not arrive out of sequence in normal operation.  An explicit timer function is to be included in FC-2.  An OPERATION is defined as a group of related exchanges.  The definition of a Path implies physical locking.  Connections are virtual, and the environment needs to be established before they are used.  Multiple virtual connections can be open simultaneously up to a limit defined by the implementation or application.  Multiple simultaneous exchanges are supported over a virtual connection, but not over a defined physical path.  A Path can be defined between an Originator and Responder, or an Originator and a switch port.  The Buffer Count can be specified at either connection or exchange time.  The multiplexing of virtual connections is on an exchange basis. Joe closed by suggesting that as Busy is a form of rejection, it could be handled by the Reject frame with a defined reason code. The general preference, however, seemed to be for maintaining a separate Busy frame. Dal Allan specifically thanked Joe for bearing the brunt of the group's questions, and for producing the summary of the discussions. Joe was followed by Ron Soderstrom of IBM, who gave an update of a previous presentation on Low Cost, High Performance Optical Components. A copy of Ron's slides is Attachment 5. Ron began by noting that the telephone companies use a large quantity of fiber, but few transmitters and receivers - and therefore most of the cost reduction efforts have been directed at the fiber. In computer links such as the Fiber Channel this will of course be reversed with the cost of transmitters and receivers dominating. He then moved on the consider the available types of optical source. He noted that some companies have been able to operate longwave leds at reduced optical power levels at Gigabaud rates by using high dopant concentrations, but this may lead to lower device reliability. In contrast the shortwave laser was developed for use in Compact Disk (CD) audio products, and although there are no speed requirements in the CD application the device structure intrinsically allows performance at Gigabit rates. John Severyn of Lawrence Livermore Labs. asked if the package limited the speed attainable from the laser, and was told that IBM has been able run the devices at 2 Gigabaud in the laboratory, and that Japanese reports in the literature support that this is feasible. Kevin Able of Corning asked if there was a reliability degradation in driving the devices at those high rates, and Ron replied that the reliability is a function of the average power dissipated and not the data rate. Ron then noted that from mid 1988 to mid 1989 it is estimated that 150 million shortwave lasers were produced. This compares with a yearly volume for longwave lasers of 3000 parts maximum. He noted that the prices for these longwave lasers have decreased from $1K- 2K two years ago to $200-250 today, but cautioned that they were still fabricated in Gallium Arsenide in a standalone package whereas a led is typically found in a connectorized package with integrated drive electronics. Ron stated that the optical power versus temperature curve for the shortwave laser is so steep that variations of only a few degrees cause wide fluctuations in optical power output, and therefore a closed loop control system that makes use of the inherent back facet monitor is a must for stable, reliable operation. Phillip Fraley of AT&T asked if packages with an integrated silicon photodiode were available, and Ron replied that they had been reported at technical conferences but that as yet no products were available. Ron identified other advantages of the shortwave laser as being a low drive current and operation from a single +5 volt supply. Ron then moved on to describe three different types of laser - namely the Self-Pulsating, Multi-Longitudinal Mode and Single Longitudinal Mode types. He described the automated testing that is being performed at IBM Rochester, in which 2500 lasers are being tested in various environments, and demonstrating an average failure rate of less than one 0.01% per thousand hours. Terry Anderson asked how the shortwave lasers performed over 62.5 þm fiber, and was told that as the fiber is tuned for longer wavelengths the performance is "terrible". Kevin Able amplified this by noting that Corning does not characterize any fiber at 780 nm, and that 9 þm fiber does not ensure single mode operation at that frequency. He also noted that, in operating a fiber away from its zero dispersion wavelength, chromatic dispersion and absorption becomes an issue. Kevin stated that Bellcore had left open the use of 780 nm components in "fiber to the home" applications due to their potential low cost, and so components and fiber may in the future be characterized for 780 nm operation, but today the knowledge base is insufficient to support standardization. Ron then closed by considering receptacle designs and detector parameters. He noted that the use of a large area Silcon Pin photodetector meant that alignment could be successfully obtained without the necessity for lenses etc. Ron was followed by John Renwick, who listed the options for FC- 2 Error Handling as follows:  Receiver Notification Only:  Leave it up to FC-3 a Timer  No Acknowledge  Notify Sender and Receiver  Recovery in FC-3 with faster trigger  Ack and FC-2 Timer or Nack  ECC in FC-2, Error Correction at Destination  Retransmission in FC-2  From point of error (go back N) -OR-  Selective retransmission John noted that reordering of frames received out of sequence could be made an implementation choice (assuming that frames have sequence numbers), and suggested that the standard should not dictate this matter. The group then reviewed Dal Allan's list of 1988 Fiber Channel parameters that is contained in Attachment 1. The revised list that was generated is as follows: Cost Cheap (SCSI low end), Competitive w/copper, Premium acceptable Speed 10, 25, 50, 100, 200 Mbytes/s Distance 20 m, less than 2 Km, greater than 2 Km Error Rates 10-9 detected, 10-16 undetected Performance Low Overhead, 80% data payload, low latency Connections Up to 32 MTBF 200K Hours up to FC-2 Environment 0-65 degrees C, 10-99% humidity, +5 volts, shock resistant, tight cable bend Fault Tolerant Redundant Paths Baud Rate 1062.5 Megabaud rate at less than 2 Km Coding IBM 8B/10B Dal Allan then closed the meeting by suggesting that the major subject for the December meeting be topology, and asking all attenders to bring to that meeting the configurations that they require to be supported by the Fiber Channel.