December 15, 1989 TO: X3T9.3 Fiber Channel Working Group Members FROM: Roger Cummings SUBJECT: FIBER CHANNEL WORKING GROUP MINUTES Please find attached a draft of the minutes of the ANSI X3T9.3 Fiber Channel Working Group of December 5 and 6, 1989. Note that there are also sixteen Attachments to the minutes that relate to presentations at the meeting. The next Fiber Channel Working Group will be hosted by Sun Microsystems at the Red Lion Hotel in San Jose, CA on January 11 and 12, 1990 as part of the January Working Group week. A meeting notice for this week is attached along with a schedule of X3T9.3 meetings (both plenaries and working groups) for 1990. It is recommended that reservations be made at the Red Lion as soon as possible, because the number of rooms available is limited. Note that in addition to the regularly scheduled Working Group meetings, it is intended to hold the following working meetings during the week beginning January 8, 1990: Monday SCSI Cable Working Meeting Monday and Tuesday HPPI Data Framing Working Meeting Wednesday IPI Device Generic Protocols on HPPI Working Meeting Thursday and Friday IPI Device Specific Disk Command Set Working Meeting If there are any corrections required to, or omissions noted from, the minutes I can be reached as follows: Phone: (303) 673-6357 (Business) (303) 665-0761 (Home) Internet: 70106.307@compuserve.com Fax: (303) 673-5891 Regards Roger Cummings Senior Engineer Subsystems Controller Development MD4271 #neb0/rc 1990 SCHEDULE OF MEETINGS DATES EVENT LOCATION 01/08 - 01/12 Working Groups San Jose, CA 02/19 - 02/21 Plenaries and Working Groups Austin, TX 03/05 - 03/09 Working Groups Irvine, CA 04/23 - 04/25 Plenaries and Working Groups St. Pete., FL 05/07 - 05/11 Working Groups Chicago, IL 06/18 - 06/20 Plenaries and Working Groups Wichita, KS 07/09 - 07/13 Working Groups Rochester, MN 08/20 - 08/22 Plenaries and Working Groups Seattle, WA 09/04 - 09/07 Working Groups Denver, CO 10/15 - 10/17 Plenaries and Working Groups Valley Forge, PA 10/29 - 11/02 Working Groups TBD 12/03 - 12/05 Plenaries and Working Groups San Diego, CA The format of the week for months containing only working groups is as follows: Monday SCSI Common Access Method Group Tuesday and Wednesday SCSI Working Group Thursday and Friday Fiber Channel Working Group The format of the week for the months containing both plenaries and working groups is as follows: Monday Separate X3T9.2 (SCSI) and X3T9.3 plenaries (Monday portion of X3T9.3 devoted to IPI and HPPI) Tuesday Separate X3T9.2 (SCSI) and X3T9.3 plenaries (Tuesday portion of X3T9.3 devoted to Fiber Channel) Wednesday Fiber Channel Working Group. X3T9.2/89-141 MEETING ANNOUNCEMENTS CAM MEETING JANUARY 8, 1990 SCSI WORKING GROUP MEETING JANUARY 9 AND 10, 1990 FIBER CHANNEL MEETING JANUARY 11 AND 12, 1990 SUN MICROSYSTEMS WELCOMES YOU TO ATTEND THE MEETINGS IN SAN JOSE, CALIFORNIA TO BE HELD AT Red Lion Inn 2050 Gateway Place San Jose, California 95110 Phone: (800) 547-8010 (408) 453-4000 Sun Microsystems is hosting the CAM meeting, the SCSI Working Group meeting, and the Fiber Channel Meeting from January 8 through January 12, 1990. A block of rooms has been laid aside at a cost of $95.00 per night, single occupancy, not including taxes. When making reservations, refer to the Sun Microsystems/SCSI meeting to be sure you get the correct room rate. The hotel is sorry to report that they do not have data line connections yet and that you will have to use acoustic couplers. Free shuttle service is available from the San Jose Airport, about three minutes away from the hotel. If you are arriving by car, take the First Street exit off highway 101 toward the airport. The hotel is clearly visible on your right just before the airport. If you have any questions, you can contact Bob Snively at Sun Microsystems for further information. His phone is (415) 336-5332. Lifeco Meeting Services has a toll-free number, (800) 821-1342, to help you with any travel arrangements if you want to take advantage of their service. American Airlines has made available a STAR number providing significant discounts (5% to 45%) on fares for people coming to the meeting. The number can be booked and ticketed by any travel agency. Please refer to STAR # S15Z0VL. The booking must be made by telephone to 1-800-433-1790. A United group fare is also available. The account number is 0167N and the telephone number 1-800-521-4041. The Northwest group fare organized by ENDL may also still be available. The account number is #15201 and the telephone number 1-800-328-1111. I hope that this helps those who will be coming to San Jose, American Airlines' newest hub. Bob Snively MINUTES OF THE THIRTEENTH FIBER CHANNEL WORKING GROUP MEETING The Thirteenth meeting of the ANSI X3T9.3 Fiber Channel Working Group was held on the afternoon of December 5 and on December 6 at the Catamaran Resort Hotel in San Diego, CA as part of an X3T9 plenary week hosted by Fred Meadows of NCR. A total of 64 people attended as follows: ADVANCED MICRO DEVICES Jim Kubinec Tom Slykhouse Yun-Che Wang AMDAHL Rich Taborek AMDAHL/KEY COMPUTER LABS. Ed Cardinal AMP Charles Brill Bob Southard AMPHENOL INTERCONNECT John LaShier ANCOR COMMUNICATION Terry Anderson AT&T James E Morris BT&D TECHNOLOGIES Ray Johnson CANSTAR Kumar Malavalli CONTROL DATA Wayne Sanderson CRAY RESEARCH INC. John Renwick Wayne Roiger DATA GENERAL Bill Mathrani DIGITAL EQUIPMENT CORP. Chris Baldwin ENDL Dal Allan FORD AEROSPACE Gary Waldeck FUJITSU AMERICA Bob Driscal Kohji Mohri HARRIS SPACE SYSTEMS CORP. Allen Williams HEWLETT PACKARD Kurt Chan Del Hanson HITACHI DATA SYSTEMS Robert Freund IBM Henry Brandt Ed Calkins K. C. Chennappan Robert Dugan Charlie Martin Joseph R. Mathis Jerry Radcliffe Frank Shott Horst L Truestedt Carl Zeitler IBM RESEARCH Hans R. Muller IPITEK INC. Joel Steinberg LASER DIODE INC. Henry J. Wojtunik LASERTRON Kenneth Nill LAWRENCE LIVERMORE NATIONAL LABS. John Severyn LOS ALAMOS NATIONAL LAB Richard Thomsen Don Tolmie LYTEL John DeAndrea MADISON CABLE Bob Bellino Brian Morissette MAXTOR CORP. Larry Lamers NATIONAL SEMICONDUCTOR Sam Laymoun NCR John Lohmeyer ORTEL Israel Ury SAPPHIRE SYSTEMS Don Deel SHELL DEVELOPMENT CO. Patric Savage SIEMENS FIBER OPTIC COMPONENTS Schelto Van Doorn SPECTRAGRAPHICS Scott Palmquist STORAGETEK Roger Cummings Steve Zanowick SUMITOMO ELECTRIC Harry Takada TANDEM COMPUTERS Mary Henske Pete Tobias THINKING MACHINES CORP. Edward Parish THOMAS AND BETTS Harvey Waltersdorf ULTRA NETWORK TECHNOLOGIES Kent Bossange Mark Bryers UNISYS Doug Morrissey VITESSE SEMICONDUCTOR Michael Millhollan The meeting was opened by the Chairman of the Working Group, Dal Allan of ENDL Consulting, who identified the four major agenda items as being: a) To detail existing configurations using copper interfaces that are candidates for an upgrade using the Fiber Channel. b) To discuss requirements for a 250 Megabaud FC-0. c) To continue to work on the FC-2 protocol d) To hear presentations by AMD and Ultra. Dal then asked for details of the existing configurations, and learned that only Rich Taborek of Amdahl had prepared slides on the subject. A copy of Rich's presentation is Attachment 1. Rich began by describing a current Amdahl configuration in which a number of cpu complexes are connected via multiple paths to control units which connect to multiported disk devices. He stated that the cpu complex may contain up to 8 cooperating cpu's in a manner similar to the IBM 3090, and that it is possible to connect up to 16,000 block multiplexer channels to each complex. Each channel supports a rate of 4.5 Megabytes/s (limited by the channel protocol) and operates over a distance of 400 feet. Each control unit may connect to 4-16 channels, and a typical configuration is 16 control units each with four datapaths per cpu complex. He noted that some control units can support up to sixteen concurrent operations, and stated that there are interfaces between control units to support cache coherency. He emphasized that although some static switches (such as T-Bars) are used in Amdahl systems, they are not used with the channels that connect to disks. Wayne Sanderson of Control Data noted that the topology of an IPI- based system would be identical to that described by Rich, and Henry Brandt of IBM asked if the Fiber Channel needed to support a similar "daisy-chain" topology, and was told that it was being considered. Dal Allan noted that Rich's description of the control units incorporated the functionality of the Storage Director in the IBM architecture. Rich then moved on to consider possible configurations based on the Fiber Channel. He noted that the current half-duplex copper channels would be replaced by full-duplex fiber ones, and that the copper "daisy-chains" could well be replaced by a switched configuration. There was then a short recess while other people drew the existing configurations to which they saw Fiber Channel being applied in the future, and these were then presented to the group. The first of these configurations was a SCSI system presented by Dal Allan. A copy of Dal's slide is Attachment 2. He noted that SCSI provides a two level addressing scheme and that a 32 bit wide system is the "latest and greatest". Dal was followed by John Renwick of Cray Research, who presented four configurations based on HPPI. A copy of John's slides is Attachment 3. He noted that the symmetrical nature of HPPI is very useful in executing diagnostics (configuration b)), and that in configuration c) in which three cpus are interconnected, the HPPI channel is still simplex. In addition, Configuration d) shows two cpus with an asymmetric dataflow between them. John stated that average configuration today there are four HSX channel pairs per cpu, and that he expects this to increase in the future. Don Tolmie then described the present Los Alamos configuration. A representation and description of this configuration is given in Attachment 4. Don stated that the central switch could be a store and forward packet or circuit switch type, and noted that the broadcast function to the workstations is today performed over Ethernet. He described the switch as supporting a mix of types of application and noted that it does not use HPPI today. In the future he expects the switch type to change, to perhaps use the Fiber Channel as the interface to the switch, and also perhaps to use FDDI as the interface to the many workstations. Roger Cummings of Storagetek then described a configuration consisting of multiple workstations with centralized disk storage. A copy of the figure with a description is Attachment 5. Robert Freund of Hitachi Data Systems then defined a configuration, which is shown in Attachment 6. He noted that the separation between the mainframe and the peripherals was defined by asset control requirements, and justified the length of 5 Km by stating that their standard product covers 2 Km today and that most of the "specials" that they are asked to quote are for 5 Km. He saw this requirement arising from the increasing cost of downtown real estate, which is forcing many companies to fragment their facilities across multiple sites in a geographic area of perhaps ten city blocks. He named this configuration a network of sub- campuses. Horst Truestedt of IBM then defined and described a typical IPI configuration. Horst's configuration and description is Attachment 7. Dal Allan then illustrated the concept of dynamic pathing that is a feature of the IPI Device Generic Command sets, and is being proposed for inclusion in SCSI-3. A copy of Dal's slide is Attachment 8. Allen Williams of Harris Space Systems then described the CORE system, which is intended to provide real-time monitoring and control of launches at the Kennedy Space Center. The system is intended to be used with heavy lift launch vehicles for missions to the Moon and to Mars into the next century. It will also be used for missions associated with the Space Station, which represents its biggest payload. CORE is required to support up to 50,000 changes per second on the remotely-monitored controls for these missions. Allen described two system configurations and these descriptions are contained in Attachment 9. The first configuration (the Baseline) is based on an FDDI configuration, and uses a special deterministic version of Ethernet for some of the lower speed connections. Allen noted that while this configuration had adequate bandwidth, which was determined primarily by the data input architecture, it could not meet the requirements for latency due to the command and control processing which had to take place in the application processors. Therefore they were investigating alternative architectures including the one shown, which is very similar to the existing system. It is based on the use of a common data buffer with direct links to both the data input and application processing subsystems. Kent Bossange of Ultra Network Technologies asked if the addition of repeaters to the architectures had been considered, and was told that they had been avoided because of the intelligence that they would have to contain to return trouble reports etc. Dal Allan noted that ICL had included repeaters in their Macrolan system for reasons of future growth, and had sold many more systems with repeaters than they expected. Hans Muller of IBM Research noted that latency is an issue that should be carefully considered in the derivation of the Fiber Channel architecture, and the meeting was reminded that Gary Stephens of IBM had suggested the provision of a separate CRC for the header portion of the FC-2 frame as a method of reducing latency. Kent Bossange stated that Ultra had looked at this tradeoff in the design of their UltraNet and decided against the separate CRC because of the predicted low frequency of errors. Terry Anderson of Ancor Communications then presented an architectural view of his company's switching products. A copy of Terry's slide is Attachment 10. Kent Bossange questioned Terry's reference to a broadcast mode, and was told that it merely meant that one node should be capable of communicating with multiple other nodes. The general feeling was that such an ability would be useful, but that a definitive multicast mode of operation is not a requirement at this stage. However this lead into a more general discussion on the subject of the method of FC-2 addressing. Don Deel of Sapphire Systems identified the key issue as being one of "name space" i.e. whether the name of a node is equivalent to the path name to a node. It was noted that this may mean that the interconnection fabric be intelligent enough to perform the implied routing. As an alternative, Richard Thomsen of Los Alamos described a "source routing" scheme in which the "address" of a destination node is different from each point interconnected by the fabric. Bob Beach of Ultra Network Technologies stated that network management must be considered in determining the addressing scheme. Doug Morrissey of Unisys asked that, regardless of the method chosen, the addressing scheme be standardized such that one set of software would work with all "Fiber Channel compatible" switches. The discussion was then postponed until the following day. Wednesday began with a presentation by Jim Kubinec of AMD. A copy of Jim's slides is Attachment 11. Given the fact that it is well known that the 4B/5B code used in FDDI has a dc content, Jim's presentation was an attempt to quantify the effect of this dc imbalance. To do this AMD had performed an experiment using a random data generator, the data dependent jitter pattern specified for FDDI, and worst case dc offsets. Jim defined a receiver equivalent circuit, and noted that in a balanced code one tenth of the capacitance is needed to handle the dc offset as for the 4B/5B code. He showed why a 20 mV offset is the worst case for a 4 Kilobyte frame and why with the worst case pattern the offset saturates at 100 mV. Don Tolmie asked what effect two good characters at the end of the 4K frame has, and was told that the dc offset might recover to 20% but that three idle characters and a sync character were required for complete recovery. Jim then evaluated the effect of the dc offset on the data dependent jitter. John Severyn of Lawrence Livermore Labs. noted that the contribution of the receiver to the data dependent jitter also needs to be evaluated, and Jim noted that white gaussian noise had been used in the model to simulate this effect. In closing Jim noted that there is an economic factor in the definition of a coding scheme for a lower performance FC variant in that TAXI chips are already in volume use with several hundred customers for both point-point links and FDDI. The question to the group is therefore whether the cost edge that using TAXI chips would provide is worth the complexity of having different coding schemes for different FC variants. Jerry Radcliffe of IBM complimented Jim on a "nice linear analysis", but questioned if some non-linear effects need to be considered. Chris Baldwin of Digital Equipment noted that the effect of the dc offset on edge jitter (the uncertainty in the decision point at the receiver) has to be considered. Jim Morris of AT&T amplified this point by noting that Jim Kubinec's analysis was predicated upon the model of a single capacitor into a resistive load and that in datacom applications some "tricks" have been used which would complicate the model. Dal Allan asked that the analysis be reworked using what has already been established of the FC frame structure. The next presentation was given by Mark Bryers of Ultra Network technologies on the subject of the architecture of their UltraNet. A copy of Mark's presentation is Attachment 12. Mark began by describing the topology of UltraNet and the structure and operation of some of the components. He stated that the cards that are mounted in the UltraBus backplane are hot-pluggable. Don Tolmie inquired about the speed of the backplane, and was told that it carries a gigabit of user data and that the arbitration cycle takes 200 nanoseconds. Wayne Sanderson asked for the size of the packets, and was told that a 32 Kilobyte packet is normally used, and that the data highways are 64 bits wide and incorporate byte parity (as well as the ISO checksum that is used throughout the system). Mark then showed a multi-hub configuration in which up to four serial cables can be ganged to support a user data rate of up to a gigabit/s between hubs. He noted that the protocol of the inter- hub link is designed to be independent of the media, and that both fiber and copper interconnections are used. He stated that various data rates are supported per links, and that as it was a design goal that all link errors be detected, a 32 bit polynomial CRC was used in addition to the 4B/5B encoding (and the polynomial was handled completely in hardware). Del Hanson of Hewlett-Packard asked if the same media was required for all links in a ganged group, and was told that the only requirement was that all links be the same speed. Mark then moved on to describe some of the practical experience that had been gained with UltraNet. He noted that they had initially had a connector mating problem, and that although 75% of the problems that this caused were bit errors they did also see some burst errors. He also stated that they had found that their idle pattern had to have a random nature for two reasons. The first of these was that at power on the clock recovery circuitry would lock on to a harmonic close in frequency to the expected frequency. This was also a problem during diagnostic loopback testing where the circuitry would lock on to a harmonic of the number of bits in the loop. Jerry Radcliffe noted that part of this problem could be caused by overdriving the receiver and reminded everyone of the need to limit the received power in short cables. The second problem was that without the random idle it proved impossible to pass FCC testing with the links that used coaxial cables. Horst Truestedt questioned if this problem would be seen in a Fiber Channel coax version which is likely to only be used within a cabinet as its maximum operating distance will probably be in the order of 10 meters. Also the problem was noted to be a cost-related as the use of a more expensive double-shielded coax rather than the low-cost single-shielded coax would have eliminated the problem. Wayne Sanderson questioned if there would be interest in using this more expensive coax to allow the FC coax version to operate over a longer distance. Jerry Radcliffe asked what Ultra's experience with errors had been in stable systems and Mark replied that in the lab no errors are seen. Dal Allan then asked why forward error correction had been considered, and was told that it was not an issue in this environment but that a cross-country link by T1 or T3 would have been different. Mark then concluded by reviewing the different types of fiber and coax cables that Ultra had tested. He noted that the Corning fiber supports the highest rate, but that chromatic dispersion is also significant - and that only Corning had committed to a factor for this parameter. Del Hanson asked why multiple fiber types were supported, and was told that it was simply because of the multiplicity of the fiber types that customers already had installed. Dal Allan asked for a price comparison of high-quality coax versus fiber and Kent Bossange replied that his gut feel was that the coax is much cheaper. John Severyn pointed out that the single versus multiple ganged links decision also has to be considered and Kent replied that a fiber transceiver is four times expensive as a coax one. Gary Waldeck of Ford Aerospace asked for the basis of a fiber versus coax decision ,and was told that it was simply the link distance. Don Deel inquired if Ultra used the AMD TAXI chips, and was told that they had initially been used for low speed debug, but that now Ultra used their own gate array to obtain the higher speeds. John Severyn asked why the TAXI chip had not been retained for the lower speed applications, and was told that the approach had been to design a common board into which was plugged a pod containing the transceiver for a particular speed. Sam Laymoun of National Semiconductor asked what kind of gate array was used, and was told that the shift register portion uses an ECL type while the rest of the control logic is contained in a standard CMOS array. Patric Savage of Shell Development asked where Ultra would use a Fiber Channel in this scheme, and was told that they would replace the four fiber gigabit link by a single fiber one, and that they would also benefit by the replacement of copper on the interface side. Dal Allan asked if they had encountered any problems in qualifying transceivers at 312.5 Megabaud (for the 250 Megabit link), and was told that two vendors had been qualified but that only one was presently active. The meeting then returned to consideration of the FC-2 addressing scheme that had been postponed on the previous day, and another extended discussion resulted. A number of the participants drew comparison to the operation of the telephone system, and expressed the desire to keep the routing out of the host software. The discussion came to a head with the formulation of the following resolution: "I move that within a single interconnect fabric FC-2 use a fixed address space where reach Responder's address is unique, is the same from any Originator, and is not dependent on the route taken. Switches which require explicit routing information in the address field are not compliant". This resolution was proposed by Don Tolmie, and seconded by Dal Allan. A vote on a per company basis was then called, and passed with 22 for and 1 against. The meeting then continued with a presentation by Joe Mathis of IBM on FC-2. Joe began by giving a summary of the decisions relating to FC-2 that had been taken at the November FC Working Group meeting. A copy of this slide is Attachment 13. Joe noted that the frame size had been limited to 4 Kilobytes partially on the basis of the coverage of the 32 bit CRC, but that this issue may have to be revisited because Ultra have stated that they cover a 32 Kilobyte frame with the same CRC. Kent Bossange agreed to look into this subject. Joe then moved on to expand the FC-2 definition, and a copy of his presentation is Attachment 14. He began by redefining the frame types and expanding on the types of frame delimiter. He introduced the concept of a switch controller, and noted that it should not be completely transparent i.e. it should be addressable to obtain statistics. Joe then moved on to consider Connections and Paths, and Patric Savage asked if the routing changed between frames in a virtual connection and was told that within an exchange the path was fixed. Doug Morrissey then raised the issue of whether both a port address and a station address were required, and Ed Parish of Thinking Machines suggested that a port address was necessary for diagnostics. Terry Anderson stated that the Ancor system operates on port addresses, that a port can be dedicated to be receive only, and that they can define "hunt groups" that operate in much the same way as a rotary does with telephone lines. Joe emphasized that what an address is, and how that it is viewed, is an important subject. Joe Mathis then distributed an updated version of his FC-2 document for review by the group. A copy of the document is Attachment 15. The document was then given a cursory review by the group, and the detail comments were recorded by Carl Zeitler of IBM as Joe had to leave. However the following significant points were noted: a) The FC-2 document should contain no reference to data rates. b) The string topology still merits investigation. c) The issues of multicast and broadcast remain to be resolved. d) The definition of a number of Idle characters (or an Idle sequence) needs further definition. Dal Allan then closed the meeting by stating that the agenda for the January Fiber Channel Working Group meeting would be: a) Continuation of the discussion on the 250 Megabaud FC version, and a decision point on whether it should be pursued. b) Continuation of work on FC-2 and the 1062.5 Megabaud FC version. Kent Bossange accepted an action item to investigate the coverage of a 32 Kilobyte frame with a 32 bit CRC. An IBM presentation on the options for the FC-0 is included as Attachment 16 for information.