From: macher@atmkn.UUCP (K.Eimermacher) Newsgroups: comp.sys.intel,comp.realtime,eunet.misc,comp.misc Subject: New concepts on Multibus II (long) Keywords: Multibus II,Transport performance,new enhancements,Realtime UNIX Message-ID: <3440@atmkn.UUCP> Date: 22 Dec 92 11:27:49 GMT Lines: 373 New concepts on Multibus II Klaus Eimermacher, ATM Computer GmbH 1. Introduction Approximately 10 years ago INTEL introduced a complete new bus concept - the MULTIBUS II system. It consists of 4 different bus systems, which can be implemented in one environment: - local bus extension (LBX II) - parallel system bus (PSB II) - serial system bus (SSB) - single board extension (SBX) The most commonly used bus of Multibus II is the parallel system bus (PSB II). This bus is optimized for communication between intelligent bus agents and has numerous LAN - like features. This backplane-LAN is commonly called a "LAN in a box". Today, boards are usually connected to Multibus II using the Message Passing Coprocessor (MPC), which is available from INTEL and VLSI. This ensures that boards from different vendors have the same hardware- and firmware - interface to the bus. Therefore, no compatibility problems arise when connecting to Multibus II. The Message Passing Coprocessor needs a DMA controller for Multibus II-transfers. The DMA controller moves data from memory to MPC or vice versa (see ff). Currently the Multibus Manufacturers Group (MMG) is extending the Multibus II specification from 40 MBytes to provide a bus bandwidth of 80 Mbyte/sec and "hot board insertion". These new features have already been shown in demo systems. The work on these new technologies has shown that new concepts are needed in the future in order to take advantage of these new Multibus II features. One reason for this is that the currently available DMA controllers have reached their performance limits, thus limiting board performance. Another disadvantage of current board designs is the high percentage of CPU performance needed to use the on-board I/O. ATM Computer GmbH now has solved both of these problems on the new MBS/433-ST board by using additional new RISC processors on a 486 single board computer. The result is a design which achieves higher I/O-rates and higher Multibus II data rates than can be achieved by standard CPU/DMA designs. 2. Message Passing on Multibus II Multibus II supports, besides the standard address ranges "memory" and "I/O", 2 other address ranges, which are not utilised in traditional bus systems: 1) the geographic addressing of agents and of configuration registers on these agents, the so called interconnect address range. The interconnect address range consists of up to 512 control- or configuration registers per bus agent. These registers can be defined to be accessible "globally" - that is, they can be accessed by any busagent - or "locally". The definition of the interconnect registers and of valid values can be found in Intel's "interconnect specification". It is accepted world-wide as a defacto Multibus II standard. The registers allow self recognition of system configurations by software and - by reprogramming the interconnect registers with different parameters - other configurations can very easily be programmed into a board. 2) the Message address range. The message address range" supports 2 different types of messages: - the unexpected or unsolicited message, and - the expected or solicited message. The unsolicited message is used as a high speed interrupt message and used to transport small amounts of data. The unsolicited message is also used for negotiation of a long data transfer - the solicited message. The solicited message supports data transfers of up to 16 Mbytes of data. The data is packed and unpacked using a special "packet protocol". This is all done by the MPC during send or receive operations. A message consists of a header, which has to be compliant with the Multibus II "Message Transport Protocol", and of user data. Error conditions, such as parity error or timeout, are automatically handled by the MPC without involving the CPU. 2.1. "Traditional" board designs Most of the available Multibus II boards are designed as follows: +---------+ +---------+ +-----------+ | Memory | | CPU | | (A)DMA- | +----+----+ +----+----+ | Controller| | | +-----+-----+ | | | -------+----------------+-------------+-------------- local CPU bus --------+-------------------------------------------- | +------+------+ +---------------+ | Message - | | Interconnect- | | Passing - +--------+ Controller | | Coprocessor | | 87xx | +------+------+ +---------------+ | -----------+------------- Multibus II (PSB II) ------------------------- The local bus is shared by the CPU and a DMA controller. The main memory which holds data and messages is also connected to the local bus. To send a message to another PSB bus agent, the CPU has to assemble the message in memory, to program its DMA controller, and to program the MPC. The CPU then writes an unsolicited message with type "buffer request" into the MPC registers which is sent by the MPC to an MPC on a receiving agent. After receiving the "buffer grant" message, the data transfer is automatically performed by the MPC and DMA controller, without involving the CPU on the transmitting or receiving agents. If the transfer was successful, a completeness-interrupt is signalled to the CPU. Error conditions are signalled immediately, to give the CPU the chance to handle the error condition. Some kinds of error conditions are handled independently by the MPC using retry mechanisms, e.g. PSB parity error. Most of today's Multibus II boards use the 82258(A) ADMA controller, which is manufactured by SIEMENS and INTEL. This controller has a 16 bit bus interface and is available in CMOS up to 20 MHz. Older board designs use the 10 or 12 MHz NMOS version. With a special board design, the controller can perform 32 bit bus cycles from memory to MPC or vice versa, so called "flyby cycles". This works only if data is 32 bit aligned. Otherwise, the DMA controller has to do two 16 bit cycles, the "two cycle mode". The maximum address range for the ADMA without additional on-board circuitry is 16 Mbytes. For operating systems, this limits the accessible memory for message transfers to 16 MB on most of the boards. The software protocol for Multibus II message passing and programming of the DMA and MPC are all handled by software drivers, which are run by the CPU. To do this, the CPU has to service a lot of MPC and DMA interrupts. The task switch overhead involved - saving the old system state, creating a new environment, servicing the interrupt, restoring the old system state consumes a significant amount of the available CPU performance, thus unnecessarily slowing down application programs and important tasks inside the operating system. Although the message passing protocol is the most important example for the drawbacks of current designs, the above statements hold true for any kind of I/O handled by the main CPU. 3. A New Concept To achieve higher throughput, the CPU has obviously to be freed from most of the overhead involved in handling message passing and I/O in general. To achieve this, ATM Computer has implemented the following concept: - use of an additional, independent RISC - I/O processor, which is running the complete Multibus II transport software. - decoupling of this RISC processor from the CPU memory bus and implementation of an independent I/O bus (two bus system). - separate code memory for the I/O processor on the I/O bus - use of a high performance 32 bit DMA controller This results in a configuration as follows: +---------+ +---------+ | Memory | |486 CPU | +----+----+ +----+----+ | | | | -------+----------------+---------------------------- local CPU bus ----------+------------------------------------------ | +---------+--------+ +---------------+ | Busarbiter | | Memory | +---------+--------+ +-------+-------+ | | ----------+----------------------------+------------- E/A Bus --------+--------------------------------------+----- | | +------+------+ +---------------+ +------+---------+ | Message - | | Interconnect- | | 80960 CA | | Passing - +--+ Controller | | 33 Mhz RISC - | | Coprocessor | | 87xx | | Processor/DMA | +------+------+ +---------------+ +----------------+ | --------+------------- Multibus II (PSB II) ---------------------- (simplified block diagram of MBS/433-ST) 4. Implementation For the I/O processor, ATM Computers chose INTEL's high performance 80960 CA RISC processor. It not only gives the necessary CPU power to run message passing and I/O, but also solves the problem of implementing a very high performance 32-bit DMA controller. The chip has an integrated 4 channel 32 bit DMA controller with data rates up to 53 Mbyte/sec. 256 KB of SRAM memory are connected to the I/O bus, this memory holds the programs and data of the 960 CPU. With this hardware configuration, it is now possible to run the complete Multibus II Transport protocol software independently on the RISC I/O processor, without influencing the main CPU (486). The main CPU gets only one single interrupt, either after successful completion of a full message transfer, or if an unresolvable error has been detected. Most of the minor errors or special conditions can be resolved by the transport software run by the 960, with only a few exceptions, e.g. a "receiver not ready" condition. Communication between the 486 main CPU and the 960 I/O CPU is implemented using a 32 bit communication register and queues in memory. The objective was to implement a very fast interface. Commands to do message transfers or I/O are given to the 960 I/O processor via a command queue. The 960 reads and executes the commands, and writes a completion status in a status queue. Then, the 486 is interrupted. The format of the queues is an enhanced version of INTELs "Shared Memory Interface specification". The communication register is used during initialization to pass information to the 960, e.g. the start address of the 960 initialization code. After power up, the 486 CPU downloads the 960 I/O program code into the 960 memory. The programs are either stored in EPROM or on disc. A bus arbiter manages the bus cycles from the 486 bus to the I/O bus and vice versa. The bus priority can be fixed or "round robin". Due to the instruction caches inside both processors, normally those accesses do not collide with the program execution on the processors. The software on the 960CA RISC processor is based on the realtime kernel iRMK 960. Message passing is implemented as tasks within iRMK 960. The software can easily be extended with additional I/O tasks. ATM Computer GmbH sells a configured ready to use version of the 960 software including full message passing transport software in combination with its real- time operating system REAL/IX. 4.1 Features of the single board computer MBS/433-ST In addition to the above mentioned I/O-processor, the MBS/433-ST board offers the following I/O interfaces: 4.1.1 I/O connected to the 486 CPU bus - two channel RS232 (V24) serial controller using Z85230 (20MHz) - two 8259A compatible interrupt controllers with up to 15 interrupt sources - one 8254 compatible timer with three 16 bit timers - one interrupt multiplexer, which allows a flexible interrupt configuration without jumpering or wrapping. - an 8 KByte non volatile data RAM 4.1.2 I/O connected to the 960 I/O bus The MBS/433-ST is designed to run operating systems which use high SCSI performance, such as REAL/IX. Therefore, the 2 MIPS SCSI controller 53C710 (33 MHz) is connected to the I/O bus. This device has a RISC CPU kernel and allows for very flexible control of SCSI devices by executing "scripts". These scripts are written in a special programming language tailored to the requirements and the special behaviour of the SCSI bus. The scripts have to be stored in the SRAM connected to the I/O bus. Thus, code fetches by the SCSI controller do not interfere with the main CPU. The controller supports data rates up to 10 Mbytes/sec on the SCSI bus in synchronous mode. The 53C710 has a timer designed to detect timeout conditions on the SCSI bus of greater than 250ms. For a good realtime system, this interval is too long. To meet the realtime system requirements, an additional 8254 timer was connected to program shorter SCSI timeout intervals. Normal SCSI devices respond 8-10ms after a command. Shorter timeouts avoid long bus locks on the SCSI-bus. Additionally the 8254 supplies the timer to interrupt for iRMK960. In order to connect additional I/O devices, the board offers two extension interfaces: a) a 32 bit interface compliant to OME Version 2 on connector P2 The OME 2 interface is connected via the backplane, compliant with the OME specification version 2 using a 128 pin DIN connector on P2. The backplane is available with 3, 4, and 5 slots. This is more suitable for Industrial requirements than the "piggy-back" interface which was defined in OME version 1.Connecting via P2 on the backplane brings many advantages, in particular easier board handling during maintenance of a system. ATM Computer offers a broad range of extension modules on OME 2. b) a serial fiber optic interface with currently 50 Mbit/s net data rate (increasable to 125 Mbit/s) The serial fiber optic interface is based on the AMD TAXI technology. The interface has 1 Kbyte transmit- and receive-FIFOs on each side and can be operated with interrupts. The net data rate of 50Mbit/s is sufficient to connect I/O subsystems, or to directly link two boards. Access to the interface is via standard ST connectors. 5. Future Multibus II enhancements The technical committee of the Multibus Manufacturers Group (MMG) is currently working on a specification for Multibus II performance enhancement. The proposal will set the maximum bus burst data rate to 80 Mbyte/sec. Technologically, this is no big deal, because new faster technologies for MPC chips, and faster ABT bus drivers can be used. These components are available now, and therefore it is easy to double the bus clock today. With "traditional" board concepts, the Multibus II performance enhancement will half the packet transfer time on the bus, but will not change the overall system performance. The potential bottleneck are the slow DMA controllers, which limit data throughput. ATMs new concept using the 960 RISC CPU/DMA will be able to take advantage of the increased Multibus II performance. Another MMG subcommittee is working on the specification of "hot board insertion", that is disconnecting boards from and connecting boards to the PSB with "power on". To make this work, electrical problems have to be solved, as well as problems like "booting and self-testing replaced boards". The main goal is to be able to use currently available boards without any, or with only minor changes. A demo system was shown at the BUSCON WEST in February 1992 at Long Beach. This system included a special backplane which could run with 20 MHz, and which also supported "hot board insertion". Currently, the MMG plans to submit these two extensions to the Multibus II specification, and to obtain the approval of IEEE, in 1993. 6. Conclusion Enhancements in chip technology today enable the performance of Multibus II to be doubled. To take advantage of the performance of this "new" Multibus II, new concepts in board development and board design are necessary. Only by using new processor technology based on RISC, in combination with standard CISC processors like the 80x86 family and their operating systems, and by implementing advanced on-board bus concepts, can the system profit by the doubled bus speed. The "hot board insertion" feature will extend the range of Multibus II applications in the industrial, and communications market. ATM's MBS/433-ST and Concurrent Technologies' CP 486/960 single board computers are the first high performance Multibus II CPUs to designed to take advantage of the new Multibus II features! used abbreviations: OME - On Board Module Extension CPU - Central Processing Unit MMG - Multibus Manufacturers Group I/O - Input / Output RISC - Reduced Instruction Set CPU CISC - Complex Instruction Set CPU MPC - Message Passing Coprocessor FIFO - First-in, First-out DMA - Direct Memory Access Trademarks: iRMK960 - Fa. INTEL UNIX - Fa. AT&T REAL/IX - Fa. Modcomp / ATM TAXI - Applied Micro Devices ----------- If you have any questions regarding the new Multibus II concepts or regarding the boards please do not heasitate to contact me! --- ------------------------------------------------------------------------snip- Klaus Eimermacher | ATM Computer GmbH | Email: macher@atmkn.uucp _ _____ _ _ | Abt. E53 | (..!unido!atmkn!macher) / \ |_ _|| \/ | | Buecklestr. 1 - 5 | Voice: 07531/807-332 | AEG | | | | | | W-7750 Konstanz | Fax: 07531/807-280 |_| |_| |_| |_| |_| | Germany | Ampr.net: 44.130.55.1 | | dl1gek.ampr.org -----------------------------------------------------------------------------