PEX 8680, PCI Express Gen 2 Switch, 80 Lanes, 20 Ports
Features
PEX 8680 General Features
o
80-lane, 20-port PCIe Gen2 switch
-
Integrated 5.0 GT/s SerDes
o
35 x 35mm
2
, 1156-ball FCBGA package
o
Typical Power: 9.0 Watts
The ExpressLane
TM
PEX 8680 device offers Multi-Host PCI Express
switching capability enabling users to connect multiple hosts to their
respective endpoints via scalable, high bandwidth, non-blocking
interconnection to a wide variety of applications including
servers,
storage systems, and communications platforms.
The PEX 8680 is
well suited for
fan-out, aggregation, and peer-to-peer
applications.
Multi-Host Architecture
The PEX 8680 employs an enhanced version of PLX’s field tested PEX 8648
PCIe switch architecture, which allows users to configure the device in
legacy single-host mode or multi-host mode with up to six host ports capable
of 1+1 (one active & one backup) or N+1 (N active & one backup) host
failover. This powerful architectural enhancement enables users to build
PCIe based systems to support high-availability, failover, redundant and
clustered systems.
High Performance & Low Packet Latency
The PEX 8680 architecture supports packet
cut-thru with a maximum
latency of 176ns (x16 to x16).
This, combined with large packet memory,
flexible common buffer/FC credit pool and non-blocking internal switch
architecture, provides full line rate on all ports for performance-hungry
applications such as
servers
and
switch fabrics.
The low latency enables
applications to achieve high throughput and performance. In addition to low
latency, the device supports a packet payload size of up to 2048 bytes,
enabling the user to achieve even higher throughput.
Data Integrity
The PEX 8680 provides
end-to-end CRC
(ECRC) protection and
Poison bit
support to enable designs that require
end-to-end data integrity.
PLX also
supports data path parity and memory (RAM) error correction circuitry
throughout the internal data paths as packets pass through the switch.
Flexible Configuration
The PEX 8680’s 20 ports can be configured to lane widths of x1, x2, x4, x8,
or x16. Flexible buffer
x4
x8
allocation, along with the
device's
flexible packet
flow control,
maximizes
PEX 8680
PEX 8680
throughput for applications
where more traffic flows in
the downstream, rather than
6 x8 6 x4
19 x4
upstream, direction. Any
port can be designated as the
x16
x8
upstream port, which can be
changed dynamically.
Figure 1 shows some of the
PEX 8680
PEX 8680
PEX 8680’s common port
configurations in legacy
Single-Host mode.
4 x8 10 x4
8 x8
Figure 1. Common Port Configurations
PEX 8680 Key Features
o
Standards Compliant
-
PCI Express Base Specification, r2.0
(backwards compatible w/ PCIe r1.0a/1.1)
-
PCI Power Management Spec, r1.2
-
Microsoft Vista Compliant
-
Supports Access Control Services
-
Dynamic link-width control
-
Dynamic SerDes speed control
o
High Performance
♦
performancePAK
Read Pacing (bandwidth throttling)
Multicast
Dynamic Buffer/FC Credit Pool
-
Non-blocking switch fabric
-
Full line rate on all ports
-
Packet Cut-Thru with 176ns max packet
latency (x16 to x16)
-
2KB Max Payload Size
o
Flexible Configuration
-
Ports configurable as x1, x2, x4, x8, x16
-
Registers configurable with strapping
pins, EEPROM, I
2
C, or host software
-
Lane and polarity reversal
-
Compatible with PCIe 1.0a PM
o
Multi-Host & Fail-Over Support
-
Configurable Non-Transparent (NT) port
-
Failover with NT port
-
Up to Six upstream/Host ports with 1+1
or N+1 failover to other upstream ports
o
Quality of Service (QoS)
-
Eight traffic classes per port
-
Weighted round-robin source
port arbitration
o
Reliability, Availability, Serviceability
♦
visionPAK
Per Port Performance Monitoring
Per port payload & header counters
SerDes Eye Capture
Error Injection and Loopback
-
4 Hot Plug Ports with native HP Signals
-
All ports hot plug capable thru I
2
C
(Hot Plug Controller on every port)
-
ECRC and Poison bit support
-
Data Path parity
-
Memory (RAM) Error Correction
-
INTA# and FATAL_ERR# signals
-
Advanced Error Reporting
-
Port Status bits and GPIO available
-
Per port error diagnostics
-
JTAG AC/DC boundary scan
© PLX Technology, www.plxtech.com
Page 1 of 1
5/14/2009, Version 1.1
PEX 8680, PCI Express Gen 2 Switch, 80 Lanes, 20 Ports
The PEX 8680 can also be configured in Multi-Host
mode where users can choose up to six ports as
host/upstream ports and assign a desired number of
downstream ports to each host. In Multi-Host mode, a
virtual switch is created for each host port and its
associated downstream ports inside the device. The
traffic between the ports of a virtual switch is completely
isolated from the traffic in other virtual switches. Figure
2 illustrates some configurations of the PEX 8680 in
Multi-Host mode where each ellipse represents a virtual
switch inside the device.
The PEX 8680
x8
x8
x8 x8 x8
also provides
several ways to
configure its
PEX 8680
PEX 8680
registers. The
device can be
2 x8, 4 x4 2 x8, 4x4
4 x4 4 x4 4 x4
configured
through
4 x4s
6 x4s
strapping pins,
I
2
C interface,
host software,
or
PEX 8680
PEX 8680
an optional
serial
EEPROM.
16 x4s
12 x4s
Figure 2. Common Multi-Host Configurations
This allows
for easy debug during the development phase,
performance monitoring during the operation phase, and
driver or software upgrade.
Dual-Host & Failover Support
In Single-Host mode, the PEX 8680 supports a
Non-
Transparent (NT) Port,
which enables the
implementation of
dual-host systems
for redundancy
and host failover capability.
Primary Host
Secondary Host
Primary Host
Secondary Host
CPU
CPU
The NT port allows systems
to isolate host memory
domains by presenting the
Root
Complex
processor subsystem as an
endpoint rather than
NT
another memory system.
Base
PEX 8680
address registers are used
Non-Transparent
Port
to translate addresses;
End
End
End
doorbell registers are
Point
Point
Point
used to send
interrupts between the
Figure 3. Non-Transparent Port
address domains; and scratchpad registers (accessible by
both CPUs) allow inter-processor communication (see
Figure 3).
Multi-Host & Failover Support
In Multi-Host mode, PEX 8680 can be configured with
up to six upstream host ports, each with its own
dedicated downstream ports. The device can be
configured for 1+1 redundancy or N+1 redundancy. The
PEX 8680 allows the hosts to communicate their status
to each other via special door-bell registers. In failover
mode, if a host fails, the host designated for failover will
disable the upstream port attached to the failing host and
program the downstream ports of that host to its own
domain. Figure 4a shows a two host system in Multi-
Host mode with two virtual switches inside the device
and Figure 4b shows Host 1 disabled after failure and
Host 2 having taken over all of Host 1’s end-points.
Host
1
Host
2
Host
1
Host
2
PEX 8680
PEX 8680
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
Figure 4a. Multi-Host
Figure 4b. Multi-Host Fail-Over
Hot Plug for High Availability
Hot plug capability allows users to replace hardware
modules and perform maintenance without powering
down the system. The PEX 8680 hot plug capability
feature makes it suitable for
High Availability (HA)
applications.
Four downstream ports include a Standard
Hot Plug Controller. If the PEX 8680 is used in an
application where one or more of its downstream ports
connect to PCI Express slots, each port’s Hot Plug
Controller can be used to manage the hot-plug event of
its associated slot. Every port on the PEX 8680 is
equipped with a hot-plug control/status register to
support hot-plug capability through external logic via the
I
2
C interface.
SerDes Power and Signal Management
The PEX 8680 supports software control of the SerDes
outputs to allow optimization of power and signal
strength in a system. The PLX SerDes implementation
supports four levels of power – off, low, typical, and
high. The SerDes block also supports
loop-back modes
and
advanced reporting of error conditions,
which
enables efficient management of the entire system.
Interoperability
The PEX 8680 is designed to be fully compliant with the
PCI Express Base Specification r2.0, and is backwards
compatible to PCI Express Base Specification r1.1 and
© PLX Technology, www.plxtech.com
Page 2 of 2
5/14/2009, Version 1.1
PEX 8680, PCI Express Gen 2 Switch, 80 Lanes, 20 Ports
r1.0a. Additionally, it supports
auto-negotiation, lane
reversal,
and
polarity reversal.
Furthermore, the PEX
8680 is tested for Microsoft Vista compliance. All PLX
switches undergo thorough interoperability testing in
PLX’s
Interoperability Lab
and
compliance testing at
the PCI-SIG plug-fest.
visionPAK
TM
Another PLX exclusive,
visionPAK
is a debug
diagnostics suite of integrated hardware and software
instruments that users can use to help bring their systems
to market faster.
visionPAK
features consist of
Performance Monitoring, SerDes Eye Capture, Error
Injection, SerDes Loopback, and more.
Performance Monitoring
The PEX 8680’s real time performance monitoring
allows users to literally “see” ingress and egress
performance on each port as traffic passes through the
switch using PLX’s Software Development Kit (SDK).
The monitoring is completely passive and therefore has
no affect on overall system performance. Internal
counters provide extensive granularity down to traffic &
packet type and even allows for the filtering of traffic
(i.e. count only Memory Writes).
SerDes Eye Capture
Users can evaluate their system’s signal integrity at the
physical layer using the PEX 8680’s SerDes Eye
Capture feature. Using PLX’s SDK, users can view the
receiver eye of any lane on the switch. Users can then
modify SerDes settings and see the impact on the
receiver eye. Figure 5 shows a screenshot of the SerDes
Eye Capture feature in the SDK.
performancePAK
TM
Exclusive to PLX,
performancePAK
is a suite of unique
and innovative performance features which allows
PLX’s Gen 2 switches to be the highest performing Gen
2 switches in the market today. The
performancePAK
features consists of the Read Pacing, Multicast, and
Dynamic Buffer Pool.
Read Pacing
The Read Pacing feature allows users to throttle the
amount of read requests being made by downstream
devices. When a downstream device requests several
long reads back-to-back, the Root Complex gets tied up
in serving that downstream port. If that port has a narrow
link and is therefore slow in receiving these read packets
from the Root Complex, then other downstream ports
may become starved – thus, impacting performance. The
Read Pacing feature enhances performances by allowing
for the adequate servicing of all downstream devices.
Multicast
The Multicast feature enables the copying of data
(packets) from one ingress port to multiple (up to 19)
egress ports in one transaction allowing for higher
performance in dual-graphics, storage, security, and
redundant applications, among others. Multicast relieves
the CPU from having to conduct multiple redundant
transactions, resulting in higher system performance.
Dynamic Buffer Pool
The PEX 8680 employs a dynamic buffer pool for Flow
Control (FC) management. As opposed to a static buffer
scheme which assigns fixed, static buffers to each port,
PLX’s dynamic buffer allocation scheme utilizes a
common pool of FC Credits which are shared by other
ports. This shared buffer pool is fully programmable by
the user, so FC credits can be allocated among the ports
as needed. Not only does this prevent wasted buffers and
inappropriate buffer assignments, any unallocated
buffers remain in the common buffer pool and can then
be used for faster FC credit updates.
Figure 5. SerDes Eye Capture
Error Injection & SerDes Loopback
Using the PEX 8680’s Error Injection feature, users can
inject malformed packets and/or fatal errors into their
system and evaluate a system’s ability to detect and
recover from such errors. The PEX 8680 also supports
Internal Tx, External Tx, Recovered Clock, and
Recovered Data Loopback modes.
© PLX Technology, www.plxtech.com
Page 3 of 3
5/14/2009, Version 1.1
PEX 8680, PCI Express Gen 2 Switch, 80 Lanes, 20 Ports
Applications
Suitable for
host-centric
as well as
peer-to-peer traffic
patterns,
the PEX 8680 can be configured for a wide
variety of form factors and applications.
Host Centric Fan-out
The PEX 8680, with its symmetric or asymmetric lane
configuration capability, allows user-specific tuning to a
variety of host-centric applications. Figure 6 shows a
server
design where, in a quad or multi processor
system, users can assign endpoints/slots to CPU cores to
distribute the system load. The packets directed to
different CPUs will go to different (user assigned) PEX
8680 upstream ports, allowing better queuing and load
balancing capability for higher performance.
CPU
CPU
CPU
CPU
Embedded or Communications Systems
The PEX 8680’s 80 lanes can come in handy for
embedded or communications applications requiring
heavy processing and/or connectivity to multiple
endpoints. Figure 8a shows an embedded system where
the PEX 8680 is being used to fan-out to eight endpoints
using x8 and x16 links. Figure 8b shows a
communications system where the PEX 8680 is using
x16 and x8 links to fan out to I/Os and three CPUs
which have been configured as endpoints. These CPUs
will run as endpoints, conducting different processing
tasks while the host CPU (connected to the PEX 8680
via a x16 upstream link) manages them.
CPU
CPU
Chip
Set
Memory
Chip
Set
Chipset
Endpoint
x16
x16
x4
x16
x8
x8
PEX 8680
x8s
x16s
& x8s
PEX 8680
x16s
PEX 8680
Endpoint
I/O
I/O
x8s
x4s
I/O
I/O
I/O
I/O
I/O
CPU
Endpoint
CPU
Endpoint
I/O
CPU
Endpoint
PCIe Gen1 or PCIe Gen2 slots
Figure 8a. Embedded System
Figure 8b. Comms System
Figure 6. Host Centric Dual Upstream
Multi-Host Systems
In multi-host mode, the PEX 8680 can support up to six
hosts at once. By creating six virtual switches, the PEX
8680 allows six hosts to fan-out to their respective
endpoints. This reduces the number of switches required
for fan-out, saving precious board space and power. In
Figure 7, the PEX 8680 is being shared by six different
CPU cores (hosts) on three servers, with each CPU core
running its own applications (I/Os). The PEX 8680
assigns the endpoints to the appropriate host and isolates
them from the other hosts. In Figure 7, the endpoints are
assigned to the CPU core of the same color.
CPU
CPU
CPU
CPU
CPU
CPU
N+1 Fail-Over in Storage Systems
The PEX 8680’s Multi-Host feature can also be used to
develop storage array clusters where each host manages
a set of storage devices independent of others (Figure 9).
Users can designate one of the hosts as the failover-host
for all the other hosts while actively managing its own
endpoints. The failover-host will communicate with
other hosts for status/heartbeat information and execute a
failover event if/when it gets triggered.
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
x4
x4
x8
x8
PEX 8680
x4
x4
x8
x8
Chip
Set
Chip
Set
Chip
Set
PEX 8680
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
PEX
PEX 8616 8616
PEX 8612
x4
x4
x4
x4
PEX 8612
x4
x4
FC
FC
x4
x4
FC
FC
FC
FC
FC
FC
8 Disk Chassis
8 Disk Chassis
8 Disk Chassis
8 Disk Chassis
Figure 7. Multi-Host System
Figure 9. N+1 Failover
Page 4 of 4
5/14/2009, Version 1.1
© PLX Technology, www.plxtech.com
PEX 8680, PCI Express Gen 2 Switch, 80 Lanes, 20 Ports
Software Usage Model
From a system model viewpoint, each PCI Express port
is a virtual PCI to PCI bridge device and has its own set
of PCI Express configuration registers. It is through the
upstream port that the BIOS or host can configure the
other ports using standard PCI enumeration. The virtual
PCI to PCI bridges within the PEX 8680 are compliant
to the PCI and PCI Express system models. The
Configuration Space Registers (CSRs) in a virtual
primary/secondary PCI to PCI bridge are accessible by
type 0 configuration cycles through the virtual primary
bus interface (matching bus number, device number, and
function number).
Interrupt Sources/Events
The PEX 8680 switch supports the INTx interrupt
message type (compatible with PCI 2.3 Interrupt signals)
or Message Signaled Interrupts (MSI) when enabled.
Interrupts/messages are generated by PEX 8680 for hot
plug events, doorbell interrupts, baseline error reporting,
and advanced error reporting.
ExpressLane
PEX 8680 RDK
The PEX 8680 RDK (see Figure 10) is a hardware
module containing the PEX 8680 which plugs right into
your system. The PEX 8680 RDK can be used to test
and validate customer software, or used as an evaluation
vehicle for PEX 8680 features and benefits. The PEX
8680 RDK provides everything that a user needs to get
their hardware and software development started.
Software Development Kit (SDK)
PLX’s Software Development Kit is available for
download at
www.plxtech.com/sdk.
The software
development kit includes drivers, source code, and GUI
interfaces to aid in configuring and debugging the PEX
8680. For more information, please refer to the PEX
8680 RDK Product Brief.
Both
performancePAK
and
visionPAK
are supported by
PLX’s RDK and SDK, the industry’s most advanced
hardware- and software-development kits.
Product Ordering Information
Part Number
PEX8680-AA50BC F
PEX8680-16U8D BB
RDK
Description
80-Lane, 20-Port PCI Express Switch,
Pb-Free (35x35mm
2
)
PEX 8680 Rapid Development Kit with
x16 Upstream and Eight x8 Downstream
Figure 10. PEX8680-16U8D BB RDK
PLX Technology, Inc. All rights reserved. PLX, the PLX logo, ExpressLane,
Read Pacing and Dual Cast are trademarks of PLX Technology, Inc. All other
product names that appear in this material are for identification purposes only
and are acknowledged to be trademarks or registered trademarks of their
respective companies. Information supplied by PLX is believed to be accurate
and reliable, but PLX assumes no responsibility for any errors that may appear
in this material. PLX reserves the right, without notice, to make changes in
product design or specification.
Development Tools
PLX offers hardware and software tools to enable rapid
customer design activity. These tools consist of a
hardware module (PEX 8680 RDK), hardware
documentation (available at
www.plxtech.com),
and a
Software Development Kit (also available at
www.plxtech.com).
Visit
www.plxtech.com
for more information.
© PLX Technology, www.plxtech.com
Page 5 of 5
5/14/2009, Version 1.1
