Ticket #40 (assigned task)

Opened 2 years ago

Last modified 5 months ago

Need clarification on comparison PCI-X and PCI-e max performance

Reported by: jpr@uab.edu Assigned to: jpr@uab.edu (accepted)
Priority: normal Milestone:
Component: researchnet Version:
Keywords: pci-e pci-x wikipedia specifications Cc: jpr@uab.edu, fran@cis.uab.edu

Description

I'm trying to better understand our performance expectations for our various clusters using PCI-X and PCI-e.

The speed specs in the Wikipedia entry for PCI-e seem accurate. PCI-e is described as an assembly of 2.5Gbs (Gigabit per second) channels. Assuming a 10bit byte, that gives 250MBs (mega-bytes per second). The article seems consistent with these numbers. The channels are assembled in x1 x2 x4 x8 x16 or x32 configurations. So our Dell 1950 with an x8 PCI-e card will support 2.5Gbs x 8 = 20Gbs (20 Gigabits per channel). That is, 2 x 10Gbs. In otherwords the PCI-e segment is twice as fast as our 10Gbs ethernet backplane. And, since PCI-e cards can negotiate their speed independently of other cards, we shouldn't see any negative impacts for other cards that might be connected to the same PCI-e segment.

The entry that describes PCI-X is not as clear, at least to me. There is a helpful side bar that list the top speed of PCI-X as 1GBs, that is giga-bytes per second. If we're assuming a 10bit byte, that would mean PCI-X can support 10 giga-bits per second. Is this accurate?

If PCI-X can indeed do 10Gbs, then we should be able to consider 10Gbs cards for our PCI-X hosts as well. PCI-X is a parallel bus and its max performance is impacted by the weakest spec'd card on the bus. If we put a 10Gbs card in those hosts, we should be sure to configure the PCI-X bus to maximize performance.

I need clarification on these specs, and if they are accurate we can adjust our assumptions. If they are not, we may need to adjust Wikipedia. ;)

Change History

(follow-up: ↓ 2 ) 03/06/08 11:21:22 changed by fabrizio@uab.edu

PCI-X is bus-based, parallel communication architecture and therefore shares the PCI bus not only with other peripheral cards but potentially also with all of the integrated PCI devices on the motherboard. Common bus speeds are 133Mhz or 100Mhz. It negotiates down to lowest common denominator, and you can have a max of one device at 133Mhz speed (two at 100Mhz speed). Max theoretical speed of a 133Mhz/64-bit PCI-X device is 1GB/s.

PCI-e is a switched, point-to-point serial communication architecture so there are no bus bandwidth concerns. It's also bi-directional, meaning a single channel actually can go to 5Gbps, 2.5 Gbps in each direction. Doing the math, an x8 device would be 2GBps in each direction, or 4GB/s total.

In short, PCIe offers far better performance than PCI-X, and less contention.

(in reply to: ↑ 1 ) 03/06/08 11:30:18 changed by jpr@uab.edu

  • cc set to jpr@uab.edu, fran@cis.uab.edu.
  • owner changed from nobody to jpr@uab.edu.
  • status changed from new to assigned.

This is all true, but if the answer to the question is "PCI-X can do 10Gbs", then we should consider direct 10Gbs connectivity for our PCI-X hosts, instead of creative use of multiple 1Gbs connections.

I think it would be worth verifying the max speed of PCI-X and then seeing if there's any chance our PCI-X hosts could come close to 10Gbs on the PCI-X segment, given the limitations of other devices on the same bus. That is, can we have a dedicated PCI-X bus on these hosts and maximize it's performance?

Replying to fabrizio@uab.edu:

In short, PCIe offers far better performance than PCI-X, and less contention.

03/06/08 12:52:12 changed by fabrizio@uab.edu

Well, theoretical max on PCI-X 133MHz/64bit is 1GB/s. In practice, it would be something less. So it would fall anywhere from "significantly short" to "not even close to" 10Gbps. :-)

(follow-up: ↓ 5 ) 03/06/08 12:53:50 changed by fabrizio@uab.edu

Also, note that I am not looking at 10Gbps vs. multi 1Gpbs in this case. I'm talking about Olympus, and the options would be to either try a PCI-X 10Gig card (do they even exist?) or swap out the riser for the PCIe version.

Where multi 1Gbps comes into play is strictly on the systems that have core architectures that don't support 10Gbps speeds (Everest, in our case).

(in reply to: ↑ 4 ) 03/06/08 14:07:32 changed by jpr@uab.edu

Agreed. I'm only talking about PCI-X for Everest and Cheaha. I'm just saying we need to determine if a 10Gbs card is feasible for these hosts. If it is, we don't need to play around with multiple 1Gbs connections.

I'm still assuming that the 1GB/s peak theoretical performance actually means 10Gbs, I haven't seen anything to the contray but no one has confirmed it with a "yes." ;)

Provisioning a 10Gbs network connection on these hosts could improve the data transfer rate if the card can be isolated on that bus.

If PCI-X max speed of 1GB/s is in fact 10Gbs, it seems the next steps are:

  1. What is the speed of these systems PCI-X (66Mhz or 133Mhz)
  2. Is there a dedicated PCI-X bus available?
  3. Are their 10Gbs PCI-X cards?

Replying to fabrizio@uab.edu:

Also, note that I am not looking at 10Gbps vs. multi 1Gpbs in this case. I'm talking about Olympus, and the options would be to either try a PCI-X 10Gig card (do they even exist?) or swap out the riser for the PCIe version.

Where multi 1Gbps comes into play is strictly on the systems that have core architectures that don't support 10Gbps speeds (Everest, in our case).

03/06/08 15:14:09 changed by fabrizio@uab.edu

Remember too that Everest is not standard form factor, nor is there currently room for expansion cards.

It may not be worth the effort or cost to retrofit the Verari clusters.

06/24/09 10:19:30 changed by root

  • milestone deleted.

Milestone 10GigErnet deleted