Table of Contents

  1. Project Structure
  2. Updates
  3. Systems
  4. Specifications
  5. Security
  6. Links

10GigE Research Network

The 10GigE Research Network is a high-speed inter-cluster network to improve data transfers between clusters. The network is currently wired between the UAB Shared Computing Facility in BEC and CIS Data Center. The network will facilitate operation of the UABgrid Meta-cluster and is proposed to link directly to the campus core network with support for high bandwidth connectivity directly to research labs on campus and collaborator resources across the country via the NLR.

Project Structure

The initial phase of the project, begun in February 2007, included installing and testing the Foundary FastIron SuperX switches in BEC and CH. This phase was completed August 2007.

The remainder of the project is being approached in two phases:

  1. 10GigE Interconnect - The first phase is exclusive to 10GigE interconnection and involves identifying the capabilities of the target systems and the requirements to acquire 10GigE cards for the servers and corresponding 10GigE interface modules for the switches.
  2. Meta-cluster Operation - The second phase enables the Globus interfaces on the clusters to support job staging via Globus and operation of the UABgrid Meta-cluster

Updates

2008-11-03 - The 10GigE Interconnect Phase was completed in September 2008 when the upgraded Cheaha head-node was connected to the 10GigE network. During the planning stages, it was decided that existing non-10GigE clusters would not be upgraded to 10GigE cards because of the cost of the cards and the relative performance bottle-necks on the hosts internal bus speeds. Instead, we will ensure that new systems were acquired with 10GigE networking built-in. At present, these systems include cheaha.uabgrid.uab.edu and ferrum.cis.uab.edu which are both connected at 10GigE.

Existing clusters with standard 1GigE network cards are still targeted for direct connection to the 1GigE ports on the research network, so that they can leverage the high-bandwidth interconnects on the research network. As of November 2008, adding these systems is still on the to-do list. Retaining connectivity with existing networks, in addition to adding the research network, implies a multi-homed configuration for the head node (3 networks: internal cluster network, departmental network, and research network). This configuration is sometimes limited by the number of network ports available on a head node.

2008-11-03 - The Meta-cluster operation phase is in progress. With the resolution of the firewall issues and identifying a usable multi-home configuration for the Globus software compements, Cheaha and Ferrum have both been configured to communicate across the research network. This involves installation, configuration, and testing of Globus interfaces on both hosts. Preliminary performance tests for data transfers have begun. Initial transfer rates are around 300Mbs/TCP channel. We need more investigation to see if this is expected or if improvements (TCP stack tuning, GridFTP tuning) are needed.

Systems

The following systems are targeted for connection to the 10GigE Research Network:

  • Olympus - PCI-X bus
  • Coosa - PCI-e interface, need clarification on number of channels
  • Cheaha - PCI-X bus. Upgraded system infrastructure supports 10GigE connectivity.
  • Everest - PCI-X bus
  • Ferrum - already has 10GigE card (need card specs)
  • UABgrid meta-cluster file store - PCI-e x8 half-length slot (Meta-cluster and file store functionality is being integrated with cheaha as part of the 2008 upgrade)
  • UABgrid core services VM host - PCI-e x8 full size slot (Resource not currently targeted for inclusion)

Additional clusters or research computing resources placed in either BEC or CH will also be connected.

Specifications

Cheaha and Ferrum, clusters with native 10GigE cards, are being used.

Proposed test environment using RAM disks: 

mkdir /var/tmp/ramdisk
mount -t tmpfs none /var/tmp/ramdisk -o size 4096MB

cp /path/to/large/file  /var/tmp/ramdisk
globus-url-copy -tcp-bs 107500 -p 4 -vb file:///var/tmp/ramdisk/file gsiftp://ferrum.cis.uab.edu/var/tmp/ramdisk

PCI-e and PCI-X are the interfaces available on our hosts. We need to determine where 10Gbs cards can be inserted. There is some confusion of the PCI-X performance, see ticket:40.

Security

There are several layers to the security architecture: network connectivity, firewall configurations, resource authorization. Cheaha and Ferrum are currently protected by network based firewalls on their public interfaces and host-based firewalls on the research network interfaces. Documentation of the firewall configuration and impact to Globus services is pending?.

The Globus Firewall Requirements document is a an excellent description of the interfaces exposed by Globus, how to secure them, and what impacts various configuration choices have on services.

Links collected for background material. (Needs integration)

Attachments