Last modified 6 years ago Last modified on 11/18/13 11:45:26

10GigE Research Network

The 10Gig Ethernet Research Network (a.k.a. research network or rnet) is a high-speed network connecting campus data centers to improve data transfers between compute and storage resources. The network facilitates the operation of the Research Computing System (RCS).

Network History

2007-02-01 - The initial phase of the project, begun in February 2007, included installing and testing the Foundary FastIron SuperX switches in BEC and CH. This phase was completed August 2007.

2008-11-03 - The 10GigE Interconnect Phase was completed in September 2008 when the upgraded Cheaha head-node was connected to the 10GigE network. During the planning stages, it was decided that existing non-10GigE clusters would not be upgraded to 10GigE cards because of the cost of the cards and the relative performance bottle-necks on the hosts internal bus speeds. Instead, we will ensure that new systems were acquired with 10GigE networking built-in. At present, these systems include and which are both connected at 10GigE.

Existing clusters with standard 1GigE network cards are still targeted for direct connection to the 1GigE ports on the research network, so that they can leverage the high-bandwidth interconnects on the research network. As of November 2008, adding these systems is still on the to-do list. Retaining connectivity with existing networks, in addition to adding the research network, implies a multi-homed configuration for the head node (3 networks: internal cluster network, departmental network, and research network). This configuration is sometimes limited by the number of network ports available on a head node.

2008-11-03 - The Meta-cluster operation phase is in progress. With the resolution of the firewall issues and identifying a usable multi-home configuration for the Globus software compements, Cheaha and Ferrum have both been configured to communicate across the research network. This involves installation, configuration, and testing of Globus interfaces on both hosts. Preliminary performance tests for data transfers have begun. Initial transfer rates are around 300Mbs/TCP channel. We need more investigation to see if this is expected or if improvements (TCP stack tuning, GridFTP tuning) are needed.

2013-11-18 - Since then the network has been extended to include the RUST computing center and has supported the construction of the RCS which is now adding storage and cloud resources to the campus HPC compute fabric. This network has also been extended to Huntsville to support incorporation of off-site storage fabrics.

Historical Testing Environment

Early performance test were performed using two clusters Cheaha and Ferrum, clusters with native 10GigE cards.

Proposed test environment using RAM disks:

mkdir /var/tmp/ramdisk
mount -t tmpfs none /var/tmp/ramdisk -o size 4096MB

cp /path/to/large/file  /var/tmp/ramdisk
globus-url-copy -tcp-bs 107500 -p 4 -vb file:///var/tmp/ramdisk/file gsi


There are several layers to the security architecture: network connectivity, firewall configurations, resource authorization. Cheaha and Ferrum are currently protected by network based firewalls on their public interfaces and host-based firewalls on the research network interfaces. Documentation of the firewall configuration and impact to Globus services is pending?.

The Globus Firewall Requirements document is a an excellent description of the interfaces exposed by Globus, how to secure them, and what impacts various configuration choices have on services.

Links collected for background material. (Needs integration)