wiki:BuildingSoftware
Last modified 12 months ago Last modified on 12/08/16 17:18:12

Building Software

Maintaining software in HPC environments is challenging. Each science domain will typically have their own unique software stacks, each of these can include multiple versions each with custom dependencies. Documenting build options and publicising tool availability only add to the complexity. Supporting reproducible research requires a record of the software environment used to conduct research. There is no magic solution that eliminates the need to maintain software environments. There are, however, tools to help manage the complexity.

This document proposes a tool chain for maintaining and documenting custom software environments. There are two important functions of a defined tool chain:

  1. reliably produce a software configuration
  2. allow others to understand build decisions so they can trust the result

Proposed Software Build Tool Chain

The basis of the proposed approach is adopting EasyBuild and using it's build conventions. To this we plan to add Git to track development of or improvements to EasyConfig files and construct other light weight automation steps, like updates to software documentation pages.

  • EasyBuild
    • Does the heavy lifting for the software build itself (run eb --tmpdir=${LOCAL_SCRATCH} --installpath=/share/apps/rc GCC to install GCC)
    • Builds software using a lightweight configuration file known as an EasyConfig
    • Supports many different toolchains (goolf, intel, gompi)
    • Many EasyConfig's already exist (list them all with eb -S *)
    • There is a community of developers across HPC centers with which to engage and learn from
    • Builds Environment modules to make using the tool easier (also possible to use Lmod an improved environment module system)
    • Most software builders can focus on using EasyBuild without undo concern for the rest of the tool chain
  • Git
    • EasyConfig files are tracked upstream on GitHub
    • Builds can reference EasyConfig on GitHub, specifically in pull requests pending integration upstream
    • Creates verifiable record of local modifications
    • Can lead to a lightweight cascading tool chain
    • Integrates with in-house GitLab and GitLab's continuous integration (CI) framework

For someone accustomed to building just one piece of software and then being done with it, the overhead of this workflow may seem too high. They can , of course, continue to follow their own preferred practices. This proposal is targeted at those maintaining a shared library of tools or users interested in maintaining a disciplined record of the software used for their research.

The primary alternatives to EasyBuild are spack from LLNL and smithy from ORNL. EasyBuild appears to be the more mature option. Please see the EasyBuild paper and references below for more background. Nonetheless, this is an evolving landscape and different requirements may require us to adapt to those changes.

Enhancements

Additional tools to support comprehensive solution. It should be possible to hide this work behind use of EasyBuild

  • GitLab
    • Helps maintain a public record of branches from the mainstream EasyConfigs?
    • Provides a light weight CI framework that can be used to solve problems of maintaining shared documentation
  • Publication (proposed outline)
    • Leverage "description" field of EasyConfig
    • Use a git commit hook on publication of RCS-specific EasyConfig to trigger an update to a shared documentation repo
    • Central documentation repo accepts documentation snippet in Markdown format
    • Parsing success leads to commit being accepted (http://pandoc.org/)
    • Committed changes get pushed to Mediawiki with mw-to-git front end
      • hides exposing specific wiki platform to builder (in our case Mediawiki)
      • dev can write in markdown format (popular on GitHub and GitLab)
      • lets users find docs easily

Issues

  • Learning EasyBuild
  • Learning the EasyConfig abbreviation conventions
    • EasyConfig file naming convention is eb software-version-toolchain-toolchainversion.eb
    • this describes the software version and the compiler and libraries that make up the software building toolchain
    • list the available tool chains with eb --list-toolchain
  • May want to uses Lmod improve module features (module hierarchies)
  • Build documentation tool chain

EasyBuild Concepts

EasyBuild simplifies managing software environments but it comes with typical challenges of learning a new tool. A general purpose getting started guide is available and is a good read. You won't typically have follow that document, however, because the steps it describes are automatically taken care of by the --robot build directive.

It's best to start by getting familiar with the concepts and terms used by EasyBuild to manage the software build steps. Keep that page handy and refer to it from time to time as you start working with easybuild. It will keep you oriented as you work with different parts of EasyBuild.

EasyBuild manages the steps to download the source code, configure, build, and install it, and will also create a modules file to make adding the software to your environment easy.

Two important concepts in EasyBuild to understand are:

  • the procedures used to encode the rules for interfacing with a particular software project (called EasyBlocks?)
  • the key-value files used to build a specific version of the software (called EasyConfigs?)

In most cases, you will only need to work with EasyConfigs?. It's only when you are adding support for a new software project that you need to worry about the EasyBlocks?.

EasyBuild Quick Start

The following steps demonstrate an build in our cluster environment.

Access EasyBuild on Cheaha

You can start using the system installed EasyBuild via a module.

module load EasyBuild/2.6.0

Use the command eb to interact with Easybuild and be careful about command parameter order, specifically the --robot parameter comes at the end of lines. eb --help can provide a quick reference but the on-line docs are more useful.

Search for an EasyConfig

Search for an existing EasyConfig file for a tool of interest. Searches are case insensitive.

eb -S {software}

Replace the string {software} with a software package you are interested in, e.g. eb -S blast.

The search result is the list of the EasyConfig's available to build. The names returned include the complete path to the EasyConfig file and follow the naming convention described above.

$CFGS1/{firstletterofsoftware}/{software}/{software}-{version}-{toolchain}-{toolchainversion}.eb

From this output you will only need to use the {software}-{version}-{toolchain}-{toolchainversion} portion of the result when you request a build.

Build the EasyConfig

The build itself is straight forward and all the tools your software needs can be built automatically with the --robot flag.

Do a quick sanity check with the --dry-run argument

eb --dry-run {software}-{version}-{toolchain}-{toolchainversion}

This will show you all the dependencies needed. If there is no '[x]' next to a listed dependency, the build will automatically build that software.

Build the software for real.

eb {software}-{version}-{toolchain}-{toolchainversion}

When it's done it will be installed in your personal software collection in your $HOME directory by default. If you are collaborating with others and building software to share, please consult your team for a valid --installpath argument to use.

Use the Software

To use modules from your personal software install location simply add it to your module path.

MODULEPATH=$MODULEPATH:$HOME/.local/easybuild/modules/all

This will cause all your personal software tools to be listed and available via normal module commands like module avail. With the above path you will see you personal software listed at the end of the listing of all modules. You can then run module load {software} to make your application available from the command line.

Developing EasyBuild Scripts

In order to add EasyBuild support for unavailable packages, you will want to set yourself up a development install of easybuild. This will enable you to work with any of the upstream parts of the code base in git clones of your own forks on github. Most of your focus will be on the easyconfig repo and maybe easyblocks on occasion but the provided setup script works with clones of all the repos in the easybuild system, so you'll need to fork them all on github.

Known Issues

EasyBuild simplifies complex tasks but when things don't go as expected you may be confronted with lots of details related to a build failure. Just remember that the EasyBuild keeps a log file while its working. If the build is not successful, it keeps the log and tells you where to find it. Look at the information near the end of that log file to understand why our build fails.

Are you using the goolf-1.4.10 toolchain?

Many default EasyConfig's, especially in for bioinformatics tools use the goolf-1.4.10 toolchain. This toolchain expects to use GCC 4.7.2, however, this version of GCC won't build on CentOS 7.x, the operating system running on our cluster. This has to do with a bug in the GCC documentation files and trying to use newer texinfo versions provided by the OS to build the documents.

Try a newer toolchain

There are two options. You can tell EasyBuild to try a newer tool chain with the --try-toolchain-version argument

eb software-version-toolchain-toolchainversion -D --try-toolchain-version=1.7.20

Contribute an Updated EasyConfig

EasyBuild is open source software and works best when users contribute fixes. EasyBuild uses GitHub for development so understanding git and GitHub helps. See our GitQuickStart for some help.

It's very easy to update an EasyConfig to use a new toolchain and the contributor workflow is well documented.

Example Builds

Set Up Environment

The following are examples of useful environment variables when using EasyBuild (either execute within the current shell or add to your ~/.bashrc file)

# EasyBuild Stuff for build user
export EASYBUILD_INSTALLPATH=/share/apps/rc    # Root installation directory for all EB components, including software and module files
export EASYBUILD_TMPDIR=$HOME/tmp                  # Temporary directory to avoid filling up /tmp
export EASYBUILD_MODULES_TOOL=EnvironmentModulesC   # Force EB to use Environment Modules instead of the default Lmod (note, Lmod is preferred if starting from scratch)
export EASYBUILD_MODULE_SYNTAX=Tcl                # Force use of TCL instead of the default Lua for Module language syntax (Lua is preferred if starting from scratch)

Example: install BLAST

Note: In the examples below we are using "--installpath=/share/apps/rc" as the installation path for the software. You must change the --installpath to reflect a path to which you have write access.

Search for an EasyConfig

Search for an existing EasyConfig file for a tool of interest. Searches are case insensitive.

$ eb -S blast
== temporary log file in case of crash /tmp/eb-1Qou69/easybuild-1K4vOK.log
== Searching (case-insensitive) for 'blast' in /share/apps/easybuild/2.3.0/software/EasyBuild/2.3.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.3.0-py2.7.egg/easybuild/easyconfigs 
CFGS1=/share/apps/easybuild/2.3.0/software/EasyBuild/2.3.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.3.0-py2.7.egg/easybuild/easyconfigs
 * $CFGS1/b/BLAST+/BLAST+-2.2.27-goalf-1.1.0-no-OFED.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.27-goolf-1.4.10.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.27-ictce-4.0.6.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.27_ictce-fixes.patch
 * $CFGS1/b/BLAST+/BLAST+-2.2.28-goolf-1.4.10-Python-2.7.3.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.28-goolf-1.4.10.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.28-ictce-4.1.13.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.28-ictce-5.3.0-Python-2.7.3.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.28-ictce-5.3.0.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.28_ictce-fixes.patch
 * $CFGS1/b/BLAST+/BLAST+-2.2.30-foss-2015a-Python-2.7.9.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.30-goolf-1.4.10.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.30-intel-2015a-Python-2.7.9.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.30_basename-fixes.patch
 * $CFGS1/b/BLAST+/BLAST+-2.2.30_ictce-fixes.patch
 * $CFGS1/b/BLAST+/BLAST+-2.2.31-foss-2015b-Python-2.7.10.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.31-intel-2015a-Python-2.7.10.eb
 * $CFGS1/b/BLAST+/BLAST+-2.2.31-intel-2015b-Python-2.7.10.eb
 * $CFGS1/b/BLAST/BLAST-2.2.26-Linux_x86_64.eb
 * $CFGS1/m/mpiBLAST/mpiBLAST-1.6.0-goalf-1.1.0-no-OFED.eb
 * $CFGS1/m/mpiBLAST/mpiBLAST-1.6.0-goolf-1.4.10.eb
 * $CFGS1/m/mpiBLAST/mpiBLAST-1.6.0-ictce-4.0.6.eb
 * $CFGS1/m/mpiBLAST/mpiBLAST-1.6.0-ictce-5.2.0.eb
 * $CFGS1/m/mpiBLAST/mpiBLAST-1.6.0-ictce-5.3.0.eb
 * $CFGS1/m/mpiBLAST/mpiBLAST_disable-ncbi-X11-apps.patch
== Tmporary log file(s) /tmp/eb-1Qou69/easybuild-1K4vOK.log* have been removed.
== Temporary directory /tmp/eb-1Qou69 has been removed.

The strings "foss", "intel", "goolf", "ictce", "Linux" refer to the dependency chain for this stack. You don't choose these blindly. Typically, "Linux" will be the lightest weight, as it relies on most of your platform supplied tools. Easybuild will only install what it needs to install as described by the config, but the first time you reference a particular tool chain you may end up install a lot of dependencies.

Also note that when building a configuration you only need to specify the file name of the EasyConfig (ending in ".eb") not the path to the file.

Use Dry-Run to Preview Install

Review the tools that will be installed with a dry run. Note the "--robot" argument should be the last argument and it means resolve dependencies. Anything without an "x" in the brackets will be need to be installed.

$ eb --dry-run --tmpdir=${LOCAL_SCRATCH} --installpath=/share/apps/rc BLAST-2.2.26-Linux_x86_64.eb  --robot

== temporary log file in case of crash /tmp/eb-jjGRjH/easybuild-bun3Fz.log
Dry run: printing build status of easyconfigs and dependencies
CFGS=/share/apps/easybuild/2.3.0/software/EasyBuild/2.3.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.3.0-py2.7.egg/easybuild/easyconfigs/b/BLAST
 * [ ] $CFGS/BLAST-2.2.26-Linux_x86_64.eb (module: BLAST/2.2.26-Linux_x86_64)
== Tmporary log file(s) /tmp/eb-jjGRjH/easybuild-bun3Fz.log* have been removed.
== Temporary directory /tmp/eb-jjGRjH has been removed.

Execute the Install

Execute the install. All software will be downloaded to, built from, and installed to ~/.local by default. Note that the references to the /tmp/ dir for crash files are the only global file locations and designed to provide insight in the event of other failures. They will automatically be removed on success.

$ eb --tmpdir=${LOCAL_SCRATCH} --installpath=/share/apps/rc BLAST-2.2.26-Linux_x86_64.eb  --robot
== temporary log file in case of crash /tmp/eb-40fCGO/easybuild-KII16H.log
== resolving dependencies ...
== processing EasyBuild easyconfig /share/apps/easybuild/2.3.0/software/EasyBuild/2.3.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.3.0-py2.7.egg/easybuild/easyconfigs/b/BLAST/BLAST-2.2.26-Linux_x86_64.eb
== building and installing BLAST/2.2.26-Linux_x86_64...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== postprocessing...
== sanity checking...
== cleaning up...
== creating module...
== permissions...
== packaging...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file /home/jpr/.local/easybuild/software/BLAST/2.2.26-Linux_x86_64/easybuild/easybuild-BLAST-2.2.26-20150921.133657.log
== Build succeeded for 1 out of 1
== Tmporary log file(s) /tmp/eb-40fCGO/easybuild-KII16H.log* have been removed.
== Temporary directory /tmp/eb-40fCGO has been removed.

Use the Tool

You can now update the module path to include the module files created as part of the build process.

$ MODULEPATH=/home/$USER/.local/easybuild/modules/all/:$MODULEPATH
$ module avail

-------------------------------------------------- /home/$USER/.local/easybuild/modules/all/ ---------------------------------------------------
BLAST/2.2.26-Linux_x86_64 EasyBuild/2.3.0           libreadline/6.3-GCC-4.9.2 SQLite/3.8.8.1-GCC-4.9.2  zlib/1.2.8-GCC-4.9.2
bzip2/1.0.6-GCC-4.9.2     GCC/4.9.2                 ncurses/5.9-GCC-4.9.2     Tcl/8.6.3-GCC-4.9.2

Load the new configuration:

$ module load BLAST/2.2.26-Linux_x86_64 

See where your command is installed.

$ which blastall
~/.local/easybuild/software/BLAST/2.2.26-Linux_x86_64/bin/blastall

Example: install Java

Note: In the examples below we are using "--installpath=/share/apps/rc" as the installation path for the software. You must change the --installpath to reflect a path to which you have write access.

Install the latest version of Java JDK which is Java-1.8.0_60

Search for an EasyConfig

Search for an existing EasyConfig file for a tool of interest. Searches are case insensitive.

$ eb -S Java-1.8.0

 * $CFGS1/j/Java/Java-1.8.0_20.eb
 * $CFGS1/j/Java/Java-1.8.0_25.eb
 * $CFGS1/j/Java/Java-1.8.0_31.eb
 * $CFGS1/j/Java/Java-1.8.0_40.eb
 * $CFGS1/j/Java/Java-1.8.0_45.eb

Oops, 1.8.0_60 isn't there. Let's create the eb file based off of a similar version

if [ ! -d ~/.local/easybuild/ebfiles_repo/Java/ ]; then
  mkdir -p ~/.local/easybuild/ebfiles_repo/Java/
fi

cp -a \
      ${EBROOTEASYBUILD}/lib/python2.7/site-packages/easybuild_easyconfigs-2.3.0-py2.7.egg/easybuild/easyconfigsj/Java/Java-1.8.0_45.eb \
      ~/.local/easybuild/ebfiles_repo/Java/j/Java/Java-1.8.0_60.eb

$ vim ~/.local/easybuild/ebfiles_repo/Java/j/Java/Java-1.8.0_60.eb

Update the version and save

name = 'Java'
version = '1.8.0_60'

homepage = 'http://java.com/'
description = """Java Platform, Standard Edition (Java SE) lets you develop and deploy
 Java applications on desktops and servers."""

toolchain = {'name': 'dummy', 'version': 'dummy'}

# download the tar.gz directly from http://www.oracle.com/technetwork/java/javase/downloads/index.html
(vp, vs) = version.split('_')
altver = '%su%s' % (vp.split('.')[1], vs)
sources = ['jdk-%s-linux-x64.tar.gz' % altver]

moduleclass = 'lang'

Download the tar file from Oracle: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html and save it into .local/easybuild/sources/j/Java/

Note that the Oracle download saves the file as a .gz file, not .tar.gz, which is proper and expected by the EasyBuild script. So, we need to rename it:

mv ~/.local/easybuild/sources/j/Java/jdk-8u60-linux-x64.gz \
  ~/.local/easybuild/sources/j/Java/jdk-8u60-linux-x64.tar.gz

Use Dry-Run to Preview Install

Review the tools that will be installed with a dry run. Note the "--robot" argument should be the last argument and it means resolve dependencies. Anything without an "x" in the brackets will be need to be installed.

$ eb --dry-run --tmpdir=${LOCAL_SCRATCH} --installpath=/share/apps/rc Java-1.8.0_60.eb --robot

== temporary log file in case of crash /scratch/local/eb-35pFCx/easybuild-VEDTTz.log
Dry run: printing build status of easyconfigs and dependencies
 * [ ] ~/.local/easybuild/ebfiles_repo/Java/j/Java/Java-1.8.0_60.eb (module: Java/1.8.0_60)
== Tmporary log file(s) /scratch/local/eb-35pFCx/easybuild-VEDTTz.log* have been removed.
== Temporary directory /scratch/local/eb-35pFCx has been removed.

Execute the Install

Execute the install. All software will be downloaded to, built from, and installed to ~/.local by default. Note that the references to the /tmp/ dir for crash files are the only global file locations and designed to provide insight in the event of other failures. They will automatically be removed on success.

Note the use of the arguments:

  • --tmpdir=${LOCAL_SCRATCH} : This is important to ensure that the build is performed on local storage (i.e. not NFS or Lustre) and that it's not performed in the space limited /tmp directory
  • --installpath=/share/apps/rc : Tells eb the installation root, under which it will create a software and modules directories
  • --robot : Tells eb to also build dependancies if they aren't already installed in the "--installpath"
    $ eb --tmpdir=${LOCAL_SCRATCH} --installpath=/share/apps/rc Java-1.8.0_60.eb --robot
    
    == temporary log file in case of crash /scratch/local/eb-JKW1xq/easybuild-UPfbMc.log
    == resolving dependencies ...
    == processing EasyBuild easyconfig ~/.local/easybuild/ebfiles_repo/Java/j/Java/Java-1.8.0_60.eb
    == building and installing Java/1.8.0_60...
    == fetching files...
    == creating build dir, resetting environment...
    == unpacking...
    == patching...
    == preparing...
    == configuring...
    == building...
    == testing...
    == installing...
    == taking care of extensions...
    == postprocessing...
    == sanity checking...
    == cleaning up...
    == creating module...
    == permissions...
    == packaging...
    == COMPLETED: Installation ended successfully
    == Results of the build can be found in the log file /share/apps/rc/software/Java/1.8.0_60/easybuild/easybuild-Java-1.8.0_60-20151030.131345.log
    == Build succeeded for 1 out of 1
    == Tmporary log file(s) /scratch/local/eb-JKW1xq/easybuild-UPfbMc.log* have been removed.
    == Temporary directory /scratch/local/eb-JKW1xq has been removed.
    
    

Use the Tool

Take a look at the automatically generated module file

$ module show Java/1.8.0_60

-------------------------------------------------------------------
/share/apps/rc/modulefiles/all/Java/1.8.0_60:

module-whatis    Description: Java Platform, Standard Edition (Java SE) lets you develop and deploy
 Java applications on desktops and servers. - Homepage: http://java.com/
conflict         Java
prepend-path     CPATH /share/apps/rc/software/Java/1.8.0_60/include
prepend-path     LD_LIBRARY_PATH /share/apps/rc/software/Java/1.8.0_60/lib
prepend-path     LIBRARY_PATH /share/apps/rc/software/Java/1.8.0_60/lib
prepend-path     MANPATH /share/apps/rc/software/Java/1.8.0_60/man
prepend-path     PATH /share/apps/rc/software/Java/1.8.0_60/bin
setenv           EBROOTJAVA /share/apps/rc/software/Java/1.8.0_60
setenv           EBVERSIONJAVA 1.8.0_60
setenv           EBDEVELJAVA /share/apps/rc/software/Java/1.8.0_60/easybuild/Java-1.8.0_60-easybuild-devel
prepend-path     PATH /share/apps/rc/software/Java/1.8.0_60
setenv           JAVA_HOME /share/apps/rc/software/Java/1.8.0_60
-------------------------------------------------------------------

Load the new configuration:

$ module show rc/all/Java/1.8.0_60

See where your command is installed.

$ which java

/share/apps/rc/software/Java/1.8.0_60/bin/java

Example: Install R

Note: In the examples below we are using "--installpath=/share/apps/rc" as the installation path for the software. You must change the --installpath to reflect a path to which you have write access.

  1. Load the eb module
    module load EasyBuild/2.6.0
    
    
  2. List the available R builds
    eb -S R-
    
     * $CFGS1/r/R-bundle-Bioconductor/R-bundle-Bioconductor-3.1-goolf-1.7.20-R-3.2.0.eb
     * $CFGS1/r/R-bundle-Bioconductor/R-bundle-Bioconductor-3.1-intel-2015a-R-3.2.1.eb
     * $CFGS1/r/R-bundle-Bioconductor/R-bundle-Bioconductor-3.1-intel-2015b-R-3.2.1.eb
     * $CFGS1/r/R-bundle-devtools/R-bundle-devtools-20150520-intel-2015a-R-3.1.3.eb
     * $CFGS1/r/R-bundle-extra/R-bundle-extra-20150323-intel-2015a-R-3.1.3.eb
     * $CFGS1/r/R-bundle-pbd/R-bundle-pbd-20150605-intel-2015a-R-3.1.3.eb
     * $CFGS1/r/R/R-3.0.1-goalf-1.1.0-no-OFED-bare.eb
     * $CFGS1/r/R/R-3.0.1-goolf-1.4.10-bare.eb
     * $CFGS1/r/R/R-3.0.1-ictce-4.0.6-bare.eb
     * $CFGS1/r/R/R-3.0.1-ictce-4.1.13-bare.eb
     * $CFGS1/r/R/R-3.0.1-ictce-5.3.0-bare.eb
     * $CFGS1/r/R/R-3.0.2-goolf-1.4.10.eb
     * $CFGS1/r/R/R-3.0.2-ictce-5.3.0-bare.eb
     * $CFGS1/r/R/R-3.0.2-ictce-5.3.0.eb
     * $CFGS1/r/R/R-3.0.2-ictce-5.5.0.eb
     * $CFGS1/r/R/R-3.1.0-ictce-5.5.0-bare.eb
     * $CFGS1/r/R/R-3.1.0-ictce-5.5.0.eb
     * $CFGS1/r/R/R-3.1.1-ictce-6.2.5-bare-mt.eb
     * $CFGS1/r/R/R-3.1.1-ictce-6.2.5-default-mt.eb
     * $CFGS1/r/R/R-3.1.1-ictce-6.3.5-bare-mt.eb
     * $CFGS1/r/R/R-3.1.1-ictce-6.3.5-default-mt.eb
     * $CFGS1/r/R/R-3.1.1-intel-2014b.eb
     * $CFGS1/r/R/R-3.1.2-foss-2015a.eb
     * $CFGS1/r/R/R-3.1.2-goolf-1.5.14.eb
     * $CFGS1/r/R/R-3.1.2-intel-2015a-bare.eb
     * $CFGS1/r/R/R-3.1.2-intel-2015a.eb
     * $CFGS1/r/R/R-3.1.3-foss-2015a.eb
     * $CFGS1/r/R/R-3.1.3-intel-2015a.eb
     * $CFGS1/r/R/R-3.2.0-foss-2015a-bare.eb
     * $CFGS1/r/R/R-3.2.0-goolf-1.7.20-bare.eb
     * $CFGS1/r/R/R-3.2.0-goolf-1.7.20.eb
     * $CFGS1/r/R/R-3.2.0-ictce-7.3.5-bare.eb
     * $CFGS1/r/R/R-3.2.0-intel-2015a-bare.eb
     * $CFGS1/r/R/R-3.2.1-foss-2015b-bare.eb
     * $CFGS1/r/R/R-3.2.1-foss-2015b.eb
     * $CFGS1/r/R/R-3.2.1-intel-2015a.eb
     * $CFGS1/r/R/R-3.2.1-intel-2015b-bare.eb
     * $CFGS1/r/R/R-3.2.1-intel-2015b.eb
    
    
  3. Attempt to build R with --robot to also build dependencies (note I had to install libX11-devel package via BrightCM cmsh in order for EasyBuild to build the TK dependency). OOPS it fails because it can't find the Java JDK source:
    eb --tmpdir=${LOCAL_SCRATCH} --installpath=/share/apps/rc R-3.2.0-goolf-1.7.20.eb --robot
    
    

    resolving dependencies

    processing EasyBuild easyconfig /share/apps/rc/software/EasyBuild/2.6.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.6.0-py2.7.egg/easybuild/easyconfigs/j/Java/Java-1.7.0_80.eb

    building and installing Java/1.7.0_80...

    fetching files

    FAILED: Installation ended unsuccessfully (build directory: /home/mhanby/.local/easybuild/build/Java/1.7.0_80/dummy-dummy): build failed (first 300 chars): Couldn't find file jdk-7u80-linux-x64.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /share/apps/rc/software/EasyBuild/2.6.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.6.0-py2.7.egg/easybuild/easyconfigs/j/Java/j/Java/jdk-7u80-linux-x64.tar.gz, /share/

  4. Download the Java JDK file (http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html) in question, jdk-7u80-linux-x64.tar.gz, to your $HOME/.local/easybuild/sources directory (this can be modified using the --sourcepath=SOURCEPATH argument). Also note that for some reason Oracle shows the jdk file named with the tar.gz suffix but when downloaded it only has a .gz suffix. Easybuild is looking specifically for tar.gz so rename it
    mv ~/.local/easybuild/sources/j/Java/jdk-7u80-linux-x64.gz \
          ~/.local/easybuild/sources/j/Java/jdk-7u80-linux-x64.tar.gz
    
    
  5. Now try running the build again and it should work
    eb --tmpdir=${LOCAL_SCRATCH} --installpath=/share/apps/rc R-3.2.0-goolf-1.7.20.eb --robot
    
    
  6. Test running R
    module load R/3.2.0-goolf-1.7.20
    
    build@login001 sources]$ which R
    /share/apps/rc/software/R/3.2.0-goolf-1.7.20/bin/R
    [build@login001 sources]$ R
    
    R version 3.2.0 (2015-04-16) -- "Full of Ingredients"
    Copyright (C) 2015 The R Foundation for Statistical Computing
    Platform: x86_64-unknown-linux-gnu (64-bit)
    
    R is free software and comes with ABSOLUTELY NO WARRANTY.
    You are welcome to redistribute it under certain conditions.
    Type 'license()' or 'licence()' for distribution details.
    
      Natural language support but running in an English locale
    
    R is a collaborative project with many contributors.
    Type 'contributors()' for more information and
    'citation()' on how to cite R or R packages in publications.
    
    Type 'demo()' for some demos, 'help()' for on-line help, or
    'help.start()' for an HTML browser interface to help.
    Type 'q()' to quit R.
    
    > quit();
    Save workspace image? [y/n/c]: n
    
    
    $ module show R/3.2.0-goolf-1.7.20
    
    -------------------------------------------------------------------
    /share/apps/rc/modules/all/R/3.2.0-goolf-1.7.20:
    
    module-whatis   R is a free software environment for statistical computing and graphics. - Homepage: http://www.r-project.org/
    conflict        R
    module  load goolf/1.7.20
    module  load libreadline/6.3-goolf-1.7.20
    module  load ncurses/5.9-goolf-1.7.20
    module  load libpng/1.6.17-goolf-1.7.20
    module  load libjpeg-turbo/1.4.0-goolf-1.7.20
    module  load Java/1.8.0_60
    module  load Tcl/8.6.4-goolf-1.7.20
    module  load Tk/8.6.4-goolf-1.7.20-no-X11
    module  load cURL/7.43.0-goolf-1.7.20
    module  load libxml2/2.9.2-goolf-1.7.20
    prepend-path    LD_LIBRARY_PATH /share/apps/rc/software/R/3.2.0-goolf-1.7.20/lib64
    prepend-path    LD_LIBRARY_PATH /share/apps/rc/software/R/3.2.0-goolf-1.7.20/lib64/R/lib
    prepend-path    LIBRARY_PATH /share/apps/rc/software/R/3.2.0-goolf-1.7.20/lib64
    prepend-path    LIBRARY_PATH /share/apps/rc/software/R/3.2.0-goolf-1.7.20/lib64/R/lib
    prepend-path    MANPATH /share/apps/rc/software/R/3.2.0-goolf-1.7.20/share/man
    prepend-path    PATH /share/apps/rc/software/R/3.2.0-goolf-1.7.20/bin
    prepend-path    PKG_CONFIG_PATH /share/apps/rc/software/R/3.2.0-goolf-1.7.20/lib64/pkgconfig
    setenv  EBROOTR /share/apps/rc/software/R/3.2.0-goolf-1.7.20
    setenv  EBVERSIONR 3.2.0
    setenv  EBDEVELR /share/apps/rc/software/R/3.2.0-goolf-1.7.20/easybuild/R-3.2.0-goolf-1.7.20-easybuild-devel
    setenv  EBEXTSLISTR base-,datasets-,graphics-,grDevices-,grid-,methods-,splines-,stats-,stats4-,tools-,utils-,abind-1.4-3,magic-1.5-6,geometry-0.3-5,bit-1.1-12,filehash-2.2-2,ff-2.2-13,bnlearn-3.7.1,bootstrap-2015.2,combinat-0.0-8,deal-1.2-37,fdrtool-1.2.14,formatR-1.2,gtools-3.4.2,gdata-2.13.3,GSA-1.03,highr-0.5,infotheo-1.2.0,lars-1.2,lazy-1.2-15,kernlab-0.9-20,mime-0.3,markdown-0.7.7,mlbench-2.1-1,NLP-0.1-6,mclust-5.0.1,RANN-2.5,rmeta-2.16,segmented-0.5-1.1,som-0.3-5,SuppDists-1.1-9.1,stabledist-0.6-6,survivalROC-1.0.3,pspline-1.0-16,timeDate-3012.100,longmemo-1.0-0,ADGofTest-0.3,ade4-1.7-2,AlgDesign-1.1-7.3,base64enc-0.1-2,BH-1.55.0-3,brew-1.0-6,Brobdingnag-1.2-4,corpcor-1.6.7,longitudinal-1.1.11,checkmate-1.5.2,cubature-1.1-2,DEoptimR-1.0-2,digest-0.6.8,fastmatch-1.0-4,ffbase-0.11.3,iterators-1.0.7,maps-2.3-9,nnls-1.4,sendmailR-1.2-1,spam-1.0-1,subplex-1.1-4,stringr-0.6.2,evaluate-0.7,logspline-2.1.5,ncbit-2013.03.29,permute-0.8-3,plotrix-3.5-11,randomForest-4.6-10,scatterplot3d-0.3-35,SparseM-1.6,tripack-1.3-6,irace-1.06,rJava-0.9-6,lattice-0.20-31,RColorBrewer-1.1-2,latticeExtra-0.6-26,Matrix-1.2-0,png-0.1-7,Rcpp-0.11.5,RcppArmadillo-0.5.000.0,plyr-1.8.2,pROC-1.7.3,quadprog-1.5-5,BB-2014.10-1,BBmisc-1.9,fail-1.2,rlecuyer-0.3-3,snow-0.3-13,MASS-7.3-40,tree-1.0-35,pls-2.4-3,class-7.3-12,e1071-1.6-4,nnet-7.3-9,nlme-3.1-120,minqa-1.2.4,RcppEigen-0.3.2.4.0,quantreg-5.11,mgcv-1.8-6,colorspace-1.2-6,robustbase-0.92-3,sp-1.1-0,vcd-1.3-2,snowfall-1.84-6,rpart-4.1-9,mice-2.22,urca-1.2-8,fracdiff-1.4-2,logistf-1.21,akima-0.5-11,bitops-1.0-6,boot-1.3-15,mixtools-1.0.3,cluster-2.0.1,gclus-1.3.1,coda-0.17-1,codetools-0.2-11,foreach-1.4.2,doMC-1.3.3,DBI-0.3.1,foreign-0.8-63,survival-2.38-1,gam-1.09.1,gamlss.data-4.2-7,gamlss.dist-4.3-4,hwriter-1.3.2,KernSmooth-2.23-14,zoo-1.7-12,xts-0.9-7,TTR-0.22-0,quantmod-0.4-4,lmtest-0.9-33,mnormt-1.5-2,mvtnorm-1.0-2,pcaPP-1.9-60,numDeriv-2012.9-1,lava-1.4.0,prodlim-1.5.1,pscl-1.4.9,RSQLite-1.0.0,BatchJobs-1.6,sandwich-2.3-3,sfsmisc-1.0-27,spatial-7.3-9,VGAM-0.9-7,waveslim-1.7.5,xtable-1.7-4,profileModel-0.5-9,brglm-0.5-9,deSolve-1.11,tseriesChaos-0.1-13,tseries-0.10-34,fastICA-1.2-0,R.methodsS3-1.7.0,R.oo-1.19.0,cgdsr-1.1.33,R.utils-2.0.1,R.matlab-3.2.0,gbm-2.1.1,dichromat-2.0-0,Formula-1.2-1,acepack-1.3-3.3,reshape2-1.4.1,gtable-0.1.2,munsell-0.4.2,labeling-0.3,scales-0.2.4,proto-0.3-10,ggplot2-1.0.1,Hmisc-3.15-0,fastcluster-1.1.16,chron-2.3-45,data.table-1.9.4,igraph-0.7.1,GeneNet-1.2.12,ape-3.2,htmltools-0.2.6,RJSONIO-1.3-0,caTools-1.17.1,gplots-2.16.0,ROCR-1.0-7,httpuv-1.3.2,R6-2.0.1,shiny-0.11.1,adegenet-1.4-2,phylobase-0.6.8,adephylo-1.1-6,animation-2.3,bigmemory.sri-0.1.3,bigmemory-4.4.6,calibrate-1.7.2,clusterGeneration-1.3.4,raster-2.3-40,dismo-1.0-12,expm-0.99-1.1,extrafontdb-1.0,Rttf2pt1-1.3.3,extrafont-0.17,fields-8.2-1,shapefiles-0.7,fossil-0.3.7,geiger-2.0.3,glmnet-2.0-2,labdsv-1.6-1,MatrixModels-0.4-0,stabs-0.5-1,mboost-2.4-2,msm-1.5,nor1mix-1.2-0,np-0.60-2,polynom-1.3-8,quantreg-5.11,polspline-1.1.9,TH.data-1.0-6,multcomp-1.4-0,gridExtra-0.9.1,rms-4.3-0,RWekajars-3.7.12-1,RWeka-0.4-24,slam-0.1-32,tm-0.6,TraMineR-1.8-9,chemometrics-1.3.9,FNN-1.1,ipred-0.9-4,yaml-2.1.13,knitr-1.10,statmod-1.4.21,miscTools-0.6-16,maxLik-1.2-4,mlogit-0.2-4,getopt-1.20.0,gsalib-2.1,reshape-0.8.5,optparse-1.3.0,klaR-0.6-12,neuRosim-0.2-12,locfit-1.5-9.1,GGally-0.5.0,beanplot-1.2,clValid-0.6-6,matrixStats-0.14.0,DiscriMiner-0.1-29,ellipse-0.3-8,leaps-2.9,nloptr-1.0.4,lme4-1.1-8,pbkrtest-0.4-2,car-2.0-25,flashClust-1.01-2,FactoMineR-1.29,modeltools-0.2-21,flexclust-1.3-4,flexmix-2.3-13,prabclus-2.2-6,diptest-0.75-6,trimcluster-0.1-2,fpc-2.1-9,BiasedUrn-1.06.1,TeachingDemos-2.9,jsonlite-0.9.16,kohonen-2.0.18,base64-1.1,registry-0.2,pkgmaker-0.22,rngtools-1.2.4,doRNG-1.6,nleqslv-2.7,RGCCA-2.0,pheatmap-1.0.2,openxlsx-2.4.0,pvclust-1.3-2,RCircos-1.1.2,VennDiagram-1.6.9,xlsxjars-0.6.1,xlsx-0.5.7,vegan-2.3-0,forecast-6.1,fma-2.01,expsmooth-2.3,fpp-0.5,XML-3.98-1.1,memoise-0.2.1,crayon-1.3.1,testthat-0.10.0,rmarkdown-0.7,curl-0.9.1,RCurl-1.95-4.7,httr-0.6.1,maptools-0.8-36,deldir-0.1-9,tensor-1.5,polyclip-1.3-0,goftest-1.0-2,spatstat-1.41-1,gdalUtils-0.3.1,pracma-1.8.3,bio3d-2.2-2
    -------------------------------------------------------------------
    
    

References