Site Configuration Guide

If you do not already have a Computation Institute (CI) account, you can request one at https://www.ci.uchicago.edu/accounts/. This page will give you a list of resources you can request access to.

If you already have an existing CI account, but do not have access to Beagle, send an email to support@ci.uchicago.edu to request access.

Once you have account, you should be able to access a Beagle login node with the following command:

Follow the steps outlined below to get started with Swift on Beagle:

step 1. Load the Swift and Sun-java module on Beagle as follows: module load swift sun-java

step 2. Create and change to a directory where your Swift related work will stay. (say, mkdir swift-lab, followed by, cd swift-lab)

step 3. To get started with a simple example running /bin/cat to read an input file data.txt and write to an output file f.nnn.out, start with writing a simple swift source script as follows:

step 4. The next step is to create a sites file. An example sites file (sites.xml) is shown as follows:

step 5. In this step, we will see the config and tc files. The config file (cf) is as follows:

The tc file (tc) is as follows:

More about config and tc file options can be found in the Swift documentation.

step 6. Run the example using following commandline:

You can further change the value of -n to any arbitrary number to run that many number of concurrent cat

step 7. Swift will show a status message as "done" after the job has completed its run in the queue. Check the output in the generated outdir directory (ls outdir)

A key factor in scaling up Swift runs on Beagle is to setup the sites.xml parameters. The following sites.xml parameters must be set to scale that is intended for a large run:

Following is an example sites.xml for a 50 slots run with each slot occupying 4 nodes (thus, a 200 node run):

In this section we will discuss some of the common issues and remedies while using Swift on Beagle. The origin of these issues can be Swift or the Beagle’s configuration, state and user configuration among other factors. We try to identify maximum known issues and address them here:

The most likely cause is the module is not loaded. Do the following to load the Swift module:

It is likely that it is set to a path where the compute nodes can not write, e.g. your /home directory. The remedy for this error is to set your workdirectory to the /lustre path where swift could write from compute nodes.

If you do not already have a Fusion account, you can request one at https://accounts.lcrc.anl.gov/request.php. Email support@lcrc.anl.gov for additional help.

In order to run a job on a Fusion compute node, you must first be associated with a project.

Each project has one or more Primary Investigators, or PIs. These PIs are responsible for adding and removing users to a project. Contact the PI of your project to be added.

More information on this process can be found at http://www.lcrc.anl.gov/info/Projects.

Before accessing Fusion, be sure to have your SSH keys configured correctly. SSH keys are required to access fusion. You should see information about this when you request your account. Check ssh FAQ or email support@lcrc.anl.gov for additional help.

Once your keys are configured, you should be able to access a Fusion login node with the following command:

Swift uses various configuration files to determine how to run an application. This section will provide a working configuration file which you can copy and paste to get running quickly. The sites.xml file tells Swift how to submit jobs, where working directories are located, and various other configuration information. More information on sites.xml can be found in the Swift documentation.

The first step is to paste the text below into a file named sites.xml.

This file will require one customization. Create a directory called swiftwork. Modify _WORK_ in sites.xml to point to a new directory. For example

The tc configuration file gives information about the applications that will be called by Swift. More information about the format of tc can be found in the Swift user guide.

Paste the following example into a file named tc

Within the Swift directory is an examples directory which contains several introductory Swift scripts. The example we will use in this section is called catsn.swift. The script copies input file’s content to another file using the Unix cat utility. Copy this script to the same directory that your sites.xml and tc.data files are located.

Finally, run the script

You should see 10 new text files get created, named catsn*.out. If you see these files, then you have succesfully run Swift on Fusion!

Fusion has two queues: shared and batch. The shared queue has a maximum 1 hour walltime and limited to 4 nodes. The batch queue is for all other jobs.

Edit your sites.xml file and edit the queue option to modify Swift’s behavior. For example:

More information on Fusion queues can be found at http://www.lcrc.anl.gov/info/BatchJobs.

The best place for additional help is the Swift user mailing list. You can subscribe to this list at swift-user. When submitting information, send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.

The following sites.xml has the basic definitions for how to run a job using the LSF scheduler.

The values for workdirectory, queue, and project will likely need to be adjusted based on the requirements of your project.

If you do not already have an account on Intrepid, you can request one here. More information about this process and requesting allocations for your project can be found here.

Accessing the Intrepid via SSH can be done with any SSH software package. Before logging in, you will need to generate an SSH public key and send it to support@alcf.anl.gov for verification and installation.

This security token uses one-time passwords for controlled access to the BG/P login systems.

When you gain access to Intrepid, you should receive a cryptocard and a temporary PIN. You must have a working cryptocard, know your PIN, and have your SSH key in place before you may login.

You can connect to Intrepid with the following command:

You will be presented with a password prompt. The first part of your password is your PIN. Enter you PIN, press the Cryptocard button, and then enter the password your crypocard generates. If this is the first time you are logging in, you will be prompted to change your PIN.

The most recent versions of Swift can be found at http://swiftlang.org/downloads/index.php. Follow the instructions provided on that site to download and build Swift.

Once you have installed Swift, add the Swift binary to your PATH so you can easily run it from any directory.

In your home directory, edit the file ".bashrc".

If you have installed Swift via a source repository, add the following line at the bottom of .bashrc.

If you have installed Swift via a binary package, add this line:

Replace <version> with the actual name of the swift directory in the example above.

Note that on Intrepid, the compute nodes can not create or write to a /home filesystem. Consequently, in order for Swift to interface correctly from login node to the compute nodes, Swift must write all internal and intermediate files to /intrepid-fs0, which is writable by the compute nodes. In order to accomplish this, export the environment variable SWIFT_USERHOME as follows:

Before you can create a Swift configuration file, there are some things you will need to know.

Now that you know what queue to use, your project, and your work directory, it is time to set up Swift. Swift uses a configuration file called sites.xml to determine how it should run. There are two methods you can use for creating this file. You can manually edit the configuration file, or generate it with a utility called gensites.

Below is the tc.data file used by Swift’s test suite.

Copy these commands and save it as tc.data.

The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.

Now that everything is in place, run Swift with the following command:

You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these files should contain the contents of what you placed into data.txt. If this happens, your job has run successfully!

The best place for additional help is the Swift user mailing list. You can subscribe to this list. When submitting information, send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.

To begin, copy the text below and paste it into your Swift distribution’s etc directory. Name the file coaster-service.conf.

Change directories to the location you would like to run a Swift script and start the coaster service with this command:

This will create a configuration file that Swift needs called sites.xml.

Next, run Swift. If you do not have a particular script in mind, you can test Swift by using a Swift script in the examples/ directory.

Run the following command to run the script:

The coaster service will run indefinitely. The stop-coaster-service script will terminate the coaster service.

This will kill the coaster service and kill the worker scripts on remote systems.

Once you have access to Midway, connect to a Midway login node.

Swift is available on Midway as a module. To load the Swift, run:

Below is an example that uses two of the queues available on Midway, sandyb and westmere. Be sure to adjust walltime, work directory, and other options as needed.

Below is an example sites.xml that is suitable for running with MPI. Jobtype must be set to single. The value you set for ppn will determine the number of cores/slots your application uses per node. The value of count will set the number of nodes to request. The example below requests 2 nodes with 12 slots per node.

A Slurm submit script has many settings and options. Swift knows about many of the basic Slurm settings, like how to define a project or a queue, but it does not know about every setting. Swift provides a simple way to pass-thru your own settings into the Slurm submit script.

The general way to do this is:

Here is one specific example. Slurm has the ability to notify users via email when a job is done. To make this happen, the Slurm submit script that Swift generates needs a line that contains "--mail-type=END". The following line will make it happen.

Any valid Slurm setting can be set in a similar way (see the sbatch man page for a list of all settings).

Below is the simplest example, where the coaster service is started, and workers are launched locally on the same machine.

First, create a file called coaster-service.conf with the configuration below.

To start the coaster service and worker, run the command "start-coaster-service". Then run Swift with the newly generated sites.xml file.

You can then run swift multiple times using the same coaster service. When you are finished and would like to shut down the coaster, run stop-coaster-service.

The start-coaster-service script can start workers on multiple remote machines. To do this, there are two main settings you need to define in your coaster-service.conf. The first is to set WORKER_MODE=ssh, and the second is set WORKER_HOSTS to the list of machines where workers should be started.

If there is no shared filesystem available between the remote machines and the local machine, you will need to enable coaster provider staging to transport files for you. Below is an example Swift configuration file to enable it:

Run start-coaster service to start coaster and workers. When you run Swift, reference the cf file to enable provider staging.

This example is for a situation where you want to start a worker on nodes that you can’t connect to directly. If you have to first connect to a login/gateway machine before you can ssh to your worker machine, this configuration is for you.

The coaster-service.conf and cf files are the same as in Example 2.

Assume that node.host.edu is the machine where you want to start your worker, and that gateway.host.edu is the machine where you must log into first. Add the following to your $HOME/.ssh/config file:

This will allow you to SSH directly to node.host.edu. You can now add node.host.edu to WORKER_HOSTS.

The coaster workers need to be able to make a connection back to the coaster service. If you are running Swift on a machine behind a firewall where the workers cannot connect, you can use SSH reverse tunneling to allow this connection to happen.

To enable this, add the following line to your coaster-service.conf:

The variable WORKER_HOSTS defines the list of hostnames where workers will be started. To set this to be the contents of a file, you can set WORKER_HOSTS as follows:

To start workers via some other script, such as a scheduler submit script, export WORKER_MODE=scheduler. Once the coaster service has been initialized, start-coaster-service will run whatever user defined command is defined in $SCHEDULER_COMMAND.

The contents of SCHEDULER_COMMAND will vary greatly based on your needs and the system you are running on. However, all SCHEDULER_COMMANDs will need to run the same command exactly once on each worker node:

Here is an example that runs on a campus cluster using the Slurm scheduler: .coaster-service.conf

The SCHEDULER_COMMAND in this case submits a Slurm job script and starts the workers via the following commands:

Below is a list of all settings that start-coaster-service knows about, along with a brief description of what it does.

The settings are defined in terms of bash variables. Below is an example of the format used

Below is a list of the options that coaster-service.conf recognizes and what they do.

It is recommended that you create a new SSH key exclusively for this purpose. In order to avoid being prompted for passwords/passphrases, your SSH passphrase will be stored in a read protected file. Run this command on the machine where you will be running Swift:

You will be prompted to create a passphrase. This will create two files: $HOME/.ssh/id_dsa-swift and $HOME/.ssh/id_dsa-swift.pub.

On the remote host where you will be running, edit or create the file $HOME/.ssh/authorized_keys. Paste in the contents of the newly created $HOME/.ssh/id_dsa-swift.pub from the previous step to the end of the file.

From the host where you will be running Swift, run the following command to verify your keys are working:

You should be prompted for your new passphrase and be able to connect.

Create a file called $HOME/.ssh/auth.defaults on the host where you are running Swift. Use the following commands to create this file:

Next, edit $HOME/.ssh/auth.defaults and add the following lines:

Replace login.remotehost.edu with the hostname you want to use, replace the values for "your_remote_username", "your_passphrase", and set the correct path of private key you generated.

Here is an example sites.xml file that will allow you to connect and transfer data to a remote host:

Since you want to data transfer via ssh, you’ll want to verify that you’re not using any other file transfer mechanisms. Make sure you have the following swift properties defined in your configuration file:

The first step of this process is to verify that you can connect to a remote site without being prompted for a password or passphrase.

Typically to make this work you will need to add your SSH public key to the $HOME/.ssh/authorized_keys file on the remote system.

This SSH connection must work without specifying a username on the command line. If your username differs on your local machine and the remote machine, you will need to add an entry like this to your local $HOME/.ssh/config:

Once you have verified that you can connect without prompt to the remote machine, you will need to create a sites.xml file that contains the host information. The example below will assume there is no scheduler on the remote system - it simply connects to the remote machine and runs work there.

If there is a shared filesystem between the two machines, you can set this as your work directory and skip this step. Otherwise, you will need to enable coaster provider staging.

To do this, add the following line to your "cf" file:

To run swift, then:

The most recent versions of Swift can be found at the Swift downloads page. Follow the instructions provided on that site to download and build Swift.

You will need to do the following steps in order to run.

The system where you run Swift needs to have the myproxy-init tool installed. Ideally it should also have globus-job-run for testing purposes.

Swift uses two environment variables in order for remote job execution to work. They are $GLOBUS_HOSTNAME and $GLOBUS_TCP_PORT_RANGE.

GLOBUS_HOSTNAME should contain the full hostname of the system where you are running. It may also contain the IP address of the machine.

GLOBUS_TCP_PORT_RANGE defines a range of ports to which a remote system may connect. You will likely need this defined if you are behind a firewall with somewhat restricted access.

Once you have verified you have everything you need on the submit host where you are running, you can obtain an XSEDE grid certificate with following command:

You may use the following example as a guide to run on Stampede. You will likely need to make a few modifications, as described below.

You will need to modify the XSEDE project name to the match the name that has been allocated to you. In most cases you’ll want to set the work directory to your Stampede scratch directory. This is defined, on Stampede, in the environment variable $SCRATCH.

You may now run your Swift script exactly as you have before.

If you are having problems getting this working correctly, there are a few places where you may look to help you debug. Since this configuration is slightly more complicated, there are several log files produced.

For help in getting this configuration working, feel free to contact the Swift support team at swift-support@ci.uchicago.edu.