How-To Tips for Specific User Communities
-----------------------------------------

Saving Logs - for UChicago CI Users
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you have a UChicago Computation Institute account, run this command
in your submit directory after each run. It will copy all your logs and
kickstart records into a directory at the CI for reporting, usage
tracking, support and debugging.

----
rsync --ignore-existing *.log *.d login.ci.uchicago.edu:/disks/ci-gpfs/swift/swift-logs/ --verbose
----

Specifying TeraGrid allocations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TeraGrid users with no default project or with several project
allocations can specify a project allocation using a profile key in the
site catalog entry for a TeraGrid site:

----
<profile namespace="globus" key="project">TG-CCR080002N</profile>
----

More information on the TeraGrid allocations process can be found here
<http://www.teragrid.org/userinfo/access/allocations.php>.

Launching MPI jobs from Swift
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There are several ways to run MPI jobs under Swift. Two will be discussed here -
calling mpiexec from a wrapper script, and using the MPICH/coasters interface.

Calling mpiexec
^^^^^^^^^^^^^^^
In this example, a single MPI program will run across two nodes. For this to
happen, sites.xml must be configured to allocate two nodes but only run a
single job on them. A wrapper script must then be used to call mpiexec.

sites.xml
+++++++++
First, we need to make sure that Swift will allocate exactly two nodes. This
can be done with the maxnodes and nodegranularity settings.

-----
<profile namespace="globus" key="nodeGranularity">2</profile>
<profile namespace="globus" key="maxnodes">2</profile>
-----

Next, we want to make sure that the MPI program is called only once on those
nodes. There are two settings we must set to get this behavior:
-----
<profile namespace="globus" key="jobsPerNode">1</profile>
<profile namespace="globus" key="jobtype">single</profile>
-----

tc.data
+++++++
The app defined in tc.data should be a shell script wrapper to the actual
program that is being called. Let's assume in this example that the MPI program
we are using is called "mpitest", and the wrapper script will be called 
"mpitest.sh". The tc.data will look like this then:

-----
host mpitest /path/to/mpitest.sh
-----

Wrapper script
++++++++++++++
The wrapper script in this example, mpitest.sh, will call mpiexec and launch
the real MPI program. Here is an example:

-----
#!/bin/bash 

mpiexec /path/to/mpitest "$@"
-----

Swift then makes an invocation that does not look any different from any other
invocation. In the code below, we pass one input file and get back one output
file. 

----
type file;

app (file output_file) mpitest (file input_file)
{
   mpitest @input_file @output_file; 
}

file input <"input.txt">;
file output <"output.txt">;

output = mpitest(input);
----

==== MPICH/Coasters

In this case, the user desires to launch many MPI jobs within a single
Coasters allocation, reusing Coasters workers for variable-sized jobs.
The reuse of the Coasters workers allows the user to launch many
MPI jobs in rapid succession with minimal overhead.

The user must access to MPICH compiled for sockets, with +mpiexec+ in
the +PATH+ environment variable.  Swift uses this MPICH installation
to launch the user processes on the remote Coasters workers, which are
able to connect back to +mpiexec+ and coordinate the job launch.  The
infrastructure must allow the user MPI processes to find each other
and communicate over sockets.

To configure the user MPI job, simply add +mpi.processes+ and
+mpi.ppn+ to the profile in the +tc.file+:

----
pbs_site my_program  /path/to/program null null globus::mpi.processes=16;globus::mpi.ppn=8
----

Coasters must be set with +jobsPerNode=1+.

This runs +mpiexec+ locally, and allocates 2 Coasters workers (2
nodes), each with 8 MPI processes.  Thus, +MPI_COMM_WORLD+ has size
16.



Running on Windows
~~~~~~~~~~~~~~~~~~

Swift has the ability to run on a Windows machine, as well as the
ability to submit jobs to a Windows site (provided that an appropriate
provider is used).

In order to launch Swift on Windows, use the provided batch file
(swift.bat). In certain cases, when a large number of jar libraries are
present in the Swift lib directory and depending on the exact location
of the Swift installation, the classpath environment variable that the
Swift batch launcher tries to create may be larger than what Windows can
handle. In such a case, either install Swift in a directory closer to
the root of the disk (say, c:\swift) or remove non-essential jar files
from the Swift lib directory.

Due to the large differences between Windows and Unix environments,
Swift must use environment specific tools to achieve some of its goals.
In particular, each Swift executable is launched using a wrapper script.
This script is a Bourne Shell script. On Windows machines, which have no
Bourne Shell interpreter installed by default, the Windows Scripting
Host is used instead, and the wrapper script is written in VBScript.
Similarly, when cleaning up after a run, the "/bin/rm" command available
in typical Unix environments must be replaced by the "del" shell command.

It is important to note that in order to select the proper set of tools
to use, Swift must know when a site runs under Windows. To inform Swift
of this, specify the "sysinfo" attribute for the "pool" element in the
site catalog. For example:

----
<pool handle="localhost" sysinfo="INTEL32::WINDOWS">
...
</pool>
----