[[configuration]]
@LIBRARY
== Configuration

Swift is mainly configured using a configuration file, typically called 
*swift.conf*. This file contains configuration properties and site descriptions.
A simple configuration file may look like this:

[listing,swiftconf]
-----
site.mysite {
    execution {
        type: "coaster"
        URL: "my.site.org"
        jobManager: "ssh:local"
    }
    staging: "local"

    app.ALL {executable: "*"}
}

# select sites to run on
sites: [mysite]

# other settings
lazy.errors: false
-----

NOTE: Swift versions 0.95 and older used a different configuration system in
which the same information was contained in three files: +sites.xml+,
+swift.properties+ and +tc.data+. The old configuration files can be converted 
to the new configuration format using the *swift-convert-config* utility that 
comes with versions of Swift newer than 0.96:

-----
swift-convert-config -sites.file sites.xml -config swift.properties \
    -tc.file tc.data -out swift.conf
-----


=== Configuration Syntax

The Swift configuration files are expressed in a modified version of JSON. The 
main additions to JSON are:

- Quotes around string values, in particular keys, are optional, unless the 
strings contain special characters (single/double quotes, square and curly 
braces, white space,
+$+, +:+, +=+, +,+, +`+, +^+, +?+, +!+, +@+, +*+, +\+), or if they
represent other values: +true+, +false+, +null+, and numbers.
- +=+ and +:+ can be used interchangeably to separate keys from values
- +=+ (or +:+) is optional before an open bracket
- Commas are optional as separators if there is a new line
- +${...}+ expansion can be used to substitute environment variable values or 
  Java system properties. If the value of an environment variable is needed, it 
  must be prefixed with +env.+. For example +${env.PATH}+. Except for include
  directives, the
  +${...}+ must not be inside double quotes for the substitution to work. The 
  same outcome can be achieved using implicit string concatenation: +"/home/"${env.USER}"/bin"+
 
Comments can be introduced by starting a line with a hash symbol (+#+) or using
a double slash (+//+):

[listing,swiftconf]
-----
# This is a comment
// This is also a comment

keepSiteDir: true # This is a comment following a valid property
-----

=== Include Directives

Include directives can be used to include the contents of a Swift configuration 
file from another Swift configuration file. This is done using the literal 
+include+ followed by a quoted string containing the path to the target file. 
The path may contain references to environment variables or system properties 
using the substitution syntax explained above. For example:

[listing,swiftconf]
-----
# an absolute path name
include "/home/joedoe/swift-config/site1.conf"

# include a file from the Swift distribution package
include "${swift.home}/etc/sites/beagle.conf"

# include a file using an environment variable
include "${env.SWIFT_CONFIG_DIR}/glow.conf"
-----

[[section-property-merging]]
=== Property Merging

If two properties with the same name are present in a configuration file, they 
are either merged or the latter one overrides the earlier one. This depends on 
the type of property. Simple values are always overridden, while objects are 
merged. For example:

[listing,swiftconf]
-----
key: 1
key: 2
# key is now 2

object {
    key1: 1
}

object {
    key2: 2
}

# object is now { key1: 1, key2: 2}
-----

This can be used to define certain template files that contain most of the
definitions for sites, and then include them in other files and override or add 
only certain aspects of those sites. For example, assume +swift-local.conf+ 
includes a definition for a site named +local+ that can be used to run 
applications on the Swift client side. Override only the work directory
could be achieved with the following +swift.conf+:

[listing,swiftconf]
-----
include "swift-local.conf"

site.local {
    # use existing definition for site.local, but override workDirectory
    workDirectory: "/tmp"
}
-----

A full override of the definition of +site.local+ can be done by first
setting it to +null+ and then providing a new definition:

[listing,swiftconf]
-----
include "swift-local.conf"

# forget previous definition of site.local
site.local: null

# define a new site.local from scratch
site.local {
    ...
}
-----

=== Configuration Search Path

By default, Swift attempts to load multiple configuration files, merging them sequentially as described in the <<section-property-merging, Property Merging>> Section. The files are:

. Distribution Configuration (*[D]*): +${swift.home}/etc/swift.conf+
. Site Configuration (*[S]*): +${env.SWIFT_SITE_CONF}+ (if +SWIFT_SITE_CONF+ is defined)
. User Configuration (*[U]*): +${env.HOME}/.swift/swift.conf+ (if present)
. Run Configuration (*[R]*): +${env.PWD}/swift.conf+ (if present)

In addition, a number of configuration properties can be overridden 
individually on the Swift command line. For a list of such configuration 
properties, please use +swift -help+ or refer to the 
<<running:running-swift, Running Swift>> Section.

The run configuration can be overridden on the Swift command line using the 
+-config <file>+ command line argument. If the +-config+ argument is specified, 
Swift will not attempt to load +swift.conf+ from the current directory and will 
use the value of +<file>+ instead.

The entire configuration search path can be replaced with a custom search path 
using the +-configpath+ command line argument. The value passed to +-configpath+
must be a list of paths pointing to various configuration files, separated by 
the standard operating system path separator (+':'+ on Linux and +';'+ on 
Windows). For example:

[listing,shell]
-----
swift -configpath /etc/swift/swift.conf:~/swift-configs/s1.conf:swift.conf
-----

If in doubt about what configuration files are being loaded or to troubleshoot 
configuration issues, Swift can be started with the +-listconfig+ command line argument. +-listconfig+ accepts tow possible values:

* +files+: will print a list of configuration files loaded by Swift
* +full+: will print a list of configuration files loaded by Swift, as well as
the final merged configuration.

=== Configuration File Structure

The contents of a Swift configuration file can be divided into a number of 
relevant sections:

- site declarations
- global application declarations
- Swift configuration properties

==== Site Declarations

Swift site declarations are specified using the +site.<name>+ property, where 
text inside angle brackets is to be interpreted as a generic label for 
user-specified content, whereas content between square brackets is optional:

[listing,swiftconf]
-----
site.<name> {
    execution {...}
    [staging: "swift" | "local" | "service-local" | "shared-fs" | "wrapper"]
    [filesystem {...}]
    workDirectory: <path>

    [<site options>]
    [<application declarations>]
}
-----

A site name can be any string. If the string contains special characters, it 
must be quoted:

[listing,swiftconf]
-----
site."$pecial-$ite" {...}
-----

==== Site Selection

Once sites are declared, they must be explicitly enabled for Swift to use them. 
This can be achieved with the +sites+ option, which accepts either an array or a comma-separated list of site names:

[listing,swiftconf]
-----
sites: ["site1", "site2"]

# alternatively:

sites: "site1, site2"
-----

The +sites+ option can also be specified on the Swift command line:

[listing,shell]
-----
swift -sites site1,site2 script.swift
-----

==== Execution Mechanisms

The +execution+ property tells Swift how applications should be executed on a 
site:

[listing,swiftconf]
-----
    execution {
        type: <string>
        [URL: <string>]
        [jobManager: <string>]
        
        [<execution provider options>]
    }
-----

The +type+ property is used to select one of the mechanisms for application execution
that is known by Swift. A comprehensinve list of execution mechanisms can be found 
in <<configuration:execution-mechanisms, Execution Mechanisms Section>>. A summary is 
shown below:

[[table-execution-mechanisms]]
.Swift Execution Mechanisms
[options="header",cols="3,2,2,2,4,10"]
|========================================================================================================================
|Type                 |URL required|Uses jobManager|Default jobManager|Staging methods supported                        | 
Description

|<<em:local,local>>   | no         | no            | -                | swift, local, wrapper                           | 
Runs applications locally using a simple fork()-based mechanism

|+coaster+            | yes        | yes           | none             | swift, wrapper, local, service-local, shared-fs, direct | 
Submits applications through an automatically-deployed Swift Coasters service

|+coaster-persistent+ | yes        | yes           | none             | swift, wrapper, local, service-local, shared-fs, direct | 
Uses a manually deployed Swift Coasters service

|+GRAM5+              | yes        | yes           | "fork"           | swift, wrapper                                  | 
Uses the http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/user/#gram5User[GRAM: User's Guide] component of 
the Globus Toolkit.

|+GT2+              4+|                                                                                                 | 
An alias for 'GRAM5'

|+SSH+                | yes        | no            | -                | swift, wrapper                                  | 
Runs applications using a Java implementation of the 'SSH' protocol

|+SSH-CL+             | yes        | no            | -                | swift, wrapper                                  | 
Like 'SSH' except it uses the command-line 'ssh' tool.

|+PBS+                | no         | no            | -                | swift, wrapper                                  | 
Submits applications to a PBS or Torque resource manager

|+Condor+             | no         | no            | -                | swift, wrapper                                  | 
Submits applications using Condor

|+SGE+                | no         | no            | -                | swift, wrapper                                  | 
Uses the Sun Grid Engine

|+SLURM+              | no         | no            | -                | swift, wrapper                                  | 
Uses the SLURM local scheduler

|+LSF+                | no         | no            | -                | swift, wrapper                                  | 
Submits applications to Platform's Load Sharing Facility

|========================================================================================================================


The execution provider +options+ are options that specify finer details on how
on application should be executed. They depend on the chosen mechanism and are detailed in
<<configuration:execution-mechanisms, Execution Mechanisms>> Section. This is where 
Coasters options, such as +nodeGranularity+ or +softImage+, would
be specified. Example:

[listing,swiftconf]
-----
execution {
    type: "coaster"
    jobManager: "local:local"
    options {
        maxJobs: 1
        tasksPerNode: 2
        workerLoggingLevel: TRACE
    }
}
-----

A complete list of Swift Coasters options can be found in <<table-em-options-coaster,Coaster Options>>

[[configuration:staging]]
==== Staging

The staging property instructs Swift how to handle application input and output
files. The +swift+ and +wrapper+ staging methods are supported universally, but 
the +swift+ method requires a +filesystem+ property to be specified. The 
+staging+ property defaults to +swift+. Support for the other choices is 
dependent on the execution mechanism. The supported staging methods for each
execution mechanism are listed in the 
<<table-execution-mechanisms,Execution Mechanisms>> Section 
above. For more details about each staging method, please refer to the 
<<runtime:application-functions-execution, Application Function Execution>> 
Section. A brief description of each staging method is provided below:

+swift+::  This method instructs Swift to use a filesystem provider to direct 
all necessary staging operations from the Swift client-side to the cluster head 
node. If this method is used, the +workDirectory+ must point to a head node 
path that is on a shared file system accessible by the compute nodes.

+wrapper+:: File staging is done by the Swift application wrapper

+local+:: Used to indicate that files should be staged in/out from/to the site 
on which Swift is running. In the case of Swift Coasters, the system proxies 
the tranfers between client side and compute nodes through the Coaster Service.

+service-local+:: This method instructs the execution mechanism provider to 
stage input and output files from the remote site where the execution service 
is located. For example, if a Coaster Service is started on the login node of a
cluster, the Coaster Service will perform the staging from a file system on the 
login node to the compute node and back.

+shared-fs+:: This method is used by Coasters to implement a simple staging 
mechanism in which files are accessed using a shared filesystem that is 
accessible by compute nodes

+direct+:: Tries to avoid moving files around as much as possible and passes 
absolute path names to the application instead. The node on which the 
application is running must have access to the filesystem on which swift data 
is located.

==== File System

The file system property is used with +staging: "swift"+ to tell Swift how to access remote file systems. It has the following form:

[listing,swiftconf]
----
filesystem {
    type: <string>
    [URL: <string>]
}
----

Valid types are described below:

[[table-filesystem-providers]]
.Swift File System Providers
[options="header",cols="3, 3, 10"]
|==========================================================================
| Type      | URL required | Description
| +local+   | no           | Copies files locally on the Swift client side
| +GSIFTP+  | yes          | Accesses a remote file system using GridFTP
| +GridFTP+ | yes          | An alias for +GSIFTP+
| +SSH+     | yes          | Uses the SCP protocol
|==========================================================================


==== Site Options

Site options control various aspects of how Swift handles application execution 
on a site. All options except +workDirectory+ are optional. The options are as 
follows. Valid values are listed first, and, if applicable, are followed by 
default values.

@CONFIGPROP OS  ("INTEL32::LINUX" | "INTEL64::LINUX" | "INTEL32::WINDOWS" |\
"INTEL64::WINDOWS" | <string>)
"INTEL32::LINUX"

Can be used to tell Swift the type of the operating system 
running on the remote site. By default, Swift assumes a UNIX/Linux type OS. 
There is some limited support for running under Windows, in which case this 
property must be set to one of +"INTEL32::WINDOWS"+ or +"INTEL64::WINDOWS"+

@CONFIGPROP workDirectory <string>

The +<string>+ is a path representing a directory in which Swift should store a 
set of files relevant to the execution of an application on the site. By 
default, applications will be executed on the compute nodes in a sub-directory 
of +<string>+. Swift must be able to create the +workDirectory+ if it does not
exist, or it must be able to create files and-subdirectories in it if it exists.

@CONFIGPROP scratch <string>

If specified, it instructs swift to run applications in a directory different 
than +workDirectory+. The value of +scratch+ must point to a file system 
accessible on compute nodes (but not necessarily a shared file system). This 
option is useful if applications do intensive I/O on temporary files created 
in their work directory, or if they access their input/output files in a 
non-linear fashion.

@CONFIGPROP keepSiteDir <boolean>
false

If set to +true+, site application directories (i.e. +workDirectory+) will not 
be cleaned up when Swift completes a run. This can be useful for debugging. 


@CONFIGPROP statusMode ("files" | "provider")
"files"

Controls whether application exit codes are handled by the execution mechanism 
or passed back to Swift by the Swift wrapper script through files. 
Traditionally, Globus GRAM did not use to return application exit codes. This 
has changed in Globus Toolkit 5.x. However, some local scheduler execution 
mechanisms, such as 'PBS', are still unable to return application exit codes. 
In such cases, it is necessary to pass the application exit codes back to Swift 
in files. This comes at a slight price in performance, since a file needs to be 
created, written to, and transferred back to Swift for each application 
invocation. It is however also the default, since it works in all cases.

@CONFIGPROP maxParallelTasks <integer>
2

The maximum number of concurrent application invocations
allowed on this site.

@CONFIGPROP initialParallelTasks <integer>
2

The limit on the number of concurrent application invocations on this site when 
a Swift run is started. As invocations complete successfully, the number of 
concurrent invocations on the site is increased up to +maxParallelTasks+.

@END

Additional, less frequently used options, are as follows:

@CONFIGPROP wrapperParameterMode ("args" | "files")
"args"

If set to +"files"+, Swift will, as much as possible, pass application 
arguments through files. The applications will be invoked normally, with their 
arguments in the +**argv+ parameter to the +main()+ function. This can be 
useful if the execution mechanism has limitations on the size of command
line arguments that can be passed through. An example of execution mechanism 
exhibiting this problem is Condor.

@CONFIGPROP wrapperInterpreter <string>
"/bin/bash" (UNIX), "cscript.exe" (Windows)

Points to the interpreter used to run the Swift application invocation wrapper

@CONFIGPROP wrapperScript <string>
"_swiftwrap" (UNIX), "_swiftwrap.vbs" (Windows)

Points to the Swift application invocation wrapper. The file must exist in the 
'libexec' directory in the Swift distribution.

@CONFIGPROP wrapperInterpreterOptions '[' [<string> [, <string>]*] ']'
[] (UNIX), ["//Nologo"] (Windows)

Command line options to be passed to the wrapper interpreter

@CONFIGPROP cleanupCommand <string>
"/bin/rm" (UNIX), "cmd.exe" (Windows)

A command to use for the cleaning of site directories (unless +keepSiteDir+ is 
set to +true+) at the end of a run.

@CONFIGPROP cleanupCommandOptions '[' [<string> [, <string>]*] ']'
["-rf"] (UNIX), ["/C", "del", "/Q"] (Windows)

Arguments to pass to the cleanup command when cleaning up site work directories

@CONFIGPROP delayBase <float>
2.0

Swift keeps a quality indicator for each site it runs applications on. This is 
a number that gets increased for every successful application invocation, and 
decreased for every failure. It then uses this number in deciding which sites 
to run applications on (when multiple sites are defined). If this number 
becomes very low (a sign of repeated failures on a site), Swift implements an 
exponential back-off that prevents jobs from being sent to a site that 
continuously fails them. +delayBase+ is the base for that exponential back-off:
asciimath:["delay" = "delayBase"^(-"score" * 100)]
           
@CONFIGPROP maxSubmitRate <positiveInteger>

Some combinations of site and execution mechanisms may become error prone if 
jobs are submitted too fast. This option can be used to limit the submission 
rate. If set to some number +N+, Swift will submit applications at a rate of at
most +N+ per second.

@END

==== Application Declarations

Applications can either be declared globally, outside of a site declaration,
or specific to a site, inside a site declaration:

[listing,swiftconf]
------
app.(<appName>|ALL) {
    # global application
    ...
}

site.<siteName> {
    app.(<appName>|ALL) {
        # site application
        ...
    }
}
------

A special application name, +ALL+, can be used to declare options for all 
applications. When Swift attempts to run an application named +X+, it will
first look at site application declarations for +app.X+. If not found, it will
check if a site application declaration exists for +app.ALL+. The search will
continue with the global +app.X+ and then the global +all.ALL+ until a match
is found. It is possible that a specific application will only be declared
on a sub-set of all the sites and not globally. Swift will then only select
a site where the application is declared and will not attempt to run the
application on other sites.

An application declaration takes the following form:

[listing,swiftconf]
-----
app.<appName> {
    executable: (<string>|"*")
    [jobQueue: <string>]
    [jobProject: <string>]
    [maxWallTime: <time>]
    [options: {...}]
    <environment variables>
}
-----

The +executable+ is mandatory, and it points to the actual location of the
executable that implements the application. The special string +"*"+ can
be used to indicate that the executable has the same name as the application
name. This is useful in conjunction with +app.ALL+ to essentially declare
that a site can be used to execute any application from a Swift script. If the
executable is not an absolute path, it will be searched using the +PATH+ 
envirnoment variable on the remote site.

The following example illustrates how options are inherited:

[listing,swiftconf]
------
    # global app options
    app.ALL {
        options {
            # use "ProjectX" on all sites
            project: "ProjectX"
        }
    }
    
    app.myapp1 {
        options {
            # this applies to all instances of myapp1
            # unless overriden on specific sites
            count: 2
        }
    }
    
    site.s1 {
        ...
        app.ALL {
            # use a default ppn of 4 for apps on this site
            ppn: 4
        }
        
        app.myapp1 {
            # use a ppn of 2 for this specific app on this site
            ppn: 2
        }
        ...
    }
}
------

Environment variables can be defined as follows:

[listing,swiftconf]
-----
    env.<name>: <value>
-----

For example:

[listing,swiftconf]
-----
    env.LD_LIBRARY_PATH: "/home/joedoe/lib"
-----

The remaining options are:

@CONFIGPROP jobQueue <string>

If the application is executed using a mechanism that submits to a queuing 
system, this option can be used to select a specific queue for the application

@CONFIGPROP jobProject <string>

A queuing system project to associate the job with.

@CONFIGPROP maxWallTime ("<mm>" | "<hh:mm>" | "<hh:mm:ss>")

The maximum amount of time that the application will take to execute on the 
site. Most application execution mechanisms will both require and enforce this 
value by terminating the application if it exceeds the specified time. The 
default value is 10 minutes.

@END

[[configuration:general-swift-options]]
==== General Swift Options

There are a number of configuration options that modify the way that
the Swift run-time behaves. They are listed below:


@CONFIGPROP sites ('[' <site> [, <site>]* ']' | "<site>[, <site]*")

Selects, out of the set of all declared sites, a sub-set of sites to run 
applications on.

@CONFIGPROP hostName <string>

Can be used to specify a publicly reacheable DNS name or IP address for this 
machine which is generally used for Globus or Coaster callbacks. Normally this 
should be auto-detected. However, if the machine does not have a public DNS 
name, this may need to be set manually.

@CONFIGPROP TCPPortRange "<lowPort>, <highPort>"

A TCP port range can be specified to restrict the ports on which certain 
callback services are started. This is likely needed if your submit host is 
behind a firewall, in which case the firewall should be configured to allow 
incoming connections on ports in this range.

@CONFIGPROP lazyErrors <boolean>
false

Use a lazy mode to deal with errors. When set to +true+ Swift will proceed with 
the execution until no more data can be derived because of errors in dependent 
steps. If set to 'false', an error will cause the execution to immediately stop

@CONFIGPROP executionRetries <positiveInteger>
0

The number of time an application invocation will be retries if it fails until 
Swift finally gives up and declares it failed. The total number of attempts will 
be asciimath:[1 + "executionRetries"].

@CONFIGPROP logProvenance <boolean>
false

If set to +true+, Swift will record provenance information in the log file.

@CONFIGPROP alwaysTransferWrapperLog <boolean>
alwaysTransferWrapperLog: false

Controls when wrapper logs are transfered back to the submit host. If set to 
+false+, Swift will only transfer a wrapper log for a given job when that job 
fails. If set to +true+, Swift will transfer wrapper logs whether a job fails 
or not.

@CONFIGPROP fileGCEnabled <boolean>
true

Controls the file garbage collector. If set to +false+, files mapped by 
collectable mappers (such as the concurrent mapper) will not be deleted when 
their Swift variables go out of scope.

@CONFIGPROP mappingCheckerEnabled <boolean>
true

Controls the run-time duplicate mapping checker (which indetifies mapping 
conflicts). When enabled, a record of all mapped data is kept, so this comes at 
the expense of a slight memory usage increase over time. If set +false+, the 
mapping checker is disabled. 

@CONFIGPROP tracingEnabled <boolean>
false

Enables execution tracing. If set to +true+, operations within Swift such as 
iterations, invocations, assignments, and declarations, as well as data 
dependencies will be logged. This comes at a cost in performance. It is 
therefore disabled by default.

@CONFIGPROP maxForeachThreads <positiveInteger>
16384

Limits the number of concurrent iterations that each 'foreach' statement can 
have at one time. This conserves memory for swift programs that have large 
numbers of iterations (which would otherwise all be executed in parallel).

@END

==== Ticker Options

@CONFIGPROP tickerEnabled <boolean>
true

Controls the output ticker, which regularly prints information about the counts
of application states on the Swift's process standard output

@CONFIGPROP tickerPrefix <string>
"Progress: "

Specifies a string to prefix to each ticker line output

@CONFIGPROP tickerDateFormat <string>
"E, dd MMM yyyy HH:mm:ssZ"

Specifies the date/time format to use for the time stamp of each ticker line. 
It must conform to Java's 
http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html[SimpleDateFormat]
syntax.

@END

==== CDM Options

@CONFIGPROP CDMBroadcastMode <string>
"file"

@CONFIGPROP CDMFile <string>
"fs.data"

@END

==== Replication Options

@CONFIGPROP replicationEnabled <boolean>
false

If enabled, jobs that are queued longer than a certain amount of time will
have a duplicate version re-submitted. This process will continue until a
maximum pre-set number of such replicas is queued. When one of the replicas
becomes active, all other replicas are canceled. This mechanism can potentially
prevent a single overloaded site from completely blocking a run.

@CONFIGPROP replicationMinQueueTime <seconds>
60

When replication is enabled, this is the amount of time that a job needs to 
be queued until a new replica is created.

@CONFIGPROP replicationLimit <strictlyPositiveInteger>
3

The maximum number of replicas allowed for a given application instance.

@END

==== Wrapper Staging Options

@CONFIGPROP wrapperStagingLocalServer <string>
"file://"

When file staging is set to +"wrapper"+, this indicates the default URL
scheme that is prefixed to local files.

@END

[[configuration:throttling]]
==== Throttling Options

@CONFIGPROP jobSubmitThrottle (<strictlyPositiveInteger> | "off")
4

Limits the number of jobs that can concurrently be in the process of being 
submitted, that is in the "Submitting" state. This is the state where the job
information is being communicated to a remote service. Certain execution 
mechanisms may become inefficient if too many jobs are being submitted 
concurrently and there are no benefits to parallelizing	submission beyond a 
certain point. Please not that this does not apply to the number of jobs that 
can be active concurrently.

@CONFIGPROP hostJobSubmitThrottle (<strictlyPositiveInteger> | "off")
2

Like +jobSubmitThrottle+, except it applies to each individual site.

@CONFIGPROP fileTransfersThrottle (<strictlyPositiveInteger> | "off")
4

Limits the number of concurrent file transfers when file staging is set to 
+"swift"+. Arbitrarily increasing file transfer parallelism leads to little
benefits as the throughput approaches the maximum avaiable network bandwidth.
Instead it can lead to an increase in latencies which may increase the chances
of triggering timeouts.

@CONFIGPROP fileOperationsThrottle (<strictlyPositiveInteger> | "off")
8

Limits the number of concurrent file operations that can be active at a given
time when file staging is set to +"swift"+. File operations are defined to be all
remote operations on a filesystem that exclude file transfers. Examples are: 
listing the contents of a directory, creating a directory, removing a file, etc.

@END

==== Global Versions of Site Options

@CONFIGPROP staging ("swift" | "local" | "service-local" | "shared-fs" | "wrapper")
"swift"

See <<configuration:staging,staging>>.


@CONFIGPROP keepSiteDir <boolean>
false

See See <<gprop:keepSiteDir, keepSiteDir>>.

@CONFIGPROP statusMode ("files" | "provider")
"files"

See <<gprop:statusMode, statusMode>>.

@CONFIGPROP wrapperParameterMode ("args" | "files")
"args"

See <<gprop:wrapperParameterMode, wrapperParameterMode>>.

@END

[[configuration:execution-mechanisms]]
=== Execution Mechanisms

Swift allows application execution through a number of mechanisms (or execution 
providers). The choice of each mechanism is dependent on the software installed
on a certain compute cluster. The following sub-sections list the available
choices together with their supported options as well as the available app
options when using the respective execution type.

@EM Local

The *local* execution mechanism can be used to run applications locally through
simple +fork()+ calls.

General Configuration::
+
[cols="1,3"]
|==========================================================================
| URL required    | no
| Job Manager     | not used 
| Staging methods | +swift+, +wrapper+, +local+, +service-local+, +shared-fs+, +direct+
|==========================================================================
        
Options:: N/A

Application Options::
+
[options="header",cols="3, 3, 3, 10"]
|==========================================================================
| Name      | Type    | Default Value | Description
| +count+   | Integer |             1 | Launch this many copies of the application for each invocation
|==========================================================================

Example::

[listing,swiftconf]
------
    site.local {
        execution {
            type: "local"
        }
        
        staging: direct
        
        app.ALL {
            executable: "*"
            count: 1
        }
    }
------

@EM GT5

Uses the http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/#gram5[GRAM] component
of the http://toolkit.globus.org[Globus Toolkit] to launch jobs on remote resources.

General Configuration::
+
[cols="1,3"]
|==========================================================================
| URL required    | yes
| Job Manager     | In GRAM, job managers instruct the GRAM service to submit 
                    jobs to specific resource managers on the server side. The
                    exact available job managers depend on the particular GRAM
                    installation. However, +"fork"+, which instructs GRAM to 
                    run jobs directly on the service node, should always be
                    available. In addition, the available job managers would
                    typically match the queuing systems installed on the server
                    side. For example, if a cluster uses Torque/PBS, then the
                    +"PBS"+ job manager should be available. The following is
                    a list of known possible job manager values: +"fork"+, +"PBS"+,
                    +"LSF"+, +"Condor"+, +"SGE"+, +"Slurm"+
| Staging methods | +swift+, +wrapper+
|==========================================================================

Options:: N/A

Application Options::
For a complete list and description of these options, please see the 
http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/user/#gram5-user-rsl[Globus GRAM documentation]
+
[options="header",cols="3, 3, 3, 10"]
|==========================================================================
| Name            | Type              | Default Value | Description
| +count+         | Integer           |             1 | Launch this many copies of the application for each invocation
| +max_time+      | Integer (minutes) | -             |
| +max_wall_time+ | Integer (minutes) | -             |
| +max_cpu_time+  | Integer (minutes) | -             |
| +max_memory+    | Integer (MB)      | -             |
| +min_memory+    | Integer (MB)      | -             |
| +project+       | String            | -             | A LRM project to associate the job with
| +queue+         | String            | -             | LRM queue to submit to

|==========================================================================

Example::

[listing,swiftconf]
------
    site.example {
        execution {
            type: "gt5"
            url: "login.example.org"
            jobManager: "PBS"
        }
        
        staging: swift
        
        app.sim {
            executable: /usr/bin/sim
            queue: "fast"
            min_memory: 120
        }
    }
------


@EM SSH

Runs jobs through a Java implementation of the SSH protocol. This mechanism
generally results in a higher throughput than using the command-line SSH tool
since it can reduce the number of authentication operations by re-using 
connections.

General Configuration::
+
[cols="1,3"]
|==========================================================================
| URL required    | yes
| Job Manager     | not used 
| Staging methods | swift, wrapper
|==========================================================================

Options:: N/A

Application Options:: N/A

Example::

[listing,swiftconf]
------
    site.example {
        execution {
            type: "ssh"
            url: "login.example.org"
        }
    }
------


@EM SSH-CL

Uses the ssh command-line tool to run jobs.

General Configuration::
+
[cols="1,3"]
|==========================================================================
| URL required    | yes
| Job Manager     | not used 
| Staging methods | swift, wrapper
|==========================================================================

Options:: N/A

Application Options:: N/A

Example::

[listing,swiftconf]
------
    site.example {
        execution {
            type: "ssh-cl"
            url: "login.example.org"
        }
    }
------



@EM PBS

Submits jobs directly to a Torque/PBS queue.

General Configuration::
+
[cols="1,3"]
|==========================================================================
| URL required    | no
| Job Manager     | not used 
| Staging methods | swift, wrapper
|==========================================================================

Options:: N/A

Application Options::
+
[options="header",cols="3, 3, 3, 10"]
|==========================================================================
| Name            | Type              | Default Value | Description
| +count+         | Integer           |             1 | Request this number of nodes for the job
| +ppn+           | Integer           |             1 | Sets the number of Processes Per Node
| +depth+         | Integer           |             1 | Only used if +mpp+ is set to +true+. Sets the
                                                        depth (number of OpenMP threads/cores to allocate
                                                        for each process)
| +pbs.mpp+       | Boolean           | false         | If set to +true+, use the mpp versions of 
                                                        +count+, +ppn+, and +depth+: +mppwidth+, +mppnppn+,
                                                        +mppdepth+ respectively.
| +pbs.properties+| String            | -             | If specified, this string will be passed verbatim
                                                        to PBS inside the +"#PBS -l"+ line.
| +project+       | String            | -             | A PBS project to associate the job with
| +queue+         | String            | -             | PBS queue to submit to
| +pbs.resource_list+ | String        | -             | WRITEME!
| +pbs.aprun+     | Boolean           | false         | If specified, use the +aprun+ tool instead of ssh
                                                        to start jobs on the compute nodes. +aprun+ is a tool
                                                        typically found on Cray systems.
|==========================================================================

Example::

[listing,swiftconf]
----
    site.pbs {
        execution {
            type: "PBS"
        }
        
        app.sim {
        	executable: "/usr/bin/sim"
        	count: 2
        	ppn: 2
        	depth: 2
        	pbs.mpp: true
        	queue: "fast"
        }
    }
----


@EM Condor

Submits jobs using the HTCondor system.

General Configuration::
+
[cols="1,3"]
|==========================================================================
| URL required    | no
| Job Manager     | not used 
| Staging methods | swift, wrapper
|==========================================================================

Options:: N/A

Application Options::
+
[options="header",cols="3, 3, 3, 10"]
|==========================================================================
| Name            | Type              | Default Value | Description
| +jobType+       | +"MPI"+, +"grid"+, +"nonshared"+, none  |         none | Specifies the job type (Condor universe). 
                                                                             +"nonshared"+ translates to the +"vanilla"+ universe.
| +holdIsFailure+ | Boolean           |         false | Treat jobs in the held state as failed.
| +count+         | Integer           |             1 | Number of machines to request for the job
| +condor.*+      | Any               |             - | Can be used to pass arbitrary properties to Condor.
|==========================================================================

Example::

[listing,swiftconf]
------
    site.condor {
        execution {
            type: "Condor"
        }
        
        app.sim {
            executable: "/usr/bin/sim"
            condor.leave_in_queue: "TRUE"
        }
    }
------

[[configuration:coasters]]
@EM Coasters

Coasters are a mechanism that packages multiple swift application invocations into larger
LRM jobs resulting in increased efficiency when running multiple small applications. 
To distinguish between the application invocations and the jobs in which Coasters
package them, the terms *task* and *job* are used, respectively.

General Configuration::
+
[cols="1,3"]
|==========================================================================
| URL required    | maybe
| Job Manager     | +"em1:em2"+, where +em1+ is an execution mechanism used to
                    start the Coaster Service and +em2+ is an execution mechanism
                    used by the Coaster Service to start jobs. If +em1+ requires
                    an URL, then the URL is required. Options specific to +em2+
                    can be specified using +options.jobOptions+.
| Staging methods | swift, wrapper, local, service-local, shared-fs, direct
|==========================================================================

Options::
+
[[table-em-options-coaster]]
[options="header",cols="4, 2, 3, 2"]
|==========================================================================
| Name            | Type              | Default Value | Description
| +maxJobs+       | Integer           |            20 | The maximum number of jobs that can be running at a time.
| +nodeGranularity+ | Integer         |             1 | If specified, the number of nodes requested for each job
                                                        will be a multiple of this number
| +tasksPerNode+  | Integer           |             1 | The maximum number of concurrent tasks allowed to run
                                                        on a node
| +allocationStepSize+ | asciimath:[[0, 1]]   |           0.1 | The Coaster service allocates jobs periodically depending
                                                        on the number of tasks queued. This number can be used to
                                                        limit the fraction of jobs out of +maxJobs+ that will
                                                        be used in each allocation step.
| +lowOverallocation+ | asciimath:[[1, infty)] |           10 | Indicates how much bigger the job wall time should be in
                                                        comparison to the task wall time for tasks that have a 
                                                        small wall time (around 1 second)
| +highOverallocation+ | asciimath:[[1, infty)]|           1 | Indicates how much bigger the job wall time should in 
                                                        comparison to the task wall time for tasks that have a
                                                        very large wall time
| +overallocationDecayFactor+ | asciimath:[(0, infty)] | 1e-3 | Used to interpolate the "overallocation" for task wall times
                                                        that are neither very large or very small. The formula used
                                                        is asciimath:["jw" = "tw" * ((L - H) * e^(-"tw" * D) + H)],
                                                        where +jw+ is the job walltime, +tw+ is the task walltime, +L+ is +lowOverallocation+, +H+ is +highOverallocation+, and
                                                        +D+ is +overallocationDecayFactor+.
| +spread+        | asciimath:[[0, 1]]        |           0.9 | When allocating jobs, the total number of nodes to allocate
                                                        can be fixed based on, for example, maximizing parallelism
                                                        for all the tasks. However, the way the nodes are distributed
                                                        to individual jobs can be arbitrary. This parameter controls
                                                        whether nodes should be uniformly distributed among jobs 
                                                        (+spread = 0+) or if the node distribution should be as diverse
                                                        as possible (+spread = 1+). A high spread could be useful
                                                        in fitting jobs better into a cluster's schedule.
| +reserve+       | Integer (seconds) |            60 | The amount of time to add to each job's wall time in order to
                                                        prevent premature termination of tasks due to various
                                                        overheads
| +maxNodesPerJob+ | Integer          |             1 | The maximum number of nodes that a job is allowed to have.
| +maxJobTime+    | "HH:MM:SS"        |             - | The maximum wall time that a job is allowed to have
| +userHomeOverride+ | String         |             - | A path that can be used to override the default user home
                                                        directory. This may be necessary on systems on which
                                                        compute nodes do not have access to the default user home
                                                        directory.
| +internalHostName+ | String         |             - | A host name or address that can be used to initiate connections
                                                        from compute nodes to the login node. Specifying this is
                                                        seldom necessary.
| +jobQueue+      | String            |             - | The LRM queue to submit the jobs to
| +jobProject+    | String            |             - | A LRM project to associate the job with
| +jobOptions.*+  | Object            |             - | Any number of LRM options used to start the Coaster jobs.
                                                        These options correspond to the execution mechanism specified
                                                        by +em2+ in the job manager setting and are described in the
                                                        *App Options* sections of the corresponding section of 
                                                        <<configuration:execution-mechanisms, Execution Mechanisms>>.
| +workerLoggingLevel+| +"ERROR"+, +"WARN"+, +"INFO"+, +"DEBUG"+, +"TRACE"+, or none
                                      |          none | If specified, the Coaster Workers produce a log file 
| +workerLoggingDirectory+ | String   |  | The directory where the worker logs will be created.
                                                               This directory needs to be accessible from compute 
                                                               nodes and is set to "~/.globus/coasters" by default.
| +softImage+     | String            |             - | WRITEME!
|==========================================================================

Application Options:: N/A

Example::

[listing,swiftconf]
------
    site.condor {
        execution {
            type: "Condor"
        }
        
        app.sim {
            executable: "/usr/bin/sim"
            condor.leave_in_queue: "TRUE"
        }
    }
------
