The swift configuration file (swift.conf) is a description of how swift should execute your swift workflow on one or more of your target computation resources. The swift config file therefore contains three aspects:
-
A description of the target resource, and what you want from it, such as:
-
URL of the site, scheduler the system uses, what queue you want to run on, how many nodes you need, etc
-
-
A description of application that you will run on these resources:
-
Executable for each app, walltimes for apps etc
-
-
Configuration for the workflow execution behavior:
-
handling errors, network port to use, leave files for debug etc.
-
Simple canned configurations work great for a vast majority of cases. Nevertheless, Swift provides the user with several different paramenters to tune and tweak your workflow to perfection.
Here’s the skeleton of a swift.conf:
site.<name> { # How to negotiate resources with the site execution { [URL : < URL_OF_COMPUTE_RESOURCE > ] type : < local | coaster | coaster-persistent > [jobmanager : < <local|ssh-cl>:<local,pbs,sge,slurm..> >] } # Directory for placing intermediate work files workDirectory: <path> # Site specific options (eg. queue to use,nodes to request) [<site options>] # How to move data to/from the compute nodes [staging: "swift" | "local" | "service-local" | "shared-fs" | "wrapper"] # Describe the application to run on this site [<application declarations>] }
Here’s a simple config file for a cluster which uses slurm scheduler:
sites : midway site.midway { execution { type : "coaster" # Use coasters to run on remote sites URL : "swift.rcc.uchicago.edu" # Midway login node | Alternatively use midway.rcc.uchicago.edu jobManager: "ssh-cl:slurm" # Use ssh-cl to connect, slurm is the Local resource manager options { jobQueue : "sandyb" # Select queue from (sandyb, westmere, ...) } } staging : "local" # Stage files from "local" system to Midway workDirectory : "/tmp/"${env.MIDWAY_USERNAME} # Location for intermediate files maxParallelTasks : 101 # Maximum number of parallel tasks initialParallelTasks: 100 # Maximum number of tasks at start app.ALL { executable: "*" } # All tasks to be found from commandline }