Executing app procedures ------------------------ This section describes how Swift executes app procedures, and requirements on the behaviour of application programs used in app procedures. These requirements are primarily to ensure that the Swift can run your application in different places and with the various fault tolerance mechanisms in place. Mapping of app semantics into unix process execution semantics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This section describes how an app procedure invocation is translated into a (remote) unix process execution. It does not describe the mechanisms by which Swift performs that translation; that is described in the next section. In this section, this example Swift script is used for reference: ---- type file; app (file o) count(file i) { wc @i stdout=@o; } file q <"input.txt">; file r <"output.txt">; ---- The executable for wc will be looked up in tc.data. This unix executable will then be executed in some application procedure workspace. This means: Each application procedure workspace will have an application workspace directory. (TODO: can collapse terms application procedure workspace and application workspace directory ? This application workspace directory will not be shared with any other application procedure execution attempt; all application procedure execution attempts will run with distinct application procedure workspaces. (for the avoidance of doubt: If a Swift script procedure invocation is subject to multiple application procedure execution attempts (due to Swift-level restarts, retries or replication) then each of those application procedure execution attempts will be made in a different application procedure workspace. ) The application workspace directory will be a directory on a POSIX filesystem accessible throughout the application execution by the application executable. Before the application executable is executed: * The application workspace directory will exist. * The input files will exist inside the application workspace directory (but not necessarily as direct children; there may be subdirectories within the application workspace directory). * The input files will be those files mapped to input parameters of the application procedure invocation. (In the example, this means that the file input.txt will exist in the application workspace directory) * For each input file dataset, it will be the case that @filename or @filenames invoked with that dataset as a parameter will return the path relative to the application workspace directory for the file(s) that are associated with that dataset. (In the example, that means that @i will evaluate to the path input.txt) * For each file-bound parameter of the Swift procedure invocation, the associated files (determined by data type?) will always exist. * The input files must be treated as read only files. This may or may not be enforced by unix file system permissions. They may or may not be copies of the source file (conversely, they may be links to the actual source file). During/after the application executable execution, the following must be true: * If the application executable execution was successful (in the opinion of the application executable), then the application executable should exit with unix return code 0; if the application executable execution was unsuccessful (in the opinion of the application executable), then the application executable should exit with unix return code not equal to 0. * Each file mapped from an output parameter of the Swift script procedure call must exist. Files will be mapped in the same way as for input files. * The output subdirectories will be precreated before execution by Swift if defined within a Swift script such as the location attribute of a mapper. App executables expect to make them if they are referred to in the wrapper scripts. * Output produced by running the application executable on some inputs should be the same no matter how many times, when or where that application executable is run. 'The same' can vary depending on application (for example, in an application it might be acceptable for a PNG->JPEG conversion to produce different, similar looking, output jpegs depending on the environment) Things to not assume: * Anything about the path of the application workspace directory * That either the application workspace directory will be deleted or will continue to exist or will remain unmodified after execution has finished * That files can be passed between application procedure invocations through any mechanism except through files known to Swift through the mapping mechanism (there is some exception here for external datasets - there are a separate set of assertions that hold for external datasets) * That application executables will run on any particular site of those available, or than any combination of applications will run on the same or different sites. How Swift implements the site execution model ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This section describes the implementation of the semantics described in the previous section. Swift executes application procedures on one or more sites. Each site consists of: * worker nodes. There is some execution mechanism through which the Swift client side executable can execute its wrapper script on those worker nodes. This is commonly GRAM or Falkon or coasters. * a site-shared file system. This site shared filesystem is accessible through some file transfer mechanism from the Swift client side executable. This is commonly GridFTP or coasters. This site shared filesystem is also accessible through the posix file system on all worker nodes, mounted at the same location as seen through the file transfer mechanism. Swift is configured with the location of some site working directory on that site-shared file system. There is no assumption that the site shared file system for one site is accessible from another site. For each workflow run, on each site that is used by that run, a run directory is created in the site working directory, by the Swift client side. In that run directory are placed several subdirectories: * shared/ - site shared files cache * kickstart/ - when kickstart is used, kickstart record files for each job that has generated a kickstart record. * info/ - wrapper script log files * status/ - job status files * jobs/ - application workspace directories (optionally placed here - see below) Application execution looks like this: For each application procedure call: The Swift client side selects a site; copies the input files for that procedure call to the site shared file cache if they are not already in the cache, using the file transfer mechanism; and then invokes the wrapper script on that site using the execution mechanism. The wrapper script creates the application workspace directory; places the input files for that job into the application workspace directory using either cp or ln -s (depending on a configuration option); executes the application unix executable; copies output files from the application workspace directory to the site shared directory using cp; creates a status file under the status/ directory; and exits, returning control to the Swift client side. Logs created during the execution of the wrapper script are stored under the info/ directory. The Swift client side then checks for the presence of and deletes a status file indicating success; and copies files from the site shared directory to the appropriate client side location. The job directory is created (in the default mode) under the jobs/ directory. However, it can be created under an arbitrary other path, which allows it to be created on a different file system (such as a worker node local file system in the case that the worker node has a local file system). image:swift-site-model.png[] Technical overview of the Swift architecture -------------------------------------------- This section attempts to provide a technical overview of the Swift architecture. Execution layer ~~~~~~~~~~~~~~~ The execution layer causes an application program (in the form of a unix executable) to be executed either locally or remotely. The two main choices are local unix execution and execution through GRAM. Other options are available, and user provided code can also be plugged in. The kickstart utility can be used to capture environmental information at execution time to aid in debugging and provenance capture. Swift script language compilation layer ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Step i: text to XML intermediate form parser/processor. parser written in ANTLR - see resources/VDL.g. The XML Schema Definition (XSD) for the intermediate language is in resources/XDTM.xsd. Step ii: XML intermediate form to Karajan workflow. Karajan.java - reads the XML intermediate form. compiles to karajan workflow language - for example, expressions are converted from Swift script syntax into Karajan syntax, and function invocations become karajan function invocations with various modifications to parameters to accomodate return parameters and dataset handling. Swift/karajan library layer ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Some Swift functionality is provided in the form of Karajan libraries that are used at runtime by the Karajan workflows that the Swift compiler generates. Ways in which Swift can be extended ----------------------------------- Swift is extensible in a number of ways. It is possible to add mappers to accomodate different filesystem arrangements, site selectors to change how Swift decides where to run each job, and job submission interfaces to submit jobs through different mechanisms. A number of mappers are provided as part of the Swift release and documented in the mappers section. New mappers can be implemented in Java by implementing the org.griphyn.vdl.mapping.Mapper interface. The Swift tutorial contains a simple example of this. Swift provides a default site selector, the Adaptive Scheduler. New site selectors can be plugged in by implementing the org.globus.cog.karajan.scheduler.Scheduler interface and modifying libexec/scheduler.xml and etc/karajan.properties to refer to the new scheduler. Execution providers and filesystem providers, which allow to Swift to execute jobs and to stage files in and out through mechanisms such as GRAM and GridFTP can be implemented as Java CoG kit providers. Function reference ------------------ This section details functions that are available for use in the Swift scripting language. @arg ~~~~ Takes a command line parameter name as a string parameter and an optional default value and returns the value of that string parameter from the command line. If no default value is specified and the command line parameter is missing, an error is generated. If a default value is specified and the command line parameter is missing, @arg will return the default value. Command line parameters recognized by @arg begin with exactly one hyphen and need to be positioned after the script name. For example: ---- trace(@arg("myparam")); trace(@arg("optionalparam", "defaultvalue")); ---- ---- $ swift arg.swift -myparam=hello Swift v0.3-dev r1674 (modified locally) RunID: 20080220-1548-ylc4pmda Swift trace: defaultvalue Swift trace: hello ---- @extractInt ~~~~~~~~~~~ @extractInt(file) will read the specified file, parse an integer from the file contents and return that integer. @extractFloat ~~~~~~~~~~~~~ Similar to @extractInt, @extractFloat(file) will read the specified file, parse a float from the file contents and return that float. @filename ~~~~~~~~~ @filename(v) will return a string containing the filename(s) for the file(s) mapped to the variable v. When more than one filename is returned, the filenames will be space separated inside a single string return value. @filenames ~~~~~~~~~~ @filenames(v) will return multiple values (!) containing the filename(s) for the file(s) mapped to the variable v. (compare to @filename) @regexp ~~~~~~~ @regexp(input,pattern,replacement) will apply regular expression substitution using the Java java.util.regexp API . For example: ---- string v = @regexp("abcdefghi", "c(def)g","monkey"); ---- will assign the value "abmonkeyhi" to the variable v. @sprintf ~~~~~~~~ @sprintf(spec, variable list) will generate a string based on the specified format. ----- Example: string s = @sprintf("\t%s\n", "hello"); ----- Format specifiers [width="100%",frame="topbot"] |====================== |%%| % sign |%M| Filename output (waits for close) |%p| Format variable according to an internal format |%b| Boolean output |%f| Float output |%i| int output |%s| String output |%k| Variable sKipped, no output |%q| Array output |====================== @strcat ~~~~~~~ @strcat(a,b,c,d,...) will return a string containing all of the strings passed as parameters joined into a single string. There may be any number of parameters. The + operator concatenates two strings: @strcat(a,b) is the same as a + b @strcut ~~~~~~~ @strcut(input,pattern) will match the regular expression in the pattern parameter against the supplied input string and return the section that matches the first matching parenthesised group. For example: ---- string t = "my name is John and i like puppies."; string name = @strcut(t, "my name is ([^ ]*) "); string out = @strcat("Your name is ",name); trace(out); ---- This will output the message: Your name is John. @strjoin ~~~~~~~~ @strjoin(array, delimiter) will combine the elements of an array into a single string separated by a given delimiter. The array passed to @strjoin must be of a primitive type (string, int, float, or boolean). It will not join the contents of an array of files. Example: ---- string test[] = ["this", "is", "a", "test" ]; string mystring = @strjoin(test, " "); tracef("%s\n", mystring); ---- This will print the string "this is a test". @strsplit ~~~~~~~~~ @strsplit(input,pattern) will split the input string based on separators that match the given pattern and return a string array. Example: ---- string t = "my name is John and i like puppies."; string words[] = @strsplit(t, "\\s"); foreach word in words { trace(word); } ---- This will output one word of the sentence on each line (though not necessarily in order, due to the fact that foreach iterations execute in parallel). @toInt ~~~~~~ @toInt(input) will parse its input string into an integer. This can be used with @arg to pass input parameters to a Swift script as integers. @toFloat ~~~~~~~~ @toFloat(input) will parse its input string into a floating point number. This can be used with @arg to pass input parameters to a Swift script as floating point numbers. @toString ~~~~~~~~~ @toString(input) will parse its input into a string. Input can be an int, float, string, or boolean. @length ~~~~~~ @length(array) will return the length of an array in Swift. This function will wait for all elements in the array to be written before returning the length. @java ~~~~~~ @java(class_name, static_method, method_arg) will call a java static method of the class class_name. Built-in procedure reference ---------------------------- This section details built-in procedures that are available for use in the Swift scripting language. readData ~~~~~~~~ readData will read data from a specified file and assign it to Swift variable. The format of the input file is controlled by the type of the return value. For scalar return types, such as int, the specified file should contain a single value of that type. For arrays of scalars, the specified file should contain one value per line. For complex types of scalars, the file should contain two rows. The first row should be structure member names separated by whitespace. The second row should be the corresponding values for each structure member, separated by whitespace, in the same order as the header row. For arrays of structs, the file should contain a heading row listing structure member names separated by whitespace. There should be one row for each element of the array, with structure member elements listed in the same order as the header row and separated by whitespace. The following example shows how readData() can be used to populate an array of Swift struct-like complex type: ---- type Employee{ string name; int id; string loc; } Employee emps[] = readData("emps.txt"); ---- Where the contents of the "emps.txt" file are: ---- name id loc Thomas 2222 Chicago Gina 3333 Boston Anne 4444 Houston ---- This will result in the array "emps" with 3 members. This can be processed within a Swift script using the foreach construct as follows: ---- foreach emp in emps{ tracef("Employee %s lives in %s and has id %d", emp.name, emp.loc, emp.id); } ---- readStructured ~~~~~~~~~~~~~~ readStructured will read data from a specified file, like readdata, but using a different file format more closely related to that used by the ext mapper. Input files should list, one per line, a path into a Swift structure, and the value for that position in the structure: ---- rows[0].columns[0] = 0 rows[0].columns[1] = 2 rows[0].columns[2] = 4 rows[1].columns[0] = 1 rows[1].columns[1] = 3 rows[1].columns[2] = 5 ---- which can be read into a structure defined like this: ---- type vector { int columns[]; } type matrix { vector rows[]; } matrix m; m = readStructured("readStructured.in"); ---- (since Swift 0.7, was readData2(deprecated)) trace ~~~~~ trace will log its parameters. By default these will appear on both stdout and in the run log file. Some formatting occurs to produce the log message. The particular output format should not be relied upon. tracef ~~~~~~ +tracef(_spec_, _variable list_)+ will log its parameters as formatted by the formatter _spec_. _spec_ must be a string. Checks the type of the specifiers arguments against the variable list and allows for certain escape characters. Example: ---- int i = 3; tracef("%s: %i\n", "the value is", i); ---- Specifiers: +%s+:: Format a string. +%b+:: Format a boolean. +%i+:: Format a number as an integer. +%f+:: Format a number as a floating point number. +%q+:: Format an array. +%M+:: Format a mapped variable's filename. +%k+:: Wait for the given variable but do not format it. +%p+:: Format variable according to an internal format. Escape sequences: +\n+:: Produce a newline. +\t+:: Produce a tab. Known issues: :: Swift does not correctly scan certain backslash sequences such as +\\+. writeData ~~~~~~~~~ writeData will write out data structures in the format described for readData. The following example demonstrates how one can write a string "foo" into a file "writeDataPrimitive.out": ---- include::../../tests/language-behaviour/IO/writeDataPrimitive.swift[] ----