ALCF Swift/Turbine Tutorial =========================== Introduction ------------- This document describes a technique to run ordinary C codes at large-scale on ALCF machines using Swift/T scripts. The scheme of integrating the C code into Swift/T described here is called _main-function wrapping_. It enables C main() programs to be defined as app() functions callable by Swift/T. Arguments to these wrapped main functions are passed as an array of strings (argv). This approach enables C programs (and in fact, programs in almost any compiled language) to be called as a function rather than executed via a POSIX fork()/exec() invocation. This permits apps to be called from Swift/T on the Blue Gene/Q (whose compute node kernels do not support fork/exec), and is more efficient than fork/exec even on systems that do support that invocation mechanism. A package that demonstrates this technique can be obtained from `svn`: ---- svn co https://svn.mcs.anl.gov/repos/exm/apps/main-wrap main-wrap cd main-wrap ---- This directory contains an example application and utilities to generate the required wrapped components (stubs and object files) from C source codes. Quickstart ---------- A synopsis of the steps we'll be describing here is: ---- ./gendata 100 5 ./genleaf vesta-gcc mockdock.[ch] user.swift ./run-cobalt.sh vesta-gcc 32 cat work/output.txt ---- Definitions ----------- Swift:: The Swift language Swift/T:: The new, high-performance version of Swift that runs under MPI Turbine:: The run time system for Swift/T (hence */T*) Leaf function:: An external foreign language function or application that is called from Swift Sample application: mockdock ---------------------------- This package contains an example docking program called `mockdock.c` and associated header file called `mockdock.h`. The program accepts two files as input (representing a protein file and peptide file), and returns a number. These files are read by mockdock, but their contents is ignored - they can contain anything. The program prints a single integer on standard out, based on the length of the input files. The code is as follows: [source,C] ---- include::mockdock.c[] ---- Sample data for the above program can be generated using the provided `gendata` script: ---- ./gendata ---- In the above commandline, ++ is the number of generated peptide files, ++ is the number of lines in a single peptide file. The generated peptide files will be in a directory called +peptides+ and the generated protein file will be +prot.txt+. For instance, to generate +1000+ peptide files with 50 lines in each file, run: ---- ./gendata 1000 50 ---- Generate and compile the wrapped-main() leaf function ------------------------------------------------------ The process of generating leaf function involves the following steps, which are automated by the script +genleaf+ for simple apps: . Create a copy of the user code and replace the main function with a Swift-accessible entry point, called +leaf_main+. . Add this +leaf_main+ to the user's header file. . Create a Tcl stub that enables the leaf function to be called by the Swift runtime. . Add the stub definition to the user's Swift script source. . Compile C source to build object code and integrate it into a single shared object library using the C compiler. The +genleaf+ script takes a machine-type and the user's C and header files as input and produces a Swift script called +user-code.swift+. The machine type specifies the BG/Q system name and the compiler to use: (vesta|mira)-(gcc|xlc). To run +genleaf+ on a typical ("vanilla") Linux system: ---- ./genleaf -v vanilla ---- To run on Vesta with gcc: ---- ./genleaf -v vesta-gcc ---- To run on Mira with xlc: ---- ./genleaf -v mira-xlc ---- For example, on Mira, to generate the required shared objects and Swift code for the +mockdock+ example: ---- ./genleaf -v mira-gcc mockdock.c mockdock.h mockdock.swift ---- This will generate multiple files, including the modified +user-code.swift+, some Tcl code, object (+.o+) and a shared-object file (+.so+) . In this tutorial example, the +user.swift+ script is: ---- include::user.swift[] ---- Run the application ------------------- On a vanilla Linux system, run: ---- turbine user-code.tcl ---- For Vesta or Mira, this command will submit a Cobalt job to run the generated user-code.swift script: ---- ./run-cobalt.sh vesta-gcc ---- ...where the machine type is the same as was specified for genleaf - (vesta|mira)-(gcc|xlc), and +PROCS+ is the number of processes. (+PROCS+ must be at least 512 on Mira.) The output appears in +./work/output.txt+ . Appendix: Further options ------------------------- Custom configuration can be set for Swift/Turbine or BG/Q via a configuration file. In our example, this file is named +cf+ . It sets environment variables such as queues, project names, and Turbine settings: ---- export MODE=BGQ export WALLTIME=00:10:00 export PROJECT=ExM export PPN=16 # Processes per node export QUEUE=default export ADLB_PRINT_TIME=1 export TURBINE_LOG=0 export TURBINE_DEBUG=0 export TURBINE_ENGINES=1 export ADLB_SERVERS=1 ---- To run the generated Turbine object code on ALCF machines, +run-cobalt.sh+ adds Turbine to you +PATH+. To set this and related properties in your environment, do: ---- $TURBINE_SCRIPTS/turbine-cobalt-run.zsh -n -s ./cf user-code.tcl ---- ...where ++ is the number of MPI processes. This will submit a +nprocs/16 node+ Cobalt job. Outputs will be sent to the directory designated by the +TURBINE_OUTPUT+ environment variable. The standard output and error from the Swift run will be located in +$TURBINE_OUTPUT/output.txt+. More detailed documentation can be found http://www.mcs.anl.gov/exm/local/guides/swift.html[here]. Appendix: Benchmark results on Mira ----------------------------------- Using the +genleaf+ technique with the +mockdock+ example as described above, a benchmark study was done on Mira for up to 1 million wrapped-main application invocations. Each app task invocation ("Tasks" column) ran for 30 seconds of real time. The number of MPI processes (on process per CPU core) was set to process 16 _waves_ of 30-second tasks. The number of load-balancing and task-distribution servers in each run ("Servers" column) was incremented as the number of tasks increased. Performance results of this study are shown in Table 1: .Performance figures [options="header,footer"] |====================================== |Tasks|Cores (Nodes) |Servers| Run time (sec) |256,000 |16,032 (1002) | 32 | 492.150 |512,000 |32,064 (2004) | 64 | 501.608 |768,000 |48,128 (3008) | 128 | 495.333 |1,000,000|62,756 (3923) | 256 | 556.021 |======================================