Swift on Diverse Infrastructures --------------------------------- Beagle ~~~~~~ Swift is available on Beagle as a software module. In this recipe, we will walk you through a very simple example running Swift on Beagle. The prerequisites required for running this example on Beagle is an access to Beagle and a project allocation. The example is a simplest form of a real application that involves reading from a disc file and writing onto another disc file. It runs the +/bin/cat+ Unix command to read an input file called +data.txt+ and write it to an output file in a folder +outdir+, called +f.nnn.out+. image::figures/catsn.png[Swift Example] The steps are as follows: *step 1.* Load the Swift module on Beagle commandline as follows: +$ module load swift+ *step 2.* Create and change to a directory where your Swift related work (source, inputs, outputs) will stay. (say, +$ mkdir swift-lab+, followed by, +$ cd swift-lab+) *step 3.* copy the folder at +/home/ketan/labs/catsn+ to the above directory. (+$cp -r /home/ketan/catsn .+ followed by +$ cd catsn+). *step 4.* In the sites file: +sites.xml+ make the following two changes: 1. change the path of the +$$$$+ element to your preferred location (this location is *required* to be on /lustre, say to +/lustre/beagle//swift-lab/swift.workdir+), and 2. Change the project name to your project (for instance, +CI-CCR000013+) . The workdirectory will contain execution data related to each run, e.g. wrapper scripts, system information, inputs and outputs. *step 5.* Run the example using following commandline (also found in run.sh): ---- $ swift -config cf -tc.file tc -sites.file sites.xml catsn.swift -n=1 ---- You can further change the value of +-n+ to any arbitrary number to run that many number of concurrent +cat+ *step 6.* Check the output in the generated +outdir+ directory (+ls outdir+) Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes will get fast prioritized execution. Good for small tests Swift supports a coasters based computing environment for Beagle. A detailed version of Swift documentation is maintained http://www.ci.uchicago.edu/swift/docs/index.php[here]. // //Intrepid-BG/P //~~~~~~~~~~~~~ //Swift on Intrepid-BG/P // PADS ~~~~ Swift on PADS To execute your Swift script on the PADS cluster use this command: ---- swift -tc.file tc -sites.file pbs.xml modis.swift ---- where the contents of a simple pbs.xml sites file could be: [xml] source~~~~ fast 00:05:00 10000 .10 /home/you/swiftwork source~~~~ OSG ~~~ This section describes how to get Swift running on the OSG Grid. We will use a manual coaster setup to get Swift running on OSG. .Coaster setup on OSG The following figure shows an abstract scheme for the manual coasters setup on OSG. image::figures/coaster_setup.png[Coaster setup] In the following steps, we will go through the process of manually setting //Swift on OSG // Bionimbus ~~~~~~~~~ This section explains a step by step procedure on getting Swift running on the Bionimbus cloud. We will use the _manual_ _coasters_ configuration on the Bionimbus cloud. **step1.** Connect to the gateway (ssh gatewayx.lac.uic.edu) **step2.** Start a virtual machine (euca-run-instances -n 1 -t m1.small emi-17EB1170) **step3.** Start the coaster-service on gateway +coaster-service -port 1984 -localport 35753 -nosec+ **step4.** Start the Swift-script from the gateway using normal Swift commandline +swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999+ .cf ---- wrapperlog.always.transfer=true sitedir.keep=true execution.retries=1 lazy.errors=true status.mode=provider use.provider.staging=true provider.staging.pin.swiftfiles=false foreach.max.threads=100 provenance.log=true ---- .tc ---- localhost modftdock /home/ketan/swift-labs/bionimbus-coaster-modftdock/app/modftdock.sh null null GLOBUS::maxwalltime="1:00:00" ---- (See below for a sample sites.xml for this run) **step5.** Connect back to gateway from virtual machines using reverse ssh tunneling as follows: .From the gateway prompt +ssh -R *:5000:localhost:5000 root@10.101.8.50 sleep 999+ WHERE: *=network interface, should remain the same on all cases localhost=the gateway host, should remain the same 5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL vary depending upon which port you want to listen to 5000(RIGHT OF localhost)=the port on target host that you want to forward root@10.101.8.50=the ip of the Virtual Machine on Bionimbus cloud, this will vary based on what ip you get for your Virtual Machine instance #On anywhere as long as provide the correct callback uri: here the "http://140.221.9.110:42195" is the callback uri of previous ones **step6.** Start the worker from the virtual machine +worker.pl http://localhost:42195 tmp /tmp # where 42195 is the port where the coaster service is listening to the workers+ .sites.xml for the above run ---- passive 4 10000 100 100 100 1 10 25.00 10000 proxy /home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir ---- //Magellan //~~~~~~~~ //Swift on Magellan //