Task Farming
What is task farming?
Task farming is a technique that allows concurrent execution of multiple serial (single processor) tasks on parallel computers. ICHEC provides a utility called taskfarm to do this. As ICHEC policies do not facilitate individual serial jobs users must use the taskfarm to run multiple serial processes in a single job.
How to access the taskfarm
The taskfarm utility is not in your default PATH but can be loaded when required using an environment module. To use taskfarm in a batch job, you can add the following line to load the module within the job submission script itself.
module load taskfarm
What is a task?
In this context a task is a single command or a group of consecutively executed commands. An example of a group of commands as a task:
cd dir01; ../my_executable input_file > output_file
This simple task changes to a specific subdirectory and runs an executable from the parent directory with an input file. The output from the executable is redirected to a file. It's important that all the components of the tasks are separated by semicolons and are all on the one line. A more complex tasks involving multiple different steps can be created. A more complex example involving extra commands:
cd dir01; pwd; date; ../my_executable input_file > output_file; ../my_postprocessor output_file > processed_output_file; date
A taskfarm example
This is an example of using taskfarm running on 40 cores on Kay. For the purpose of the example we will assume that we have a directory that contains the following:
- a Slurm script to run the taskfarm
- the taskfarm input file
- the executable(s) being run in the tasks
- a subdirectory for each task containing its input file (task01, task02, ... )
First the Slurm script (called job.sh in this example). It runs a job on 40 cores, as found on a single node on Kay, for up to one hour. For more information on batch processing see here.
#!/bin/sh #SBATCH -p ProdQ #SBATCH -N 1 #SBATCH -t 1:00:00 # Charge job to myaccount #SBATCH -A myaccount # Write stdout+stderr to file #SBATCH -o output.txt # Mail me on job start & end #SBATCH --mail-user=myemail@universityname.ie #SBATCH --mail-type=BEGIN,END # This job's working directory cd $SLURM_SUBMIT_DIR module load taskfarm taskfarm tasks
Next the taskfarm input file (called tasks in this example). Here 50 tasks are included. When the taskfarm first starts, the first 40 tasks will be started; then each time a task exits, the next available task will be started. As Kay has 40 cores per node, Taskfarm will initially start 40 tasks.
cd task01; ../my_executable input_file > output_file cd task02; ../my_executable input_file > output_file cd task03; ../my_executable input_file > output_file cd task04; ../my_executable input_file > output_file cd task05; ../my_executable input_file > output_file cd task06; ../my_executable input_file > output_file cd task07; ../my_executable input_file > output_file cd task08; ../my_executable input_file > output_file cd task09; ../my_executable input_file > output_file cd task10; ../my_executable input_file > output_file cd task11; ../my_executable input_file > output_file cd task12; ../my_executable input_file > output_file cd task13; ../my_executable input_file > output_file cd task14; ../my_executable input_file > output_file cd task15; ../my_executable input_file > output_file cd task16; ../my_executable input_file > output_file cd task17; ../my_executable input_file > output_file cd task18; ../my_executable input_file > output_file cd task19; ../my_executable input_file > output_file cd task20; ../my_executable input_file > output_file cd task21; ../my_executable input_file > output_file cd task22; ../my_executable input_file > output_file cd task23; ../my_executable input_file > output_file cd task24; ../my_executable input_file > output_file cd task25; ../my_executable input_file > output_file cd task26; ../my_executable input_file > output_file cd task27; ../my_executable input_file > output_file cd task28; ../my_executable input_file > output_file cd task29; ../my_executable input_file > output_file cd task30; ../my_executable input_file > output_file cd task31; ../my_executable input_file > output_file cd task32; ../my_executable input_file > output_file cd task33; ../my_executable input_file > output_file cd task34; ../my_executable input_file > output_file cd task35; ../my_executable input_file > output_file cd task36; ../my_executable input_file > output_file cd task37; ../my_executable input_file > output_file cd task38; ../my_executable input_file > output_file cd task39; ../my_executable input_file > output_file cd task40; ../my_executable input_file > output_file cd task41; ../my_executable input_file > output_file cd task42; ../my_executable input_file > output_file cd task43; ../my_executable input_file > output_file cd task44; ../my_executable input_file > output_file cd task45; ../my_executable input_file > output_file cd task46; ../my_executable input_file > output_file cd task47; ../my_executable input_file > output_file cd task48; ../my_executable input_file > output_file cd task49; ../my_executable input_file > output_file cd task50; ../my_executable input_file > output_file
Finally submit the job.
sbatch job.sh
Additional taskfarm options
With taskfarm the following options are available:
- TASKFARM_PPN, if this environment variable is set to an integer value between 1 and 40 on Kay the taskfarm utility will limit itself to use that number of processors on each node. This can be very useful if you need to limit the number of tasks that can run on a node because of memory requirements. By default this is set to 40 (the number of CPU cores on a node) implicitly on Kay.
- TASKFARM_SILENT, if this environment variable is set then the taskfarm utility will not produce output, however tasks may still produce their own output.
- %TASKFARM_TASKNUM%, if the proceeding string is used in a task line the utility will replace it with the number of the task, starting from zero for the first task. This can be used to create files or directories that are named based on the task number for instance.
- TASKFARM_SMT, if this environment variable is set then the taskfarm utility will allocate as many tasks per core as the number of threads it can support in SMT mode. This option can be useful if tasks are heavy on IO but light on computation and thus each core can sustain multiple threads.
Efficiency considerations
- The minimum number of tasks specified should be the number of CPUs requested for the batch job.
- The tasks (or series of sub-tasks) to be executed within the task farm should in most cases have no dependencies on each other.
- It is important to be careful that the tasks are reasonably well balanced in terms of run-times. If one task has a substantially longer runtime it may stay running long after the others have completed, leading to idle cores and waste of resources. Users who have tasks with highly variable runtimes are best advised to package a large number of tasks as input and run on fewer nodes, i.e. keep all the compute cores busy for as long as possible within the job. For example, instead of running 400 tasks in a 10-node job (400 cores), use 2-4 nodes instead and request additional walltime.
