AN INTRODUCTION TO USING

Todd Tannenbaum

ATFC5 - India - October 25, 2019

1
 University of Wisconsin-Madison
Center for High Throughput Computing
(CHTC)

2
 Agenda
› Today:
Introduction for users
Also useful for administrators 
› Plan for Tomorrow:
Session One (9:30am-11:00am)
• Architecture/Administration Overview (90min)
Session Two (11:30am-1:00pm)
• Open Question/Answer Session (45 min)
• Monitoring (20 min)
• What's New and What's Coming Up? (20 min)

3
Introduction

4
 What is HTCondor?
• Software that schedules and runs
computing tasks on computers

HTCONDOR

5
 How It Works
• Submit tasks to a queue (on a submit point)
• HTCondor schedules them to run on
computers (execute points)

execute
submit

execute

execute

6
Single Computer

execute execute

submit
execute

7
Multiple Computers

execute
submit

execute

execute

8
 Why HTCondor?
• HTCondor manages and runs work on your
behalf
• Schedule tasks on a single computer to not
overwhelm the computer
• Schedule tasks on a group* of computers
(which may/may not be directly accessible to
the user)
• Schedule tasks submitted by multiple users
on one or more computers

*in HTCondor-speak, a “pool”

9
 Why HTCondor, cont
• Open source software to enable distributed High
Throughput Computing (HTC)
• Full featured, mature production system (1M+
LOC)
• Widely deployed
– Used in production at hundreds of universities,
government labs, commercial companies to manage
compute clusters in science, engineering, finance, …
– Components used to federate compute clusters into
campus grids and wide-area computing grids, e.g.
Open Science Grid, WLCG, …

10
Open Science Grid

[Link]

11
 CMS Global Pool
• Dynamic cluster, ~200k - 300k cores pulled
in from sites worldwide

12
12
 Bursting into Google Cloud @ SC16

13
13
 User-Focused Tutorial
• For the purposes of this tutorial, we are
assuming that someone else has set up
HTCondor on a computer/computers to
create a HTCondor “pool”.
• The focus of this talk is an introduction on
how to get started running computational
work on this system.

14
Running a Job with
HTCondor

15
 Jobs
• A single computing task is called a “job”
• Three main pieces of a job are the input,
executable (program) and output

• Executable must be runnable from the

command line without any interactive input
16
 Job Example
• For our example, we will be using an
imaginary program called
“compare_states”, which compares two
data files and produces a single output file.

[Link]
compare_
[Link]
states

[Link]

$ compare_states [Link] [Link] [Link]

17
 File Transfer
• What about files? Can use a shared file system,
chirp, or file transfer mechanism.
• Our example will use HTCondor’s file transfer :

Submit Execute

(submit_dir)/ (execute_dir)/
input files output files
executable

18
 Job Translation
• Submit file: communicates everything
about your job(s) to HTCondor

executable = compare_states
arguments = [Link] [Link] [Link]

should_transfer_files = YES
transfer_input_files = [Link], [Link]
when_to_transfer_output = ON_EXIT

log = [Link]
output = [Link]
error = [Link]

request_cpus = 1
request_disk = 20MB
request_memory = 20MB

queue 1

19
 Submit File
[Link]
executable = compare_states
arguments = [Link] [Link] [Link]

should_transfer_files = YES
transfer_input_files = [Link], [Link]
when_to_transfer_output = ON_EXIT

log = [Link]
output = [Link]
error = [Link]

request_cpus = 1
request_disk = 20MB
request_memory = 20MB

queue 1

20
 Submit File
[Link]
• List your
executable = compare_states executable and
arguments = [Link] [Link] [Link] any arguments it
takes.
should_transfer_files = YES
transfer_input_files = [Link], [Link]
when_to_transfer_output = ON_EXIT compare_
states
log = [Link]
output = [Link]
error = [Link] • Arguments are
any options
request_cpus = 1
request_disk = 20MB
passed to the
request_memory = 20MB executable from
the command line.
queue 1 $ compare_states [Link] [Link] [Link]

21
 Submit File
[Link]
executable = compare_states • Indicate
arguments = [Link] [Link] [Link] your input
should_transfer_files = YES files.
transfer_input_files = [Link], [Link]
when_to_transfer_output = ON_EXIT

log = [Link]
output = [Link] [Link]
error = [Link]

request_cpus = 1
request_disk = 20MB
request_memory = 20MB [Link]

queue 1

22
 Submit File
[Link]
executable = compare_states • HTCondor will
arguments = [Link] [Link] [Link] transfer back
should_transfer_files = YES all new and
transfer_input_files = [Link], [Link]
when_to_transfer_output = ON_EXIT changed files
log = [Link] (usually
output = [Link]
error = [Link]
output) from
request_cpus = 1
the job.
request_disk = 20MB
request_memory = 20MB
[Link]
queue 1

23
 Submit File
[Link]
• log: file
executable = compare_states
arguments = [Link] [Link] [Link] created by
should_transfer_files = YES
HTCondor to
transfer_input_files = [Link], [Link] track job
when_to_transfer_output = ON_EXIT
progress
log = [Link] • output/err
output = [Link]
error = [Link] or: captures
request_cpus = 1 stdout and
request_disk = 20MB stderr
request_memory = 20MB

queue 1

24
 Submit File
[Link]
executable = compare_states
• Request the
arguments = [Link] [Link] [Link] appropriate
should_transfer_files = YES resources
transfer_input_files = [Link], [Link] for your job
when_to_transfer_output = ON_EXIT
to run.
log = [Link]
output = [Link] • queue:
error = [Link] keyword
request_cpus = 1 indicating
request_disk = 20MB
request_memory = 20MB
“create a
job.”
queue 1

25
 Submitting and Monitoring
• To submit a job/jobs:
condor_submit submit_file_name
• To monitor submitted jobs, use:
condor_q
$ condor_submit [Link]
Submitting job(s).
1 job(s) submitted to cluster 128.

$ condor_q
-- Schedd: [Link] : <[Link]:9618?... @ 05/01/17 10:35:54
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
alice CMD: compare_states 5/9 11:05 _ _ 1 1 128.0

1 jobs; 0 completed, 0 removed, 1 idle, 0 running, 0 held, 0 suspended

HTCondor Manual: condor_submit

HTCondor Manual: condor_q 26
 More about condor_q
• By default condor_q shows:
– user’s job only (as of 8.6)
• See everyone with "condor_q –allusers"
– jobs summarized in “batches” (as of 8.6)
• Constrain with username, ClusterId or full
JobId, which will be denoted[U/C/J] in the
following slides
$ condor_q
-- Schedd: [Link] : <[Link]:9618?... @ 05/01/17 10:35:54
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
alice CMD: compare_states 5/9 11:05 _ _ 1 1 128.0

1 jobs; 0 completed, 0 removed, 1 idle, 0 running, 0 held, 0 suspended

JobId = ClusterId .ProcId

27
 More about condor_q
• To see individual job information, use:
condor_q -nobatch
$ condor_q -nobatch
-- Schedd: [Link] : <[Link]:9618?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
128.0 alice 5/9 11:09 0+00:00:00 I 0 0.0 compare_states [Link] [Link]

1 jobs; 0 completed, 0 removed, 1 idle, 0 running, 0 held, 0 suspended

• We will use the -nobatch option in the

following slides to see extra detail about
what is happening with a job
28
 Job Idle
$ condor_q -nobatch
-- Schedd: [Link] : <[Link]:9618?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
128.0 alice 5/9 11:09 0+00:00:00 I 0 0.0 compare_states [Link] [Link]

1 jobs; 0 completed, 0 removed, 1 idle, 0 running, 0 held, 0 suspended

Submit Node
(submit_dir)/
[Link]
compare_states
[Link]
[Link]
[Link]
[Link]
[Link]

29
 Job Starts by doing File Transfer
$ condor_q -nobatch
-- Schedd: [Link] : <[Link]:9618?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
128.0 alice 5/9 11:09 0+00:00:00 < 0 0.0 compare_states [Link] [Link] w

1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended

Submit Node Execute Node

(submit_dir)/ (execute_dir)/
[Link]
compare_states compare_states
[Link] [Link]
[Link] [Link]
[Link]
[Link]
[Link]

30
 Job Running
$ condor_q -nobatch

-- Schedd: [Link] : <[Link]:9618?...

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
128.0 alice 5/9 11:09 0+00:01:08 R 0 0.0 compare_states [Link] [Link]

1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended

Submit Node Execute Node

(submit_dir)/ (execute_dir)/
[Link] compare_states
compare_states [Link]
[Link] [Link]
[Link] stderr
[Link] stdout
[Link] [Link]
[Link]

31
 Job Completes
$ condor_q -nobatch
-- Schedd: [Link] : <[Link]:9618?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
128 alice 5/9 11:09 0+00:02:02 > 0 0.0 compare_states [Link] [Link]

1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended

Submit Node Execute Node

(submit_dir)/ (execute_dir)/
[Link] compare_states
compare_states [Link]
[Link] [Link]
stderr
[Link] stderr
stdout
[Link] stdout
[Link]
[Link] [Link]
[Link]

32
 Job Completes (cont.)
$ condor_q -nobatch

-- Schedd: [Link] : <[Link]:9618?...

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD

0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended

Submit Node

(submit_dir)/
[Link]
compare_states
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]

33
 Log File
000 (128.000.000) 05/09 11:09:08 Job submitted from host:
<[Link]&sock=6423_b881_3>
...
001 (128.000.000) 05/09 11:10:46 Job executing on host:
<[Link]:9618&sock=5053_3126_3>
...
006 (128.000.000) 05/09 11:10:54 Image size of job updated: 220
1 - MemoryUsage of job (MB)
220 - ResidentSetSize of job (KB)
...
005 (128.000.000) 05/09 11:12:48 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
33 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
33 - Total Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1 1 1
Disk (KB) : 14 20480 17203728
Memory (MB) : 1 20 20
34
 Job States
transfer
transfer
executable
output
and input to
back to
execute
submit node
node

condor_ Idle Running Completed

submit (I) (R) (C)

in the queue leaving the queue

35
 Assumptions
• Aspects of your submit file may be
dictated by infrastructure and configuration
• For example: file transfer
– previous example assumed files would need
to be transferred between submit/execute
should_transfer_files = YES

– not the case with a shared file system

should_transfer_files = NO

36
 Shared file system
• If a system has a shared file system, where
file transfer is not enabled, the submit
directory and execute directory are the same.

Submit Execute

shared_dir/
input
executable
output

37
 Resource Request
• Jobs are nearly always using a part of a
computer, not the whole thing
• Very important to request appropriate
resources (memory, cpus, disk) for a job

whole
computer
your request

38
 Resource Assumptions
• Even with reasonable default CPU, memory
and disk requests, these may be too small!
• Important to run test jobs and use the log file
to request the right amount of resources:
– requesting too little: causes problems for your
and other jobs; jobs might by held by HTCondor
– requesting too much: jobs will match to fewer
“slots”

39
Job Matching and
Class Ad Attributes

40
 The Central Manager
• HTCondor matches jobs with computers
via a “central manager”.

execute
submit

central manager execute

execute

41
 Class Ads
• HTCondor stores a list of information about
each job and each computer.
• This information is stored as a “Class Ad”

• Class Ads have the format: can be a boolean,

number, string, or
AttributeName = value expression
HTCondor Manual: Appendix A: Class Ad Attributes
42
 Job Class Ad
RequestCpus = 1
Err = "[Link]"
WhenToTransferOutput = "ON_EXIT"
TargetType = "Machine"
executable = compare_states
arguments = [Link] [Link] [Link] Cmd =
should_transfer_files = YES "/home/alice/tests/htcondor_week/compar
transfer_input_files = [Link], [Link]
when_to_transfer_output = ON_EXIT e_states"
JobUniverse = 5
=
log = [Link]
output = [Link]
error = [Link] Iwd = "/home/alice/tests/htcondor_week"
request_cpus = 1 RequestDisk = 20480
request_disk = 20MB
request_memory = 20MB NumJobStarts = 0
queue 1 WantRemoteIO = true
OnExitRemove = true
TransferInput = "[Link],[Link]"
+ MyType = "Job"
HTCondor configuration Out = "[Link]"
UserLog =
"/home/alice/tests/htcondor_week/[Link]
g"
RequestMemory = 20
... 43
 Computer “Machine” Class Ad
HasFileTransfer = true
DynamicSlot = true
TotalSlotDisk = 4300218.0
TargetType = "Job"
TotalSlotMemory = 2048
Mips = 17902
Memory = 2048

= UtsnameSysname = "Linux"
MAX_PREEMPT = ( 3600 * 72 )
Requirements = ( START ) && (
IsValidCheckpointPlatform ) && (
WithinResourceLimits )
OpSysMajorVer = 6
TotalMemory = 9889
+ HasGluster = true
HTCondor configuration OpSysName = "SL"
HasDocker = true

...
44
 Job Matching
• On a regular basis, the central manager
reviews Job resource requests and Machine
Class Ads and matches jobs to computers.

execute
submit

central manager execute

execute

45
 Job Execution
• (Then the submit and execute points
communicate directly.)

execute
submit

central manager execute

execute

46
 Class Ads for People
• Class Ads also provide lots of useful
information about jobs and computers to
HTCondor users and administrators

47
 Finding Job Attributes
• Use the “long” option for condor_q
condor_q -l JobId

$ condor_q -l 128.0
WhenToTransferOutput = "ON_EXIT"
TargetType = "Machine"
Cmd = "/home/alice/tests/htcondor_week/compare_states"
JobUniverse = 5
Iwd = "/home/alice/tests/htcondor_week"
RequestDisk = 20480
NumJobStarts = 0
WantRemoteIO = true
OnExitRemove = true
TransferInput = "[Link],[Link]"
MyType = "Job”
UserLog = "/home/alice/tests/htcondor_week/[Link]"
RequestMemory = 20
...
48
 Some Useful Job Attributes
• UserLog: location of job log
• Iwd: Initial Working Directory (i.e.
submission directory) on submit node
• MemoryUsage: maximum memory the job
has used
• RemoteHost: where the job is running
• BatchName: attribute to label job batches
• ...and more
49
 Selectively display specific
attributes
• Use the “auto-format” option:
condor_q [U/C/J] -af Attribute1 Attribute2 ...

$ condor_q -af ClusterId ProcId RemoteHost MemoryUsage

17315225 116 slot1_1@[Link] 1709

17315225 118 slot1_2@[Link] 1709
17315225 137 slot1_8@[Link] 1709
17315225 139 slot1_7@[Link] 1709
18050961 0 slot1_5@[Link] 196
18050963 0 slot1_3@[Link] 269
18050964 0 slot1_25@[Link] 245
18050965 0 slot1_23@[Link] 196
18050971 0 slot1_6@[Link] 220

50
 Other Displays
• See the whole queue (all users, all jobs)
condor_q -all
$ condor_q -all

-- Schedd: [Link] : <[Link]:9618?...

OWNER BATCH_NAME SUBMITTED DONE RUN IDLE HOLD TOTAL JOB_IDS
alice DAG: 128 5/9 02:52 982 2 _ _ 1000 18888976.0 ...
bob DAG: 139 5/9 09:21 _ 1 89 _ 180 18910071.0 ...
alice DAG: 219 5/9 10:31 1 997 2 _ 1000 18911030.0 ...
bob DAG: 226 5/9 10:51 10 _ 1 _ 44 18913051.0
bob CMD: [Link] 5/9 10:55 _ _ _ 2 _ 18913029.0 ...
alice CMD: sb 5/9 10:57 _ 2 998 _ _ 18913030.0-999

51
 condor_q Reminder
• Default output is batched jobs
– Batches can be grouped manually using the
JobBatchName attribute in a submit file:
JobBatchName = “CoolJobs”

– Otherwise HTCondor groups jobs

automatically
• To see individual jobs, use:
condor_q -nobatch

52
 Class Ads for Computers
as condor_q is to jobs, condor_status is to computers (or “machines”)

$ condor_status
Name OpSys Arch State Activity LoadAv Mem Actvty
slot1@[Link] LINUX X86_64 Unclaimed Idle 0.000 673 25+01
slot1_1@[Link] LINUX X86_64 Claimed Busy 1.000 2048 0+01
slot1_2@[Link] LINUX X86_64 Claimed Busy 1.000 2048 0+01
slot1_3@[Link] LINUX X86_64 Claimed Busy 1.000 2048 0+00
slot1_4@[Link] LINUX X86_64 Claimed Busy 1.000 2048 0+14
slot1_5@[Link] LINUX X86_64 Claimed Busy 1.000 1024 0+01
slot1@[Link] LINUX X86_64 Unclaimed Idle 1.000 2693 19+19
slot1_1@[Link] LINUX X86_64 Claimed Busy 1.000 2048 0+04
slot1_2@[Link] LINUX X86_64 Claimed Busy 1.000 2048 0+01
slot1_3@[Link] LINUX X86_64 Claimed Busy 0.990 2048 0+02
slot1@[Link] LINUX X86_64 Unclaimed Idle 0.010 645 25+05
slot1_1@[Link] LINUX X86_64 Claimed Busy 1.000 2048 0+01

Total Owner Claimed Unclaimed Matched Preempting Backfill Drain

X86_64/LINUX 10962 0 10340 613 0 0 0 9

X86_64/WINDOWS 2 2 0 0 0 0 0 0

Total 10964 2 10340 613 0 0 0 9

HTCondor Manual: condor_status 53

 Machine Attributes
• Use same options as condor_q:
condor_status -l Slot/Machine
condor_status [Machine] -af Attribute1 Attribute2 ...
$ condor_status -l slot1_1@[Link]
HasFileTransfer = true
COLLECTOR_HOST_STRING = "[Link]”
TargetType = "Job”
TotalTimeClaimedBusy = [Link]
UtsnameNodename = ""
Mips = 17902
MAX_PREEMPT = ( 3600 * ( 72 - 68 * ( WantGlidein =?= true ) ) )
Requirements = ( START ) && ( IsValidCheckpointPlatform ) && (
WithinResourceLimits )
State = "Claimed"
OpSysMajorVer = 6
OpSysName = "SL”
...
54
 Machine Attributes
• To summarize, use the “-compact” option
condor_status -compact
$ condor_q -compact
Machine Platform Slots Cpus Gpus TotalGb FreCpu FreeGb CpuLoad ST
[Link] x64/SL6 8 8 23.46 0 0.00 1.24 Cb
[Link] x64/SL6 8 8 23.46 0 0.46 0.97 Cb
[Link] x64/SL6 11 16 23.46 5 0.00 0.81 **
[Link] x64/SL6 8 8 23.46 0 4.46 0.76 Cb
[Link] x64/SL6 1 12 23.45 11 13.45 0.00 **
[Link] x64/SL6 0 24 23.45 24 23.45 0.04 Ui
[Link] x64/SL6 24 80 1009.67 8 0.17 0.60 **

Total Owner Claimed Unclaimed Matched Preempting Backfill Drain

x64/SL6 10416 0 9984 427 0 0 0 5

x64/WinVista 2 2 0 0 0 0 0 0

Total 10418 2 9984 427 0 0 0 5

55
Submitting Multiple Jobs
with HTCondor

56
 Many Jobs, One Submit File
• HTCondor has built-in ways to submit
multiple independent jobs with one submit
file

57
 Advantages
• Run many independent jobs...
– analyze multiple data files
– test parameter or input combinations
– and more!
• ...without having to:
– start each job individually
– create separate submit files for each job

58
 Multiple, Numbered, Input Files
[Link]
(submit_dir)/
executable = [Link]
arguments = [Link] [Link] [Link]
transfer_input_files = [Link] [Link]
[Link]
log = [Link] [Link]
output = [Link]
error = [Link] [Link]

queue

• Goal: create 3 jobs that each analyze a

different input file.
59
 Multiple Jobs, No Variation
[Link]
(submit_dir)/
executable = [Link]
arguments = [Link] [Link] [Link]
transfer_input_files = [Link] [Link]
[Link]
log = [Link] [Link]
output = [Link]
error = [Link] [Link]

queue 3

• This file generates 3 jobs, but doesn’t use

multiple inputs and will overwrite outputs

60
 Automatic Variables

ClusterId ProcId
• Each job’s
ClusterId and
ProcId numbers
128 0
are saved as job
queue N 128 1 attributes
• They can be
128 2
accessed inside
... ... the submit file
using:
128 N-1 – $(ClusterId)
– $(ProcId)

61
 Job Variation
[Link]
(submit_dir)/
executable = [Link]
arguments = [Link] [Link] [Link]
transfer_input_files = [Link] [Link]
[Link]
log = [Link] [Link]
output = [Link]
error = [Link] [Link]

queue

• How to uniquely identify each job

(filenames, log/out/err names)?
62
 Using $(ProcId)
[Link]
executable = [Link]
arguments = file$(ProcId).in file$(ProcId).out
should_transfer_files = YES
transfer_input_files = file$(ProcId).in
when_to_transfer_output = ON_EXIT

log = job_$(ClusterId).log
output = job_$(ClusterId)_$(ProcId).out
error = job_$(ClusterId)_$(ProcId).err

queue 3

• Use the $(ClusterId), $(ProcId)

variables to provide unique values to jobs.*
* May also see $(Cluster), $(Process) in documentation 63
Organizing Jobs

64
 Shared Files
• HTCondor can transfer an entire directory
or all the contents of a directory
(submit_dir)/
– transfer whole directory [Link]
transfer_input_files = shared shared/
[Link]
[Link]
– transfer contents only [Link]
[Link]
transfer_input_files = shared/
[Link]

• Useful for jobs with many shared files;

transfer a directory of files instead of listing
files individually
65
Organize Files in Sub-Directories
• Create sub-directories* and use paths in
the submit file to separate input, error, log,
and output files.

log

* must be created before the job is submitted

66
 Use Paths for File Type
(submit_dir)/
[Link] [Link] input/ log/ err/
[Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link]
[Link] [Link] [Link]

[Link]
executable = [Link]
arguments = file$(Process).in file$(ProcId).out
transfer_input_files = input/file$(ProcId).in

log = log/job$(ProcId).log
error = err/job$(ProcId).err

queue 3

67
 InitialDir
• Change the submission directory for each job
using initialdir
• Allows the user to organize job files into
separate directories.
• Use the same name for all input/output files
• Useful for jobs with lots of output files

job0 job1 job2 job3 job4

68
 Separate Jobs with InitialDir
(submit_dir)/
[Link] job0/ job1/ job2/
[Link] [Link] [Link] [Link]
[Link] [Link] [Link]
[Link] [Link] [Link]
[Link] [Link] [Link]
[Link]
executable = [Link]
initialdir = job$(ProcId)
arguments = [Link] [Link] Executable should be
transfer_input_files = [Link] in the directory with
the submit file, *not*
log = [Link] in the individual job
error = [Link] directories

queue 3

69
 Other Submission Methods
• What if your input files/directories aren’t
numbered from 0 - (N-1)?
• There are other ways to submit many jobs!

70
 Submitting Multiple Jobs
executable = compare_states Replacing
arguments = [Link] [Link] [Link]
single job
transfer_input_files = [Link], [Link]
inputs
queue 1

executable = compare_states with a

arguments = $(infile) [Link] $(infile).out
variable of
transfer_input_files = [Link], $(infile)
choice
queue ...

71
 Possible Queue Statements
matching ... queue infile matching *.dat
pattern
in ... list queue infile in ([Link] [Link] [Link])

from ... file

queue infile from state_list.txt
[Link]
[Link]
[Link]
state_list.txt

72
 Queue Statement Comparison
matching .. Natural nested looping, minimal programming, use optional
pattern “files” and “dirs” keywords to only match files or directories
Requires good naming conventions,
in .. list Supports multiple variables, all information contained in a single
file, reproducible
Harder to automate submit file creation
from .. file Supports multiple variables, highly modular (easy to use one
submit file for many job batches), reproducible
Additional file needed

73
 Using Multiple Variables
• Both the “from” and “in” syntax support
using multiple variables from a list.
[Link] job_list.txt
executable = compare_states [Link], 2010
arguments = -year $(option) –input $(file) [Link], 2015
[Link], 2010
should_transfer_files = YES [Link], 2015
when_to_transfer_output = ON_EXIT [Link], 2010
transfer_input_files = $(file) [Link], 2015

queue file,option from job_list.txt

HTCondor Manual: submit file options

74
 Other Features
• Match only files or directories:
queue input matching files *.dat
queue directory matching dirs job*

• Submit multiple jobs with same input data

queue 10 input matching files *.dat

– Use other automatic variables: $(Step)

arguments = -i $(input) -rep $(Step)
queue 10 input matching files *.dat

75
Testing and Troubleshooting

76
 What Can Go Wrong?
• Jobs can go wrong “internally”:
– "job failed to run…"
– something happens after the executable
begins to run
• Jobs can go wrong from HTCondor’s
perspective:
– A job can’t be started at all ("failed to launch"),
– Uses too much memory,
– Has a badly formatted executable,
– And more...
77
 Reviewing Failed Jobs
• A job’s log, output and error files can provide
valuable information for troubleshooting

Log Output Error

• When jobs were Any “print” or Captured by the

submitted, “display” information operating system
started, and from your program
stopped
• Resources used
• Exit status
• Where job ran
• Interruption
reasons

78
 Reviewing Jobs
• To review a large group of jobs at once,
use condor_history
As condor_q is to the present, condor_history is to the past

$ condor_history alice
ID OWNER SUBMITTED RUN_TIME ST COMPLETED CMD
189.1012 alice 5/11 09:52 0+00:07:37 C 5/11 16:00 /home/alice
189.1002 alice 5/11 09:52 0+00:08:03 C 5/11 16:00 /home/alice
189.1081 alice 5/11 09:52 0+00:03:16 C 5/11 16:00 /home/alice
189.944 alice 5/11 09:52 0+00:11:15 C 5/11 16:00 /home/alice
189.659 alice 5/11 09:52 0+00:26:56 C 5/11 16:00 /home/alice
189.653 alice 5/11 09:52 0+00:27:07 C 5/11 16:00 /home/alice
189.1040 alice 5/11 09:52 0+00:05:15 C 5/11 15:59 /home/alice
189.1003 alice 5/11 09:52 0+00:07:38 C 5/11 15:59 /home/alice
189.962 alice 5/11 09:52 0+00:09:36 C 5/11 15:59 /home/alice
189.961 alice 5/11 09:52 0+00:09:43 C 5/11 15:59 /home/alice
189.898 alice 5/11 09:52 0+00:13:47 C 5/11 15:59 /home/alice

HTCondor Manual: condor_history

79
 “Live” Troubleshooting
• To log in to a job where it is running, use:
condor_ssh_to_job JobId
$ condor_ssh_to_job 128.0
Welcome to slot1_31@[Link]!
Your condor job is running with pid(s) 3954839.

HTCondor Manual: condor_ssh_to_job 80

 Held Jobs
• HTCondor will put your job on hold if there’s
something YOU need to fix.
• A job that goes on hold is interrupted and
kept from running again, but remains
submitted in the
queue in the
“H” state.

81
 Diagnosing Holds
• If HTCondor puts a job on hold, it provides
a hold reason, which can be viewed with:
condor_q -hold [ -wide]
$ condor_q -hold -af HoldReason
Error from slot1_1@[Link]: Job has gone over
memory limit of 2048 megabytes.
Error from slot1_20@[Link]: SHADOW at
[Link] failed to send file(s) to <[Link]:35110>: error
reading from /home/alice/[Link]: (errno 2) No such file or directory;
STARTER failed to receive file(s) from <[Link]:9618>
Error from slot1_11@[Link]: STARTER
at [Link] failed to send file(s) to <[Link]:9618>; SHADOW at
[Link] failed to write to file /home/alice/Test_18925319_16.err:
(errno 122) Disk quota exceeded
Error from slot1_38@[Link]: Failed
to execute '/var/lib/condor/execute/slot1/dir_2471876/condor_exec.exe' with
arguments 2: (errno=2: 'No such file or directory')

82
 Common Hold Reasons
• Job has used more memory than
requested
• Incorrect path to files that need to be
transferred
• Badly formatted bash scripts (have
Windows instead of Unix line endings)
• Submit directory is over quota
• The admin has put your job on hold

83
 Fixing Holds
• Job attributes can be edited while jobs are
in the queue using:
condor_qedit [U/C/J] Attribute Value
$ condor_qedit 128.0 RequestMemory 3072
Set attribute ”RequestMemory".

• If a job has been fixed and can run again,

release it with:
condor_release [U/C/J]
$ condor_release 128.0
Job 18933774.0 released
HTCondor Manual: condor_qedit
HTCondor Manual: condor_release
84
 Holding or Removing Jobs
• If you know your job has a problem and it
hasn’t yet completed, you can:
– Place it on hold yourself, with condor_hold [U/C/J]
$ condor_hold bob
All jobs of user ”bob" have been held

$ condor_hold 128
All jobs in cluster 128 have been held

$ condor_hold 128.0
Job 128.0 held

– Remove it from the queue, using condor_rm [U/C/J]

HTCondor Manual: condor_hold
HTCondor Manual: condor_rm 85
 Job States, Revisited

condor_ Idle Running Completed

submit (I) (R) (C)

in the queue leaving the queue

86
 Job States, Revisited

condor_ Idle Running Completed

submit (I) (R) (C)

condor_hold, or
HTCondor puts
condor_release a job on hold

Held
(H)

in the queue leaving the queue

87
 Job States, Revisited*

condor_ Idle Running Completed

submit (I) (R) (C)

condor_hold,
or job error
condor_release condor_rm Removed
(X)

Held
(H)

in the queue leaving the queue

*not comprehensive
88
Some Use Cases and
Mechanisms

89
 Interactive Jobs
• An interactive job proceeds like a normal
batch job, but opens a bash session into the
job’s execution directory instead of running
an executable.
condor_submit -i submit_file
$ condor_submit -i [Link]
Submitting job(s).
1 job(s) submitted to cluster 18980881.
Waiting for job to start...
Welcome to slot1_9@[Link]!

• Useful for testing and troubleshooting

90
 Output Handling
• Only transfer back specific files from the
job’s execution using transfer_ouput_files
transfer_output_files = [Link]

(submit_dir)/ (execute_dir)/
condor_exec.exe
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]

91
 condor_chirp
• What if you want to only read part of a file?
• What if you want to write records into an
output file?

Use condor_chirp !
[Link]

Can also edit job classad or add entries to

the job event log file!

92
 Self-Checkpointing
• By default, a job that is interrupted will
start from the beginning if it is restarted.
• It is possible to implement self-
checkpointing, which will allow a job to
restart from a saved state if interrupted.
• Self-checkpointing is useful for very long
jobs, and being able to run on
opportunistic resources.

93
 Self-Checkpointing How-To
• Edit executable:
– Atomically save intermediate states to a
checkpoint file
– Always check for a checkpoint file when starting
• Add HTCondor option that a) saves all
intermediate/output files from the interrupted
job and b) transfers them to the job when
HTCondor runs it again
when_to_transfer_output = ON_EXIT_OR_EVICT
# Optional: also checkpoint if my job exits with
# a specified exit status
CheckpointExitCode = 77
94
 Job Universes
• HTCondor has different “universes” for
running specialized job types
HTCondor Manual: Choosing an HTCondor Universe

• Vanilla (default)
– good for most software
HTCondor Manual: Vanilla Universe

• Set in the submit

file using:
universe = vanilla

95
 Other Universes
• Local
– Run jobs on the submit node
• Java
– Built-in Java support
– Executable is a jar file
• Grid
– Delegate jobs to another
scheduler
– The basis for HTCondor-CE

96
 Other Universes (cont.)
• Docker
– Run jobs inside a Docker container
• VM
– Run jobs inside a virtual machine
• Parallel
– Used for coordinating jobs across multiple
servers (e.g. MPI code)
– Not necessary for single server multi-core
jobs
97
 Multi-CPU and GPU Computing
• Jobs that use multiple cores on a single
computer can be run in the vanilla universe
(parallel universe not needed):
request_cpus = 16

• If there are computers with GPUs, request

them with:
request_gpus = 1

98
 Docker Universe
universe = docker
executable = /bin/my_executable

Executable comes either from submit machine or image

NOT FROM execute machine

99
 Docker Universe
universe = docker
executable = /bin/my_executable
docker_image = deb7_and_HEP_stack

Image is the name of the docker image stored on

execute machine

100
 Docker Universe
universe = docker
executable = /bin/my_executable
docker_image = deb7_and_HEP_stack
transfer_input_files = some_input
HTCondor can transfer input files from
submit machine into container

(same with output in reverse)

101
 Docker Universe
universe = docker
executable = /bin/my_executable
arguments = arg1
docker_image = deb7_and_HEP_stack
transfer_input_files = some_input
output = out
error = err
log = log
queue

102
Automation

103
 Automation
• After job submission, HTCondor manages
jobs based on its configuration
• You can use options that will customize job
management even further
• These options can
automate when
jobs are started,
stopped, and removed.
104
 Retries
• Problem: a small number of jobs fail with a
known error code; if they run again, they
complete successfully.
• Solution: If the job exits with the error
code, leave it in the queue to run again

max_retries = 3

105
 Retries, cont.
• Can also combine with
success_exit_code = < Integer >
retry_until = < Integer | Expression >

executable = [Link]
max_retries = 5
retry_untl = ExitCode >= 0
queue

106
 Automatically Hold Jobs
• Problem: Your job should run in 2 hours or
less, but a few jobs “hang” randomly and
run for days
• Solution: Put jobs on hold if they run for
over 2 hours, using a periodic_hold
statement job is running

periodic_hold = (JobStatus == 2) &&

((CurrentTime - EnteredCurrentStatus) > (60 * 60 * 2))

How long the job has been 2 hours

running, in seconds

107
 Automatically Release Jobs
• Problem (related to previous): A few jobs
are being held for running long; they will
complete if they run again.
• Solution: automatically release those held
jobs with a periodic_release option,
up to 5 times job is held

periodic_release = (JobStatus == 5) &&

(HoldReason == 3) && (NumJobStarts < 5)

job was put on hold job has started running

by periodic_hold less than 5 times
108
 Automatically Remove Jobs
• Problem: Jobs are repetitively failing
• Solution: Remove jobs from the queue
using a periodic_remove statement
periodic_remove = (NumJobsStarts > 5)

job has started running

more than 5 times

109
 Automatic Memory Increase
• Putting all these pieces together, the
following lines will:
– request a default amount of memory (2GB)
– put the job on hold if it is exceeded
– release the the job with an increased memory request

request_memory = ifthenelse(isUndefined(MemoryUsage),
2048,(MemoryUsage * 3/2), 2048)
periodic_hold = (MemoryUsage >= ((RequestMemory) * 5/4 )) &&
(JobStatus == 2)
periodic_release = (CurrentTime - EnteredCurrentStatus) > 180) &&
(NumJobStarts < 5) && (HoldReasonCode == 3)

110
 Relevant Job Attributes
• CurrentTime: current time
• EnteredCurrentStatus: time of last status
change
• ExitCode: the exit code from the job
• HoldReasonCode: number corresponding to a
hold reason
• NumJobStarts: how many times the job has
gone from idle to running
• JobStatus: number indicating idle, running, held,
etc.
• MemoryUsage: how much memory the job has
used
HTCondor Manual: Appendix A: JobStatus and HoldReason Codes
111
 General User Commands
• condor_submit Submit new Jobs
• condor_status View Pool Status
• condor_q View Job Queue
• condor_q -analyze Why job/machines fail to match?
• condor_ssh_to_job Create ssh session to active job
• condor_submit -i Submit interactive job
• condor_hold / release Hold a job, or release a held job
• condor_run Submit and block
• condor_rm Remove Jobs
• condor_prio Intra-User Job Prios
• condor_history Completed Job Info
• condor_submit_dag Submit new DAG workflow
• condor_chirp Access files/ad from active job

112
Describing Workflows with
DAGMan

113
 Workflows
• Problem: Want to submit download
jobs in a particular order,
with dependencies split

between groups of jobs

• Solution: Write a DAG 1 2 3 ... N

combine

114
DAG = ”directed acyclic graph”
• topological ordering of
vertices (“nodes”) is
established by directional
connections (“edges”)
• “acyclic” aspect requires
a start and end, with no
looped repetition
– can contain cyclic
subcomponents, covered
in later slides for Wikimedia Commons

workflows

[Link]/wiki/Directed_acyclic_graph 115
DAGMan in the HTCondor
Manual

116
Simple Example for this Tutorial
• The DAG input file will A
communicate the
“nodes” and directional
“edges” of the DAG
B1 B2 B3 ... BN

HTCondor Manual: DAGMan Applications > DAG Input File 117

 Basic DAG input file:
JOB nodes, PARENT-CHILD edges
[Link] A
JOB A [Link]
JOB B1 [Link]
JOB B2 [Link]
JOB B3 [Link] B1 B2 B3 ... BN
JOB C [Link]
PARENT A CHILD B1 B2 B3
PARENT B1 B2 B3 CHILD C

• Node names are used by various C

DAG features to modify their
execution by DAG Manager.
HTCondor Manual: DAGMan Applications > DAG Input File 118
 Endless Workflow Possibilities

Wikimedia Commons

[Link] 119
Endless Workflow Possibilities

[Link] 120
Submitting and Monitoring
a DAGMan Workflow

121
 Basic DAG input file:
JOB nodes, PARENT-CHILD edges
[Link] A
JOB A [Link]
JOB B1 [Link]
JOB B2 [Link]
JOB B3 [Link] B1 B2 B3 ... BN
JOB C [Link]
PARENT A CHILD B1 B2 B3
PARENT B1 B2 B3 CHILD C

HTCondor Manual: DAGMan Applications > DAG Input File 122

 Submitting a DAG to the queue
• Submission command:
condor_submit_dag dag_file

$ condor_submit_dag [Link]
------------------------------------------------------------------
File for submitting this DAG to HTCondor : [Link]
Log of DAGMan debugging messages : [Link]
Log of HTCondor library output : [Link]
Log of HTCondor library error messages : [Link]
Log of the life of condor_dagman itself : [Link]

Submitting job(s).
1 job(s) submitted to cluster 87274940.
------------------------------------------------------------------

HTCondor Manual: DAGMan > DAG Submission 123

 Jobs are automatically
submitted by the DAGMan job
• Seconds later, node A is submitted:
$ condor_q
-- Schedd: [Link] : <[Link]:9618?...
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
alice [Link]+128 4/30 18:08 _ _ 1 5 129.0
2 jobs; 0 completed, 0 removed, 1 idle, 1 running, 0 held, 0 suspended

$ condor_q -nobatch
-- Schedd: [Link] : <[Link]:9618?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
128.0 alice 4/30 18:08 0+00:00:36 R 0 0.3 condor_dagman
129.0 alice 4/30 18:08 0+00:00:00 I 0 0.3 A_split.sh
2 jobs; 0 completed, 0 removed, 1 idle, 1 running, 0 held, 0 suspended

HTCondor Manual: DAGMan > DAG Submission 124

 Jobs are automatically
submitted by the DAGMan job
• After A completes, B1-3 are submitted
$ condor_q
-- Schedd: [Link] : <[Link]:9618?...
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
alice [Link]+128 4/30 8:08 1 _ 3 5 129.0...132.0
4 jobs; 0 completed, 0 removed, 3 idle, 1 running, 0 held, 0 suspended

$ condor_q -nobatch
-- Schedd: [Link] : <[Link]:9618?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
128.0 alice 4/30 18:08 0+00:20:36 R 0 0.3 condor_dagman
130.0 alice 4/30 18:18 0+00:00:00 I 0 0.3 B_run.sh
131.0 alice 4/30 18:18 0+00:00:00 I 0 0.3 B_run.sh
132.0 alice 4/30 18:18 0+00:00:00 I 0 0.3 B_run.sh
4 jobs; 0 completed, 0 removed, 3 idle, 1 running, 0 held, 0 suspended

HTCondor Manual: DAGMan > DAG Submission 125

 Jobs are automatically
submitted by the DAGMan job
• After B1-3 complete, node C is submitted
$ condor_q
-- Schedd: [Link] : <[Link]:9618?...
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
alice [Link]+128 4/30 8:08 4 _ 1 5 129.0...133.0
2 jobs; 0 completed, 0 removed, 1 idle, 1 running, 0 held, 0 suspended

$ condor_q -nobatch
-- Schedd: [Link] : <[Link]:9618?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
128.0 alice 4/30 18:08 0+00:46:36 R 0 0.3 condor_dagman
133.0 alice 4/30 18:54 0+00:00:00 I 0 0.3 C_combine.sh
2 jobs; 0 completed, 0 removed, 1 idle, 1 running, 0 held, 0 suspended

HTCondor Manual: DAGMan > DAG Submission 126

 Status files are Created at the
time of DAG submission
(dag_dir)/
[Link] [Link] [Link]
[Link] [Link] (other job files)
[Link] [Link] [Link]
[Link] [Link] [Link]
[Link]

*.[Link] and *.[Link] describe the queued

DAGMan job process, as for all queued jobs
*.[Link] has detailed logging (look to first for errors)
*.[Link]/out contain std err/out for the DAGMan job
process
*.[Link] is a combined log of all jobs within the DAG
DAGMan > DAG Monitoring and DAG Removal 127
 Removing a DAG from the
queue
• Remove the DAGMan job in order to stop and remove
the entire DAG:
condor_rm dagman_jobID
• Creates a rescue file so that only incomplete or
unsuccessful NODES are repeated upon resubmission

$ condor_q
-- Schedd: [Link] : <[Link]:9618?...
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS
alice [Link]+128 4/30 8:08 4 _ 1 6 129.0...133.0
2 jobs; 0 completed, 0 removed, 1 idle, 1 running, 0 held, 0 suspended
$ condor_rm 128
All jobs in cluster 128 have been marked for removal

DAGMan > DAG Monitoring and DAG Removal

DAGMan > The Rescue DAG 128
 Removal of a DAG results in a
rescue file
(dag_dir)/
[Link] [Link] [Link] [Link] [Link] (other job files)
[Link] [Link] [Link]
[Link] [Link] [Link]
[Link] [Link] [Link].rescue001

• Named dag_file.rescue001
• increments if more rescue DAG files are created
• Records which NODES have completed
successfully
• does not contain the actual DAG structure
DAGMan > DAG Monitoring and DAG Removal
DAGMan > The Rescue DAG 129
 Rescue Files For Resuming a
Failed DAG
• A rescue file is created when:
– a node fails, and after DAGMan advances
through any other possible nodes
– the DAG is removed from the queue
(or aborted; covered later)
– the DAG is halted and not unhalted
(covered later)
• Resubmission uses the rescue file (if it exists)
when the original DAG file is resubmitted
– override: condor_submit_dag dag_file -f
DAGMan > The Rescue DAG 130
 Node Failures Result in DAG
Failure
• If a node JOB fails
(non-zero exit code) A
– DAGMan continues to
run other JOB nodes
until it can no longer
make progress B1 B2 B3 ... BN
• Example at right:
– B2 fails
– Other B* jobs continue
– DAG fails and exits after C
B* and before node C
DAGMan > The Rescue DAG 131
 Resolving held node jobs
$ condor_q -nobatch
-- Schedd: [Link] : <[Link]:9618?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
128.0 alice 4/30 18:08 0+00:20:36 R 0 0.3 condor_dagman
130.0 alice 4/30 18:18 0+00:00:00 H 0 0.3 B_run.sh
131.0 alice 4/30 18:18 0+00:00:00 H 0 0.3 B_run.sh
132.0 alice 4/30 18:18 0+00:00:00 H 0 0.3 B_run.sh
4 jobs; 0 completed, 0 removed, 0 idle, 1 running, 3 held, 0 suspended

• Look at the hold reason (in the job log, or with

‘condor_q -hold’)
• Fix the issue and release the jobs (condor_release)
-OR- remove the entire DAG, resolve, then resubmit
the DAG

HTCondor Manual: DAGMan > DAG Submission 132

 DAG Completion
(dag_dir)/
[Link] [Link] [Link]
[Link] [Link] (other job files)
[Link] [Link] [Link]
[Link] [Link] [Link]
[Link] [Link]

*.[Link] is a summary of events and outcomes

*.[Link] will note the completion of the DAGMan job
*.[Link] has detailed logging (look to first for errors)

DAGMan > DAG Monitoring and DAG Removal 133

 Beyond the Basic DAG:
Some Node-level Modifiers

134
 PRE and POST scripts run on the
submit server, as part of the node
[Link]
JOB A [Link] A
SCRIPT POST A [Link] POST script
JOB B1 [Link]
JOB B2 [Link]
JOB B3 [Link]
JOB C [Link] B1 B2 B3 ... BN
SCRIPT PRE C tar_it.sh
PARENT A CHILD B1 B2 B3
PARENT B1 B2 B3 CHILD C

• Use sparingly for lightweight work;

PRE script
otherwise include work in node jobs C
HTCondor Manual: DAGMan Applications > DAG Input File 135
 RETRY failed nodes to overcome
transient errors
• Retry a node up to N times if the exit code is
non-zero:
RETRY node_name N
Example: JOB A [Link]
RETRY A 5
JOB B [Link]
PARENT A CHILD B

• See also: retry except for a particular exit code

(UNLESS-EXIT), or retry scripts (DEFER)
• Note: Unnecessary for nodes (jobs) that can use
max_retries in the submit file
DAGMan Applications > Advanced Features > Retrying
DAGMan Applications > DAG Input File > SCRIPT 136
 RETRY applies to whole node,
including PRE/POST scripts
• PRE and POST scripts are included in retries
• RETRY of a node with a POST script uses the
exit code from the POST script (not from the job)
– POST script can do more to determine node success,
perhaps by examining JOB output

Example: SCRIPT PRE A [Link]

JOB A [Link]
SCRIPT POST A [Link]
RETRY A 5

DAGMan Applications > Advanced Features > Retrying

DAGMan Applications > DAG Input File > SCRIPT 137
 SCRIPT Arguments and Argument
Variables
JOB A [Link]
SCRIPT POST A [Link] [Link] $RETURN
RETRY A 5

$JOB: node name

$JOBID: [Link]
$RETURN: exit code of the job
$PRE_SCRIPT_RETURN: exit code of PRE script
$RETRY: current retry count
(more variables described in the manual)
DAGMan Applications > DAG Input File > SCRIPT
DAGMan Applications > Advanced Features > Retrying 138
 Modular Organization and
Control of DAG Components
• Splices and SubDags
• Node Throttling
• Node Priorities
• Lots more in the Manual…

139
 Additional Resources
• HTCondor-Users Email List!
[Link]

• Manual (and man pages)

[Link]

• Nice HTCondor FAQs, examples, and documentation from

our friends in Canary Islands:
[Link]

• HTCondor HOWTO Recipes has FAQ on job submission

[Link]

140
THANK YOU!

141
ADDITIONAL DAGMAN SLIDES

142
Submit File Templates via VARS
• VARS line defines node-specific values that are
passed into submit file variables
VARS node_name var1=“value” [var2=“value”]
• Allows a single submit file shared by all B jobs, rather
than one submit file for each JOB.
[Link] [Link]
JOB B1 [Link] …
VARS B1 data=”B1” opt=“10” InitialDir = $(data)
JOB B2 [Link] arguments = $(data).csv $(opt)
VARS B2 data=“B2” opt=“12” …
JOB B3 [Link] queue
VARS B3 data=“B3” opt=“14”

DAGMan Applications > Advanced Features > Variable Values 143

SPLICE groups of nodes to simplify
lengthy DAG files
[Link]
JOB A [Link]
A
SPLICE B [Link]
JOB C [Link]
PARENT A CHILD B
PARENT B CHILD C
B1 B2 B3 ... BN
[Link]
JOB B1 [Link]
JOB B2 [Link]
…
JOB BN [Link] C

DAGMan Applications > Advanced Features > DAG Splicing 144

 Use nested SPLICEs with DIR for
repeating workflow components
[Link]
JOB A [Link] DIR A A
SPLICE B [Link] DIR B
JOB C [Link] DIR C
PARENT A CHILD B
PARENT B CHILD C
[Link] B1 B2 BN
[Link]
SPLICE B1 ../[Link] DIR B1 1 1 1
SPLICE B2 ../[Link] DIR B2 ...
…
SPLICE BN ../[Link] DIR BN 2 2 2
[Link]
JOB 1 ../[Link]
JOB 2 ../[Link]
PARENT 1 CHILD 2 C
DAGMan Applications > Advanced Features > DAG Splicing 145
 Use nested SPLICEs with DIR for
repeating workflow components
[Link]
JOB A [Link] DIR A
(dag_dir)/
SPLICE B [Link] DIR B [Link]
JOB C [Link] DIR C A/ [Link] (A job files)
PARENT A CHILD B B/ [Link] [Link]
PARENT B CHILD C [Link] [Link]
B1/ (1-2 job files)
[Link] B2/ (1-2 job files)
SPLICE B1 ../[Link] DIR B1 …
SPLICE B2 ../[Link] DIR B2 BN/ (1-2 job files)
… C/ [Link] (C job files)
SPLICE BN ../[Link] DIR BN

[Link]
JOB 1 ../[Link]
JOB 2 ../[Link]
PARENT 1 CHILD 2

DAGMan Applications > Advanced Features > DAG Splicing 146

 More on SPLICE Behavior
• Upon submission of the outer DAG, nodes in the
SPLICE(s) are added by DAGMan into the overall DAG
structure.
– A single DAGMan job is queued with single set of status files.
• Great for gradually testing and building up a large DAG
(since a SPLICE file can be submitted by itself, as a
complete DAG).
• SPLICE lines are not treated like nodes.
– no PRE/POST scripts or RETRIES (though this may change)

DAGMan Applications > Advanced Features > DAG Splicing 147

What if some DAG components
can’t be known at submit time?
A
If N can only
be determined
as part of the
B1 B2 B3 ... BN work of A …

148
 A SUBDAG within a DAG
[Link]
JOB A [Link]
A
SUBDAG EXTERNAL B [Link]
JOB C [Link]
PARENT A CHILD B
PARENT B CHILD C
B1 B2 B3 ... BN
[Link] (written by A)
JOB B1 [Link]
JOB B2 [Link]
…
JOB BN [Link] C

DAGMan Applications > Advanced Features > DAG Within a DAG 149
 More on SUBDAG Behavior
• WARNING: SUBDAGs should only be used (over
SPLICES) when absolutely necessary!
– Each SUBDAG EXTERNAL has it’s own DAGMan job
running in the queue.
• SUBDAGs are nodes (can have PRE/POST scripts,
retries, etc.)
• A SUBDAG is not submitted until prior nodes in
the outer DAG have completed.

DAGMan Applications > Advanced Features > DAG Within a DAG 150
 Use a SUBDAG to achieve Cyclic
Components within a DAG
• POST script determines whether another iteration is A
necessary; if so, exits non-zero
• RETRY applies to entire SUBDAG, which may include
multiple, sequential nodes

[Link]
JOB A [Link]
SUBDAG EXTERNAL B [Link]
B
SCRIPT POST B [Link]
POST script
RETRY B 1000
RETRY
JOB C [Link]
PARENT A CHILD B
PARENT B CHILD C
C
DAGMan Applications > Advanced Features > DAG Within a DAG 151
DAG-level Control

152
 Pause a running DAG with
hold/release
• Hold the DAGMan job process:
condor_hold dagman_jobID
• Pauses the DAG
– No new node jobs submitted
– Queued node jobs continue to run (including
SUBDAGs), but no PRE/POST scripts
– DAGMan jobs remains in the queue until
released (condor_release) or removed

DAGMan > Suspending a Running DAG 153

 Pause a DAG with a halt file
• Create a file named DAG_file.halt in the
same directory as the submitted DAG file
• Pauses the DAG
– No new node jobs submitted
– Queued node jobs, SUBDAGs, and POST scripts
continue to run, but not PRE scripts
• DAGMan resumes after the file is deleted
– If not deleted, the DAG creates rescue DAG file
and exits after all queued jobs have completed

DAGMan > Suspending a Running DAG

DAGMan > The Rescue DAG 154
Throttle job nodes of large DAGs
via DAG-level configuration
• If a DAG has many (thousands or more)
jobs, performance of the submit server
and queue can be assured by limiting:
– Number of jobs in the queue
– Number of jobs idle (waiting to run)
– Number of PRE or POST scripts running
• Limits can be specified in a DAG-specific
CONFIG file (recommended) or as
arguments to condor_submit_dag

DAGMan > Advanced Features > Configuration Specific to a DAG 155

 DAG-specific throttling via a
CONFIG file
[Link]
JOB A [Link] A
SPLICE B [Link]
JOB C [Link]
PARENT A CHILD B
PARENT B CHILD C
CONFIG [Link] B1 B2 B3 ... BN
[Link]
DAGMAN_MAX_JOBS_SUBMITTED = 1000
DAGMAN_MAX_JOBS_IDLE = 100
DAGMAN_MAX_PRE_SCRIPTS = 4 C
DAGMAN_MAX_POST_SCRIPTS = 4

DAGMan > Advanced Features > Configuration Specific to a DAG 156

Other DAGMan Features

157
 Other DAGMan Features:
Node-Level Controls
• Set the PRIORITY of JOB nodes with:
PRIORITY node_name priority_value

• Use a PRE_SKIP to skip a node and mark it as

successful, if the PRE script exits with a specific
exit code:
PRE_SKIP node_name exit_code

DAGMan Applications > Advanced Features > Setting Priorities

DAGMan Applications > The DAG Input File > PRE_SKIP 158
 Other DAGMan Features:
Modular Control
• Append NOOP to a JOB definition so that its JOB
process isn’t run by DAGMan
– Test DAG structure without running jobs (node-level)
– Simplify combinatorial PARENT-CHILD statements (modular)

• Communicate DAG features separately with INCLUDE

– e.g. separate file for JOB nodes and for VARS definitions, as part of the
same DAG

• Define a CATEGORY to throttle only a specific subset

of jobs
DAGMan Applications > The DAG Input File > JOB
DAGMan Applications > Advanced Features > INCLUDE
DAGMan Applications > Advanced > Throttling by Category 159
 Other DAGMan Features:
DAG-Level Controls
• Replace the node_name with ALL_NODES to apply
a DAG feature to all nodes of the DAG

• Abort the entire DAG if a specific node exits with a

specific exit code:
ABORT-DAG-ON node_name exit_code

• Define a FINAL node that will always run, even in

the event of DAG failure (to clean up, perhaps).
FINAL node_name submit_file
DAGMan Applications > Advanced > ALL_NODES
DAGMan Applications > Advanced > Stopping the Entire DAG
DAGMan Applications > Advanced > FINAL Node 160