Controlling Xi-Batch Job Flow with Conditions and Assignments

Xi-BatchXi-Batchjob-controlschedulingvariablesbtjchangebtr

A condition is a test on a variable that must hold before a job may start; an assignment changes a variable when the job reaches a given point in its life. This article covers how to write both, what happens when a condition never holds, and how to change them on a job that already exists. What a variable is, how it is created and how it is permissioned are covered in Working with Variables in Xi-Batch.

Condition syntax

A job may carry up to 10 conditions, and all of them must hold at the same moment.

[host:]variable_name operator value

No spaces are allowed anywhere inside it. The parser reads the variable name, then expects the operator immediately, then takes everything remaining as the value. A space before the operator is rejected outright - btr stops with "Bad condition" and the offending argument - and a space after it becomes the first character of the value. Leading whitespace, before the variable name, is discarded.

# Correct
btr -c 'backup_status=Complete' post-backup.sh

# Rejected - btr: Bad condition 'backup_status = Complete'
btr -c 'backup_status = Complete' post-backup.sh

Operators:

• = or == equal to
• != not equal to
• \< less than
• \<= less than or equal
• > greater than
• >= greater than or equal

The left-hand side must be a variable name, optionally prefixed with a host name and a colon. It may be up to 19 characters and must begin with a letter or an underscore.

The right-hand side must be a literal value. There is no way to compare two variables with each other, and no expression, command or environment reference is accepted. A value that begins with a digit or a minus sign is taken as a whole number; anything else is text, up to 49 characters. Prefix the value with a colon to force a digit-leading value to be treated as text.

# Compare against the text "0012", not the number 12
btr -c 'partno=:0012' job.sh

The value in the condition decides which comparison is used. If the condition's value is a number, the variable is read as a number and a text variable counts as zero. If the condition's value is text, the variable is converted to text and the two are compared character by character - so 10 sorts before 9. Write numeric conditions against variables you keep numeric.

Simple examples

# Wait for a backup to finish
btr -c 'backup_status=Complete' post-backup.sh

# Wait for a counter to reach a threshold
btr -c 'processed_count>=100' summary.sh

# Wait for a status to be anything but Pending
btr -c 'status!=Pending' process.sh

# Numeric comparison
btr -c 'error_count<5' continue.sh

A value containing spaces is legal; it is the space before the operator that is rejected:

btr -c 'extract_status=Not Started' job.sh

Several conditions on one job

Up to 10 conditions, all of which must hold. There is no OR, and there is no grouping.

btr -c 'backup_status=Complete' \
    -c 'validation_status=Passed' \
    -c 'error_count=0' \
    final-report.sh

An eleventh condition is rejected as the command line is read, with "Sorry, you cannot have that many conditions". A ninth assignment is rejected in the same way.

A condition that never holds leaves the job waiting

The scheduler tests a job's conditions each time it reconsiders the queue, immediately before it would start the job. If any condition does not hold, the job is passed over and the scheduler moves on to the next one. Nothing is written to the job log, nothing is written to the scheduler's report file, the job's status does not change, and the job's next run time is not moved.

Two consequences follow:

• A job whose condition can never become true waits indefinitely. Nothing times it out and nothing reports it. A job waiting on a misspelt variable name, or on a value that nothing ever sets, sits in the queue looking exactly like a job that is merely waiting its turn.
• A condition on a variable that does not exist, or that the job's owner cannot read, holds the job for every operator - including != and \<. Such a variable is unknown, and unknown blocks. Deleting a variable a job depends on, or removing the owner's read permission on it, stops the job.

Finding out why a job is waiting. Print the job's conditions and then read each variable:

btjlist -H -F '%N %H %C'
btvar backup_status

If the variable name in the listing is one you do not recognise, or reading it gives "Variable does not exist", that is the answer. The article Diagnosing Why a Ready Xi-Batch Job Will Not Start covers the full set of tests the scheduler applies, of which conditions are one.

Conditions on another host's variable

A condition may name a variable on another host by prefixing it with the host name and a colon. The variable must be exported by the host that owns it.

btr -c 'server1:backup_status=Complete' local-job.sh

btr -c 'server1:data_ready=Yes' \
    -c 'server2:data_ready=Yes' \
    merge-data.sh

Criticality decides what happens when that host goes away

Criticality applies only to conditions that name another host. The flag is accepted on a condition without a host prefix and then discarded, so marking a local condition critical does nothing.

Non-critical is the default. When the owning host goes away, its variables on this machine are discarded or reduced to unreadable placeholders, and a non-critical condition on such a placeholder is skipped entirely. It ceases to exist for as long as the host is down, and the job runs. Nothing is logged when this happens, so a job that ran without its precondition is indistinguishable afterwards from one that ran with it.

Critical holds the job until the variable comes back, indefinitely and with no message - no timeout, no report-file entry, no change of status.

Choose deliberately. Mark a condition critical with -k before it, and back to non-critical with -K:

# Critical: hold the job if server1 is unreachable
btr -k -c 'server1:backup_status=Complete' post-backup.sh

# Explicitly non-critical
btr -K -c 'server1:optional_check=OK' process.sh

-k and -K apply to every -c that follows them on the command line, so a mixture is written by switching between them.

Use critical for:

• Dependencies the job must not run without
• Any condition whose whole purpose is to protect the job

Use non-critical for:

• Optional checks that should not stop work when a peer is down
• Monitoring and reporting jobs

In btq, on the conditions screen, c toggles the critical mark for remote conditions, and the assignments screen uses the same key.

btjlist does not print the critical mark. Neither the condition format nor the assignment format carries it, so btq's condition and assignment screens are the only way to see which way an existing job is set.

Assignment syntax

An assignment changes a variable's value when the job reaches a given point in its life. A job may carry up to 8 assignments in total, across all stages.

An assignment is given with -s, and its syntax is the same shape as a condition and has the same whitespace rule:

[host:]variable_name operator value

Assignment operators:

=
Assign the value

+= or +
Add to a numeric variable

-= or -
Subtract from a numeric variable

*= or *
Multiply a numeric variable

/= or /
Divide a numeric variable

%= or %
Remainder of a numeric variable

The arithmetic operators are applied by the scheduler and are skipped in three cases: when the variable holds text, when the value in the assignment is text, and when dividing or taking a remainder by zero. Nothing is logged and the job is unaffected.

The right-hand side is a literal value on the same terms as a condition: numeric if it begins with a digit or a minus sign, text otherwise, up to 49 characters, with a leading colon to force text.

The stage flags

Which stage an assignment applies to is set by a separate option, -f, taking a string of letters. -f applies to every -s that follows it on the command line and stays in force until the next -f, so the order of the options matters.

S (Start)
Apply when the job starts

N (Normal)
Apply when the job finishes with an exit code in its normal range

E (Error)
Apply when the job finishes with an exit code in its error range

A (Abort)
Apply when the job is killed by a signal, or exits with a code in neither range

C (Cancel)
Apply when the job is cancelled before it runs

R (Reverse)
Apply the opposite operation at the end of the job as well

The normal and error ranges are the job's own. By default the normal range is exit code 0 only and the error range is 1 to 255, so N is "exited 0", E is "exited non-zero" and A is "killed by a signal". Both ranges can be changed per job with -X, in which case N, E and A follow the ranges.

If no -f is given, the flags are SNEAR - the assignment is applied at start, at every one of the three end states, and in reverse. Give -f explicitly on a plain status update.

The letters differ between the command line and btq: on the btq assignments screen the normal-exit flag is O (Ok), and the others are S, E, A, C and R as here.

Simple examples

Update a status at start and at each outcome:

btr -f S  -s 'status=Running' \
    -f N  -s 'status=Complete' \
    -f EA -s 'status=Failed' \
    job.sh

A status variable set at start and on normal exit alone is left saying Running when the job fails, so cover all three end states.

Count the runs:

btr -f S -s 'job_count+=1' job.sh

Read the assignments back:

btjlist -H -F '%N %H %S'

The display prints the flag letters, a colon, then the assignment - for example SN:status=Running.

The reverse flag

An assignment carrying R is applied twice: once at the stage or stages named, and again in reverse at the end of the job. The reverse of an arithmetic operator is its inverse - += becomes -=, *= becomes /= and so on. The reverse of a plain assignment sets the variable to zero if the value was numeric, and to the empty string if it was text.

# Set the flag to 1 at start; zero it at every end state
btr -f SNEAR -s 'in_progress=1' critical.sh

That covers normal, error and abort in one assignment. Cancellation needs C added, and a job that never reaches an end state runs no end-of-job assignment at all - see Two jobs that must never overlap below.

Capturing the exit code or signal

Two literal values are recognised on the right-hand side of a plain assignment: exitcode and signal.

btr -s 'last_exit=exitcode' job.sh
btr -s 'kill_signal=signal' job.sh

Both must be used with = and with no other operator. Both ignore any -f given: they are applied at whichever of the normal, error and abort ends the job actually reaches, and never at start or cancel. The variable receives a whole number.

Changing conditions and assignments on an existing job

btjchange takes the same -c, -s, -f, -k, -K, -b and -B options as btr. It is not interactive: with no change options at all it prints "You didn't specify any changes to the jobs" and stops without looking at the job numbers. The interactive editor is btq.

btjchange adds to the existing lists. Clear them first:

# Replace the whole condition list
btjchange -y -c 'backup_status=Complete' 87001

# Replace the whole assignment list
btjchange -z -f S -s 'status=Running' -f N -s 'status=Done' 87001

-y cancels the conditions, -z the assignments.

Criticality cannot be toggled on its own. -k and -K change a flag that is applied as each condition is parsed, so a btjchange carrying only -k copies the existing conditions unchanged and does nothing. Restate the condition:

btjchange -y -k -c 'server1:backup_status=Complete' 87001

Read back what the job now carries:

btjlist -H -F '%N %H %C %S'

Assignments to another host's variable

An assignment may name a variable on another host, which must be exported by its owner. The change is forwarded to the owning host and applied there.

btr -f N -s 'server1:import_complete=Yes' export-data.sh

Remote assignments have their own criticality pair, -b (critical) and -B (not critical), on the same terms as -k and -K for conditions: not critical by default, and only ever applied to an assignment that names a host.

Two jobs that must never overlap

Two jobs that need the same tape drive, or update the same database, are the common case for a variable used as a semaphore. There are two ways to build one.

The job-level form is the one older documentation gives: both jobs carry the condition that the lock is free, decrement it at start and increment it at the end.

btvar -C -s 1 LOCK

btr -c 'LOCK>0' -f S -s 'LOCK-=1' -f NEA -s 'LOCK+=1' job-xyz.sh
btr -c 'LOCK>0' -f S -s 'LOCK-=1' -f NEA -s 'LOCK+=1' job-abc.sh

Initialising the variable to a larger number caps the group at that many concurrent jobs.

What holds. On a single host, for jobs whose conditions and assignments are all on that host's own variables, the scheduler re-tests the conditions after taking the variable lock and immediately before applying the start assignment. Two ordinary local jobs due at the same moment will not normally both see the lock free.

What does not hold:

• Across hosts nothing covers the sequence. A job on the host that owns the lock variable and a job on another host are started by two different schedulers, and the check that protects a variable against a simultaneous update applies only to requests that began on the owning machine. Both jobs can see the lock free and both can run, with nothing reported.
• Nothing releases the lock if a job never reaches an end state. Stopping the scheduler kills running jobs outright, so their end-of-job assignments never run and the lock stays taken. Nothing times it out and nothing reports it, so every job waiting on it waits indefinitely.
• A stage the release was not attached to leaves the lock taken. Use NEA, or the reverse flag, so that error and abort release it as well as a normal finish; cancellation is in neither set unless C is added.
• The arithmetic can be skipped. If anything ever stores text in the lock variable, the decrement and the increment both stop happening, with nothing logged, and the condition then reads the variable as zero for ever.

The script-level form is the sound one. btvar can test and set a variable in one atomic operation - the write happens only if nobody has changed the variable since btvar read it. Take the lock inside the job:

#!/bin/sh
btvar -s 1 -eq 0 update_lock || { echo "lock held"; exit 1; }
trap 'btvar -s 0 update_lock' EXIT INT TERM
do-the-work

This is guarded by the scheduler, it reports failure to take the lock as an exit status the script can act on, and the trap releases it on paths a job assignment would miss. Its own limits - owning host only, a retry ceiling, and no protection on a conditional delete - are set out in Working with Variables in Xi-Batch.

Two other ways to express the constraint: make the second job conditional on the first job's completion status, so a missed release leaves one job waiting rather than a stale lock blocking the whole group; or give the jobs load levels that make two of them exceed LOADLEVEL, so the scheduler will not run them together.

Where a counter must be correct, let the scheduler do the arithmetic with += in an assignment. Reading and writing it from a script with two separate btvar calls leaves an unguarded gap between them.

Workflow patterns

Sequential job chain

btr -f N -s 'step1_done=Yes' step1.sh

btr -c 'step1_done=Yes' -f N -s 'step2_done=Yes' step2.sh

btr -c 'step2_done=Yes' step3.sh

Reset the flags at the start of each cycle, or the second run of the chain starts with every step already marked done. Reset at the start rather than at the end of the previous cycle: an aborted run leaves an end-of-cycle reset unexecuted.

Fan-out

btr -f N -s 'parent_done=Yes' parent.sh

btr -c 'parent_done=Yes' child1.sh
btr -c 'parent_done=Yes' child2.sh
btr -c 'parent_done=Yes' child3.sh

Fan-in

btr -f N -s 'input1_ready=Yes' input1.sh
btr -f N -s 'input2_ready=Yes' input2.sh
btr -f N -s 'input3_ready=Yes' input3.sh

btr -c 'input1_ready=Yes' \
    -c 'input2_ready=Yes' \
    -c 'input3_ready=Yes' \
    merge-all.sh

Counter with a threshold

btvar -C -s 0 file_count

btr -f N -s 'file_count+=1' process-file.sh

btr -c 'file_count>=100' -f S -s 'file_count=0' generate-summary.sh

Keep the counter numeric. If anything stores text in it the additions stop happening, with nothing logged, and the summary job then waits indefinitely.

State machine

btvar -C -s Initialise workflow_state

btr -c 'workflow_state=Initialise' -f N -s 'workflow_state=Extract'   initialise.sh
btr -c 'workflow_state=Extract'    -f N -s 'workflow_state=Transform' extract.sh
btr -c 'workflow_state=Transform'  -f N -s 'workflow_state=Load'      transform.sh
btr -c 'workflow_state=Load'       -f N -s 'workflow_state=Complete'  load.sh

A state machine driven this way stalls permanently at whichever step fails, because a failing job does not advance the state and nothing reports the stall. Add an error assignment at each step that moves the state somewhere an alert job is waiting on.

A production workflow using all of it

# Variables
btvar -C -s "Not Started" extract_status
btvar -C -s 0 extract_count
btvar -C -s "Not Started" transform_status
btvar -C -s "Not Started" load_status

# Extract job, daily at 01:00
btr -T '01:00' -r Days:1 \
    -f S  -s 'extract_status=Running' -s 'extract_count+=1' \
    -f N  -s 'extract_status=Complete' \
    -f EA -s 'extract_status=Failed' \
    extract-data.sh

# Transform job, waits for extract
btr -c 'extract_status=Complete' \
    -f S  -s 'transform_status=Running' \
    -f N  -s 'transform_status=Complete' -s 'extract_status=Not Started' \
    -f EA -s 'transform_status=Failed' \
    transform-data.sh

# Load job, waits for transform
btr -c 'transform_status=Complete' \
    -f S  -s 'load_status=Running' \
    -f N  -s 'load_status=Complete' -s 'transform_status=Not Started' \
    -f EA -s 'load_status=Failed' \
    load-data.sh

# Alert jobs, one per failure state
btr -c 'extract_status=Failed' \
    -f N -s 'extract_status=Not Started' send-alert.sh

btr -c 'transform_status=Failed' \
    -f N -s 'transform_status=Not Started' send-alert.sh

btr -c 'load_status=Failed' \
    -f N -s 'load_status=Not Started' send-alert.sh

-T sets the start time. -t is the delete time - a number of hours after which the job is removed if that long has passed since it was queued or last ran - and using it here would set no start time at all.

Each job here uses two -s options after one -f; that is what -f persisting means in practice, and it is why the four -s options in the transform job fall into two pairs.

The limits are 10 conditions and 8 assignments per job, and a job needing more of either is usually two jobs.

Related articles

All articles · Release notes · Contact support