Identifying and Removing Redundant Xi-Batch Jobs

Xi-BatchXi-Batchbackup-restorehousekeepingjob-controlbtjchangebtjdelbtjlist

The queue holds one-off jobs left in Done state, jobs that stopped for good years ago, and work the site still depends on, and in a listing the three look alike. This article covers reading the queue, deciding which jobs are redundant on the evidence the product holds, taking a copy, and removing them.

What stays in the queue

A job that runs once, has no repeat specification and is not retained removes itself when it finishes. The standing contents of the queue are therefore the site's timed and repeating work, plus whatever has been retained.

The queue is the schedule. Deleting a job deletes the site's only copy of it, because the delete also unlinks the job's script file from the spool directory. There is no undo and no archive unless you take one first.

Reading the queue with btjlist

The btjlist command is the primary tool for reviewing jobs from the command line. By default it shows only jobs owned by this host in a terse format. Add -H for column headings and -R to include jobs owned by remote hosts:

btjlist -H
btjlist -HR

The shipped default display shows the job number, user, title, command interpreter, priority, load level, time, condition variables and progress state. Both the default format and the default heading setting can be changed per site or per user through the option-defaults files, so a script should always name the format it wants rather than relying on the default.

The format codes for housekeeping

The -F option lets you specify exactly which fields to display. The format string uses % codes, each representing a job attribute, and case matters throughout.

Code	Meaning
%N	Job number, zero-padded to the width of the widest one; host:number for a job owned by another host
%U	User
%H	Job title; the queue prefix is stripped only when -q is also given
%h	Job title with the queue prefix always stripped
%q	Queue name - the part of the title before the colon, empty if the title has no colon
%P	Progress state
%T	Next run time in full, dd/mm/yyyy hh:mm
%t	Next run time, hh:mm if today else dd/mm
%o	Time originally submitted
%W	Last or next time, depending on the progress state
%b	Time the last run started
%f	Time the last run ended; blank if the job has never run
%r	Repeat specification, or Delete or Retain
%d	Auto-delete time in hours; 0 means never
%c	Condition variable names only
%C	Conditions in full, with comparisons and values
%s	Assignment variable names only
%S	Assignments in full
%x	Exit code from the last run
%y	Signal number from the last run
%L	Load level
%i	Process id, blank when the job is not running
%e	Export scope - Export, Rem-runnable, or blank for local
%O	Originating host

A format string tailored for housekeeping review might look like this:

btjlist -HR -F "%N %U %h %P %r %f %d"

This shows the job number, owner, title, progress state, repeat specification, the time the job last finished, and its auto-delete time.

To filter by queue name, user, or group:

btjlist -H -q "nightly*" -F "%N %h %P %W %r"
btjlist -H -u olduser -F "%N %h %P %f"
btjlist -H -g finance -F "%N %h %P %W"

The -q pattern is a shell-style glob matched against the whole title, and jobs whose title has no colon in it are included as well; -Z excludes those. All three restrictions are applied by the listing itself.

What the product records about a job's last run

Xi-Batch keeps two per-job timestamps that survive a restart: the time the last run started and the time it ended. Both are shown by btjlist. Neither is displayed with a year. The time fields render as hh:mm when the moment is within the next day and as dd/mm otherwise, so a job that last ran a fortnight ago and one that last ran in 2011 produce the same five characters. A blank %f means the job has never run, and that is the only unambiguous reading available from btjlist.

The one place a full date with a year is recorded is the job log, and only if the LOGJOBS system variable has been set. Out of the box it is empty and nothing is logged.

Unless LOGJOBS was already in use, the product holds the day and month of a job's last run and no year. Every conclusion of the form "this has not run for years" is an inference from context: the job's title, the queue it belongs to, who owns it, whether the account still exists. Take a copy first in every case where the answer is not obvious.

The progress states

The scheduler looks for work only among jobs whose progress state is blank. A job showing anything else is examined only to see whether its auto-delete time has arrived, and is then passed over, however many times its scheduled time comes round.

The states are:

blank (no state)
The job is runnable. It is waiting for its scheduled time, for its conditions to be satisfied, or for capacity. This is the state of a repeating job between runs.

Init, Strt
The job is being started. Init is the first phase, Strt the second.

Run
The job is executing. For a job running on another host the field reads Run:hostname; for a job owned elsewhere and running here it reads Local Exec.

Fin
The job's process has ended and the scheduler is completing it.

Done
The job ran and finished cleanly, and was retained on the queue. It will not run again.

Err
The job's last run exited with a status outside the range configured as normal for it. It will not run again.

Abrt
The job was killed - by an operator, by the maximum-run-time guard, or by the scheduler being stopped while the job was running. It will not run again.

Canc
The job was set to cancelled. It will not run again until someone sets it back.

A repeating job in Err, Abrt or Canc is a fault. When a repeating job ends, the scheduler advances its next-run time either way, and resets the progress state to blank only if the job finished cleanly. A job that failed months ago shows tomorrow's date in the time column and reads as healthy apart from three letters in the progress column, and it is work the site still expects to happen. Set it runnable again with btjchange -N, or r in btq, and find out why it stopped.

Which jobs are candidates for removal

Use a delimiter in the format string and split on it: the last field of a btjlist line receives no padding, and a title can contain anything.

btjlist -HR -F "%N|%U|%h|%P|%r|%f" | awk -F'|' '$4 == "Done"'

Jobs in Done state with no repeat specification
These ran once and were retained. If the %d column is 0 they will stay for ever. This is the largest and safest category.

Jobs in Canc state
A job set to cancelled and forgotten. Check who owns it and whether the cancellation was meant to be temporary before removing it.

Jobs belonging to users who have left
Filter by user, and check the account still exists:

btjlist -HR -u departed_user -F "%N %h %P %r %f"

Jobs whose conditions can never be satisfied
A condition on a variable that has been deleted, or that the job's owner cannot read, holds the job for ever with nothing reported. Compare each job's conditions against the variable list:

btjlist -HR -F "%N %h %C"
btvlist -H

Jobs owned by hosts that are no longer in service
These appear only with -R, and %O gives the originating host. They are the remote host's property; deleting the local copy does not remove the job at its owner.

A job with no scheduled time and no conditions runs on the next pass of the scheduler, and had it no repeat specification it would already have run and removed itself. A job of that shape sitting in the queue is either in a non-blank progress state, in which case see above, or is being held back by something else.

Dating a job's last run from the job log

If the LOGJOBS system variable has been set, the scheduler appends one line to the named file for every job event. Set it with btvar, giving the value with -s and the name as the trailing argument:

btvar -S -s /var/log/xi/jobs.log LOGJOBS
btvar LOGJOBS

Each line is pipe-separated: date, time, job number, job title, event, user, group, priority, load level. The date carries a four-digit year, which makes this the only durable record of when a job last ran. The events are lower-case words; the ones that matter for housekeeping are started, completed, error, abort, cancel, auto delete, exceeded runtime and exceeded grace period. Where a request came from another host the event field is prefixed with that host name and a colon.

To find the last time a job completed:

awk -F'|' '$3 == "1420" && $5 == "completed"' /var/log/xi/jobs.log | tail -5

The job number in the log is written without leading zeroes, unlike the zero-padded %N column, so the two do not compare as strings without stripping the padding.

Three limits on the log. The day and month are written the other way round on machines whose timezone is four or more hours west of UTC. Nothing is recorded when a job is passed over, so silence in the log proves that the job did not run and never why. The log begins when LOGJOBS was set, and covers no period before that.

Variables held by the jobs you remove

Variables and jobs are linked through conditions and assignments. List the variables with btvlist:

btvlist -H

To see which jobs reference a particular variable:

btjlist -HR -F "%N %h %C %S" | grep VARIABLE_NAME

The scheduler refuses to delete a variable that is in use, answering "Variable %s in use in jobs.", and the same refusal blocks changing the variable's export or cluster flags. So the safe order is to remove the jobs first and then try each variable in turn:

btvar -D VARIABLE_NAME

The check counts every job in the queue whatever its progress state, so a long-finished job that will never run again still pins a variable. The scheduler also refuses to delete any of the five system variables - LOADLEVEL, CLOAD, MACHINE, LOGJOBS and LOGVARS. In btq, V switches to the variable list, C creates a variable and D deletes one.

Taking a copy before removing anything

Before removing anything, take a copy. Xi-Batch ships four converters that read the scheduler's binary state files and write shell scripts of the commands that would recreate them. On a packaged installation they carry an xb- prefix and live on the ordinary command path.

The scheduler's state files are in the spool directory, by default /var/spool/xi/batch. On a build with XML support - which is every packaged build - the job and variable files carry a suffix, and the converters choose their input format from that suffix, so the name must be given exactly:

cd /var/spool/xi/batch
ls btsched_jfile* btsched_vfile*

To back up jobs, giving the job file, the output script and a directory for the job scripts:

mkdir -p /var/backups/xi/batch/Scripts
xb-cjlist btsched_jfile.xbjl6 /var/backups/xi/batch/Jcmd /var/backups/xi/batch/Scripts

Give the plain name btsched_jfile instead if the installation has no XML support. To produce a single self-contained script with the job scripts embedded as here-documents, use -I with a delimiter word and omit the third argument.

To back up variables, command interpreters and user permissions:

xb-cvlist btsched_vfile.xbvl6 /var/backups/xi/batch/Vcmd
xb-ciconv -D /var/spool/xi/batch cifile /var/backups/xi/batch/Cicmd
xb-btuconv -D /var/spool/xi/batch btufile6 /var/backups/xi/batch/Ucmd

xb-btuconv has no default source directory, so -D is required for it. The others default to the configured spool directory.

When restoring, the order is user permissions, command interpreters, variables, then jobs: btr rejects a job whose command interpreter or referenced variables do not yet exist.

Two gaps in the backup. xb-cvlist skips system variables, so LOADLEVEL, LOGJOBS and LOGVARS are not in the script and must be set again by hand. xb-cjlist does not carry the last-run and last-start times, so restoring from Jcmd loses whatever run history the queue held.

Removing the jobs

Deleting a job unlinks its script file from the spool directory as well as removing the queue entry. Take the copy above first.

Using btq interactively
Move to the job and press D to delete it. If the job is running, kill it first with K, which offers a choice of signal, and wait for it to stop.

Using btjdel from the command line
btjdel takes a list of job numbers, so one invocation handles the whole batch:

btjdel 1420 1421 1422

For a job owned by another host, include the host prefix:

btjdel avon:24918

A running job is skipped with a message unless -y is given, and a job the caller has no permission to delete is skipped with a message as well. Neither of those sets a non-zero exit status, so check the queue afterwards.

Taking a copy of individual jobs first
btjdel -u -k writes a command file and a job file for each job named, into the directory given by -D, and leaves the queue entry in place. This is the per-job equivalent of the whole-queue backup above.

Cancelling before deleting
To stop a job running without removing it yet:

btjchange -C 1420

This sets the progress state to Canc, which takes the job out of the scheduler's consideration while keeping it on the queue for review. btjchange -N puts it back.

Unqueueing for archival
In btq, U on the job saves the job script and a command file that could resubmit it. This is the safest approach when you are unsure whether a job might be needed again.

Bulk identification
To list just the job numbers of a user's Done jobs:

btjlist -R -u olduser -F "%N|%P" | awk -F'|' '$2 == "Done" { print $1 }'

That output can be passed to btjdel with xargs. Review the list first.

Removing a decommissioned remote host

Xi-Batch reads its host file, by default /etc/xi/batch-hosts, once, when the scheduler starts, and attaches to every host in it that is not marked manual, external or roaming. There is no retry: a host that was unreachable at that moment stays disconnected until either it initiates the connection itself or the scheduler is restarted.

Two consequences for housekeeping. Nothing reports whether a host is connected. There is no host-listing command, and the evidence is indirect: a disconnected host's jobs and variables disappear from btq and from btjlist -R. And the connect and disconnect operations are reached by invoking btstart under the names btconn and btdisconn, but the packages do not create those links, so on a packaged installation a scheduler restart is the only way to re-attach a host.

Removing a decommissioned host is therefore an edit to the host file followed by a scheduler restart, scheduled for a window in which no jobs are running - stopping the scheduler kills every running job and leaves it in Abrt.

What deleting jobs returns to the shared-memory tables

The job table lives in a shared-memory segment sized by btstart -j, and the variable table by btstart -v. Neither is a hard ceiling: when the table fills, the scheduler allocates a larger segment, copies the contents across and writes a warning to btsched_reps reading "Increased size of job segment to accommodate ...". A submission fails only when the operating system refuses the larger segment.

Deleting jobs returns their slots to the free list at once, so the space is reusable immediately. The segment itself never shrinks, so the memory is not returned until the scheduler is restarted. btstart's sizing arguments are ignored where the scheduler is already running; resizing means btquit followed by btstart.

After the removal

List the queue again with btjlist -HR -F "%N %U %h %P %r %f %d" and confirm that what remains is the work the site still expects. A job the caller had no permission to delete, and a job that was running, are both still there.

Every job that was running when the scheduler stopped comes back as Abrt, so check the progress states after a scheduler restart as well.

Where the jobs removed shared a queue name prefix, review the whole queue as a group. The members of a queue commonly depend on one another through conditions and assignments, and removing one job can leave the rest unable to run.

Related articles

All articles · Release notes · Contact support