Tracking Xi-Batch Job Activity with LOGJOBS

Xi-BatchXi-Batchloggingmonitoringvariables

Xi-Batch writes a job log only while the LOGJOBS system variable holds a destination, and a new installation starts with it empty. This article says what the log records, how to set the destination and its permissions, and how to rotate the file without losing entries.

LOGJOBS names the destination

LOGJOBS is an Xi-Batch system variable, set with btvar. While it holds the empty string no job log is written and nothing reports that fact.

The variable lives in the scheduler's saved variable file alongside LOADLEVEL, CLOAD and LOGVARS, so its value is kept across a normal stop and start: the scheduler reopens the destination as it reloads the variable file. Two situations reset it to empty, switching logging off with no message:

A scheduler started with no saved variable file
a new installation, or one whose variable file has been removed or reinitialised. The system variables are recreated with their defaults, and LOGJOBS comes back empty.

A migration to another system
the variable export tool skips every system variable, so LOGJOBS, LOGVARS, LOADLEVEL and CLOAD are not carried by the export. Job logging is off on the new machine until it is set again. See the Xi-Batch migration article, and check LOGJOBS as a routine post-migration step.

What the scheduler records

The scheduler writes an entry when:

A job is created
submitted to the queue.

A job starts, or cannot be started
the scheduler begins execution, or the attempt fails.

A job finishes
as completed, error or abort, decided by the job's exit-code ranges (see Status Codes).

A job is cancelled, deleted or killed
by an operator, or automatically when it outlives its run-time limit or its grace period, or when it is auto-deleted after the retention interval.

A job is forced
started immediately, with or without advancing its scheduled time.

A job's ownership, group, mode or details change
including the daylight-saving adjustment applied to scheduled times.

The machine running a job is lost
the scheduler marks the job aborted on behalf of the failed host.

Variable assignments are logged separately, through LOGVARS. Scheduler starts, stops, licence messages and panics go to the scheduler's own report file, btsched_reps, which is a different file with a different format.

Setting the destination

LOGJOBS is set with btvar, which requires a running scheduler. The value is the argument to -s; the variable name is the positional argument, and the order matters.

Log to file:

btvar -s jobaudit.log LOGJOBS

A relative name is resolved against the scheduler's working directory, which is the spool directory - by default /var/spool/xi/batch - so jobaudit.log becomes /var/spool/xi/batch/jobaudit.log. An absolute path is used as given.

Log to a program:

btvar -s '|/usr/local/bin/job-logger.sh' LOGJOBS

A value whose first character is \| is treated as a shell command line rather than a file name. See "Logging to Programs" below for what this involves and who may configure it.

Disable logging:

btvar -s '' LOGJOBS

The empty string turns logging off and closes the current destination.

Who may set it. The variable is owned by the Xi-Batch system user, batch, and grants write permission to its owner only. Root and the batch user both hold full Xi-Batch privileges and can set it; other users are refused.

A new value takes effect immediately: the scheduler closes the previous destination and opens the new one as the assignment is applied. Where the destination cannot be opened - a bad path, a directory the batch user cannot write, a command that fails to start - the scheduler makes no report and logging is off, so confirm afterwards that entries are arriving.

The log file's owner and mode

When logging to a file, the scheduler sets the file's owner, group and mode from the LOGJOBS variable each time it opens the file:

File owner and group
the owner and group of the LOGJOBS variable - by default the batch system user and its group.

File mode
built from the variable's read and write permissions alone. Read on the variable gives r on the file, write gives w, for user, group and others independently. The variable's other permission bits take no part, and no execute bit is ever set. The shipped defaults are read and write for the owner and read for the group, giving mode 0640.

Show what is in force:

btvlist -F '%N %U %G %M %V' LOGJOBS

The mode column reads as U:, G: and O: groups, in which R is read and W is write.

To let your own tooling read the log, grant read to the group that tooling runs as - or to others - on the variable:

btvar -M 'G:+R' LOGJOBS

A mode or ownership change does not reach the existing log file on its own. The scheduler applies owner and mode only when it opens the file, and changing the variable's mode does not reopen it. Re-assign the value to make the change effective:

btvar -s jobaudit.log LOGJOBS

The log line

Each log entry is a single line with pipe-delimited fields:

05/01/2026|10:22:43|13741|date|completed|jmc|users|150|1000

Field order:

1. Date - dd/mm/yyyy, with a four-digit year. In time zones four or more hours behind UTC the first two fields are exchanged, giving mm/dd/yyyy
2. Time - HH:MM:SS, local time
3. Job Number - the job number, or machine:jobnumber for a job belonging to another host
4. Job Title - the job's title, or \ if it has none
5. Status Code - the event (see below), prefixed with machine: when the request came from a remote host
6. User - job owner user name
7. Group - job group name
8. Priority - job priority value
9. Load Level - job load level value

There is no separator after the load level; the line ends there. Each entry is flushed as it is written, so a file sink is up to date as soon as the event happens.

The status field

The status field is one of the following. They are written in lower case, and several contain a space, which matters when writing a parser. The strings are taken from the installed help file, so a site with a customised or translated help file may see different words.

abort
the job finished with a status outside both its exit-code ranges, or was killed by a signal

auto delete
the job was removed automatically after its retention interval expired

cancel
a queued job was cancelled before it started

chgrp
the job's group was changed

chmod
the job's permissions were changed

chown
the job's owner was changed

completed
the job finished with a status in its normal exit-code range

create
the job was submitted to the queue

delete
the job was deleted from the queue

dst adjust
the job's next start time was moved by a daylight-saving adjustment

error
the job finished with a status in its error exit-code range, or the scheduler could not start it at all

exceeded grace period
the job was still running after its grace period and was killed outright

exceeded runtime
the job passed its run-time limit and was signalled

force-run
the job was forced to run without advancing its scheduled time

force-start
the job was forced to start immediately

jdetails
other job details were modified (conditions, assignments, times and so on)

manual kill
a running or starting job was killed on request

network-aborted
the machine the job was running on was lost, and the job was marked aborted

started
job execution began

Which of completed, error and abort a job gets is a property of the job. Each job carries a normal range and an error range, set with btr -X. The shipped defaults are normal 0:0 and error 1:255, so by default exit 0 gives completed and any other exit status gives error; a job killed by a signal gives abort whatever the ranges are.

Reading the log

Match the status field case-sensitively in lower case, and allow for the machine: prefix on entries driven from a remote host.

Count jobs by status:

awk -F'|' '{print $5}' /var/spool/xi/batch/jobaudit.log | sort | uniq -c

Find failed jobs, local and remote:

awk -F'|' '$5 ~ /(^|:)error$/' /var/spool/xi/batch/jobaudit.log

Track a specific user's jobs:

awk -F'|' '$6 == "jsmith"' /var/spool/xi/batch/jobaudit.log

Daily job completion count:

awk -F'|' '$5 == "completed" {print $1}' jobaudit.log | sort | uniq -c

Jobs with high load levels:

awk -F'|' '$9 > 5000' /var/spool/xi/batch/jobaudit.log

Logging to a program

A LOGJOBS value beginning with \| is a shell command line. The scheduler starts it through the shell and writes each log line to its standard input.

Four properties govern how such a sink must be written and who may install one.

Only a local administrator may set it. The scheduler refuses a \| value that arrives from a network peer or from a job's own variable assignment, and refuses it before storing it, so the value cannot be left behind to be opened at the next start. A sink is configured on the machine, by root or the batch user, with btvar. Cluster-wide configuration by broadcasting the variable does not work.

It runs with the scheduler's privileges. The command is run by the scheduler process, not by the job's owner and not by the user who set the variable. Treat the script as system software: own it by root, keep it out of directories other users can write, and give the absolute path - the command inherits the scheduler's environment, whose PATH is that of the service that started it.

The program is restarted periodically. When the scheduler next rewrites its queue files - at most every 300 seconds while there is activity - it closes the pipe and starts the command again. A sink that reads its input in a loop will therefore see end-of-file and be re-run several times an hour. Write it so that repeated invocation is harmless: append rather than truncate, and do any per-run setup idempotently.

A sink that stops reading loses entries. If the program exits, the scheduler is not told; entries are discarded until the next queue rewrite starts a fresh copy. If the program reads slowly enough for the pipe to fill, the scheduler blocks writing to it, and it waits for the program to finish when it closes the pipe. Keep the sink fast and non-blocking; do no network calls in it.

Example logger script:

#!/bin/sh
# /usr/local/bin/job-logger.sh
# Runs from the scheduler. Keep it fast; it is re-run periodically.

while IFS='|' read -r date time jobnum title status user group priority load; do
    # Strip any machine: prefix from the status field
    event=${status##*:}

    # Note high-priority failures in syslog
    if [ "$event" = "error" ] && [ "$priority" -gt 150 ]; then
        logger -t xibatch "Job failed: $title (job $jobnum, user $user)"
    fi

    # Append every entry to a file of our own
    printf '%s|%s|%s|%s|%s|%s|%s|%s|%s\n' \
        "$date" "$time" "$jobnum" "$title" "$status" \
        "$user" "$group" "$priority" "$load" \
        >> /var/log/xibatch-jobs.log
done

Rotating the log

Job logs grow without limit. Xi-Batch never rotates the log, and it never reopens it. The scheduler holds the file open from the moment the value is set until the value changes or the scheduler stops, and it ignores SIGHUP, so there is no signal that will make it pick up a new file.

So renaming the log and creating a new one leaves the scheduler appending to the renamed - or deleted - file, and the new file stays empty. Nothing reports this.

Two arrangements work. The first uses logrotate and is therefore Linux only; the second is a shell script and runs on every platform Xi-Batch ships for.

Rotate by copying and truncating. The scheduler opens the file in append mode, so truncation is safe and writing resumes at the start of the empty file:

# /etc/logrotate.d/xibatch-jobs
/var/spool/xi/batch/jobaudit.log {
    weekly
    rotate 52
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Do not add a create stanza here; copytruncate and create are alternatives, and create is the one that loses entries. Entries written between the copy and the truncation are lost.

Rotate by moving the file and re-assigning the variable. Re-assigning LOGJOBS closes the old file and opens a new one with the correct owner and mode, so nothing is lost:

#!/bin/sh
# Run as root or as the batch user
LOG=/var/spool/xi/batch/jobaudit.log
DATE=$(date +%Y%m%d)

if [ -f "$LOG" ]; then
    mv "$LOG" "${LOG}.${DATE}"
    btvar -s jobaudit.log LOGJOBS
    gzip "${LOG}.${DATE}"
fi

Forwarding events to another system

Forward job events from a separate process that reads the log file: log to a file, rotate it as above, and have an independent agent tail or poll that file. Make no network calls from a LOGJOBS sink, which runs inside the scheduler's logging path, where a slow or unreachable endpoint stalls job operations and where the program is restarted at every queue rewrite. Where a sink is wanted, keep it to a local, non-blocking hand-off - appending to a spool file, or writing to the local syslog daemon - and forward from there.

Confirming entries are arriving

Check the LOGJOBS value:

btvar LOGJOBS

btvar with no operation options prints the variable's current value; empty output means logging is off.

Test logging:

# Submit test job
echo "echo test" | btr -h "Test logging"

# Check log file
tail /var/spool/xi/batch/jobaudit.log

You should see create, started and completed entries for the job.

What goes wrong

No log entries appearing
Check the value with btvar; an empty value means logging is off, and a fresh variable file or a migration is the usual reason. If the value is set, confirm the batch user can create and write the file in the spool directory. A destination the scheduler cannot open produces no error message anywhere, and logging stays off.

Log program not receiving data
Give the command as an absolute path; the scheduler's PATH is not an administrator's. Confirm the program is executable by the batch user and that it does not exit on end-of-file expecting to run once - it is restarted at every queue rewrite. Failure to start the command is not reported in btsched_reps or anywhere else.

Log file grows too large
Rotate with copytruncate, or move the file and re-assign LOGJOBS. Do not rotate with a plain create.

Entries stopped after a rotation
The scheduler is still writing to the rotated file. Re-assign LOGJOBS to reopen the current path.

Permission denied reading the log
The file's owner, group and mode come from the LOGJOBS variable. Grant read on the variable with btvar -M, then re-assign the value so the change reaches the file.

Related articles

All articles · Release notes · Contact support