Managing Xi-Batch IPC Resources

Xi-BatchXi-Batchipcschedulertroubleshootingxb-ripc

Xi-Batch holds its live queue in System V interprocess-communication objects, and a scheduler that was killed rather than stopped leaves them behind. This article says which objects the product uses, how to list them, and how to clear the ones left by an unclean stop.

The IPC objects Xi-Batch creates

Xi-Batch coordinates the scheduler btsched and the client commands through a small set of objects. Every build Xi Software packages uses the same configuration:

• Two System V shared memory segments - one holding the live job queue, one holding the live variables. This is the queue state the commands read.
• Two lock files - btjob.lock and btvar.lock in the spool directory, used with flock to serialise access to those segments.
• One Unix domain socket - btsched_req, in the sockets directory (default /var/spool/xi/btuds, overridable with SPUDSOCKS in /etc/xi/batchconfig). Client commands send their requests to the scheduler over this socket.
• One semaphore set of three - created only when the licence enables networking, and used solely to coordinate variable locking between hosts.

On a standard installation, ipcs shows two shared memory segments belonging to Xi-Batch, no message queues, and one semaphore set on a networked licence or none on a standalone one.

A small number of installations run a variant built to Xi's order, and those use different mechanisms: memory-mapped files btmm_jobs and btmm_vars in the spool directory in place of shared memory, so ipcs shows no Xi-Batch shared memory at all; a semaphore set in place of the two lock files; or a System V message queue in place of the request socket, with an extra shared memory segment alongside it. Run xb-ripc and it reports what it finds on your system.

The keys, and the range they can appear in

The keys are compiled-in constants, so they are the same on every host; none is derived with ftok. All of them begin 0x5869b:

• Job shared memory segment - 0x5869b002
• Variable shared memory segment - 0x5869b003
• Network-locking semaphore set - 0x5869b003 (the same number in the separate semaphore key space)
• Message queue, on an installation that uses one - 0x5869b000, with its transport segment at 0x5869b100
• Lock semaphore set, on an installation that uses one in place of the lock files - 0x5869b001

The two segment keys are a starting point. If a key is already taken, or when a segment is replaced by a larger one, the scheduler steps the key on by two and tries again, wrapping after a hundred keys. A segment can therefore appear anywhere from 0x5869b002 to 0x5869b065.

Multiple environments. Where several independent Xi-Batch environments run on one host, the environment number in XIBATCH_ENV shifts every key: the value added is (environment number + 1) multiplied by 1024. Environment 0 puts the job segment at 0x5869b402, environment 1 at 0x5869b802, environment 2 at 0x5869bc02, and environment 3 at 0x5869c002 - past the 0x5869b prefix altogether. On such a host, select entries by their owning user, or let xb-ripc work the keys out for you.

What a clean run looks like

The scheduler creates these objects when it starts and removes them when it stops cleanly, whether it is stopped with btquit or sent SIGTERM. The lock files are left in place between runs and truncated at the next start.

The initial job and variable allocations come from NUMJOBS and NUMVARS in /etc/xi/batchconfig, or from the -j and -v arguments to btstart. When the queue outgrows its allocation the scheduler creates a larger segment at the next key, copies the contents across and removes the old one, logging Increased size of job segment to btsched_reps. Three segments appear for a moment during that changeover.

Listing what is there

# What Xi-Batch itself finds. Run as root or as the batch user.
xb-ripc

# The shared memory segments, selected by owner
ipcs -m | grep batch

# The request socket. Its presence is how every Xi-Batch command
# decides whether the scheduler is running.
ls -l /var/spool/xi/btuds/btsched_req

# The lock files
ls -l /var/spool/xi/batch/btjob.lock /var/spool/xi/batch/btvar.lock

Select by owner: that filter is unaffected by the key stepping described above, and it keeps Xi-Text out of the listing (Xi-Text keys begin 0x58691). The administration manual writes ipcs -o. On Solaris, AIX and HP-UX that option adds the outstanding-usage figures - for a shared memory segment, the number of processes attached to it - and those platforms print the owner by default. Linux has no -o at all, and a plain ipcs -m gives you both the owner and the attach count in its nattch column.

One pair of entries belongs to something else. A shared memory segment at 0x5869b200 and a two-member semaphore set at 0x5869b201 belong to btfilemon, the file monitor, which is a separate program run by an ordinary user. Those two keys carry no environment offset and are owned by whoever started the monitor, so an owner filter will miss them and a key filter will catch them. Leave them alone: xb-ripc ignores them, and the monitor removes them itself when the last one exits. The file monitor is installed only by the tarball installation.

The symptoms of resources left behind

After a power failure, or after btsched has been killed with SIGKILL, the shared memory segments, the network semaphore set and the btsched_req socket file are all left behind.

The socket file is the one that prevents a restart. The scheduler binds that path at startup, and if the file already exists the bind fails and btsched exits immediately with status 0 and no message. btstart reports that it is restarting the scheduler, and nothing comes up.

Stale shared memory leaves the scheduler free to start, since it allocates at the next free key. It confuses the client commands instead, because they scan the key range for a segment and can find the abandoned one ahead of the live one.

The symptoms are:

• btstart reports that it is restarting the scheduler, but no btsched process appears
• Commands report that the scheduler is not running when it should be
• Commands show a stale or empty job queue, or report Panic! Cannot read job queue
• More than two Xi-Batch shared memory segments are present while no scheduler is running

Clearing them

Stop the scheduler first: xb-ripc -d removes whatever it finds, including the live queue.

# On a packaged Linux install, stop the service rather than the
# program: the unit runs btquit -y for you, and it is configured to
# restart btsched on failure if you kill it behind systemd's back.
systemctl stop xibatch

# On other platforms, or where the service is not managed by systemd
btquit -y

# Check it has gone
ps -ef | grep btsched

# If a btsched process remains, send it SIGTERM - never SIGKILL.
# SIGTERM runs the same clean shutdown as btquit and removes the IPC
# objects; SIGKILL is what leaves them behind.
kill PID

# Remove anything left over. Run as root or as the batch user.
# This also deletes the stale btsched_req socket.
xb-ripc -d -o /dev/null

# On a host running multiple environments, name the environment
xb-ripc -d -S 1 -o /dev/null

# Confirm nothing of ours remains
ipcs -m | grep batch
ls -l /var/spool/xi/btuds/btsched_req

# Restart
systemctl start xibatch

# Verify: the request socket is back and the queue reads
ls -l /var/spool/xi/btuds/btsched_req
btjlist -H

At that start the scheduler creates fresh objects and reloads the queue from the saved job and variable files.

Where xb-ripc is unavailable, ipcrm will remove the segments by identifier. Take the identifier from the second column of the ipcs listing on Linux and from the ID column on Solaris, AIX and HP-UX, remove only entries owned by batch, leave the two file-monitor entries described above in place, and delete the btsched_req socket file by hand afterwards. An entry cannot be removed while a process is still attached to it, so make sure the user commands have exited as well as the scheduler.

Related articles

All articles · Release notes · Contact support