Sizing the Xi-Batch Shared Memory Queue
What the start-up numbers allocate, when the segments grow, and what happens when they cannot
Xi-Batch allocates its job and variable queues in shared memory when the scheduler starts, and the number of slots is a setting. This article says where that number comes from, what happens when the slots run out, and how to change it.
What the shared memory holds
Xi-Batch keeps its live queue in System V shared memory so that every command can read it without going through the scheduler. The segments are created when the scheduler starts and removed when it stops cleanly.
In normal operation there are two: a job segment and a variable segment, each holding its own header, hash tables and slot array. A third appears while the file monitor is running, and a further one appears briefly while a segment is being enlarged.
The segments are sized at start-up, and they do grow
The scheduler allocates a number of slots - jobs or variables - and the byte size of the segment follows from that number. When the slots run out, the scheduler creates a larger segment at the next key, copies the contents across and removes the old one. Growth is a routine operation.
It copies the whole segment while the queue is locked, and a grow that cannot be completed means work is refused.
Setting the initial allocation
btstart takes options:
| Option | Long form | Meaning |
|---|---|---|
| -j | --initial-job-size | Initial number of job slots |
| -v | --initial-var-size | Initial number of variable slots |
| -l | --initial-load-level | Initial maximum load level - a scheduling limit, not a memory setting |
The shipped service unit runs:
/usr/local/bin/btstart -j 2000 -v 500 -l 10000
An option given as zero or omitted falls back to the compiled-in default.
Making the setting permanent
The site's own setting is a system variable in the master configuration file, /etc/xi/batchconfig. Every program reads that file, so the value applies however the scheduler is started. Editing the service unit or the start-up script also works.
| Setting | Name in /etc/xi/batchconfig |
|---|---|
| Initial jobs | NUMJOBS |
| Initial variables | NUMVARS |
| Growth step, jobs | INCJOBS |
| Growth step, variables | INCVARS |
Each line is a name, an equals sign and a value:
NUMJOBS = 20000
NUMVARS = 2000
The command-line values override the file, so a service unit that passes numbers wins over anything set here. Set the numbers in one place.
The compiled-in defaults
The compiled-in default for variables is 300. The job default is not a single published number: it is derived when the product is built from the shared-memory limit of the build machine, and capped at 20,000. On a current Linux build it reaches the cap.
The value a given binary carries cannot be read from it, so set the job number explicitly. Starting the scheduler with no numbers gives a queue likely to be far larger than the 2,000 the shipped service unit passes.
One further rule: a saved queue larger than the number you asked for raises the allocation. At start-up the scheduler sizes the segment from the saved queue file if that is bigger than your figure, so an installation recovering a large queue keeps every entry.
How much memory a slot takes
A slot is a fixed-size structure, so the segment is roughly the slot count multiplied by the slot size, rounded up. An Xi-Batch job slot is a large structure, because it carries the job's environment space inside the slot, so a given number of jobs takes considerably more memory here than the same number would in a print queue.
The bytes column of ipcs -m is the actual allocation on your own system:
ipcs -m
Xi-Batch's segments have keys beginning 0x5869b.
The size column is a Linux one. Solaris, AIX and HP-UX print only the type, identifier, key, mode, owner and group by default; add -b there to get the segment size. ipcs -mb shows the size on all four platforms.
On any machine with a few gigabytes of memory, an allocation covering the site's peak is a small part of it. Sizing below the peak brings a queue-locking copy, and a refused job where the copy cannot be made. Set the figure generously, from your own peak.
What grows, when, and by how much
| Segment | Grows when | Grows by |
|---|---|---|
| Job | A job is created and the slots are full | INCJOBS, by default half the initial-jobs default |
| Variable | A variable is created and the free chain is empty | INCVARS - 20 slots by default |
The default growth step for jobs is derived from the compiled-in default, so it is large. A site that starts with -j 2000 and fills it grows by the full INCJOBS step. Set INCJOBS explicitly where the step matters.
The variable step of 20 is small enough that a site adding variables steadily provokes a whole-segment copy every twenty variables. Size NUMVARS to cover every variable the site will ever hold.
Xi-Batch records each grow in the scheduler report file /var/spool/xi/batch/btsched_reps:
*** Warning: Increased size of job segment to accommodate 12000 jobs (was 2000)
*** Warning: Increased size of variable segment to accommodate 520 vars (was 500)
Those lines are the evidence that the initial figure is too small. Search for them before changing anything else.
What happens when the slots run out
Xi-Batch declines the request and keeps running. If the job segment cannot be grown, the job is refused and the program that asked for it prints:
Cannot allocate any more jobs
If the variable segment cannot be grown:
No more space for variables.
I cannot allocate any more shared memory space to hold
new variables - sorry. Please check your kernel configuration.
The scheduler stays up and everything already queued continues to run. What is lost is the work that was refused, which may be a submission from a script that does not check its exit status.
Client programs do not always follow a growth
In the shipped release, client programs do not reliably follow every kind of segment growth. Specifically, the address published for readers after a growth is not always the address of the new segment; on Xi-Batch this affects the variable segment. The observable effect is that after a growth of that segment, commands that read the variables may show stale contents or fail to open it, until the scheduler is restarted.
Set the initial allocation to cover your peak, so that the segments do not grow in normal running. Restarting the scheduler with a larger initial figure is the reliable way to enlarge a queue.
Checking what is allocated now
List the segments:
ipcs -m
Identify them by key: 0x5869b002 and 0x5869b003, plus 0x5869b200 if the file monitor has run. Keys move up by two while a segment is being replaced, so a segment may appear at the next key up after a growth.
The segments are owned by the batch system user, so an owner filter works as well:
ipcs -m | grep batch
The IPC tool reports the same segments and names each one:
xb-ripc
Check what the scheduler was started with:
ps -ef | grep btsched
Kernel parameters
Xi-Batch does not read SHMMAX, SHMALL or SHMMNI, and it has no tuning file of its own. On a current Linux kernel the shared-memory limits are effectively unlimited by default and there is nothing to change; the allocations are megabytes, not gigabytes.
Where a limit does bite - an older kernel, or a system where the limits have been lowered deliberately - the behaviour is:
- If the kernel rejects the size as too large, the scheduler halves its request and tries again. You end up with fewer slots than you asked for, with no message.
- If the kernel is out of memory or out of segments, the request fails outright and you get the refusal messages above.
- If a segment already exists at the key, the scheduler steps to the next key and retries, up to fifty times. Stale segments from an unclean shutdown therefore do not usually block a start - but they do confuse the client programs, so clear them with xb-ripc -d.
The other Unix platforms Xi-Batch supports - Solaris, AIX and HP-UX - do have shared-memory limits that can be set low, and there the kernel's own documentation is the place to raise them.
Changing the allocation
A change to the initial allocation takes effect at the next start of the scheduler. Stop the product, change the figure, start it again.
btquit -y
btstart -j 20000 -v 10000 -l 10000
Make the same change in /etc/xi/batchconfig or in the service unit, or the next reboot will go back to the old figure. Stopping the scheduler kills every running job.
Related articles
Sizing the Xi-Text Shared Memory Queue
What the start-up numbers allocate, when the segments grow, and what happens when they cannot
Controlling the Xi-Batch Job Start Rate with STARTLIM and STARTWAIT
Two settings that bound how many jobs the scheduler starts at once, where they live, and why a change needs a restart
Managing Xi-Batch IPC Resources
What the scheduler creates, how to inspect it, and how to clear it safely with xb-ripc after a crash