Controlling the Xi-Batch Job Start Rate with STARTLIM and STARTWAIT

Xi-BatchXi-Batchcapacityconfigurationscheduler

When many jobs become ready at the same moment - scheduled for the same time, or released together when one variable changes - the scheduler can find itself starting a large number of jobs simultaneously. STARTLIM and STARTWAIT bound how many it has in the middle of starting at any one time. This article says what each does, how to read the value in force, and how to change it.

Both are settings in the master configuration file

STARTLIM and STARTWAIT live in the Xi-Batch master configuration file, /etc/xi/batchconfig. They are read once, when a program starts, so a change reaches the scheduler only when the scheduler is restarted.

They are not system variables. Older documentation presents STARTLIM and STARTWAIT as variables that btvar can read and set, and says that external packages can query and modify them that way. The scheduler creates no variables of those names, and it never consults a variable when deciding how many jobs to start. A command of the form btvar -s 10 STARTLIM fails with a variable-not-found error; adding -C creates an ordinary user variable of that name, which has no effect whatever on job starting. Use the configuration file.

STARTLIM limits jobs that are being started. The number of jobs running concurrently is governed by the LOADLEVEL system variable and the load value carried by each job, so a machine saturated by work already under way is a LOADLEVEL case.

How STARTLIM works

STARTLIM
The maximum number of jobs the scheduler will have in the process of starting at any one moment

Default value: 15

Starting a job takes work. For each one the scheduler locks the variables the job depends on, hands the job to one of its execution processes, and that process forks and execs the job. The job itself then does whatever it does at startup - opening files, connecting to a database, contacting another host.

The scheduler keeps a count of jobs it has begun to start and not yet had confirmed as started. A job joins that count when the scheduler hands it to an execution process, or - for a job that may run on another host - when the scheduler proposes it to that host. It leaves the count when the job is confirmed started, when the proposal is answered, or when the start fails.

Each time the scheduler reviews the queue it walks the jobs in queue order, which is priority order, and for each job that is otherwise ready to run:

1. If the count of jobs already being started has reached STARTLIM, the scheduler stops reviewing the queue there and arranges to come back to it.
2. Otherwise it starts the job, adds it to the count, and moves on to the next job.

A job normally leaves the count within a fraction of a second, so on an unloaded system the limit is reached only in bursts, and the scheduler resumes as soon as the first of those jobs reports that it has started.

How STARTWAIT works

STARTWAIT
The longest the scheduler will wait, in seconds, before reviewing the queue again after STARTLIM stopped it

Default value: 30 seconds

STARTWAIT is a ceiling on the wait. The scheduler reviews the queue again whenever anything happens that might change it - and a job reporting that it has started is exactly such an event. In ordinary operation the scheduler therefore carries on well within a second of jobs clearing the count, and STARTWAIT never elapses.

STARTWAIT is the backstop: if the jobs being started are slow to report back, or something has gone wrong and they never do, the scheduler still returns to the queue after this many seconds. Other pending work - a job due to run, a time limit about to expire - can bring it back sooner.

Reading the values in force

No Xi-Batch command reports these values. They are settings rather than variables, so btvar and btvlist do not show them, and no btq screen displays them. Read them from the configuration file:

grep STARTLIM /etc/xi/batchconfig
grep STARTWAIT /etc/xi/batchconfig

The installation process writes both settings into the file with their descriptions. A setting whose value is the built-in default is written commented out, so a freshly installed file looks like this:

# Start limit.
# This is a limit to the number of jobs which can be started at once.
# STARTLIM = 15

# Start wait.
# This gives a wait time between each block of jobs to be started.
# STARTWAIT = 30

Read it as follows:

• A line STARTLIM = 20 - the value in force is 20.
• A line # STARTLIM = 15, or no line at all - the built-in default is in force, which is 15 for STARTLIM and 30 for STARTWAIT.

The file records what the next scheduler start will use. If the file has been edited since the scheduler was started, the running scheduler is still using the previous value, and nothing reports it.

Applying a change

Edit /etc/xi/batchconfig as root and uncomment or add the setting. Use an equals sign: a colon is accepted by the running product, and the installation process recognises the equals form alone, so a setting written with a colon is lost the next time a package upgrade rewrites the file.

STARTLIM = 8
STARTWAIT = 30

The value must be a positive whole number. Zero, a negative number or anything that is not a number causes every Xi-Batch program - the scheduler and the user commands alike - to print a message naming the file and the line and exit immediately. Check the file before restarting.

The scheduler reads the file only at startup, so the change takes effect when it is restarted. On a packaged Linux installation:

systemctl restart xibatch

Otherwise stop and start it by hand:

btquit -y
btstart

Stopping the scheduler kills the jobs that are running. btquit sends SIGKILL to every locally running job with no grace period; the queue entries remain, the work in progress does not. Confirm with btjlist that nothing is running, and restart during a quiet period. Batch several edits into one restart.

Keep your own record of the value and the reason, outside /etc/xi. Nothing in the product records it, a package upgrade rewrites the configuration file from its own template and discards any comment you added to it, and a full package removal deletes the file, so a reinstallation writes a fresh one carrying the defaults.

Choosing a value

What decides the value is how long each of your jobs takes to get from being handed to an execution process to reporting that it has started, and that depends on the job. Work from observation.

Raise STARTLIM when:

• Jobs are lightweight and complete their startup quickly
• Bursts of ready jobs take noticeably long to all reach the running state
• No resource contention is observed at start times

Lower STARTLIM when:

• Load spikes sharply at the moment a group of jobs becomes ready
• Network or storage saturates during a burst of starts
• Jobs fail during their own initialisation when started in a group

STARTWAIT has an effect only when jobs are slow to confirm that they have started, since otherwise the scheduler resumes before it elapses. Lowering it makes the scheduler retry sooner in that case; raising it makes it retry later. How many jobs start at once is STARTLIM alone. Leave STARTWAIT at its default unless job startup on the machine is genuinely slow and the scheduler is seen to be idle waiting, or the retry itself is unwelcome.

Step 1: Watch a real burst. Identify a time when a group of jobs becomes ready together, and observe the system across it.

uptime
btjlist

Step 2: Change one setting at a time. Halve STARTLIM if starts are swamping the machine; the effect is on the size of the burst, and it should be visible on the next occurrence.

Step 3: Watch the same burst again, across several days before concluding anything. Job mixes vary by day of the week and by month end.

Two related settings in the same file bear on the same behaviour:

• EXECPROCS - the number of processes the scheduler runs to start jobs, three by default. This is what limits how quickly the jobs allowed by STARTLIM are actually got under way.
• LOADLEVEL - a system variable rather than a configuration setting, and the control over how many jobs run concurrently. See the article on load levels.

Example: a large group of network jobs

400 jobs are scheduled for midnight, each of which opens a connection to another host. At the defaults, the scheduler starts jobs from the front of the queue until 15 of them are in the process of starting, then stops. As each of those confirms that it has started, the scheduler is woken and starts another. The 400 jobs enter the running state in a rolling stream of at most 15 concurrent startups.

How long that stream takes depends on how long these jobs take to start - with the connection setup on the far side of it, that may be seconds each rather than milliseconds. Where the far host or the network is what is being overwhelmed, lowering STARTLIM narrows the stream. Where 400 jobs all running at once is the problem, LOADLEVEL is the setting to use.

Troubleshooting

Jobs are ready but not starting
Check LOADLEVEL and CLOAD first - the load level is the more common cause. STARTLIM bounds the burst alone.

btvar LOADLEVEL
btvar CLOAD

Jobs still all start at once
Confirm the setting is uncommented in /etc/xi/batchconfig, and that the scheduler has been restarted since the edit. Until it is restarted the running scheduler is using the previous value, and nothing reports that.

An Xi-Batch command exits immediately with a configuration error
The value given to STARTLIM or STARTWAIT is not a positive whole number. Every Xi-Batch program reads this file and refuses to run, so this affects the whole system. Correct the line named in the message.

btvar shows a STARTLIM variable
Someone has created an ordinary variable of that name, most likely by following older advice to set it with btvar. It has no effect on job starting and can be deleted with btvar -D STARTLIM.

Related articles

All articles · Release notes · Contact support