Why an Xi-Batch Job Will Not Start When CLOAD Is Below LOADLEVEL

Xi-BatchXi-Batchcapacityjob-controlschedulingtroubleshootingbtjchangebtjlistbtulist

Jobs start minutes or hours after their scheduled time, start at a different time each day, or fail to run at all, while CLOAD sits well below LOADLEVEL and raising LOADLEVEL changes nothing. The system-wide load level is one of nine tests the scheduler applies to each job on every pass through the queue, and it is the only one of the nine that any command reports. A job held back by any of the other eight looks exactly like a job waiting its turn: nothing is written to the report file, nothing is written to the job log, and no message is sent to anyone.

What the scheduler tests, in order

The scheduler walks the whole queue in working-priority order and applies these tests to each job in turn. The first one that fails ends the scheduler's interest in that job until the next pass.

#	Test	Effect when it fails	Visible as
1	The progress code is blank	Job skipped	%P is Done, Err, Abrt or Canc
2	The job's load level plus CLOAD is at most LOADLEVEL	Job skipped, scan continues	%L against CLOAD and LOADLEVEL
3	The job is not already proposed to, or held for, another host	Job skipped	Nothing
4	A job owned by another host is marked remote runnable and names a command interpreter this machine has	Job skipped	%e and %I
5	The scheduled time has arrived	Job skipped until then	%W
6	The owning user's running jobs, plus this one, come to no more than that user's total load level	Job skipped	Nothing
7	Every condition is satisfied	Job skipped	%c and %C against btvlist
8	Fewer jobs are in the process of starting than STARTLIM allows	The rest of the scan is abandoned	Nothing
9	After the job starts, CLOAD is still below LOADLEVEL	The rest of the scan is abandoned	CLOAD

Two properties of this list:

Test 2 skips the job and carries on
A job that would take CLOAD over the limit is passed over, but the scan continues, so a smaller job further down the queue starts ahead of it. The bigger job can be overtaken repeatedly. Tests 8 and 9 abandon the rest of the scan altogether, so jobs behind them are not looked at until the next pass.

Nothing is recorded when a job is passed over
There is no log entry, no report-file line and no message for any of the nine tests. The job log records events, and a job that was passed over produced none. A job can be skipped every minute for a year and leave no trace.

The second load limit: the owner's total load level

Test 6 is a load limit independent of CLOAD and LOADLEVEL, and no command reports it.

Each time the scheduler considers a job it re-scans the whole queue and adds up the load levels of every job with the same owning user id that is currently in progress state Init, Strt, Run or Fin and running on this machine. If that total plus the candidate job's own load level exceeds the owner's total load level, the job is skipped.

Three per-user limits exist. The shipped defaults are:

Limit	btulist code	Column	Default	What it does
Maximum load level	%x	Maxll	1000	Caps the load level of any one job, checked at submission
Total load level	%t	Totll	10000	Caps the combined load of that user's running jobs - test 6
Special load level	%s	Specll	1000	The load level a job gets when it does not take one from its command interpreter

With the shipped defaults a job takes 1000 and a user's total is 10000, so every user is capped at ten concurrent jobs whatever LOADLEVEL says. On a stock installation LOADLEVEL is also 10000, so the two limits bind together for a single busy user, and the per-user limit is the one reached first as soon as a second user is active.

The total load level is held as an unsigned 16-bit value, so it cannot be set above 65535. On a system whose LOADLEVEL has been raised into the hundreds of thousands, no user can ever be given a matching allowance, and the per-user limit becomes the binding constraint permanently.

Read the three limits with btulist:

btulist -H -F "%u %x %t %s"

The first line of the listing is the DEFAULT entry, which applies to every user without an entry of their own. Change a user's total with btuchange:

btuchange -T 20000 fred

The interactive program btuser shows the same table, with t and m to set a selected user's total and maximum, and T and M to set the defaults applied to new users. Changes are saved on exit.

A job whose progress code is not blank never starts

Test 1 is the cause of a job that did not run at all, and the one most often taken for a load problem.

The scheduler looks for work only among jobs whose progress code is blank. A job showing Done, Err, Abrt or Canc is examined only to see whether its auto-delete time has arrived, and is then passed over. That is true however many times its scheduled time comes round.

The schedule keeps advancing. When a repeating job ends, the scheduler advances its next-run time either way, and resets the progress code to blank only if the job finished cleanly. A job that exits outside its normal range is left in Err with tomorrow's date in the time column, so btjlist shows a job that reads as scheduled and healthy, and it will never run again.

Forcing such a job with f or g is refused for any job whose progress code is not blank, and the refusal message is "Job is running" whatever the code holds.

To find them:

btjlist -HR -F "%N %U %h %P %W %r %x"

Any job with a repeat specification in the %r column and Done, Err, Abrt or Canc in the %P column is parked. The %x column gives the exit code from the run that parked it. To make one runnable again:

btjchange -N 1420

In btq, r on the job does the same thing, and P offers the full list of progress codes.

How a job comes to be parked

Every one of these leaves the job with a non-blank progress code and no further attempt to run it:

• The job exited with a status outside the range configured as normal for it, giving Err
• The job was killed, by an operator or by the maximum-run-time guard, giving Abrt
• The scheduler could not fork a process for the job, giving Err
• Someone set it to cancelled and did not set it back, giving Canc
• The job ran once, had no repeat specification, and was retained, giving Done
• The scheduler was stopped, or the machine was rebooted, while the job was running

Stopping the scheduler sends SIGKILL to every job running on this machine. When the scheduler is started again it reads the saved job file, and every job that was recorded as starting or running is loaded back as Abrt. Each scheduler restart therefore parks every job that was running at the time, permanently, and the queue afterwards reads as normal apart from the progress column. Check the progress codes as part of the restart procedure.

A restart zeroes CLOAD and resets LOADLEVEL

CLOAD is a counter held in the shared-memory segment, and it is set to zero every time the scheduler starts, after the saved variable file has been read. Jobs that were running come back as Abrt, which also removes them from the per-user total, since that sum counts only progress states Init, Strt, Run and Fin.

A CLOAD figure that never falls to zero at a quiet period is work the scheduler holds as in progress within this run. The candidates are a job whose process never exits - a service submitted as a batch job - or a job proposed to a remote host that has not yet answered, since an outstanding proposal is counted in CLOAD too.

LOADLEVEL is also reset at every start. The installed start command supplies it explicitly, so the value in force after a restart is the one on the start command line rather than anything set with btvar since. This is covered in the load-levels article.

Working down the nine tests

The order below finds the answer fastest.

1. Rule out the parked-job case first.

btjlist -HR -F "%N %U %h %P %W %r %x"

A job with a repeat specification and a non-blank progress code has stopped for good, whatever the load figures say.

2. Read the two system variables. Each btvar command takes one variable name and prints the bare value:

btvar CLOAD
btvar LOADLEVEL

Where either reports "Unknown variable", the account you are using lacks read access on it: the two variables are created with the batch daemon's default variable modes, which give read access to the owner and the primary group only.

3. Do the per-user arithmetic. List what the scheduler counts against the owner, which is that user's jobs in Init, Strt, Run or Fin running locally:

btjlist -HR -u fred -F "%N %h %P %L %i %b"

Add the %L column for the rows whose %P column is Init, Strt, Run or Fin, and compare the total with that user's Totll from btulist. If the remaining headroom matches the concurrency you observe in practice, test 6 is the limit being hit.

4. Check the conditions. A job with conditions waits until every one is satisfied:

btjlist -HR -F "%N %h %C"
btvlist -H

Compare each condition's variable name and comparison against the variable's current value. A condition on a variable that no longer exists, or that the job's owner cannot read, holds the job for ever.

5. Check the scheduled time and the missed-occurrence rule. A repeating job whose due time passes while it is held back is handled according to its reschedule option. Where that option is set to skip, and the delay exceeds the margin the scheduler allows, the occurrence is abandoned: the time advances to the next occurrence, the job moves to the back of the queue, and nothing is recorded. That is the mechanism behind "daily jobs occasionally do not run at all" on a system that is otherwise healthy.

6. Consider the start rate last. STARTLIM caps the number of jobs in the process of starting at one instant, and when the cap is reached the scheduler abandons the rest of the scan. It delays starts; it does not prevent them. It is a master-configuration setting rather than a variable, and no command reports the value the running scheduler is using - see the article on STARTLIM and STARTWAIT.

Confirming from the job audit trail

Where LOGJOBS is set the log settles several of these questions, because it is the only place in the product that records a full date including the year.

Each line is pipe-separated: date, time, job number, job title, event, user, group, priority, load level. The events relevant here are started, completed, error, abort, cancel, exceeded runtime and exceeded grace period.

To read one job's history, reading the file whose name is the value of LOGJOBS:

awk -F'|' '$3 == "1420"' `btvar LOGJOBS`

The job number in the log is written without leading zeroes, which is a different form from the zero-padded %N column btjlist prints. A remote job appears as host:number. Matching the two by string comparison therefore needs the padding stripped.

The log records what happened, so a job that was passed over produces no line and its absence proves nothing on its own. The scheduler writes it from the moment LOGJOBS is set, and it covers no period before that.

It does settle concurrency: count how many of one user's jobs are between a started and a completed line at any moment. A count that never exceeds a fixed number over days of operation is the user's headroom.

Clearing the block and keeping the queue moving

Reset a parked job
btjchange -N on the job number, or r in btq. Where the progress code is Run, confirm the job's process is absent first: resetting a job whose process is still alive can result in two copies running.

Release capacity in a controlled window
Making a large number of jobs runnable at once will let them all start together. Either work in small batches, or lower LOADLEVEL temporarily so that the backlog drains gradually.

Set a maximum elapsed run time
In btq, u opens the process parameters for a job and R sets the maximum permitted elapsed run time. An over-running job is then killed automatically with the signal set by K, after the grace period set by g. This is the only mechanism that ends a job which never exits.

Set the reschedule option deliberately
A repeating job set to skip missed occurrences drops one whenever it is held back past the margin, with nothing recorded. Where every occurrence matters, choose catch-up.

Enable the job audit trail
The LOGJOBS log is the only durable record of when a job last ran. The backup tool xb-cvlist omits system variables, so LOGJOBS is set again by hand after a restore.

Keep services out of the queue
A process intended to run continuously holds its load level, and its owner's allowance, for as long as it runs. It belongs under a service manager.

The commands in one place

Find parked jobs:

btjlist -HR -F "%N %U %h %P %W %r %x"

Make a parked job runnable again:

btjchange -N <jobnumber>

Read the two system variables:

btvar CLOAD
btvar LOADLEVEL

Read the three per-user limits:

btulist -H -F "%u %x %t %s"

Raise a user's total load level:

btuchange -T <n> <user>

List one user's running jobs with their load levels:

btjlist -HR -u <user> -F "%N %h %P %L %i"

Test one job's progress state in a script:

btjstat -s Err,Abrt,Canc <jobnumber>

Related articles

All articles · Release notes · Contact support