Diagnosing Xi-Batch Variable Propagation Between Hosts

Xi-BatchXi-Batchnetworkingtroubleshootingvariablesbtconnbtvlist

A variable set on one host shows an old value, or no value, on another. This article says how a value travels between hosts, what becomes of it when a host goes away, and how to tell which of those you are looking at. Article 41 covers setting the hosts file up; this article assumes it already exists.

Each variable has one owning host

Every variable has exactly one owning host - the machine it was created on. That host holds the master copy and pushes it to the other hosts. Each push is a one-way write made at the moment of the change.

A variable's export state decides who gets a copy. There are three:

State	What it means	Where the value lives
Local only	The default. The variable is never sent anywhere.	One copy, on its own host. Only jobs on that host can use it.
Exported	The owning host sends the variable, and every later change to it, to the hosts it is connected to.	One value, held by the owner. Every other host holds a read-only replica, referred to as ownerhost:NAME.
Cluster	Exported, and additionally every host is expected to have its own variable of that name. A job's reference is resolved at run time to the copy belonging to the machine the job runs on.	One value per host, with no master.

Cluster implies exported. Setting the cluster marker on a variable that is not exported is refused with "Cannot set NAME local and clustered". A cluster name must be unique on each machine - two users cannot each have one of that name - and a clustered variable cannot be renamed.

The three states are what the variable log records when the flag is changed, as the words Cluster, Exported and Local only. See article 18 for the variable log, and article 31 for variables in general.

How a change reaches the other hosts

For an ordinary change made with btvar or btq:

1. The change is applied to the master copy on the owning host.
2. As part of applying it, the owning host writes the new value to every host it is currently connected to, in one pass.
3. Each receiving host stores it in its replica. Nothing is sent back and nothing is acknowledged.

A change made against another host's variable is forwarded. Running btvar on host B against hostA:COUNTER sends the request to hostA, which applies it and then broadcasts the result to everyone, including B. A replica is never written locally, and where the owning host is not connected the request fails at once with "Specified host is off line".

Three consequences:

• Nothing polls. If a value looks old, waiting will not refresh it. Either the change never reached this host or it never happened.
• There is no catch-up. A host that was disconnected when a change was made never receives that change. It receives the variable's current value when a connection is next established, and nothing in between.
• A change made by a job is propagated as an instruction. The host that ran the job tells every other host to carry out the same assignment list itself, and every host that holds the job arrives at the same answer independently. A host that was disconnected at that moment does not, and keeps the old value until a connection sync replaces it.

Bulk transfer when a connection is established

When two schedulers connect - at startup, or when an operator runs btconn - each sends the other every exported variable it owns, together with its exported jobs. This is the only mechanism that brings a stale replica up to date. It is throttled: the scheduler pauses for the number of seconds in the MSGTXDELAY setting of /etc/xi/batchconfig after every MSGTXSIZE messages (defaults 2 seconds and 20 messages), so a large exported set takes a noticeable time to transfer.

Simultaneous changes on two hosts

The last change applied wins, and neither party is told. The owning host applies requests one at a time in the order they arrive, and the value everyone ends up with is the last one applied.

One guard covers requests made on the owning host itself. Such a request carries the change sequence number the requester read, and is refused where the variable has been changed since:

Some other user attempted to update variable COUNTER
at the same time. Please check and perhaps try again.

A request arriving from another host is not sequence-checked; the sequence number is not sent over the network. A read-modify-write done as two commands (read the value, compute, set it) from two different hosts at the same moment loses one of the updates, with no error and no log entry.

Where a counter must be correct, do the read and the write on the variable's own host, or give each host its own counter with a cluster variable and add the values up afterwards.

What happens when a host becomes unreachable

A peer is treated as gone when its scheduler shuts down cleanly, or when a read or write on the connection fails. Nothing times a peer out for silence: the keepalive is sent only after twice the per-host timeout has passed with no traffic (default 1000 seconds per host, so about 33 minutes), and an idle connection to a machine that has vanished may not be noticed until then.

When the local scheduler does decide a peer has gone, it goes through its variable table and, for every variable owned by that peer:

• if no local job refers to it, the variable is deleted. It disappears from btvlist entirely.
• if a local job refers to it in a condition or an assignment, it is kept as a placeholder: the name and ownership remain, the permissions are set to nothing, and the value can no longer be read.

Neither of those is written to the variable log or to the scheduler report file.

Jobs are then affected in three ways:

• A condition or assignment on such a placeholder that is not marked critical is skipped. The job is treated as though the condition were satisfied, and it runs. This is the default: a condition is not critical unless it was created as one.
• A condition or assignment marked critical holds the job indefinitely. The scheduler passes over it on every scan. The job's progress state does not change, no message is written anywhere, and there is no timeout. It looks exactly like a job waiting for its time.
• A job of yours that was running on the departed host is marked Abrt and appears in the job log with the event network-aborted. Its start-of-job assignments have already been made and its end-of-job assignments never will be.

The hold is released when the host returns and the real variable arrives. Where the returning variable's permissions no longer allow the job's owner the access the job needs, the condition or assignment is deleted from the job with no message, and the job then runs without it.

Marking a condition critical

btr -k marks the conditions given after it critical, and -K marks them not critical; -b and -B do the same for assignments. Each applies to the -c or -s options that follow it on the command line.

# Do not run this job at all if server2 is unreachable
btr -k -c 'server2:BACKUP_DONE=Yes' -h "Nightly report" report.sh

The marker only takes effect on a variable written with a host prefix. On a plain local variable name it is accepted and discarded.

Tracing where the value stops

Step 1: Confirm this host is running networked at all

Networking is part of the licence. Where it is not licensed the scheduler starts normally and runs local jobs with no error, and no variable can be exported. The symptom is that an export flag will not stay set.

Two checks. The licence itself:

xb-checklic

A networked licence prints the line "Validated for networks" among its output. If that line is absent, the licence does not cover networking.

Then whether the running scheduler is networked, which is the MACHINE system variable - it is created only when networking is available:

btvlist -F '%N %V' | grep '^MACHINE'

Where MACHINE is absent, this scheduler is not networked and nothing else in this article applies. The exported and cluster markers are stripped from every variable as the saved variable file is read at startup, and a later btvar -E reports success while leaving the flag off. Re-licensing the machine is the fix, and every variable that was exported has to be exported again afterwards.

Step 2: Check the variable's export state

The default btvlist format shows the export column but not the cluster column, so an exported and a clustered variable look identical in a plain listing. Ask for both:

# Name, value, export marker, cluster marker
btvlist -H -F '%N %V %E %K'

The export column shows the word Export or nothing; the cluster column shows Cluster or nothing. Remote hosts' variables are included by default and are shown with a host: prefix on the name; btvlist -L limits the listing to local ones.

To set the state:

# Exported: one value, held here, pushed to the other hosts
btvar -E -k COUNTER

# Cluster: every host has its own COUNTER
btvar -E -K COUNTER

# Local only
btvar -L COUNTER

There is no btvar -e and no btvar -v. Reading a variable takes no option at all - btvar COUNTER prints the value and nothing else.

Step 3: Check the hosts file

The hosts file is /etc/xi/batch-hosts. Older installations had /etc/Xibatch-hosts, and that name still appears in some of the product's own messages and in the manuals; the file the scheduler reads is the first one.

One host per line. Fields are separated by spaces or tabs, in this order: host name or IP address, alias, flags, timeout in seconds. Blank lines and lines beginning with # are ignored. A line may carry the host name alone. An alias of - means no alias, and a line giving a dotted IP address must supply an alias. Flags are a comma-separated list; the two that matter here are:

• probe - find the host with a UDP probe before connecting to it.
• manual - do not connect to this host automatically. It is connected only when an operator runs btconn.

If the file contains a localaddress line, giving the address this machine should present to the others, it must be the first line that is not blank or a comment. Anywhere else it is discarded and the file is recorded as containing errors. Because each machine needs its own localaddress line, the files on the different hosts are not copies of one another.

Errors in the file are reported once, when the scheduler starts, in the scheduler report file:

btsched: Warning: There are errors in the /etc/Xibatch-hosts file

A hosts file that is missing altogether produces no message; the scheduler connects to nothing.

The scheduler reads this file once, at startup. An edit has no effect until btsched is restarted, except that btconn will connect to a single host named in the edited file without one.

Step 4: Check that the two schedulers are connected

The connectivity check is btconn:

btconn server2

btconn asks the local scheduler to connect to that host now. It takes exactly one host name and no options, needs the stop scheduler privilege, prints nothing on success and returns exit status 0; on failure it prints "btconn - connection to server2 failed" and returns a non-zero status. btdisconn is the reverse.

ping and ssh are not the test. They exercise the network and the login path, and a host can answer both while having no Xi-Batch connection at all - because it is flagged manual, because its scheduler is stopped, or because the connection was lost and never retried.

A host connects to its peers once, at scheduler startup, and never retries. A peer that was down at that moment is not connected to again by this machine. Recovery depends on the other end starting up and connecting inwards, or on an operator running btconn. In a pair of machines that were rebooted in the wrong order this is the whole of the fault.

A periodic btconn against each peer, from a job or a startup script, recovers a peer that was down at start of day. It does nothing where the host is already connected.

The schedulers talk over the service entries installed with the product. Confirm they are present on both machines:

getent services xibatch
getent services btq

xibatch 2050/tcp        # Scheduler-to-scheduler connection
xibatch 2050/udp        # Probe, for hosts flagged probe
btq     2150/tcp        # Feeder port used by btq and the listing commands

A firewall between the two machines must permit TCP 2050 in both directions, and UDP 2050 as well where the probe flag is used. On a networked host, a missing xibatch entry stops the scheduler during startup.

Step 5: Compare the value on each host

btvar server2:BACKUP_STATUS

This reads the local copy. btvar attaches to the local scheduler's shared memory and does not contact server2, so it reports what this host holds. The prefix names the variable's owning host. Two failures look alike:

• "btvar: Unknown host name in 'server2:BACKUP_STATUS'." - server2 is not in the hosts file and is not resolvable. This is a naming fault.
• "btvar: Unknown variable server2:BACKUP_STATUS" - this host has no copy. Either the variable was never exported, or server2 went away and the copy was discarded.

To find out what the owning host holds, run btvar on the owning host. Comparing the two answers is the diagnosis:

# On the owner
btvar BACKUP_STATUS

# On the other machine
btvar server2:BACKUP_STATUS

Same value: the variable is propagating. Different values: the two schedulers are not connected, or were not connected when the change was made. Absent on the second machine: it is not exported, or the connection has been lost since.

Step 6: Read the scheduler report file

tail -50 /var/spool/xi/batch/btsched_reps

What is recorded there: the hosts-file warning at startup, scheduler start and stop, probe messages from an unexpected address, a reconnection to a host that was already connected, and fatal errors.

What is not recorded there: nothing about variables at all. A value arriving from another host is applied with no entry - not in this file and not in the LOGVARS variable log. A host connecting or disconnecting is not recorded. A dead host's variables being discarded is not recorded. Finding nothing here about a variable is the normal case and tells you nothing about whether the network is working. Use btconn and the value comparison in step 5 instead.

What each fault looks like, and its fix

The variable is not exported

Symptom: the variable is listed on its own host and is absent everywhere else.

# On the owning host
btvar STATUS
Ready

# On the other host
btvar hostA:STATUS
btvar: Unknown variable hostA:STATUS

Solution:

# On the owning host
btvar -E -k STATUS

# Confirm
btvlist -H -F '%N %V %E %K' | grep STATUS

The export flag can only be changed on the owning host; on any other machine the attempt is refused. Setting it broadcasts the variable to the connected hosts at once, so it should appear on the other machine immediately.

Where the flag will not stay set, go back to step 1.

The two schedulers are not connected

Symptom: the other host's variables have disappeared from btvlist, or an attempt to set one fails with "Specified host is off line".

Solution:

# Ask the scheduler to connect now
btconn server2

# Then check the value arrived
btvar server2:BACKUP_STATUS

If btconn fails, check in this order: the scheduler is running on server2; server2 is in /etc/xi/batch-hosts on this machine and this machine is in the file on server2; the xibatch service entry exists on both; TCP 2050 is open between them.

Where server2 is flagged manual in the hosts file, btconn after every restart is how it is meant to be used.

The value on the other host is old

Symptom: the owning host shows a new value, another host still shows the old one.

One of two things has happened: the two hosts were not connected when the change was made, or they are not connected now.

Solution:

# On the host with the old value
btconn hostA
btvar hostA:STATUS

Establishing the connection transfers the current value. Restarting a scheduler has the same effect, since it connects on startup, and it stops every job the scheduler is running.

Two hosts update the same variable and one update is lost

Symptom: a counter incremented by jobs on two machines advances by one where it should have advanced by two.

The cause is the read-modify-write. Two shell scripts that each read the value, add one and write it back will overlap, and the cross-host case is not detected.

Solution: let the scheduler do the arithmetic. A job assignment using += is applied by the scheduler as a single operation:

btr -h "Counted job" -f S -s 'counter+=1' work.sh

Here -f S makes the assignment happen at job start; without -f an assignment is made at start and reversed at the end of the run. Article 33 covers conditions and assignments in full.

Where mutual exclusion is needed, put a condition and an assignment on the same job, so that the scheduler tests and sets in one step.

A job ran when its remote condition should have stopped it

Symptom: server2 was down, and a job whose condition tests a variable on server2 ran anyway.

This is the default behaviour. When server2 went away, the condition's variable became an unreadable placeholder, and a condition on a placeholder that is not marked critical is skipped - the job is treated as though the condition were met.

Solution: mark the condition critical. Criticality is set when the condition list is stated, so the list has to be restated:

# Replace the job's conditions with the same one, marked critical
btjchange -y -k -c 'server2:BACKUP_DONE=Yes' 4207

-y clears the existing conditions and -k marks the ones that follow as critical. Check the result with btjlist -F '%N %C', which prints the conditions in full.

The job then waits instead of running. Waiting suits a job that must not run on stale data; running suits one whose purpose is to run every night.

A job with a critical condition never runs

Symptom: the job sits in the queue with an unremarkable state and never starts, while other jobs run normally.

A critical condition on a variable belonging to an unreachable host holds the job with no timeout, no message and no visible change of state. There is no command that shows this state directly, so it has to be inferred: list the job's conditions, look for a host prefix, and check whether that host is connected.

# What does the job depend on?
btjlist -H -F '%N %H %C' | grep 4207

# Is the named host's copy still there?
btvar server2:BACKUP_DONE

A placeholder has its permissions set to nothing, so btvar reports it as an unknown variable exactly as a deleted one would. Either answer means this host no longer holds a usable copy. Confirm with btconn: where it connects, the host was gone.

Solution: bring the host back and connect it, which releases the job:

btconn server2

Where the host is not coming back, either restate the condition without -k so the job runs regardless, or remove the condition:

# Drop the conditions entirely
btjchange -y 4207

Listing variables across the network

List the exported and clustered variables

btvlist -H -F '%N %U %V %E %K'

This lists every variable this host knows about, its own and its replicas of other hosts'. The name carries a host: prefix where the variable belongs to another machine.

List one host's variables

# Just this host's own
btvlist -L -H -F '%N %V %E %K'

# Just server2's, as this host sees them
btvlist -N -F '%N %V %E %K' | grep '^server2:'

The second of these shows this host's replicas, which are as fresh as the last connection.

What the variable log records

The LOGVARS variable log records changes that a host applies as a result of a local or forwarded request. It does not record a value arriving from another host, so the log on the receiving machine stays silent while the value changes underneath it. A network-wide picture cannot be assembled from one host's log; enable LOGVARS on the owning host, where every change to that variable is applied. Article 18 covers the log in full.

Confirming propagation after a change

# On the owning host
btvar -C -E -k -s "checked $(date '+%H:%M')" SYNC_TEST
btvar SYNC_TEST

# On the other host, immediately
btvar hostA:SYNC_TEST

The value is identical straight away: the change is written to the connected hosts as part of applying it. Where the second command shows the previous value, the two schedulers are not connected - run btconn and try again.

Note the argument order: btvar -s takes the value, and the variable name is the last argument.

Related articles

All articles · Release notes · Contact support