Setting Up Xi-Batch Network Connectivity

Xi-BatchXi-Batchconfigurationlicensingnetworkingbtconn

Xi-Batch schedulers on two or more Unix hosts can be connected so that each one knows about the others' jobs and variables. This article covers the licence, the service entries, the hosts file and the start sequence. Article 23 covers the symptoms once it is set up - a variable that has not reached another host, a job held by a remote condition, a host that has gone away.

What the connected hosts can do

Once they are connected:

• A variable owned by one host is pushed to the others as it changes, so a job on one machine can be held by a condition on another machine's variable.
• A job marked remote runnable can be started by whichever connected host has the spare load capacity, rather than only by the host it was submitted on.
• A job can be submitted from one host into another host's queue.
• The Windows clients and the API library can reach the scheduler, because the daemon that serves them is only started in network mode.

A standalone machine running only the shell commands, btq and the X11 clients needs none of this.

The network licence

Networking is a licensed capability. Check it with:

xb-checklic

A network-enabled licence prints an extra line:

Licensed to:    Acme Systems
Serial is:      123456
Start time:     27/04/2013
End time:       No limit
Validated for networks

If Validated for networks is absent, nothing reports an error and everything below appears to work:

• The scheduler starts normally and never opens a listening socket.
• The daemon that serves the Windows clients and the API is never started.
• The export and cluster markers are stripped from every variable and every job as the saved files are read, so a previously networked installation loses all of them with no message.
• btvar -E reports success and does nothing.
• The MACHINE system variable is not created, which is the quickest test on a running scheduler - it is absent from btvlist.

Obtaining a network licence is a re-licensing operation through the customer portal; there is no setting that turns networking on.

The service entries in /etc/services

Xi-Batch resolves its ports by name from /etc/services, so the entries must be present on every host. The installation adds seven lines:

xibatch     2050/tcp    # Connection port
xibatch     2050/udp    # Probe port
btq         2150/tcp    # Feeder port
xbnetsrv    2250/tcp    # External job submission
xbnetsrv    2250/udp    # Client access
xbapi       2260/tcp    # Xi-Batch API
xbapi       2260/udp    # Xi-Batch API

What each one is for:

• xibatch 2050/tcp - the connection between two schedulers. This is the one that carries jobs and variables.
• xibatch 2050/udp - the probe, used to ask whether a host is there before a TCP connection is attempted.
• btq 2150/tcp - the feeder. It carries file contents between hosts: the job's script when one host runs a job another host owns, and that job's standard output and error files on the way back.
• xbnetsrv 2250/tcp and /udp - the external server, used by the Windows clients and by btr -Q submitting into this host's queue from elsewhere.
• xbapi 2260/tcp and /udp - the API. The same daemon serves it.

All hosts must use the same port numbers. When one scheduler connects out to another it uses the port number it resolved for itself, not one it negotiates, so a mismatch produces a connection that is refused with nothing written anywhere.

If the numbers have to be moved, change them consistently on every host and restart every scheduler. The scheduler will also accept the older names btsched (in place of xibatch) and btfeeder (in place of btq) if the preferred name is not found; if no feeder entry exists at all, the feeder port becomes the connection port plus one.

The hosts file

The hosts file is:

/etc/xi/batch-hosts

A HOSTFILE setting in /etc/xi/batchconfig overrides that path. Older installations may still have the file at /etc/Xibatch-hosts; the product's own warning message still prints that name, which is a known documentation defect.

The installation creates the file with a local-address line already in it. There is no editor program for it - edit it with a text editor as root. It is read only by the scheduler and by btconn, so a syntax mistake is put right with a restart.

The line format

Lines are blank, comments beginning with #, or entries of up to four whitespace-separated fields:

<host name or IP address>  <alias>  <flags>  <timeout>

• Field 1 - a host name that resolves, or a dotted IP address. A line consisting of this field alone is legal and means "connect to this host with no options".
• Field 2 - an alias you can use in place of the name in commands and in host:variable references. A single - means no alias. A line whose first field is a dotted IP address must supply an alias, or the line is rejected.
• Field 3 - flags, comma-separated with no spaces.
• Field 4 - a timeout in seconds, between 1 and 30000. The default is 1000. This is the keepalive interval: the scheduler sends a keepalive after twice this long with no traffic, so it is also roughly how long a peer that has died silently goes unnoticed. It has no bearing on how long a connection attempt waits.

The flags

Flag	Meaning
probe	Send a UDP probe and wait for a reply before attempting the TCP connection. Omit it if UDP is blocked between the hosts.
manual	Do not connect at scheduler start. The host is connected only by btconn.
dos, client	The entry is a client machine, not another scheduler. It connects inwards; this host never connects to it.
dosuser(name), clientuser(name)	A roaming user rather than a fixed host, mapped to the named Unix account.
external	An external client system.
pwchk	Require the connecting client to supply a password.
trusted	Accepted by the parser and read by nothing in this release. Setting it has no effect.

Any other word in the flags field is an error. It does not stop the file being read: the entry is kept, the unrecognised flag is discarded, and a warning goes to the scheduler's report file, /var/spool/xi/batch/btsched_reps, at the next start. The warning names the whole file rather than the line, so check the file by eye after editing it.

Across a firewall that blocks UDP, omit the probe flag: the probe is sent, no reply arrives, and the entry is dropped after its timeout with nothing logged. Without the flag the TCP connection is attempted directly.

The localaddress line comes first

Jobs and variables are identified partly by the IP address of the host that owns them, so the scheduler has to know which of its own addresses the other hosts see. Without a hosts file it works this out by resolving its own host name, which on many systems yields a loopback address and breaks everything that follows.

The localaddress line fixes it:

localaddress    192.0.2.17

The second field may be an IP address, a host name that resolves to one, or the form gsn(host,port), which opens a TCP connection to that host and port and reads the local address off the socket - useful when the machine has several addresses and you want the one it presents to the outside world.

It must be the first line in the file that is not blank and not a comment. If any entry precedes it, the line is discarded, the file is marked as having errors, and the scheduler falls back to resolving its own name - with no message beyond the general warning.

This is also why the hosts files on the different machines are not copies of each other: each one names its own local address first, then the other hosts.

A three-host example

Three hosts, prod1, prod2 and prod3. On prod1:

# Xi-Batch hosts file for prod1

localaddress    192.0.2.11

prod2           p2      probe
prod3           p3      probe
192.0.2.90      wanbox  probe,manual    300

On prod2 the same file begins with its own address, localaddress 192.0.2.12, and lists prod1 and prod3. Every host needs an entry for every other host it is to connect to.

Starting network mode

Nothing separate is started. On a packaged Linux installation:

systemctl start xibatch

which runs btstart, which starts the scheduler and, on a network licence, the external server as well.

Three properties of the start:

• The hosts file is read once. The scheduler reads it at startup and never again; it ignores SIGHUP. An edit takes effect at the next restart, or - for a single host - by running btconn.
• A failed connection is never retried. Each host is attempted once. A peer that is down at that moment is not connected to by this host at all until either the peer connects inwards or an operator runs btconn.
• A missing hosts file is silent. The scheduler starts, connects to nothing, and reports nothing.

Entries flagged manual, dos or client are skipped at startup.

The reliable sequence for a cluster is therefore to start every host, then run btconn for each peer from one of them. Run it again after any peer restarts: the restarted host attempts its own connections once, and a pair whose attempts both landed before the far side was listening stays unconnected until someone intervenes.

Connecting and disconnecting by hand

btconn prod2
btdisconn prod2

Each takes exactly one host name or alias and no options. Success is silent with exit status 0; a failure prints btconn - connection to prod2 failed. Both require the stop scheduler privilege, and both need a running local scheduler.

btconn will connect to a host that is not in the hosts file at all, as long as the name resolves. It refuses an entry flagged as a client, because a client must connect inwards.

Connecting triggers a bulk transfer in both directions of every locally-owned exported variable and exported job, which is why btconn is also the cure for two hosts that are running but disagree about a value.

On a packaged installation these two commands are not installed. They are alternative names for btstart, and only the tarball installer creates the links; the RPM and the Debian package create only the btdst link. If btconn reports "command not found" on an RPM or deb installation, create the links as root:

ln /usr/local/bin/btstart /usr/local/bin/btconn
ln /usr/local/bin/btstart /usr/local/bin/btdisconn

Use a hard link rather than a symbolic one and keep them beside btstart, because the program identifies itself from the name it was invoked as and needs its set-user-id bit.

Making jobs and variables visible

Each job and each variable carries its own export state, set independently of the host connection.

Variables have three states - local only, exported, and cluster:

btvar -E NAME          # export
btvar -L NAME          # back to local only
btvar -E -K NAME       # exported and cluster
btvar -E -k NAME       # exported, not cluster

Read the current state with btvlist -H -F '%N %V %E %K'; the default listing format does not show the cluster column. The export state can only be changed on the host that owns the variable.

Jobs have three states too:

btjchange -n <job number>    # local only
btjchange -F <job number>    # exported: visible on other hosts, runs here
btjchange -G <job number>    # exported and remote runnable

Or in btq, select the job, press u for the process parameters screen, and change the Export field. btr takes the same three options at submission.

Read the current state with btjlist -H -F '%N %H %e', where %e prints nothing, Export or Rem-runnable.

An exported job may only refer to exported variables. If it refers to a local one the change is refused.

Confirming the setup on each host

Work through these in order on each host.

1. The licence is networked.

xb-checklic

2. The service names resolve, and to the same numbers everywhere.

getent services xibatch
getent services btq
getent services xbnetsrv
getent services xbapi

3. The peer is connected. Running btconn against an already-connected host is harmless and is the only direct test there is:

btconn prod2

4. The other host's jobs are visible.

btjlist -R -H -F '%N %O %U %H %e'

%O is the originating host, so remote jobs are the ones whose value is not this machine.

5. The other host's variables are visible. Remote variable names carry a host: prefix:

btvlist -N -F '%N %V %E %K' | grep ':'

6. A job can be submitted into the other host's queue.

btr -Q prod2 -h "connectivity test" /bin/true

This route goes through the external server on port 2250 rather than through the scheduler connection, so it tests a different path from steps 3 to 5.

What to open in the firewall, and what crosses it

Which ports have to be open. Between two schedulers, 2050/tcp and - if the hosts carry the probe flag - 2050/udp; add 2150/tcp if either host is to run the other's jobs or view their text. Add 2250/tcp and /udp only for hosts that run the Windows clients or submit with btr -Q, and 2260 only where the API is used. Opening the whole 2050-2260 range everywhere is more than the product needs.

Traffic is not encrypted and not authenticated between schedulers. A host that can reach port 2050 and is named in the hosts file is treated as a peer. Use a VPN or a private network segment where the path between hosts is not trusted.

A job's owner and group cross the network as names, and are resolved on the receiving host, so a user who is to own jobs on more than one host needs an account of the same name on each; the numeric ids need not match. A name the receiving host cannot resolve falls back to the Xi-Batch service account, so a job whose owner is absent there runs as batch. Keep the account sets aligned.

Related articles

All articles · Release notes · Contact support