Run lock and side-by-side versions

Several YTCC versions can be installed on one server, each in its own folder with its own ports and databases (see Install versions side by side). Their process managers can run at the same time, but two rules keep them from driving the field together:

• On one computer, only one project runs at a time, in whichever version line.
• The same project runs in only one version at a time, also across computers that share a PostgreSQL server. This is the run lock.

Two versions driving the same field devices could otherwise send conflicting commands.

One project per computer

The rule always applies; there is no setting to turn it off. A concurrency = "multi" line in an older config\projects.toml is read but not applied, and the process manager logs a warning at start.

Before a project starts, the process manager takes a machine lock in PostgreSQL and keeps it until the last process of the project has stopped. The lock is bound to the computer name, so two computers that share a PostgreSQL server do not block each other. Of two projects started at the same moment, for example when the services of two lines start together with Windows, only one runs. If the process manager ends unexpectedly, the lock is released by itself.

The process manager also asks the other YTCC installations on the computer which projects they run, to cover installations that do not take the machine lock. It asks only the installations registered by the setup program in the Windows registry (HKLM\SOFTWARE\YTControl\YTCC), under other installation roots too, and an installation made with a setup program from before version lines (C:\YTCC). A folder that only looks like an installation, for example one made by hand under C:\ytcc, is not counted. If an installation does not answer, the start is refused rather than allowed: the message says that the state of that installation could not be read. Check that installation in Settings › YTCC installations on this computer on the home page, or stop its service in Services, then start again.

When several projects are marked to start with the stack (autostart), only the first that can start is started; the others are skipped and logged.

If another project runs, the start is refused and nothing is stopped. The message names the installation and the running project. Stop that project first, on the home page of its installation (open it with the shortcut of that line, for example YTCC v15 (Edge)), then start the new one. A refusal because of another installation is recorded like the other refusals (see Records). On the home page, starting a project of the same installation while another one runs stops the running project first, after the confirmation; see Start, stop and restart a project.

A takeover is the exception: Stop it there and start here… in the interface may start the project while the same project still runs in another version, because the takeover stops it there. Another project in another version still blocks the start.

The project is also refused when a port that one of its drivers listens on is in use, by another program or by another YTCC installation. No process of the project is started, and the port is not moved to another number. Free the port, or change it in the driver settings, and start again.

How the run lock works

Every project has a run lock in PostgreSQL. Before the first process of a project starts, the process manager of the version takes the lock. It releases the lock only after the last process of the project has stopped. If a process does not stop, the lock stays held until it has exited.

• The lock is held on a connection of its own to the PostgreSQL server. If the process manager ends unexpectedly, the connection closes and the lock is released by itself.
• The lock belongs to the project's identity, not to its name. A project taken over into a new version keeps its identity, so the old and the new copy share one lock. A project imported with --new-identity is a separate project with its own lock.
• The lock works between all YTCC installations that use the same PostgreSQL server, also when they run on different computers.

If the project already runs in another version, the start is refused. No process of the project is started, and no other running project is stopped because of it.

What the interface shows

When you start or open a project that runs in another version, the window Project is running in another version opens. It names the version, its folder, the process ID of its process manager and the time since when the project runs there. Cancel closes the window.

Server administrators also see Stop it there and start here…. After a confirmation (Communication is interrupted for a few seconds; commands are not sent during that time., buttons Cancel and Stop and start), YTCC asks the other version to stop the project and shows how long it has been waiting. It waits up to 15 seconds.

• If the other version stops the project in time, the project starts here and Project started in this version. appears.
• If it does not stop in time, the window says that the project was not started here and that you should close it in the other version. Ask again sends the request once more.
• If the project runs on another computer, no request is sent. Stop the project on that computer first.

See Server administrators for who is a server administrator.

Taking a project over from the command line

The interface does the same with Stop it there and start here…. On the command line, for example in a script, a support engineer uses, in a command prompt on the server:

.\scada-manager start-project <project> --takeover
.\scada-manager start-project <project> --takeover --wait-secs 30

--takeover sends the other version a request to stop the project and waits for the lock, by default 15 seconds. The other version stops the project itself and records this. Nothing is ever stopped by force: if the other version does not answer or does not stop in time, the project is not started here. A version accepts such a request for the same project at most once every 30 seconds.

project adopt --takeover and project reclaim use the same request when a project moves between versions (see Upgrade to a new version and roll back).

When the previous version still seems to run

A version that holds the lock writes a heartbeat for the project every 2 seconds. If the lock has just been taken but the previous version's heartbeat for the project is still fresh, the project is not started at once. YTCC keeps the lock and starts the project by itself as soon as the heartbeat stops. The interface shows Waiting for the previous version to stop; the project will start by itself within N s. If the other version keeps running the project for 5 minutes, the start is cancelled, the lock is released and a refusal is recorded.

When the lock is lost

The process manager checks the lock connection every 5 seconds, and again before it starts or restarts a process of the project. If the connection is lost, for example because PostgreSQL restarted or the network failed:

1. From that moment, no process of the project is started or restarted. A process that would be restarted waits with the status blocked.
2. The process manager tries to take the lock again for 30 seconds. Support can change this time with run_lock_grace_secs in config\manager.toml.
3. If it gets the lock back, the waiting processes start again by themselves.
4. If it does not get the lock back, or another version has taken it, the project is stopped and the event is recorded as lock_lost.

Older YTCC versions

YTCC v1.5.0 and earlier do not take the run lock. If such an installation runs on the same server, support enters its control port in config\manager.toml of the newer version, in a text editor opened as administrator:

legacy_control_ports = [5000]

Before it starts a project, the newer version then asks the older installation. If the older one runs the project, the start is refused with the message that the project runs in an older YTCC installation. An installation made with a setup program from before version lines is still found by the one-project rule above when it runs any project.

Records

Every start, stop, refusal, takeover and lock loss is written to the project database (t_run_events) with the time, the user and the versions involved. Requests from the command line are recorded with the user cli:<Windows user>.