Station run features

How to choose run options, read a run that uses several browsers, understand the time limit, and refresh the test list on the Station.

Who this is for

Testers (QA Engineers and above) who start runs on the Station. A short section at the end is for workload authors.

Before you start

• The Station is the program on your Windows PC that runs the tests. You open it in a browser. The Console is the web site that holds projects, results and reports.
• You must be signed in on the Station with an account that has the QA Engineer role (or higher) on the project. A Viewer can look but not run.
• A workload is one test repository. A suite is a folder of tests inside it. The profile is a named set of defaults for a run (for example smoke), defined in the repository.

Run options

The run options appear in three places: the Explorer (where you tick tests), the Run page, and the Re-run… dialog. On the Run page the fields are part of the page. In the Explorer and the Re-run dialog they sit in the Run options panel, which is closed by default: the line under its title sums it up, for example “smoke · 2 workers · headed”. Choose it to open it.

Field	What it does
Workers	How many browsers the run may keep open together. Each suite gets its own group of workers. Tests with a lock tag in the same suite always run one at a time.
Retries	How many times a failed test runs again (0 to 5).
Time limit	Auto (the Station works it out) or Fixed in minutes (1 to 720).
Browser	Visible opens the browser windows on this PC so you can watch. Hidden runs do not take over your screen. It does not change results.

Every field starts from the profile, and says so (“from smoke”). When you change one, it says changed with a reset link that puts the profile value back. The panel title also shows a changed tag, and a line at the bottom names the changes (“1 change from smoke. Changed runs are out of the quality gates and metrics.”) with Reset all.

The number of workers is also capped by the Station. See Settings, Runs, Workers on this Station, which is shared by all runs on this Station. If your run asks for more, the field says “capped at N by this Station”.

“Overridden” and why it matters

If you change Workers, Retries or Time limit, the run is marked Overridden in the Console (“Overridden: … (excluded from quality gates)” on the run page). Overridden runs are left out of quality gates, release readiness and trend metrics, because their numbers are not comparable with normal runs. They still appear in the run list, with the names of what you changed.

• Browser (Visible or Hidden) does not make a run overridden.
• An Auto time limit does not make a run overridden. A Fixed one does.
• Overriding plain variables for one run (on the Run page) also marks the run as overridden.

If the run is meant to count (for example for a release), leave the options on the profile values.

A run with several suites

When a run covers more than one suite, the live view shows more:

• The strip of cells has one pulsing cell for each test running now.
• Running now lists each running test with its id, a short title, the suite and how long it has been running (for example MM-AR-013 · Rule toggle… · 02:41).
• Suites has one row per suite: progress (finished of total), how many failed, its state, how many workers it uses and a Report link.

A suite’s state is one of:

State	Meaning
Waiting for a free worker	The Station’s workers are all in use. It starts when one is free.
Waiting for lease …	Another run holds the lock this suite needs. It starts when that run finishes.
Running	Tests are running.
Done	All its tests finished.
Stopped	The run ended before all its tests finished.
Did not start	The run ended while it was still waiting.

The Log shows one line per event, with the suite name in brackets. When there are several suites, filter the log with the suite buttons above it (All or one suite).

Reports: each suite has its own report. In the live view and History use Open report: for each one. A run with one suite has a single Open report. A run with several suites does not upload a report to the Console; the reports stay on this Station.

The automatic time limit

Every run stops at a time limit, so a stuck run cannot hold a lock for ever.

• How it is worked out. For each test, the Station looks at how long it took in its last 10 runs on this Station. It adds retries, plans the suites on the available workers, and gets an estimate. The limit is the estimate plus half again, but never less than the profile’s own limit, and never more than 720 minutes (12 hours). With no history it assumes 2 minutes per test.
• Before the run the Run page shows, for example, “Estimated 38 min, limit 57 min”, and the Time limit field explains its Auto value. If you choose a Fixed limit below the estimate, you see “The limit is below the estimate”. You can still start the run.
• Only working time counts. The clock pauses while the run is only waiting for a lock or a free worker.
• At 80 % the live view shows “Most of the time limit is used”. The run keeps going.
• At the limit the Station stops the run. It ends with the reason run-timeout.

“Not run” is not a failure

Tests that were cut off, or never started, are recorded as not finished (the status Not run). They are not failures and do not count against the quality gates. Finished tests plus not-run tests always equal what you selected. The same applies when you cancel a run: the tests that were running are recorded as not finished.

What to do when a run is cut off

1. Look at the notice “N of M tests did not finish”. It says the run stopped at its time limit.
2. Start the missing tests again: tick them in the Explorer, or use Re-run… on the run.
3. If it happens often, choose Fixed and a longer time limit (this marks the run Overridden), or ask for a longer timeoutMinutes in the profile.
4. If one test hangs, look at its log first. A longer limit does not fix a hang.

Refresh from main

The Explorer lists the tests of the last list this Station made. It runs exactly that commit, so the tests you tick are the ones that run. The list is only updated when a run prepares, so it can be behind the repository.

To update it:

1. Open the Explorer and choose the workload.
2. Choose Refresh from main. The button shows it is working; you can keep using the page.
3. A message tells you the result:
   • “Tests updated: N tests, now at abc1234”: the list is new.
   • “Already up to date (abc1234)”: nothing to do.
   • “Offline: no newer commit could be fetched (using abc1234)”: you are offline, so the list is unchanged.
4. The line above the list shows the commit and when the list was made.

Things to know:

• Ticked tests that still exist stay ticked. Ones that disappeared are listed as no longer in the catalog.
• A refresh does not start a run, does not use a queue slot and has no run number.
• The Console learns about the new list at your next run.
• It can take a few minutes the first time, because the tests’ dependencies may need to install. It stops after 10 minutes.
• Refresh when you know the repository changed, for example after a colleague merged new tests or a manifest change, and before you start a run that depends on them.

If it fails, the Explorer shows “The test list could not be refreshed” with the reason. Errors you may see: refresh-running (a refresh of this workload is already working: wait), catalog-refresh-failed (read the reason; often the Playwright config needs a variable that is not set) and catalog-refresh-timeout (try again; check the network).

Account expiry on the Station

When your account expires you have 3 days of view only. In the Explorer, the environment list shows “view only” and the page says “Your account has expired. An expired account can only view. Ask an admin to extend it, then refresh the configuration.” Settings, Account shows “Expired; view only until …”. After the 3 days the account can do nothing. Ask a Project Admin or System Admin to extend the expiry (see People and roles).

For workload authors: locks

A lock tag, for example @mutates-config, marks tests that change shared configuration in the system under test. In orbit.workload.yaml:

locks:
  - { tag: "@mutates-config", scope: suite }

• scope: environment (the original rule): while one run holds the lock, no other run on that environment can run those tests.
• scope: suite: the lock only keeps runs from using the same suite at the same time. Two different suites can run together.
• A suite is the first folder under suites/ that contains the test file, for example suites/AUTO_MM01_ASSIGNMENT_RULE. A repository without suites/ uses the file’s top folder.
• Inside a suite that has lock-tagged tests, tests always run one at a time.
• When the Station lists tests, it loads your Playwright config with plain variables only. Never read a secret while the config loads.

Current Stations and the manifest validator know scope: suite. If a very old Station or validator rejects it, ask the person who runs your Orbit installation to update them.

If something goes wrong

What you see	What to do
“You cannot run on this workload”	You only have view access. Ask a Project Admin for the QA Engineer role.
“Your account has expired”	See Account expiry above.
Run ended with run-timeout	See “What to do when a run is cut off”.
A suite says “Waiting for lease …” for a long time	Another run holds the lock. Open Live or History to see it, or wait.
The Explorer lists old tests	Choose Refresh from main.
“Offline: no newer commit could be fetched”	Connect to the network and refresh again.
The run is marked Overridden and you did not mean to	You changed workers, retries, the time limit or a variable. Run again with the profile values (Reset all).