Onboard a customer project

How to bring a new customer’s tests into Orbit, from an empty project to a first run.

Who this is for

• System Admins, who create the project.
• Project Admins, who finish the setup, or add another environment later.

If you only run tests, you do not need this guide. Read Station run features.

Before you start

You need:

• A System Admin account with the authenticator (the code app) set up. Creating a project asks for the code once per session.
• The name of the customer and a short name for the project (lowercase letters, digits and dashes, 2 to 50 characters), for example example-customer.
• A test repository in the Orbit GitHub organisation, with the file orbit.workload.yaml in its root on the default branch. The next section says what that file needs.
• The address of the system under test (for example its base URL) and the names of its login secrets.

The words used here

• Console: the web site where you set things up and read results.
• Station: the program on a Windows PC that actually runs the tests.
• Customer: the company you test for. Optional. It only groups projects.
• Project: one customer’s testing space. It holds environments, workloads, people and results.
• Environment: one system under test, for example dev or qas. Tests run against it.
• Repository and workload: one test repository (Playwright tests). The setup steps say Repository (Add a repository); once registered, the repository is listed as a workload on Project settings, Workloads. One repository tests one environment.
• Manifest: the file orbit.workload.yaml in the repository. It tells Orbit the name of the workload, its team, and which variables and secrets the tests need.
• Team and role: people get access through a team. The team has a role on the project (Viewer, QA Engineer, QA Lead or Project Admin). See People and roles.

The order, and why

Project settings (in the menu), section Overview, lists the steps under Steps. Above them, under the progress bar, a strip says Next: , with one button that opens the right place. It always points at the first step not done and updates after every save. When nothing is left it says Ready: run a test on with Open Executions. If the next step is one you may not do, the strip says who to ask instead.

The list starts with Project (no number). It is done as soon as the project exists and does not count toward “ready”. It also shows the Customer: “Customer set” or “No customer yet” (optional; it only groups projects). A System Admin sees Set customer there, which opens Admin, Projects and customers, tab Customers. The numbered steps follow:

1. Environments. The repository check needs to find one.
2. Repository. Orbit reads the manifest, so it knows which names the tests need. You add it with Add a repository (see below).
3. Configuration. You can only enter the right variables and secrets once Orbit knows their names. Under it, a status line says Ready to run on when compatibility is fine, or says what is missing, with Open the compatibility matrix (the Workloads by environment table on Project settings, Environments). Compatibility is not a numbered step, but it still counts toward “ready”.
4. People. The first person also creates the team named in the manifest, if it does not exist yet.

Below the numbered list, under Next steps (marked with an arrow instead of a number), are the optional steps First run, an ad-hoc run on a Station (no release or cycle is needed), and Release and cycles.

You can do the steps in any order. The guide below walks them in this order and shows what is still missing.

Create the project and open the guide

1. Sign in as a System Admin. Open Admin, Projects and customers, tab Projects, and choose New project. The New project panel opens. (New project on the Project row of Project settings, Overview opens the same panel.)
2. Fill in:
   • Name: the display name, for example Example Customer.
   • Short name: lowercase, for example example-customer. It is filled in from the name; change it if you want.
   • Customer: pick one, or New customer… and type New customer name, or leave No customer yet. A System Admin can set it later in Details on Project settings, Overview.
   • Keep artifacts for: 7, 30 or 90 days. Traces, screenshots and videos are deleted after this. Results stay.
   • Create the project team: leave it ticked. It creates the team <short-name>-team with the QA Engineer role on the project.
3. Choose Create project. A message says “Created … and its team …”.
4. Project settings, Overview opens in guided mode (Guide me). It shows the steps across the top and one step at a time.

The Project row shows the project name, its short name and, when the project has its default team <short-name>-team, the team name. The row does not show the slug. Find the slug in the guide’s box (below), on Admin, People and access, Teams, tab Project teams (System Admins), or in the team: line of a manifest.

Write down the team slug. Right after you create the project, the guide’s Repository step shows a box that says “Put team: <slug> in orbit.workload.yaml of the repository”, with a Copy button. Usually the slug is <short-name>-team. If another team already holds that slug, a number is added (-2), so always copy the one shown.

Values to record

Key	Where it is found	Keep
Project short name	The Short name field; shown under the project name	Not secret
Team slug	The box in the guide’s Repository step, Admin, People and access, Teams (tab Project teams), or the team: line of a manifest (the Project row shows only the team name)	Not secret. You need it in the manifest.
Customer name	The Customer field	Not secret

Using the guide

• Only System Admins see New project. A Project Admin sees the Project row read-only.
• Each step has Back, Next and, for steps that are not done, Skip for now. Next works only when the step is done.
• Everything you save stays saved. You can leave with Leave guide and return later with Guide me on Project settings, Overview. Skipped steps are forgotten when you leave, but the page still lists what is open.
• The setup page shows each step as Done, To do, Blocked or Optional. The project is Ready when environments, workloads, configuration, compatibility and people are done.

What the manifest needs

The manifest is the file orbit.workload.yaml in the repository root, on the default branch. The person who writes the tests owns it. The parts that matter for onboarding:

schemaVersion: 1
id: example-customer-dev        # the workload name; unique across all projects; cannot change later
name: Example Customer Dev
team: example-customer-team     # the team slug shown after creating the project
environment:
  required: [BASE_URL]          # names of plain variables the tests read
  secrets: [APP_USER, APP_PASS] # names of secrets the tests read

(The real file has more keys: runner, install, profiles, metadata. Start from the template repository and run the validator, npx github:<your-org>/orbit-workload-tools validate.)

Rules:

• team: must be the team slug shown after you created the project. Orbit refuses to register a repository whose team has no role on the project.
• List every variable and secret by name, never by value. Orbit stops a run before it starts if one is missing.
• One repository per environment. A project can have several repositories (for example dev and qas), but each one tests one environment.
• Secret and variable names use capital letters, digits and underscores, and cannot start with ORBIT_.

environment.slug. The manifest can also say which environment the repository tests (environment: { slug: qas }). With the slug in the manifest, the stepper preselects the environment (or offers to create it); without it you pick the environment in the stepper’s Environment step. Current Stations and the validator know the key. If a very old Station or validator rejects it, ask the person who runs your Orbit installation to update them.

Test owners

Orbit shows an owner for every test (coverage by owner, owner filters, owner notifications). The Station works it out when it lists the tests of a commit, in this order:

1. The orbit.owner annotation of the test:

   test('creates a material', {
     tag: ['@id:MM-001', '@priority:P1'],
     annotation: { type: 'orbit.owner', description: '@anna.tran' },
   }, async ({ page }) => { /* … */ });

   The annotation must be static: in the test’s details (as above) or on a test.describe. An annotation added while the test runs (test.info().annotations.push(…)) is not seen when the tests are listed, so it is ignored. A test inherits the owner of its test.describe, and its own annotation overrides it; with several orbit.owner annotations, the last non-empty one wins. When a test runs in several Playwright projects, the first project in the config’s order that has an annotation decides.

2. CODEOWNERS. The entry for the spec file in .github/CODEOWNERS, else CODEOWNERS, else docs/CODEOWNERS at the repository root, read like GitHub reads it: the last matching line wins, and several owners are kept as one text (@org/qa @lead). A line without owners leaves the file without a CODEOWNERS owner.

3. The manifest team:.

The owner is text: Orbit does not check it against people or teams. A Station sends it when it lists the tests of a run’s commit, and sends the catalog again when the tests or their owners changed. So the first run of each workload after a Station upgrade fills in the owners of existing tests, without a new commit.

The Quality debt category No explicit owner counts the tests that have neither an orbit.owner annotation nor a CODEOWNERS entry. The manifest team: alone does not count, so add annotations or a CODEOWNERS file to bring it down.

Step by step in the guide

1. Environment

1. Choose Add an environment (or Add environment on Project settings, Environments). The Add an environment form opens.
2. Fill in Name, Kind (Development, QA, Staging, UAT or Production-like), and optionally Time zone, Language and Health check URL. The Short name (for example dev) is under Advanced; it is made from the name when you leave it empty, and it cannot change later.
3. Offline grace (days) is how long a Station may run without reaching the Console. 0 turns offline runs off.
4. Tick Restricted only for systems close to production. Then only Project Admins may run there, unless a person is given run access (see People and roles).
5. Choose Add environment.

2. Repository

You can add a repository at any time, also when the project is already ready: choose Add a repository on the Repository step of Overview, or at the top of Project settings, Workloads. The Add a repository stepper opens above the workloads list. The guide uses the same stepper. The stepper has four steps, Repository, Environment, Register and Configuration. Configuration comes last because Orbit can only list the names a repository needs once it is registered, and registering needs no values. Back and Next sit under the first two steps; Next is disabled until the step is complete, with the reason shown below the buttons. Once the repository is registered, Back is disabled. Close leaves the stepper.

1. Repository. In GitHub repository, pick the repository from the list (type to search the repositories Orbit can see; one that is already registered says “Already registered”), or choose Type owner/name instead and type it as owner/repo, for example <your-org>/example-customer-dev. Then choose Check repository. Nothing is saved yet. Read the lines (below), fix each Fix line, and check again.
2. Environment. Under Which environment does this repo test?, pick one environment of the project. There is no “None yet” option: a repository must test an environment. If the manifest names an environment.slug the project does not have, a box offers Create environment : you pick only the Kind, and the other values take the defaults (offline grace 3 days, not restricted). Otherwise use Create a new environment.
3. Register. Check the summary (Repository, Workload, Team, May run on) and choose Register repository. It is enabled when there is no blocking problem. Registering needs no values, and the stepper moves on to Configuration by itself.
4. Configuration. The last step. It shows the configuration panel (see “Configuration” below) for the chosen environment: one row per name the manifest needs. Skip it if you like (leave with Finish; names stay Missing and the Next strip on Overview keeps pointing at them). When you are done, choose Add another repository to start again, or Finish to close the stepper.

How the check lines read

Each line has a badge: OK, Fix (blocks registering) or Note (does not block).

Line	What to do
“… is not visible to the Orbit GitHub App.”	Check the spelling. The repository must be in the Orbit organisation and the GitHub App must have access to it.
“orbit.workload.yaml was not found on the default branch.”	Commit the manifest to the default branch, then check again.
“The manifest is not valid.”	The first five errors are listed. Fix them in the repository, then check again.
“A workload with this id already exists (…).”	The id must be unique in all projects. Change id in the manifest, or archive the other workload.
“Team … does not exist.” / “Team … has no role on this project.”	If it is the project’s own <short-name>-team: add the first person on Project settings, People with the QA Engineer role. The link Open People on the line takes you there, and adding the person creates the team. If it is another team: ask a System Admin to give that team a role on the project in Admin, People and access, Teams.
“The manifest does not say which environment it tests.” (Note)	The manifest has no environment.slug. Choose the environment in the next step, or add the slug (see above).
“Environment … is archived.”	Restore it on Project settings, Environments first.
“Environment … does not exist in this project.”	Use the Create environment … button on the line, or fix the slug in the manifest.
“Values the manifest asks for are not set yet.” (Note)	Lists the missing names per environment. It does not block registering. Fill them in at the last step, Configuration, after registering.
“Verify your authenticator first”	System Admins: choose Verify now, enter the code, try again.

If GitHub does not answer, wait a minute and try again.

Values to record (Add a repository)

Stepper step	Key	Where it is found	Keep
Repository	Repository owner/repo	GitHub	Not secret
Repository	Team slug	Manifest team:, or Team in the Register step	Not secret. It must match the manifest.
Environment	Environment short name	Manifest environment.slug, or Project settings, Environments	Not secret
Environment	Kind (Development, QA, Staging, UAT, Production-like)	Your decision for a new environment	Not secret
Register	Workload id	Manifest id	Not secret
Configuration	Variable and secret names	The rows of the panel (from the manifest)	Names are not secret
Configuration	Variable values (for example the base URL)	The system under test	Not secret, but readable by project members and kept in the version history
Configuration	Secret values	The system under test (its admin or the customer)	Secret, write-only, never shown again. Keep an offline backup (password manager) before you enter them.

3. Configuration

You enter the values in one of two places:

• The configuration panel: the last step (Configuration) of the Add a repository stepper, and the Configuration step of the guide. If the project has more than one environment, the guide shows an Environment picker above the panel; it opens on the first environment that still lacks something (chosen once, so saving never moves you to another environment).
• Project settings, Variables and secrets: Fill in configuration (on the Configuration step of Overview, or on the Next strip when it is the next step) opens this section.

In the configuration panel:

1. The panel has one row per name the manifest needs for that environment (environment.required and environment.secrets of its repositories). Orbit already knows each name, so you only type values.
2. The type is fixed by the manifest. A name listed under required is a Variable and gets a text box. A name listed under secrets is a Secret and gets a password box, which is write-only. You cannot change the type in the row.
3. Each row shows Set or Missing (a variable that comes from the whole project shows Set from project). A secret that is set shows “Set, version N” with Replace, never its value. Names that are set but that no manifest asks for are listed under Not asked for by any manifest.
4. The values apply to the environment of the repository. For the other scopes (the whole project, or one repository on the environment) and the history of changes, open Advanced and choose Open Variables and secrets.
5. If a name that the manifest declares as a secret was saved earlier as a plain variable, the row is flagged (“Set as a plain variable”) and offers Move to Secrets. It writes the secret and removes the variable. Change the password afterwards: the old value stays readable in the variable’s version history. Orbit also refuses to save a plain variable whose name the manifest declares as a secret.
6. Choose Save. The Next strip then updates and points at the next thing to do.

Secrets must go under Secrets. Passwords, tokens and keys belong in secret rows. A value typed as a plain variable is readable by project members and stays in the variable’s history after you delete or replace it, so the only fix is to change the password at the source. Keep the real value in your password manager before you enter it.

On Project settings, Variables and secrets:

• Applies to chooses the scope, named plainly: “Whole project (every environment)”, “Environment X (every repository on it)”, “Repository R on X (only this repository)”. A repository value overrides the environment value, which overrides the project value. The section opens on the environment of the project’s repository (the whole project only when there is no repository yet).
• Variables lists the names the manifests ask for first, marked “from the manifest” with Set or Missing. Type the values (Add variable for a name no manifest asks for) and choose Save variables. History and changes shows the earlier versions.
• Secrets belong to an environment, so they show only when Applies to is an environment or a repository. Each secret shows “Not set” or “Set, version N”, never its value. Choose Set value, type the value and choose Save value (or Save new version to change it later). Add a custom secret adds one that no manifest asks for. A secret that is also saved as a plain variable is flagged “Also set as a plain variable”: set the secret, then remove the variable in the Variables list.

On the run page, the collapsed Configuration used section lists the variable and secret names (with secret versions) the Station received for that run, never their values, so you can tell whether the configuration reached it.

Values to record

Key	Where it is found	Keep
Variable names (for example BASE_URL)	Manifest, environment.required (the panel rows)	Not secret. Values are visible to project members and stay in history.
Secret names (for example APP_USER, APP_PASS)	Manifest, environment.secrets (the panel rows)	Names are not secret.
Secret values	The system under test (its admin or the customer)	Secret, write-only, never shown again. Keep an offline backup (password manager) before you enter them.
Repository owner/repo	GitHub	Not secret
Workload id	Manifest, id	Not secret

Compatibility (status line)

Under Configuration, a status line reports whether every repository has all it needs on its environment. Done reads Ready to run on . If something is missing, the line names the environment and the missing keys, with Open the compatibility matrix. Fix it in Configuration. It is not a numbered step, but the project is not Ready until it is done.

4. People

On Project settings, People (the step’s Open People), choose Add person. The panel opens on a Person box that lists every active account that is not on the project yet:

• Someone who already has an account: click the box, type part of the name, email or job title, and choose the person from the list.
• Someone new: choose Invite someone new instead, then enter the Email and the Name (needed for a new person). Choose someone who already has an account takes you back to the list.

Then choose the Role and, for a new person, when the Account expires (a person you picked from the list keeps their own expiry). The first person you add with the QA Engineer role creates the team <short-name>-team if it is not there yet. The full rules, including role limits and run access, are in People and roles.

5. First run

1. Open the Station on a PC that has access to the repository.
2. Start an ad-hoc run of the workload on your environment.
3. Come back to Executions to see the result.

No release or cycle is needed for a first run.

Summary

The Summary lists what is Set up and what is Still open. The text “Everything required is set up” means the project is ready for runs. Customers, releases and cycles are optional and can be added any time.

What “ready” means

The project is Ready when all of these are done: an environment exists, a workload is registered, every variable and secret named in the manifest is set, no workload lacks something on its environment, and at least one active person has a role. The customer, the first run and releases are not needed for “ready”.

Add another environment later (for example QAS)

1. Open Project settings. Use its sections, or Guide me on Overview.
2. Add the environment (Project settings, Environments, Add environment), or create it inline in the Add a repository stepper (its Environment step).
3. Make a second repository for it, with its own manifest (same team:, its own id, for example example-customer-qas). Choose Add a repository (it is always available, also when the project is ready), pick the new environment, and Register repository.
4. Fill in the new variables and secrets for that environment: the stepper’s last step, Configuration, or later Project settings, Variables and secrets with Applies to set to the new environment. Adding an environment re-opens Configuration and Compatibility until they are done.

Undoing mistakes

• Archive instead of delete. Archiving an environment, workload or project hides it, stops new runs, and keeps all history. Restore brings it back. Delete works only when nothing depends on it.
• Editable later: project name, time zone, retention and customer (Details on Project settings, Overview, then Save details; System Admins); an environment’s fields except its short name (Edit on Project settings, Environments); which environments a workload may run on (tick May run on on Project settings, Workloads); variables (new version each save); secrets (Replace or Set value, a new version); a person’s role and expiry.
• Not editable: a workload’s id and an environment’s short name. Archive and create again instead.
• A wrong customer can be changed in Details on Overview, or with Set customer on the Project row (System Admins).

If something goes wrong

• “Verify your authenticator first”: choose Verify now, enter the code, repeat the action.
• Register repository is greyed out: read the text next to it. It names the first blocking problem.
• The check says the team has no role on the project: see the Team row above. Adding the first person is the usual fix.
• A run fails at once with missing variables: open Project settings, Overview and follow the Next strip (Fill in configuration). Variables and secrets lists the names that are Missing (secrets: “Not set”). Set them and run again.
• A secret was typed as a variable: use Move to Secrets in the configuration panel, then change the password at the source. The old value stays in the variable history.
• The guide shows a step as “Blocked”: finish the step it depends on (for example, an environment before a workload).
• Project settings shows only Environments, Workloads and Test accounts: the other sections (Overview, Variables and secrets, Integrations, People, Data reset) are for Project Admins and System Admins. A section opened by a link says “Only Project Admins open …” and names who to ask.