CI runner setup — spec, install steps, switchover

CI runner setup

The Keystone CI pipeline (.gitlab-ci.yml) runs on a shell-executor GitLab Runner — no Docker images, real PHP + Composer + Node on the host. Three jobs feed off it: test:php (Pest + deptrac + pint), test:frontend (npm ci && npm run build), and security:audit (composer audit + npm audit). A fourth, test:e2e, is manual-trigger only and needs Playwright + Chromium when you wire it up.

This article covers standing up a replacement or additional runner: hardware sizing, install steps, registration, and switchover from an existing runner.

When to do this

You want a new runner if:

• The current runner is memory-constrained. Symptoms: test:frontend failing with Killed mid-vite build (the Linux OOM-killer landing on Node); test:php hanging mid-run with no trace updates for tens of minutes (stdout buffer wedging under the parallel-tests dot-storm).
• You want to reclaim ops budget by moving from a hosted VM to local hardware.
• You want a stand-by runner so a failed primary doesn’t block every pipeline.

If neither applies, resizing the existing runner is cheaper than building a new one.

Recommended spec

Sized against the workload this pipeline actually generates. The current runner has hit Vite OOM four times across v0.8.x; the sizing below removes that failure mode and re-enables --parallel test runs.

Resource	Minimum	Recommended	Why
vCPU	2	4	php artisan test --parallel --processes=2 needs 2 cores; Vite + Composer install benefit from headroom.
RAM	8 GB	16 GB	Vite OOMs at ~1,600 modules: Node heap defaults to ~4 GB but RSS during transform spikes higher. Postgres + parallel PHP workers each need 250–500 MB.
Disk	40 GB SSD	80 GB NVMe	vendor/ ~500 MB, node_modules/ ~1 GB, Postgres data ~500 MB, plus the runner’s local cache for Composer + npm + Vite.
Bandwidth	50 GB/mo	100 GB/mo	Cold-cache composer install + npm ci ~300 MB per build; warm runs are much smaller.
OS	—	Ubuntu 24.04 LTS	Matches the existing runner’s mental model; ondrej/php PPA ships PHP 8.4.

Equivalents:

• Hetzner Cloud CPX31 (4 vCPU shared / 8 GB / 160 GB) — €16/mo. Closest like-for-like.
• Hetzner Cloud CCX13 (2 dedicated vCPU / 8 GB) — €15/mo. Slightly tighter on CPU but more consistent under load.
• A local VM at the same shape on Proxmox / Hyper-V / VMware — same recipe, no provider lock-in.

Network note. GitLab Runner polls jobs outbound over HTTPS. You don’t need to expose any inbound ports, so a runner behind a home/office NAT is fine. Stable outbound internet matters more than IP-fixedness.

Install walkthrough (Ubuntu 24.04)

Each block lives on the runner host as root or via sudo.

1. Base OS

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl ca-certificates gnupg lsb-release \
    git zip unzip jq build-essential software-properties-common
sudo timedatectl set-timezone Europe/London

2. PHP 8.4 + extensions

sudo add-apt-repository -y ppa:ondrej/php
sudo apt update
sudo apt install -y \
    php8.4-cli php8.4-pgsql php8.4-mbstring php8.4-xml php8.4-zip \
    php8.4-bcmath php8.4-intl php8.4-curl php8.4-tokenizer php8.4-redis
php -v

ext-ldap is deliberately skipped — the directorytree/ldaprecord packages only load when an LDAP auth provider is configured at runtime, and composer install in CI uses --ignore-platform-req=ext-ldap. Install php8.4-ldap only if you want to drop that flag.

3. Composer

curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer
composer --version

4. Node 22 + npm

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs
node --version && npm --version

5. Postgres

sudo apt install -y postgresql postgresql-contrib

# Create the test role + database with CREATEDB privilege (needed for
# `php artisan test --parallel` to spawn per-worker DBs).
sudo -u postgres createuser keystone_ci -P    # password: keystone_ci
sudo -u postgres createdb -O keystone_ci keystone_ci
sudo -u postgres psql -c "ALTER USER keystone_ci CREATEDB;"

# Local md5 auth so the CI connection string works without trust.
echo "host keystone_ci keystone_ci 127.0.0.1/32 md5" \
  | sudo tee -a /etc/postgresql/16/main/pg_hba.conf
sudo systemctl restart postgresql

6. GitLab Runner binary

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt install -y gitlab-runner
gitlab-runner --version

7. Register the runner

In GitLab → Project → Settings → CI/CD → Runners, click New project runner:

• Tags: alresford-ubuntu (must match the tag used in .gitlab-ci.yml, otherwise no job will pick this runner up)
• Run untagged jobs: off
• Lock to current projects: on
• Maximum job timeout: 1h (covers the slowest job: test:php with sequential tests at ~12 min, with margin)

Copy the registration token, then on the runner host:

sudo gitlab-runner register --non-interactive \
    --url "https://gitlab.com/" \
    --registration-token "GLRTX_<your_token>" \
    --executor "shell" \
    --description "keystone-runner-vm" \
    --tag-list "alresford-ubuntu" \
    --run-untagged="false" \
    --locked="true"

8. Tune output limits + concurrency

This is the bit that prevents the --parallel stdout-buffer wedge we hit on the old runner. Edit /etc/gitlab-runner/config.toml:

concurrent = 2            # 2 jobs at a time; matches 4 vCPU comfortably
check_interval = 5

[[runners]]
  name = "keystone-runner-vm"
  url = "https://gitlab.com/"
  token = "..."
  executor = "shell"
  output_limit = 16384    # 16 MB (default 4 MB); covers `--parallel`
                          # dot-storm + verbose composer output
  [runners.cache]
    Type = "local"
    Path = "/var/cache/gitlab-runner"

sudo systemctl restart gitlab-runner
sudo gitlab-runner verify   # confirms registration is live

9. Smoke-test

Trigger a manual pipeline on main from GitLab → CI/CD → Pipelines → Run pipeline. Confirm all three jobs (test:php, test:frontend, security:audit) land on the new runner and pass.

Switchover plan

When the new runner is healthy:

1. In GitLab → Project → Settings → CI/CD → Runners, click the old runner and Pause it (don’t remove yet — pause keeps the registration, stops it picking new jobs). Pipelines now flow only to the new runner.
2. Watch the next 5–10 pipelines on the new runner. Look for: any jobs hanging without trace updates, any non-deterministic test failures, any disk pressure on the runner host.
3. Once you’re satisfied, either Remove the old runner from GitLab (registration gone, can’t be re-enabled without re-registering) or leave it paused as a hot-standby.

If the new runner fails during the trial, un-pause the old one — the change is fully reversible until you remove the old registration.

Re-enabling --parallel tests

After the new runner is in place and output_limit = 16384, re-enable parallel test runs:

# .gitlab-ci.yml — restore the original test:php script line
- php artisan test --parallel --processes=2 --testsuite=Unit,Feature

Expected wall-clock drop: ~660s → ~400s for the test stage. If the trace stalls again, double output_limit and try once more before reverting.

Tuning Postgres for parallel tests

--parallel --processes=N spawns N PHP workers, each holding several Postgres connections. With the default Postgres config on Ubuntu 24.04 (max_connections = 100, shared_buffers = 128MB), --processes=4 runs out of shared memory and ~120 tests fail with SQLSTATE[53200]: Out of memory: 7 ERROR: out of shared memory. --processes=2 works at the defaults.

To raise to 4+ workers, tune /etc/postgresql/16/main/postgresql.conf:

max_connections          = 200          # default 100; ~50 per worker headroom
shared_buffers           = 1GB          # default 128MB; ~25% of system RAM
max_locks_per_transaction = 256         # default 64; parallel tests use lots of FKs
work_mem                 = 16MB         # default 4MB; speeds up SortMergeJoin in tests

Restart Postgres after editing:

sudo systemctl restart postgresql

Then raise --processes in .gitlab-ci.yml and verify a green pipeline before merging. With 12 GB RAM and 8 vCPU, the runner can comfortably go to --processes=6 once tuned.

Routine maintenance

• OS patches — monthly. sudo apt update && sudo apt upgrade -y && sudo systemctl restart gitlab-runner.
• PHP version bumps — when composer.json bumps the minimum, sudo apt install -y php8.x-... for the new version, then sudo update-alternatives --set php /usr/bin/php8.x.
• Node version bumps — re-run the NodeSource installer for the new version.
• Cache pruning — /var/cache/gitlab-runner accretes. Reset once a quarter or when disk is >80% full: sudo rm -rf /var/cache/gitlab-runner/* && sudo systemctl restart gitlab-runner.
• Disk monitoring — wire up a simple df -h alert. The runner fills disk slowly but predictably; you don’t want to discover this during a release.

When to add a second runner

A single runner is fine for normal flow; pipelines just queue when busy. Add a second runner (same tag, same project) when:

• You’re regularly waiting on pipelines because two MRs are in flight at once.
• You want the e2e job (Playwright + Chromium) on a dedicated runner so a flaky browser run can’t block test:php.
• You want geographic redundancy.

GitLab will round-robin jobs across runners with matching tags. No CI config change needed.