Installation Manual - 04 GitLab installation

How the gf-ocp GitLab VM is created, installed, routed through HAProxy, validated, and prepared as the operational source of truth.

GitLab is the first control-plane application in the greenfield build. It comes after MinIO, PowerDNS, and HAProxy because it needs object storage, stable DNS, and an edge route before it can become the operational source of truth for Terraform, Ansible, Packer, GitOps, and CI/CD.

NetBox is still part of the platform plan, but GitLab comes first so NetBox, Vault, registry, runners, and OpenShift automation can be provisioned from GitLab-managed repositories instead of becoming isolated manual installs.

Target State

The DNS record points the public GitLab name to HAProxy, not directly to the GitLab VM:

gitlab.v7.comptech-lab.com A 59.153.29.102

The GitLab VM is private-only. It has no real public interface. Package installation and future updates use the 30-block gateway for outbound access.

Creation Flow

The bootstrap flow is:

1. Create or confirm the governance issue and phase.
2. Reserve 30.30.200.20.
3. Add gitlab.v7.comptech-lab.com to PowerDNS, pointing at HAProxy 59.153.29.102.
4. Create a cloud-init seed ISO for gf-ocp-gitlab-01.
5. Create a 100 GiB qcow2 VM disk from the Ubuntu 24.04 cloud image.
6. Boot the VM with one private virtio NIC on br33 for 30.30.200.20/16.
7. Install GitLab CE from the GitLab package repository.
8. Configure UFW so only HAProxy can reach GitLab HTTP/HTTPS.
9. Add the HAProxy gitlab.v7.comptech-lab.com backend.
10. Preserve the generated root password in local secret custody without printing it.
11. Validate UI, redirect behavior, GitLab internal checks, SSH hardening, and firewall posture.

Human Operator Scripts

The repository provides a small operator script entry point so humans do not have to copy long command blocks from the manual:

./scripts/gfctl.sh preflight
./scripts/gfctl.sh prepare-cloud-init --execute <vm-name>
./scripts/gfctl.sh cloud-init-iso --execute <vm-name>
cp scripts/vms/<vm-name>.env.example scripts/vms/<vm-name>.env
./scripts/gfctl.sh create-vm --execute scripts/vms/<vm-name>.env
./scripts/gfctl.sh validate-vm --execute ze@<vm-ip>

Commands that create local artifacts or infrastructure default to dry-run. Operators must pass --execute and type EXECUTE at the prompt after the matching issue, phase, and ADR gate are approved.

Prepared cloud-init files and evidence logs stay in ignored local paths under artifacts/ and evidence/. Secrets must stay out of scripts, env files, and Git history.

First Infrastructure Pipeline

The first GitLab CI infrastructure automation pipeline lives in:

platform/infrastructure/ansible-automation

It is intentionally validation-only. It does not create infrastructure, modify hosts, read secrets, or call Vault with a token.

The pipeline uses the dedicated runner tags:

gf, infra, ansible

Current stages:

Protected non-secret CI variables:

These values are not secrets. Future secret-bearing jobs must retrieve values from Vault through an approved short-lived pattern and must be protected/manual until the related service phase is approved.

Validated pipeline:

platform/infrastructure/ansible-automation pipeline 4

Both jobs passed:

lint:shell-and-yaml-hygiene
validate:ansible

NetBox Validation Pipeline

The first service-specific validation job is:

validate:netbox-inventory

It lives in the same GitLab project:

platform/infrastructure/ansible-automation

Planned NetBox metadata:

The job validates only inventory and schema intent. It does not create a VM, connect to a NetBox host, read secrets, or modify any live service.

Validated pipeline:

platform/infrastructure/ansible-automation pipeline 9

Successful jobs:

lint:shell-and-yaml-hygiene
validate:ansible
validate:netbox-inventory

NetBox Gated Plan Pipeline

NetBox now has a gated plan path in:

platform/infrastructure/ansible-automation

Pipeline jobs:

Plan artifact:

evidence/plans/netbox-plan.yml

Public planning examples:

bootstrap/cloud-init/gf-ocp-netbox-01.user-data.example.yaml
bootstrap/cloud-init/gf-ocp-netbox-01.network-config.example.yaml
scripts/vms/gf-ocp-netbox-01.env.example

Validated pipeline:

platform/infrastructure/ansible-automation pipeline 10

Pipeline 10 reached the expected manual state:

lint:shell-and-yaml-hygiene  success
validate:ansible            success
validate:netbox-inventory   success
plan:netbox                 success
apply:netbox-blocked        manual

No NetBox VM was created. Enabling real apply requires a separate live-change issue and explicit operator approval.

Network Config

version: 2
ethernets:
  private0:
    match:
      macaddress: "52:54:00:70:07:11"
    set-name: enp2s0
    dhcp4: false
    addresses:
      - 30.30.200.20/16
    routes:
      - to: default
        via: 30.30.0.1
    nameservers:
      addresses:
        - 30.30.200.53
        - 8.8.8.8
      search:
        - v7.comptech-lab.com

Cloud-Init Install Script

The cloud-init script installs the operating-system baseline, hardens SSH, enables UFW, adds the GitLab package repository, and installs GitLab CE. The private-only VM uses the 30.30.0.1 default gateway and DNS from network config for package access.

#!/usr/bin/env bash
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive

systemctl enable --now qemu-guest-agent
systemctl enable --now chrony
systemctl enable --now fail2ban
systemctl restart ssh

ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow from 30.30.200.102 to any port 80 proto tcp
ufw allow from 30.30.200.102 to any port 443 proto tcp
ufw --force enable

curl -fsSL https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | bash
install -d -m 0755 /etc/gitlab
cat > /etc/gitlab/gitlab.rb <<'GITLAB_RB'
external_url "http://gitlab.v7.comptech-lab.com"
# Greenfield bootstrap: disable bundled Alertmanager. External platform observability will be added later.
alertmanager["enable"] = false
GITLAB_RB
apt-get install -y gitlab-ce=18.11.3-ce.0
gitlab-ctl reconfigure

Preseed external_url and alertmanager["enable"] = false before installing the GitLab package. Appending the Alertmanager setting only after package install can make the first omnibus converge fail on Alertmanager advertise-address detection. This is not the platform observability plan; external observability will be added later.

HAProxy Backend

HAProxy routes the public GitLab host to the private GitLab backend:

frontend fe_http_public
    bind 59.153.29.102:80
    mode http
    acl host_gitlab hdr(host) -i gitlab.v7.comptech-lab.com
    acl path_health path /healthz
    http-request return status 200 content-type text/plain string "gf-ocp-haproxy-01 ready\n" if path_health
    http-request return status 503 content-type text/plain string "No backend configured yet\n" if !path_health !host_gitlab
    use_backend be_gitlab_http if host_gitlab

backend be_gitlab_http
    mode http
    option httpchk
    http-check send meth GET uri /users/sign_in ver HTTP/1.1 hdr Host gitlab.v7.comptech-lab.com
    http-check expect rstatus (2|3)[0-9][0-9]
    http-request set-header X-Forwarded-Proto http
    http-request set-header X-Forwarded-Ssl off
    server gitlab01 30.30.200.20:80 check

The health check must send the GitLab host header and the /users/sign_in URI in the same http-check send line. A malformed host/header string can make HAProxy mark the backend down and return 503 even when direct access to 30.30.200.20 returns 200.

Secret Custody

The generated initial root password is not printed and is not committed. It has been onboarded into Vault:

secret/greenfield/gitlab/admin/root

The original local custody file remains only as a bootstrap fallback:

/home/ze/secrets/greenfield/gitlab-root-password.txt

Rotate the root password after the first operational login and update Vault in the same change. The first human operator account is zahid; its password follows the default human login custody path:

secret/greenfield/bootstrap/default-human-login

The bootstrap service account is svc-greenfield-bootstrap. Its initial API token is stored in Vault:

secret/greenfield/gitlab/pats/svc-greenfield-bootstrap

MinIO Backups

GitLab backups upload to the greenfield MinIO object store through scoped credentials stored in Vault:

secret/greenfield/object-storage/minio/users/gitlab-backup

The backup target is:

The GitLab object-storage backup settings live in a root-only include file:

/etc/gitlab/gitlab-backup-object-storage.rb

/etc/gitlab/gitlab.rb includes that file:

from_file "/etc/gitlab/gitlab-backup-object-storage.rb"

Do not print or commit the include file because it contains the scoped MinIO secret key.

Run a manual backup:

ssh ze@30.30.200.20 'sudo gitlab-backup create STRATEGY=copy'

Validated backup object:

gitlab-backups/1778793572_2026_05_14_18.11.3_gitlab_backup.tar

Important restore note: GitLab backup archives do not include /etc/gitlab/gitlab.rb, /etc/gitlab/gitlab-secrets.json, or the root-only backup include file. Those files must be backed up through the infrastructure secret/config workflow before GitLab is considered restore-ready.

Config And Secrets Restore Custody

The required non-archive restore files are preserved separately as an encrypted MinIO object:

The encrypted archive contains:

/etc/gitlab/gitlab.rb
/etc/gitlab/gitlab-secrets.json
/etc/gitlab/gitlab-backup-object-storage.rb

Validation performed:

• the encrypted object exists in MinIO;
• the object checksum matches the Vault metadata;
• decrypting with the Vault-held passphrase succeeds;
• tar member listing shows the expected files without printing file contents.

Do not print the Vault value at secret/greenfield/gitlab/restore-custody/config-archive; it contains the encryption passphrase.

Operational GitLab Structure

The first operational GitLab structure was initialized under issue #308.

Users:

Groups:

Projects:

Every initial project is private, has an initialized main branch, and has main protected with maintainer-level push and merge access. Force-push is disabled.

Runner VM

Do not register CI runners directly on the GitLab VM. The first runner is a separate private-only VM:

Tags are gf, infra, ansible, tofu, packer, and openshift.

Installed baseline:

• GitLab Runner 18.11.3;
• qemu guest agent;
• chrony;
• fail2ban;
• UFW default deny incoming, allow outgoing, SSH only inbound;
• SSH password and root login disabled;
• ansible, git, jq, make, unzip, and Python tooling.

Do not run gitlab-runner list in shared logs. It prints runner authentication tokens from /etc/gitlab-runner/config.toml.

The first smoke validation was committed to platform/ci/gitlab-runner-config:

runner-smoke:
  stage: validate
  tags:
    - gf
    - infra
  script:
    - hostname
    - gitlab-runner --version
    - ansible --version | head -1
    - git --version
    - jq --version

Validation passed:

• initial smoke pipeline succeeded on runner id 1;
• that runner token was rotated immediately after a local validation command printed it;
• stale local runner config was cleared;
• rotated runner token is stored in Vault at version 2;
• post-rotation smoke pipeline succeeded on runner id 2.

Start with this shell executor for infrastructure automation. Add Docker or Kubernetes executors only after registry, certificate trust, and runner isolation rules are defined.

Restore Drill

A full restore drill was completed under issue #307 into a throwaway VM:

The drill restored /etc/gitlab/gitlab.rb, /etc/gitlab/gitlab-secrets.json, and /etc/gitlab/gitlab-backup-object-storage.rb before running the GitLab backup restore.

High-level restore command sequence:

sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
sudo gitlab-backup restore BACKUP=1778793572_2026_05_14_18.11.3 force=yes
sudo gitlab-ctl restart
sudo gitlab-rake gitlab:check SANITIZE=true

Validation passed:

• restore and source package versions matched 18.11.3-ce.0;
• restore and source inventory counts matched users=1 groups=0 projects=0;
• local restore VM sign-in returned HTTP 200 after Puma warmed up;
• local restore VM root path returned HTTP 302 to /users/sign_in;
• gitlab-rake gitlab:check SANITIZE=true completed with healthy core checks;
• production GitLab route still returned HTTP 200 after the drill;
• restore VM remained private-only on 30.30.200.21/16.

Observed non-fatal restore details:

• PostgreSQL extension ownership warnings appeared for pg_trgm, btree_gist, and amcheck, but the restore task exited 0 and completed successfully.
• Immediate HTTP checks can return transient 502 while Puma recreates its socket after restore; wait and recheck before treating this as a failure.
• Cloud-init should seed an explicit operator SSH public key. Reusing the first key from another host can create an inaccessible VM.

Validation

Confirm DNS:

dig @59.153.29.101 gitlab.v7.comptech-lab.com A +short
dig @30.30.200.53 gitlab.v7.comptech-lab.com A +short
dig @1.1.1.1 gitlab.v7.comptech-lab.com A +short

Expected result:

59.153.29.102

Confirm sign-in works through HAProxy:

curl -sS -o /tmp/gitlab-signin.out -w '%{http_code}\n' \
  http://gitlab.v7.comptech-lab.com/users/sign_in

Expected result:

200

Confirm root redirects to sign-in:

curl -sS -o /tmp/gitlab-root.out -w '%{http_code} %{redirect_url}\n' \
  http://gitlab.v7.comptech-lab.com/

Expected result:

302 http://gitlab.v7.comptech-lab.com/users/sign_in

Run GitLab internal checks:

ssh ze@30.30.200.20 \
  'sudo gitlab-rake gitlab:check SANITIZE=true'

Expected high-level result:

• GitLab Shell internal API is available.
• Redis is available.
• Gitaly is OK.
• Sidekiq is running.
• Database migrations are complete.
• Active users reports 1 before onboarding.

Confirm firewall posture:

ssh ze@30.30.200.20 'sudo ufw status verbose'

Expected posture:

• default incoming deny;
• SSH allowed;
• HTTP/HTTPS allowed from 30.30.200.102 only.

Confirm the VM is private-only:

ssh ze@30.30.200.20 'ip -br addr; ip route'
ssh ze@30.30.200.2 \
  'virsh -c qemu:///system domiflist gf-ocp-gitlab-01'

Expected posture:

• only the private 30.30.200.20/16 address is present;
• default egress route is default via 30.30.0.1 dev enp2s0;
• libvirt shows only the br33 interface;
• no br-real interface is attached to the GitLab VM.

Confirm private egress:

ssh ze@30.30.200.20 \
  'dig @30.30.200.53 gitlab.v7.comptech-lab.com A +short && curl -4 -fsSI https://packages.gitlab.com | sed -n "1p"'

Expected result:

• internal DNS returns 59.153.29.102 for the GitLab route;
• package repository HTTPS returns HTTP/2 200.

Bootstrap Gaps

Before using GitLab as the durable operational source of truth:

• define protected CI variables.

• TLS is not enabled yet.

• Root password must be rotated after first operational login.

• CI variables still need to be created.