Installation Manual - 03 HAProxy installation

How the gf-ocp HAProxy edge VM is created with cloud-init, configured as the first single-node edge endpoint, issued a wildcard certificate, and validated.

HAProxy is the third greenfield foundation service after MinIO and PowerDNS. It provides the first stable edge endpoint for the platform while later services are built behind it.

This chapter documents the single-node bootstrap design. Production HA is intentionally deferred until the second edge node and VIP plan are approved.

Target State

The DNS record already exists in the v7.comptech-lab.com zone:

haproxy.v7.comptech-lab.com A 59.153.29.102

Creation Flow

The VM is created from the Ubuntu 24.04 cloud image on the greenfield hypervisor. The first build uses cloud-init directly; the long-term target is to move this into GitLab-driven automation.

The bootstrap sequence is:

1. Confirm 59.153.29.102 and 30.30.200.102 are unused.
2. Prepare cloud-init user-data, meta-data, and network-config.
3. Create a NoCloud seed ISO with cloud-localds.
4. Create the qcow2 VM disk from the Ubuntu cloud image.
5. Create the VM with two virtio NICs:
   • br-real for 59.153.29.102/27
   • br33 for 30.30.200.102/16
6. Boot the VM and wait for cloud-init status --wait.
7. Validate SSH key access, service health, HAProxy config, DNS, and the public health endpoint.

Cloud-Init Network Config

The VM uses fixed MAC addresses so netplan can assign stable interface names and static IPs:

version: 2
ethernets:
  public0:
    match:
      macaddress: "52:54:00:70:07:02"
    set-name: enp1s0
    dhcp4: false
    addresses:
      - 59.153.29.102/27
    routes:
      - to: default
        via: 59.153.29.97
    nameservers:
      addresses:
        - 30.30.200.53
        - 1.1.1.1
      search:
        - v7.comptech-lab.com
  private0:
    match:
      macaddress: "52:54:00:70:07:03"
    set-name: enp2s0
    dhcp4: false
    addresses:
      - 30.30.200.102/16

Cloud-Init User Data

The user-data file sets the host identity and disables password-based SSH:

#cloud-config
hostname: gf-ocp-haproxy-01
fqdn: haproxy.v7.comptech-lab.com
manage_etc_hosts: true
ssh_pwauth: false
disable_root: true

The operator account is key-only:

users:
  - name: ze
    groups: [adm, sudo]
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    lock_passwd: true
    ssh_authorized_keys:
      - "ssh-ed25519 REPLACE_WITH_PUBLIC_KEY"

The first package set is intentionally small:

packages:
  - haproxy
  - qemu-guest-agent
  - cloud-guest-utils
  - dnsutils
  - chrony
  - curl
  - jq
  - fail2ban
  - net-tools
  - vim

HAProxy Bootstrap Config

The initial HAProxy config exposed a public readiness response and a private stats page. The current bootstrap config also routes GitLab, MinIO, and NetBox through the private backend network.

scripts/services/haproxy/haproxy.cfg

The stats listener is private only. Add authentication or restrict it further before exposing it outside the management network.

Wildcard Edge Certificate

The v7 edge uses one Let’s Encrypt wildcard certificate for all current and future app hostnames:

Certificate: *.v7.comptech-lab.com
Additional SAN: v7.comptech-lab.com
HAProxy PEM: /etc/haproxy/certs/v7-wildcard.pem

The certificate is issued with DNS-01 against PowerDNS. Certbot runs on gf-ocp-haproxy-01, publishes _acme-challenge.v7.comptech-lab.com TXT records on gf-ocp-pdns-01, then cleans them up after validation.

Hook files:

/usr/local/sbin/certbot-pdns-auth.sh
/usr/local/sbin/certbot-pdns-cleanup.sh
/etc/letsencrypt/pdns-known-hosts
/etc/letsencrypt/renewal-hooks/deploy/haproxy-v7-wildcard-pem.sh

The PowerDNS VM allows the hook user to run only /usr/bin/pdnsutil through sudo. The hook SSH key, ACME account material, and private certificate key stay on the servers and are not committed to Git.

Issue the wildcard certificate:

ssh ze@30.30.200.102 'sudo certbot certonly \
  --manual \
  --preferred-challenges dns \
  --manual-auth-hook /usr/local/sbin/certbot-pdns-auth.sh \
  --manual-cleanup-hook /usr/local/sbin/certbot-pdns-cleanup.sh \
  --cert-name v7-wildcard \
  --non-interactive \
  --agree-tos \
  --register-unsafely-without-email \
  -d "*.v7.comptech-lab.com" \
  -d v7.comptech-lab.com'

Validate renewal:

ssh ze@30.30.200.102 \
  'sudo certbot renew --cert-name v7-wildcard --dry-run'

Only the wildcard PEM should remain active in HAProxy’s certificate directory:

/etc/haproxy/certs/v7-wildcard.pem

Older per-host PEMs belong in /etc/haproxy/certs-disabled/, and older per-host Certbot renewal configs belong in /etc/letsencrypt/renewal-disabled/.

Validation

Confirm DNS points to the HAProxy VM:

dig @59.153.29.101 haproxy.v7.comptech-lab.com A +short
dig @30.30.200.53 haproxy.v7.comptech-lab.com A +short
dig @1.1.1.1 haproxy.v7.comptech-lab.com A +short

Expected answer:

59.153.29.102

Confirm public health:

curl -fsS http://59.153.29.102/healthz
curl -fsS http://haproxy.v7.comptech-lab.com/healthz

Expected answer:

gf-ocp-haproxy-01 ready

Confirm private stats:

curl -fsS http://30.30.200.102:8404/stats

Confirm HAProxy configuration:

ssh ze@haproxy.v7.comptech-lab.com \
  'sudo haproxy -c -f /etc/haproxy/haproxy.cfg'

Confirm SSH hardening:

ssh ze@haproxy.v7.comptech-lab.com \
  'sudo sshd -T | egrep "^(passwordauthentication|permitrootlogin|pubkeyauthentication|kbdinteractiveauthentication) "'

Expected values:

permitrootlogin no
pubkeyauthentication yes
passwordauthentication no
kbdinteractiveauthentication no

Operations

Reload HAProxy safely:

ssh ze@haproxy.v7.comptech-lab.com \
  'sudo haproxy -c -f /etc/haproxy/haproxy.cfg && sudo systemctl reload haproxy'

Check listeners:

ssh ze@haproxy.v7.comptech-lab.com \
  'sudo ss -ltnp | egrep ":(22|80|443|8404)"'

Check core services:

ssh ze@haproxy.v7.comptech-lab.com \
  'systemctl --no-pager --full status haproxy ssh fail2ban qemu-guest-agent chrony'

Frontend Backlog

Add concrete frontends and backends when each target service exists:

• Vault
• Quay or Harbor
• OpenShift API
• OpenShift apps wildcard

Production Hardening Backlog

• Add gf-ocp-haproxy-02.
• Add Keepalived or an equivalent VIP mechanism.
• Move HAProxy config into GitLab-managed automation.
• Add backend health checks, logs, metrics, and alerting.
• Protect the stats endpoint with authentication or tighter network controls.