feat(proxy,deploy): split client cert/key, and let the chart issue both

[cubeorgdev]

Fixes #110
Fixes #112

Two halves of the same thing: the proxy learns to read a cert-manager-shaped Secret, and the chart learns to produce one.

1. Split client cert/key in the proxy (#110)

The proxy could only take its Control API client identity as one combined PEM. That file is something you have to build yourself, so it doesn't fit automated issuers: cert-manager only emits it via additionalOutputFormats (off by default before 1.15, and enabling it earlier needs a cluster-wide feature gate on the controller and the webhook), and kubectl create secret tls can't emit it at all. Vault and SPIRE write plain tls.crt + tls.key.

This adds HARNESS_PROXY_CLIENT_CERT_FILE/_PEM + HARNESS_PROXY_CLIENT_KEY_FILE/_PEM next to the existing identity vars, named after the pair the control-api already takes. Nothing changes on the server side and existing client.pem deployments render and behave identically.

To be clear, this isn't a bug fix — Identity::from_pem sorts PEM sections by kind, so the combined path already works with cert-manager's key-then-cert output. It's about dropping the 1.15 floor and working with any issuer.

Notes for review

• The two shapes are mutually exclusive, and half a pair is an error naming the missing half — a half-migrated deployment fails loudly rather than silently ignoring one setting.
• Identity::from_pkcs8_pem (the two-buffer constructor) is native-tls-gated and we're rustls-only, so the pair is joined in memory and handed to the same from_pem. No new deps, no crypto-provider ordering hazard.
• Blank env values now read as unset. Needed because a template that says "not configured" writes an empty string, and taking that literally would trip the new all-or-nothing check and the insecure-http guard. It also means the insecure-http guard no longer panics on an explicitly-empty var, which I think is the right reading of "configured".
• The reload fingerprint hashes whichever identity files are set, so a key-only rotation still advances it. The spawn gate is now "there is something file-backed to watch", which is the same condition the old two-var check was approximating.
• Torn writes across two files aren't a concern on k8s — Secret volume projection swaps the whole ..data directory atomically — and the existing last-known-good behaviour covers bind mounts.

2. Chart-issued certificates (#112)

Reading a cert-manager Secret still left the operator hand-writing the ~60 lines that produce one, including four values the chart already derives: the internal Service DNS name in the server SAN, the namespace, the two Secret names, and the fact that the client Secret is then split. Each of those renders fine when wrong and fails at handshake time.

So the chart now declares the Issuer and Certificate resources itself:

blue:
  internalTransport:
    mode: mtls
    # Ignored when certManager.enabled; the chart names its own Secrets then.
    serverSecret: ""
    clientSecret: ""
    clientSecretFormat: combined
    certManager:
      # Requires cert-manager in the cluster. The chart only declares Issuer and
      # Certificate resources; cert-manager generates and renews the keys.
      enabled: false
      # Leave issuerRef empty and the chart bootstraps a self-signed CA scoped to
      # this release. Point it at your own Issuer/ClusterIssuer to use an
      # existing PKI; it must be a CA-type issuer, because Blue needs the
      # `ca.crt` key that ACME issuers do not write.
      issuerRef: {}
      #  name: corporate-pki
      #  kind: ClusterIssuer
      duration: 2160h      # 90 days
      renewBefore: 360h    # 15 days
      caDuration: 87600h   # 10 years; only for the chart-created CA
      privateKey: { algorithm: RSA, size: 2048 }

A whole gateway install is then two lines, with no pre-made Secrets:

blue:
  internalTransport:
    mode: mtls
    certManager:
      enabled: true

which renders the self-signed → CA chain plus both leaves. The server SAN is filled in from the Service name:

kind: Certificate
metadata:
  name: blue-blue-internal-tls-server
spec:
  secretName: blue-blue-internal-tls-server
  commonName: blue-blue-control-api-internal
  dnsNames:
    - blue-blue-control-api-internal
    - blue-blue-control-api-internal.default.svc
  usages: [digital signature, key encipherment, server auth]
  issuerRef: { name: blue-blue-internal-ca, kind: Issuer }

Point it at an existing PKI and the bootstrap chain disappears entirely, both leaves carrying your issuerRef:

certManager:
  enabled: true
  issuerRef: { name: corporate-pki, kind: ClusterIssuer }

Notes for review

• Sprig genCA/genSignedCert was the road not taken. It re-rolls the CA on every helm upgrade unless you bolt on a lookup hack, and it puts key material in the release. Declaring intent and letting cert-manager own the crypto is the conventional path, and it's the same one Lago and LiteLLM's charts take.
• CA-type issuers only. Both sides read ca.crt out of their own Secret, and cert-manager writes that key only when the CA is known — true for SelfSigned and CA, false for ACME. Documented, not enforced; the chart can't tell what an issuerRef points at.
• Values-gated, not .Capabilities.APIVersions-gated. That helper is false under helm template with no cluster, which would silently drop every Certificate from rendered output and from verify-deployment.sh. The CRD prerequisite is documented instead. No cert-manager version floor either — this path uses no additionalOutputFormats, so it needs only cert-manager.io/v1, GA since 1.0.
• clientSecretFormat is derived, not validated. Helm can't distinguish "user set combined" from "user left the default", so failing on certManager + combined would reject every ordinary cert-manager install. It's forced to split and documented as ignored.
• Three named helpers (serverSecretName, clientSecretName, clientSecretFormat) mean no consumer template branches on the flag.
• Two new validations: certManager under insecure-http fails (nothing consumes the certificates), and issuerRef.name without kind fails — cert-manager defaults it to Issuer, which looks only in the release namespace and leaves the Certificate stuck Pending if you meant ClusterIssuer.
• No NetworkPolicy change. cert-manager writes Secrets through the API server and never dials a Blue pod; kubelet mounts them from outside NetworkPolicy scope.
• Flipping this on for a running mTLS install is a CA cutover, not an in-place upgrade — both Secrets get new content from a new CA and don't swap atomically. The 30s reload and last-known-good keep it from becoming an outage, but expect a short window of rejected handshakes. Documented as a one-time operator action rather than engineered around.

Tests

There were no Rust tests for any of the identity code. Added 8 covering the precedence table, each error case, split/combined equivalence, and the missing-trailing-newline join. One asserts the insecure-http guard lists every var client_identity_pem reads, so a future var can't escape the guard.

The tests use placeholder PEM stand-ins rather than real key material — gitleaks runs over the repo and the functions under test only select and join strings. I did verify against a real cert/key out-of-tree that Identity::from_pem accepts the concatenated bundle, including when the cert file has no trailing newline.

tests/e2e/docker-compose.yml can now point the proxy at the split pair (BLUE_E2E_PROXY_CERT_FILE/_KEY_FILE, documented in the e2e README) against files the fixture already generates. Default stays combined.

verify-deployment.sh gets six new renders: cert-manager on with no Secret names at all (3 Certificates, 2 Issuers, both derived secretNames, both SANs, split env vars and none of the combined one), an explicit clientSecretFormat=combined that must still render split, an external ClusterIssuer that must drop the bootstrap chain, and three negatives — issuerRef without kind, certManager + insecure-http, and the pre-existing "mtls with no Secret names" guard, which is the regression proving the relaxation didn't disable it.

I also diffed old-chart against new-chart renders for combined, split, and governance-only value sets: byte-identical, so existing installs are untouched.

cargo fmt --all --check, cargo clippy --all-targets, cargo test --workspace (38 binaries), helm lint, and scripts/verify-deployment.sh all pass.

🤖 Generated with Claude Code

[cubeorgdev]

This pull request was created from a Blocks session.

View in dashboard | View on desktop