[kaya-abdullah]

Description

LibreDB Platform installs on Kubernetes with Helm. charts/libredb-platform is the installer's Kubernetes mode driven by Helm: a bootstrap hook and the chart's jobs run the installer's own code, bundled into the chart as files/libredb-jobs.cjs.gz, so a Helm install generates and converges the same secret store, internal CA, PostgreSQL roles, Keycloak realm and app environments as a Docker install. The Studio chart comes in as a pinned dependency with governance on.

Beyond a first install, the chart covers upgrades with refused downgrades, Argo CD, OpenShift's restricted-v2 SCC, seal key and secret rotation through values, backup and restore, and an install without network access. Each of these was driven end to end on a real cluster; the testing section lists what each run did and measured.

The largest problem the tests found was in seal key rotation: the bootstrap job writes Studio's keyring from the secret store on every upgrade, so a key added by hand, as the runtime guide described, would have been removed by the next helm upgrade. The chart now drives the installer's own rotation through platform.sealKeys, with the same refusals as libredb-installer apply.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Code refactoring
• Performance improvement
• Test addition or update

Related Issue

No issue was opened; this delivers the "Kubernetes" item of ROADMAP.md.

Changes Made

The chart (charts/libredb-platform)

• templates/bootstrap-hook.yaml: the bootstrap Job as a pre-install, pre-upgrade and pre-rollback hook (Argo CD PreSync), with a ServiceAccount and Role that may create Secrets and ConfigMaps and replace only the installation's own. It renders the installation document from the values, converges the secret store and the internal CA, writes every Secret and ConfigMap the workloads read, records which workloads must restart, and refuses a downgrade or an immutable change. Exit codes 2, 3 and 6 fail the Job at once through podFailurePolicy, so a refusal takes seconds instead of the retries.
• templates/postgres.yaml: managed PostgreSQL 18 with verified TLS from the internal CA and a pg_hba.conf that accepts only hostssl with SCRAM, and the provisioning job that runs the installer's provisioning script. With postgresql.mode: external, a DBA runs the script the chart writes to Secret libredb-postgres-dba-script.
• templates/keycloak.yaml: managed Keycloak behind the installer's Caddy edge, which forwards only the realm's public paths; the Admin API stays in the cluster.
• templates/jobs.yaml: the migrate and keycloak-configure jobs, named after the revision and what they run (Argo CD Sync hooks), and the restart job (post-install, post-upgrade, post-rollback, Argo CD PostSync), which restarts only the workloads whose inputs changed and then records the release as the one the installation runs.
• templates/platform.yaml, templates/networkpolicy.yaml, templates/NOTES.txt, values.yaml, Chart.yaml, Chart.lock, README.md: Platform with its Ingress and upload volume, NetworkPolicies for PostgreSQL and Keycloak's Admin API, and every value documented.
• templates/_helpers.tpl: render guards that name the value when a required value is missing or two copies of a host or the issuer disagree.
• Every pod runs under the restricted Pod Security Standard; the Keycloak edge adds only NET_BIND_SERVICE, which its binary carries.

OpenShift

• global.compatibility.openshift.adaptSecurityContext (auto, force, disabled), the value the Studio chart already reads, leaves out the pods' fixed runAsUser, runAsGroup and fsGroup when the API server serves security.openshift.io/v1, so restricted-v2 assigns them from the namespace's range. Off OpenShift the rendered chart is byte for byte what it was without the value.

Seal key and secret rotation

• platform.sealKeys.issueNew, activeKid and retire become bridge.sealKeys of the installation document, one helm upgrade per step of the seal key runbook.
• installer/src/kubernetes/bootstrap.ts: rotateSealKeys applies the installer's planSealRotation and sealKidsToRetire: a new live key, a retired key kept in the store, a refusal with exit 6 for the active kid, an unknown kid, or a kid a connection still uses.
• Before a retirement, the bootstrap hook's seal-envelopes init container runs the installer's envelopeUsesScript as libredb_platform; after the release applied, the restart job's seal-canary init container runs deleteCanaryScript for each retired kid. A unit test holds the chart's SQL to the installer's text, and a kid that is not 16 hex digits fails the render.
• platform.authSecretPrevious.existingSecret mounts the previous Platform AUTH_SECRET as platform.authSecretPreviousFile, which the installer already supported.
• Every generated secret rotates by deleting it from Secret libredb-secrets and upgrading.

Scheduling

• The Platform image is built for linux/amd64 only, so Platform and the four jobs that run its image get kubernetes.io/arch: amd64 unless platform.nodeSelector names another arch, and follow platform.nodeSelector and platform.tolerations.

Installer

• src/kubernetes/ (bootstrap, keycloak job, names), src/adapters/kubernetes-api.ts (the in-cluster API client), src/cli/kubernetes-job.ts (the jobs' entry point and exit codes), and scripts/build-helm-jobs.ts (bun run build:helm-jobs), which writes the gzipped bundle the chart mounts.
• No existing module or the Docker path changed behaviour: the Kubernetes code reuses the Docker renderers and the rotation planner. e2e/helpers/studio.ts only reads an optional LIBREDB_E2E_STUDIO_URL, with the same default.
• package.json: the build:helm-jobs script.

CI

• .github/workflows/installer.yml also runs on charts/** and lints the chart with helm lint --strict, with the setup-helm action pinned by SHA.
• .gitignore: the chart's downloaded dependency and the test output.

Docs

• New docs/deploy/install-kubernetes.md: requirements, values, install, license, CA, first sign-in, back up and restore, external PostgreSQL and Keycloak, upgrade (and the two Helm behaviours to know when something goes back), rotate secrets and the seal key, differences from Docker, security notes, Argo CD, OpenShift, without network access, and the end-to-end tests.
• One-line links in README.md, ROADMAP.md, docs/deploy/install-docker.md, docs/deploy/platform-runtime.md and installer/README.md.

Testing

• I have tested this locally
• I have added/updated tests
• All existing tests pass

Gates

• Platform: lint (0 errors; the 80 warnings are the ones on main), typecheck, unit 904/904.
• Installer: typecheck, unit 949 (4 skipped, as on main), 100% line coverage (bun run coverage:check), bun run build, bun run release:check, the contracts tests against the pinned app images (5/5).
• helm lint charts/libredb-platform --strict with the values of the CI job.

New unit tests (installer/test/unit)

• kubernetes/chart.test.ts (59): renders the chart with helm template and holds it to the installer: render guards, the installation document against the schema for every mode, the release manifest's image and chart pins, Studio's and Keycloak's environments against the installer's renderers, the bootstrap Role against every object the job writes, the pods' security contexts, OpenShift adaptation in its three modes, amd64 scheduling, the rotation values, the init containers' SQL against envelopeUsesScript and deleteCanaryScript, the init containers run against a psql that refuses its first connections (they wait, and run the script once), the ingress-nginx buffer annotation, the AUTH_SECRET mount.
• kubernetes/bootstrap.test.ts (34): first install, external PostgreSQL and Keycloak, upgrades that change only what changed, restart requests, downgrade refusal, the seal key runbook step by step, the exit 6 refusals and their unchanged store.
• kubernetes/keycloak.test.ts (9), adapters/kubernetes-api.test.ts (12), kubernetes/job-exit.test.ts (1), and kubernetes/bundle.test.ts (1), which fails when the committed bundle is stale.

End-to-end runs

Each script is under charts/libredb-platform/tests/e2e/ and documented in its header. The browser suite (installer/e2e/kubernetes/helm-install.k8s.ts) reuses the Docker suite's helpers: the first admin's first sign-in with a new password and TOTP, the license page, a connection to a target PostgreSQL with its credentials sealed to Studio's key, a read grant to a Keycloak user, the user reading the table in Studio and failing to write, the revocation reaching Studio, and the user's statement in Platform's audit log.

• run-e2e.sh (kind, namespace enforcing the restricted Pod Security Standard): install in 63 to 70 s; Platform, Studio and the realm answer over TLS from the internal CA; the edge answers 404 for /admin/ and /realms/master/ and 400 for /realms/libredb/..;/master/; a pod of another namespace reaches Platform but not Keycloak's Admin API or PostgreSQL; license active; Studio's seal canary ok; browser suite 7/7 with the revocation reaching Studio in 262 to 942 ms; an upgrade with a changed value keeps the seal keys and the CA and reaches the running Platform; a rollback restores the earlier value; uninstall keeps the Secrets and volumes, and a reinstall reuses the seal keys, the CA, the database and the realm.
• run-external.sh (kind): a PostgreSQL of a simulated customer with its own CA, provisioned with the DBA script, the site's Keycloak configured by the chart's job with an admin user, the customer's TLS Secrets and a license Secret; endpoints, license and canary as above.
• run-lifecycle.sh (kind): PostgreSQL and ingress leaves set to expire in 9 days are renewed by one upgrade (824 days left), PostgreSQL restarts on its own, Platform and Studio stay ready; licenses expiring, expired and active again each reach the running Platform with an upgrade.
• run-upgrade.sh (kind): with a 1.0.2 build of this branch, an upgrade that fails after its hooks under --atomic rolls back cleanly to 1.0.1, which the installation still counts as its own; the upgrade to 1.0.2 runs the migrate job and Platform on 1.0.2 and keeps the seal keys and the CA; an upgrade back to 1.0.1 and helm rollback to the 1.0.1 revision are both refused at once with Platform still on 1.0.2; the documented recovery from the refused rollback's pending-rollback record lets the next upgrade through.
• run-argocd.sh (kind, Argo CD 3.5.5 with label resource tracking, the namespace enforcing restricted): the packaged chart served from a Helm repository in the cluster syncs to Synced and Healthy in 99 s with automated prune and self-heal; the jobs' Secrets carry no tracking label and Prune=false; a new sync with prune runs the jobs again and keeps every Secret and the seal keys; a changed value reaches the running Platform through the PostSync restart job.
• run-openshift.sh (OpenShift 4.21.30, Kubernetes 1.34.9, on AWS, through a Red Hat Developer Sandbox: one namespace, no cluster rights): install in 158 to 162 s; every pod under restricted-v2 with the namespace's uid (1002000000); edge Routes made from the Ingresses; endpoints, edge rules, license and canary; browser suite 7/7; upgrade, rollback, uninstall and reinstall as in run-e2e.sh. Run before and after the rotation and scheduling changes.
• run-rotation.sh (kind, browser driven, the granted user querying between every step): a new key reaches every Studio heartbeat while Platform keeps the old one; the same values again issue no second key; Platform switches with activeKid and its canary turns ok; retiring the old key before the re-entry fails the upgrade at once with the bootstrap log naming the connection, and nothing changes; the admin re-enters the credentials in the edit form after Platform's warning; the retirement then takes the key out of every keyring and its canary row out of Platform's table, and the store keeps it as retired. Then Studio's JWT secret (every Studio session refused, the user signed in again through the live Keycloak session without a password), Platform's AUTH_SECRET with its previous value (the admin's session survives), Studio's Keycloak client secret and libredb_platform's password (a new sign-in and Platform's reads work). 4/4 in 6.3 minutes.
• run-restore.sh (kind): a connection, a grant and a read in Studio; the backup set of the guide (the three Secrets and a pg_dump -Fc of libredb and keycloak); the namespace deleted with its volumes; the set restored into a new namespace of the same name; the seal keys and the CA are the backup's, the license is active and the canary ok; the first admin signs in with the password and TOTP of the backup, Platform holds the connection with its envelope unchanged byte for byte, and the user reads the table through the credential sealed before the backup.
• run-resilience.sh (kind): after a seed like the restore test, PostgreSQL killed with no grace period (every endpoint back in 5 s), Platform, Studio and Keycloak killed at once (66 s, Keycloak's start being the longest), the node container restarted as a power cut (97 s); after each, the connections with their envelopes, the grants and the realm's users fingerprint the same; the final browser check passes.
• run-offline.sh (kind with a local registry): helm package --dependency-update holds the Studio chart; the five images copied to the mirror by digest; the node's copies removed; Helm run with no repository and every proxy pointing at a closed port; installed from the archive in 81 s, every container's image ID from the mirror; endpoints, license and canary as above.
• run-soak.sh (kind): one installation left running, endpoints checked every 30 s, each container's memory and CPU, the restart counts and the databases' and logs' sizes every 5 minutes, and the browser check every 30 minutes. A 48-hour run was stopped at 32 hours; its clean first 28.7 hours: 10179/10179 checks answered 200 (median 10 to 14 ms, slowest 21 ms), 60/60 browser checks, no restart; memory after the first 12 hours, in MiB per hour: Platform +0.35 (206 to 220 MiB), Studio -0.04 (114 to 119), PostgreSQL -0.39, Keycloak +0.98 and slowing. An unattended Docker upgrade on the host restarted the node at hour 29; every pod came back on its own in 72 s.
• kubescape 4.0.16 on the rendered chart: score 81. The findings left are design choices written into the guide's security notes (no CPU limits, NetworkPolicies for PostgreSQL and the Admin API only, the jobs' Secret access, Recreate for the single-replica Platform and Keycloak) or false positives (GOVERNANCE_PRIVATE_KEYS_FILE is a path). trivy 0.76.0 on the five images: see Additional Notes.
• After the last chart change (the seal key rotation), run-e2e.sh, run-external.sh, run-lifecycle.sh, run-upgrade.sh (with a 1.0.2 build of the final branch), run-argocd.sh, run-restore.sh, run-rotation.sh, run-resilience.sh, run-soak.sh and run-openshift.sh all ran again and passed; run-offline.sh ran before that change.
• Rebased onto the current main; no conflicts.

Follow-up: Dokploy and CapRover (commits fc8e5d3, b11f365, 2c594ad)

Each in a fresh Ubuntu 24.04 KVM VM (8 vCPU, 14 GB, Docker 29.9.2), with a site CA and leaf for the three .libredb.test hosts and a resolver that points them at the VM, as a customer's DNS would. Docker mode ran first, then the chart on k3s v1.36.5+k3s1 on the same host behind the same PaaS.

• Docker mode, Dokploy 0.30.10: preflight with the defaults refuses port 443, naming dokploy-traefik; with tls.mode: external behind Dokploy's Traefik (a dynamic file router and certificate, trustedProxies the gateway of libredb_edge), apply in 2 min 38 s, 19 checks passed, and the installer's browser suite (--project hostnames) 22/22. A VM reboot and a Traefik restart came back on their own.
• Docker mode, CapRover 1.15.4: the same install behind CapRover's nginx (an app with the three custom domains and a custom nginx configuration with the site's certificate), apply in 2 min 41 s with 19 checks passed, but the first sign-in ended in a 502: upstream sent too big header on Platform's sign-in callback, whose session cookie chunks hold the Keycloak ID and refresh tokens. 4k fails, 8k passes; with proxy_buffer_size 16k the suite passed 22/22. Every verify check passes on the broken install.
• Chart, Dokploy: k3s's Traefik on node ports behind Dokploy's Traefik with SNI passthrough. run-e2e.sh first stopped at helm dependency build with no repository definition for https://libredb.org/libredb-studio/, which a fresh Helm cannot build from Chart.lock without helm repo add; CI copies the embedded chart, so it never ran the guide's command. With the repository added: PASS, browser suite 7/7, revocation in 576 ms.
• Chart, rotation on k3s: the retirement step hung, the bootstrap job failing 6 times with seal-envelopes: ... port 5432 failed: Connection refused and the release left pending-upgrade. k3s's kube-router refuses a new pod's first connection for about 100 ms until it holds the pod's address, and the init container made one psql call; the podFailurePolicy covers only the bootstrap container, so each retry raced again. With the wait of fc8e5d3: run-rotation.sh 4/4 on both clusters, and run-restore.sh PASS.
• Chart, CapRover: ingress-nginx 4.15.1 on node ports behind CapRover's nginx re-encrypting to it. Install in 170 s and the edge rules as on kind, but the first sign-in ended in the same 502 from ingress-nginx (4k buffers by default), and helm-install.k8s.ts reported its first test passed, since the callback URL is not /login, with the second test failing on the lost session. With the chart's new proxy-buffer-size: 16k annotation, 7/7; without it, the hardened first test now fails (Received: undefined). PostgreSQL killed: endpoints back in 1 s; Platform, Studio and Keycloak killed at once: 68 s; VM reboot: 77 s; the suite 7/7 afterwards.
• After the three commits, on the Dokploy VM: run-e2e.sh PASS (install 64 s, revocation 574 ms) and run-argocd.sh PASS (Synced and Healthy in 81 s).
• PaaS operations: Dokploy's daily Docker cleanup (on by default, 23:50) and CapRover after every app deploy run docker container prune, which removed the containers of stopped LibreDB services; docker compose start studio then started nothing and exited 0. Dokploy's "Clean unused volumes" (docker volume prune --all) during the restore runbook's stop deleted libredb_platform-uploads, and the next apply recreated it empty with every check passing. The kill switch runbook now uses up -d, which the runbooks drill checks after pruning the stopped Studio container (7/7 on CapRover), and the installer README describes both cleanups.
• tls.mode: external: a DOCKER-USER rule on port 8080 alone also drops the apps' calls to keycloak:8080 and fails 4 verify checks; the README now gives a rule on the external interface. A wrong trustedProxies breaks Studio's sign-in only, which studio-oidc-chain caught as an http:// redirect_uri; its cause now names trustedProxies and the gateway of libredb_edge.
• Not run in this follow-up: run-upgrade.sh and run-lifecycle.sh (no next-release chart or lifecycle licenses at hand), run-offline.sh, run-soak.sh, run-openshift.sh, run-external.sh; run-resilience.sh restarts a kind node, so the k3s checks above were done by hand.

Test Environment

• LibreDB Platform Version: main at 394a81b plus this branch; images Platform 1.0.1, Studio 1.0.0, Keycloak 26.8.1, PostgreSQL 18.6, Caddy 2.11.7, by digest
• Browser: Chromium (Playwright 1.58.2, headless)
• OS: Linux
• Node.js/Bun Version: Bun 1.4.2, Node 24.11.0
• Database Type: PostgreSQL 18.6 (managed, external, and as the governed target)
• Kubernetes: kind 0.33.0 with Kubernetes 1.37.0 and Traefik; OpenShift 4.21.30 (Kubernetes 1.34.9); Helm 3.21.3; Argo CD 3.5.5; k3s v1.36.5+k3s1 with Traefik 41.7.1 and ingress-nginx 4.15.1
• PaaS: Dokploy 0.30.10 (Traefik 3.6.25), CapRover 1.15.4 (nginx 1.31), Docker 29.9.2

Screenshots (if applicable)

Not attached; the browser suite saves a screenshot per step under LIBREDB_E2E_SCREENSHOTS, and the kind and OpenShift runs are available on request.

Checklist

• My code follows the project's code style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• The installer's 100% line-coverage gate passes (bun run coverage:check in installer/)
• Any dependent changes have been merged and published

Additional Notes

Decisions worth a look during review:

• The jobs bundle is a committed, generated file, held fresh by bundle.test.ts, so the chart runs the installer's exact code without a separate jobs image.
• No CPU limits by default and memory limits everywhere, so a busy pod is never throttled; NetworkPolicies cover PostgreSQL and Keycloak's Admin API only, because the ingress controller's namespace differs per cluster.
• The jobs' objects carry the label install.libredb.org/installation and no Helm or Argo CD tracking label, so a prune never deletes the seal keys; uninstall leaves them and the volumes on purpose.
• feat(installer): add the opt-in libredb-mobile Keycloak client #22 changes installer modules the Kubernetes jobs reuse (the schema, Keycloak's desired state, the env renderers). Whichever merges second needs a rebase and bun run build:helm-jobs.

Found along the way and not changed here:

• The Platform image is linux/amd64 only; an arm64 build would let the chart run on Graviton or Ampere nodes.
• trivy, HIGH and CRITICAL: Keycloak clean; Caddy 4 HIGH in Go's stdlib and x/net, with 2.11.7 already the latest; PostgreSQL's fixable ones are all in gosu, which runs only as root, and these pods are not; Platform 1.0.1 has a CRITICAL perl-base CVE from its Debian base; Platform and Studio both ship npm, whose brace-expansion and undici are flagged and unused at runtime. No image carries a cosign signature.
• The Studio chart has no preStop delay, so during a Studio restart one request can reach the stopping pod and end in a 504 after 30 seconds; the rotation test waits for the ingress to settle.
• Keycloak exits once on a cold start (Connection to libredb-postgres:5432 refused) before PostgreSQL is ready, on the first install and after a node reboot, and recovers on its own; an init wait would remove the restart.
• chart.test.ts falls back to helm dependency build when no embedded chart exists, which fails on a fresh machine without helm repo add.
• The acceptance checklist still starts Platform and Studio with docker compose start, which a PaaS prune between its steps would turn into a no-op.
• Bind mounts under /opt/libredb for postgres-data and platform-uploads would put them out of reach of docker volume prune.

Not verified: EKS, GKE and AKS with their own load balancers, and an upgrade to a release with new migrations, which needs the next release.