Skip to content

fix(tls): drop distrusted eMudhra roots; take NSS roots from x509roots, build with go1.27.3 - #4883

Merged
slayerjain merged 1 commit into
mainfrom
fix/emudhra-ca-roots-go1273
Oct 22, 2026
Merged

slayerjain merged 1 commit into
mainfrom
fix/emudhra-ca-roots-go1273

Conversation

@slayerjain

Copy link
Copy Markdown
Member

Why

The private keys of eight eMudhra intermediate CAs were found recoverable on 2026-10-17 (Mozilla, BleepingComputer). Browsers block them; software with its own root store or an unpatched Go does not.

Keploy had both gaps:

  1. Stale vendored roots. pkg/agent/proxy/tls/data/mozilla_roots.pem (vendored in fix(agent): embed Mozilla NSS fallback so agent always produces a valid CA bundle #4177, Mozilla data from Feb 2026, never refreshed) is the agent's fallback trust store — written to /tmp/keploy-tls/ca.crt for recorded apps whose image has no CA store, and used for upstream verification in distroless agents. It still contained emSign ECC Root CA - G3, emSign Root CA - C1, emSign ECC Root CA - C3, which Mozilla removed in NSS 3.131.1.
  2. Release binaries built with go1.27.0. release.yml uses setup-go with go-version-file: go.mod, so go 1.27.0 installs exactly 1.27.0 — v3.6.158 reports go1.27.0, missing go1.27.3's crypto/x509 key-block for the compromised intermediates (and the 1.27.1/1.27.2 security fixes).

What

  • Fallback roots now come from golang.org/x/crypto/x509roots/fallback/bundle (Go team's Mozilla NSS extract, 2026-10-20) instead of a vendored PEM, built at startup; the vendored file and its manual REFRESH.md are deleted.
  • Upstream verification honours NSS constraints (e.g. distrust-after, which a PEM cannot carry) via AddCertWithConstraint when the compiled-in bundle is the fallback.
  • refresh-nss-roots.yml: weekly go get …/x509roots/fallback@latest, runs the root-store tests, opens a PR (or an issue if Actions may not open PRs). Dependabot can't do this — the module only publishes pseudo-versions. cla.yml: allow github-actions[bot], run on reopened (GITHUB_TOKEN PRs start no workflow).
  • go 1.27.3 in go.mod; Dockerfile and e2e build images pinned to golang:1.27.3 by digest (golang images run GOTOOLCHAIN=local, so an older cached image fails instead of silently building a vulnerable binary); golangci-lint v2.14.0 (v2.13.x can't read go1.27.2+ export data and type-checks nothing).

⚠️ Behaviour change: root set 145 → 117

This matches Mozilla's current trust store (curl's cacert.pem of 2026-10-20 is identical except one root Go excludes by hand). Besides the three emSign roots it removes roots Mozilla had already retired: Entrust ×3, DigiCert Global Root CA / High Assurance EV / Assured ID, COMODO Certification Authority, QuoVadis 2 & 3, GTS Root R2, AffirmTrust ×4, Trustwave ×3, GLOBALTRUST 2020, SecureSign CA12, certSIGN, Chunghwa ePKI, and others; adds SECOM TLS 2024 ×2 and Telia v3 ×2. Only affects apps/agents that rely on Keploy's fallback (no CA store in the image): a server still chaining to a retired root will now fail verification there — the same as in Firefox.

Tests

  • TestLoadSystemCABundle_EmbeddedFallback_ExcludesDistrustedRoots — fails on the old bundle (lists the three emSign roots), passes now; guards against downgrades.
  • TestLoadUpstreamRootCAs_RealEmbeddedRoots_UsesNSSRootPool / …_EnforcesNSSConstraints — mutation-checked: disabling the NSS path or dropping the constraint fails them.
  • go build ./..., go vet, tls/config tests pass on go1.27.3; golangci-lint v2.14.0 clean. (TestInstallJavaCA_BackwardCompat_NoJavaIsNoop fails locally on macOS on unmodified main too — /usr/bin/java stub — unrelated.)

Companion: keploy/docker-images#38 (go1.27.3 in every CI image, released v1.2.48); keploy/enterprise follows (images, go line, notices rule for bundle.der).

…s, build with go1.27.3

The private keys of eight eMudhra intermediate CAs were found recoverable
(2026-10-17); Mozilla removed the emSign roots they chain to (NSS 3.131.1).
The agent's fallback trust store - written to /tmp/keploy-tls/ca.crt for
recorded apps whose image has no CA store, and used for upstream
verification in distroless agents - was curl's cacert.pem vendored in May
from Mozilla's February data and never refreshed. It still trusted
emSign ECC Root CA - G3, emSign Root CA - C1 and emSign ECC Root CA - C3.

- Build the fallback PEM at startup from
  golang.org/x/crypto/x509roots/fallback/bundle (Mozilla NSS, 2026-10-20)
  and delete the vendored mozilla_roots.pem. The root set now matches
  Mozilla's current one: 145 -> 117 roots. Besides the three emSign roots
  this drops roots Mozilla had already retired (Entrust, the 2006
  DigiCert Global/High Assurance/Assured ID roots, COMODO CA, QuoVadis 2/3,
  GTS R2, AffirmTrust, Trustwave, ...) and adds SECOM 2024 and Telia v3.
- Keploy's own upstream verification builds its pool from the NSS bundle
  when it is the fallback, so per-root constraints (distrust-after) are
  enforced via AddCertWithConstraint.
- refresh-nss-roots.yml bumps that module weekly and opens a PR: it only
  publishes pseudo-versions, which Dependabot never proposes. CLA allows
  github-actions[bot] and runs on reopen, since a GITHUB_TOKEN PR starts
  no workflow.
- go 1.27.3: its crypto/x509 fix key-blocks the compromised
  intermediates. release.yml's setup-go reads go.mod, so release
  binaries were built with exactly go1.27.0 (v3.6.158 reports go1.27.0).
  Dockerfile and e2e images pinned to golang:1.27.3 by digest;
  golangci-lint v2.14.0, the first that reads go1.27.2+ export data.

Tests: the embedded bundle must not contain the distrusted emSign roots
(fails on the old bundle), and NSS constraints are enforced upstream.

Signed-off-by: Shubham Jain <shubhamkjain@outlook.com>
@github-actions

Copy link
Copy Markdown

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run P50 P90 P99 RPS Error Rate Status
1 2.8ms 3.52ms 5.1ms 100.02 0.00% ✅ PASS
2 2.67ms 3.3ms 4.92ms 100.01 0.00% ✅ PASS
3 2.67ms 3.4ms 4.71ms 100.00 0.00% ✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

@slayerjain
slayerjain merged commit 92acc28 into main Oct 22, 2026
408 of 414 checks passed
@slayerjain
slayerjain deleted the fix/emudhra-ca-roots-go1273 branch October 22, 2026 08:28
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 22, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant