Repository navigation
fix(tls): drop distrusted eMudhra roots; take NSS roots from x509roots, build with go1.27.3 - #4883
Merged
Merged
Conversation
…s, build with go1.27.3 The private keys of eight eMudhra intermediate CAs were found recoverable (2026-10-17); Mozilla removed the emSign roots they chain to (NSS 3.131.1). The agent's fallback trust store - written to /tmp/keploy-tls/ca.crt for recorded apps whose image has no CA store, and used for upstream verification in distroless agents - was curl's cacert.pem vendored in May from Mozilla's February data and never refreshed. It still trusted emSign ECC Root CA - G3, emSign Root CA - C1 and emSign ECC Root CA - C3. - Build the fallback PEM at startup from golang.org/x/crypto/x509roots/fallback/bundle (Mozilla NSS, 2026-10-20) and delete the vendored mozilla_roots.pem. The root set now matches Mozilla's current one: 145 -> 117 roots. Besides the three emSign roots this drops roots Mozilla had already retired (Entrust, the 2006 DigiCert Global/High Assurance/Assured ID roots, COMODO CA, QuoVadis 2/3, GTS R2, AffirmTrust, Trustwave, ...) and adds SECOM 2024 and Telia v3. - Keploy's own upstream verification builds its pool from the NSS bundle when it is the fallback, so per-root constraints (distrust-after) are enforced via AddCertWithConstraint. - refresh-nss-roots.yml bumps that module weekly and opens a PR: it only publishes pseudo-versions, which Dependabot never proposes. CLA allows github-actions[bot] and runs on reopen, since a GITHUB_TOKEN PR starts no workflow. - go 1.27.3: its crypto/x509 fix key-blocks the compromised intermediates. release.yml's setup-go reads go.mod, so release binaries were built with exactly go1.27.0 (v3.6.158 reports go1.27.0). Dockerfile and e2e images pinned to golang:1.27.3 by digest; golangci-lint v2.14.0, the first that reads go1.27.2+ export data. Tests: the embedded bundle must not contain the distrusted emSign roots (fails on the old bundle), and NSS constraints are enforced upstream. Signed-off-by: Shubham Jain <shubhamkjain@outlook.com>
🚀 Keploy Performance Test ResultsMulti-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.
Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1% ✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2) P50, P90, and P99 percentiles naturally filter out outliers |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The private keys of eight eMudhra intermediate CAs were found recoverable on 2026-10-17 (Mozilla, BleepingComputer). Browsers block them; software with its own root store or an unpatched Go does not.
Keploy had both gaps:
pkg/agent/proxy/tls/data/mozilla_roots.pem(vendored in fix(agent): embed Mozilla NSS fallback so agent always produces a valid CA bundle #4177, Mozilla data from Feb 2026, never refreshed) is the agent's fallback trust store — written to/tmp/keploy-tls/ca.crtfor recorded apps whose image has no CA store, and used for upstream verification in distroless agents. It still contained emSign ECC Root CA - G3, emSign Root CA - C1, emSign ECC Root CA - C3, which Mozilla removed in NSS 3.131.1.release.ymlusessetup-gowithgo-version-file: go.mod, sogo 1.27.0installs exactly 1.27.0 —v3.6.158reportsgo1.27.0, missing go1.27.3'scrypto/x509key-block for the compromised intermediates (and the 1.27.1/1.27.2 security fixes).What
golang.org/x/crypto/x509roots/fallback/bundle(Go team's Mozilla NSS extract, 2026-10-20) instead of a vendored PEM, built at startup; the vendored file and its manual REFRESH.md are deleted.AddCertWithConstraintwhen the compiled-in bundle is the fallback.refresh-nss-roots.yml: weeklygo get …/x509roots/fallback@latest, runs the root-store tests, opens a PR (or an issue if Actions may not open PRs). Dependabot can't do this — the module only publishes pseudo-versions.cla.yml: allowgithub-actions[bot], run onreopened(GITHUB_TOKEN PRs start no workflow).go 1.27.3in go.mod;Dockerfileand e2e build images pinned togolang:1.27.3by digest (golang images runGOTOOLCHAIN=local, so an older cached image fails instead of silently building a vulnerable binary); golangci-lint v2.14.0 (v2.13.x can't read go1.27.2+ export data and type-checks nothing).This matches Mozilla's current trust store (curl's
cacert.pemof 2026-10-20 is identical except one root Go excludes by hand). Besides the three emSign roots it removes roots Mozilla had already retired: Entrust ×3, DigiCert Global Root CA / High Assurance EV / Assured ID, COMODO Certification Authority, QuoVadis 2 & 3, GTS Root R2, AffirmTrust ×4, Trustwave ×3, GLOBALTRUST 2020, SecureSign CA12, certSIGN, Chunghwa ePKI, and others; adds SECOM TLS 2024 ×2 and Telia v3 ×2. Only affects apps/agents that rely on Keploy's fallback (no CA store in the image): a server still chaining to a retired root will now fail verification there — the same as in Firefox.Tests
TestLoadSystemCABundle_EmbeddedFallback_ExcludesDistrustedRoots— fails on the old bundle (lists the three emSign roots), passes now; guards against downgrades.TestLoadUpstreamRootCAs_RealEmbeddedRoots_UsesNSSRootPool/…_EnforcesNSSConstraints— mutation-checked: disabling the NSS path or dropping the constraint fails them.go build ./...,go vet, tls/config tests pass on go1.27.3; golangci-lint v2.14.0 clean. (TestInstallJavaCA_BackwardCompat_NoJavaIsNoopfails locally on macOS on unmodified main too —/usr/bin/javastub — unrelated.)Companion: keploy/docker-images#38 (go1.27.3 in every CI image, released v1.2.48); keploy/enterprise follows (images, go line, notices rule for
bundle.der).