Repository navigation
fix(security): keep bulk transfers and the telemetry migrator on TLS - #96
Conversation
File bytes, Starter export dumps and update artifacts went to server-chosen URLs with http.DefaultClient and no scheme check, and the telemetry migrator opened TIDB_DSN without the verified-TLS rule the API enforces. - api/transport: CheckSecureURL, SecureTransferClient and DoSecure allow https (plain http only to loopback), check every redirect hop, keep the 10-redirect limit, and keep presigned queries out of error messages. - fs presigned uploads/reads and the download redirect, Starter export downloads, and ti update (archive, checksums, ti-drive9 companion) use them. - telemetrybackend.ValidateTiDBDSN is shared by the API and the migrator; an unset TELEMETRY_ENVIRONMENT means production in both. The telemetry e2e passes TELEMETRY_ENVIRONMENT to the migrator as it does to the API.
net/http wraps every failed request in a *url.Error that repeats the full URL, so a timeout or refused connection on a presigned transfer printed the signature. DoSecure now drops user info, query, and fragment from that URL for all errors, not only refused redirects. The redirect-limit test gets a timeout so a missing limit fails fast.
srstack
left a comment
There was a problem hiding this comment.
Review: fix(security): keep bulk transfers and the telemetry migrator on TLS
I reviewed the full 14-file diff and verified it locally on top of current main (patch applies clean):
go build ./...— okgo vet ./internal/api/... ./internal/update/... ./internal/telemetrybackend/... ./cmd/...— cleango test ./... -count=1— all passgofmt -l cmd internal e2e— clean
Verdict: no blocking issues. The approach is right (one URL rule, one redirect policy composer next to the existing SameOriginRedirect/SameOriginThen, no new dependencies), the redaction story is genuinely thought through, and the tests cover the failure modes rather than just the happy path. I checked the two easy-to-get-wrong details and both are correct: MaxRedirects re-implementation matches net/http's defaultCheckRedirect exactly (len(via) >= 10, so 10 requests are sent and the 11th is refused), and copying http.Client in SecureTransferClient leaves http.DefaultClient untouched.
I also traced every transfer site: all four presigned fs calls, the fs download redirect, Starter export downloads, and ti update metadata/checksum/archive/companion are now paired with a secure client and DoSecure. No http.DefaultClient or bare .Do(req) is left outside internal/db/sqlhttp (which is unrelated and already same-origin).
Nice catches in particular:
- Refactoring the migrator into
run(args, getenv, logger) int— that is what makes the refusal actually testable, and it also stopsdefer db.Close()being skipped byos.Exit. - Fixing
e2e/telemetry_test.goin the same PR. Without itmake telemetry-e2ewould have started failing on the next run, and it would have looked unrelated to this change.
Suggestions (non-blocking)
[MINOR] DoSecure and SecureTransferClient are two halves of one rule, but only one enforces redirects — internal/api/transport/secure.go:63,77
DoSecure checks the request URL; the redirect check lives on the client. Today all four call sites happen to be paired, but nothing stops a future caller from doing DoSecure(http.DefaultClient, req) and silently getting only half the protection — the failure is invisible and looks like it works. Could we make the pairing impossible to get wrong? Options I'd be happy with: fold the initial check into the wrapper (DoSecure calling SecureTransferClient(client) itself — the struct copy is cheap and the policy wrap is idempotent), or expose a single Transfer type that owns both, or at minimum a doc line on DoSecure saying "the client must come from SecureTransferClient" (the comment currently only describes error redaction).
[MINOR] A malformed server URL still leaks the presigned query through the pre-request build errors — internal/db/product/starter/export.go:303, internal/api/fs/client.go:238, internal/update/update.go:396
The PR states "no presigned signature ends up in an error", but these three wrap/return the error from http.NewRequestWithContext before DoSecure ever sees the URL. That error embeds the whole URL — verified: parse "http://bucket.example.com/a.csv?X-Amz-Signature=secret-signature": net/url: invalid control character in URL. Low likelihood (the server has to hand back a malformed URL), but it is the same leak class the PR closes. Since redactURL already exists in secure.go, maybe export it as transport.RedactURL and apply it to those three, or run CheckSecureURL/url.Parse before building the request (export.go and update.go would then also get the https refusal instead of a generic invalid_url).
[MINOR] A redirect loop is reported as a connectivity problem — internal/api/fs/client.go:249-258
ErrTooManyRedirects falls into the generic api.network_error ("check network connectivity and try again") branch. internal/api already distinguishes this with api.redirect_loop (internal/api/client.go:284); reusing that here would keep the two download paths consistent and save users from chasing a network issue that does not exist.
[MINOR] A misspelled TELEMETRY_ENVIRONMENT silently disables the TLS rule — internal/telemetrybackend/config.go:152
strings.EqualFold(valueOrDefault(environment, "production"), "production") means only the exact word counts: prod, prd, prduction all take the non-production path. This predates the PR on the API side, but this change extends it to the one-shot migrator, where a typo in the deploy .env means a silent cleartext migration instead of a refusal — the worst place to fail open. An allowlist (production / staging / dev / telemetry-e2e) that rejects unknown values would make it fail closed, and it would not break the e2e value this PR adds.
[MINOR] Docs are not updated for the new migrator behavior — docs/telemetry-backend-design.md:46
The design doc says "TELEMETRY_ENVIRONMENT=production enables this strict TLS validation", which now understates the rule: the migrator applies it too, and unset means production in both. One sentence there covers it. Related: the redirect-policy paragraph in AGENTS.md (around line 1082) enumerates the credential-bearing client rules that future changes must preserve; server-chosen transfer URLs now have their own rule (SecureTransferClient + DoSecure, https-only, cross-origin allowed), and that is exactly the kind of rule that gets lost without a line there.
Nits
[NIT] Duplicate loopback helper — internal/update/update.go:362 already had isLoopbackHost with the same body as the new one in secure.go. testReleaseAPIBaseURL could now call apitransport.CheckSecureURL (or an exported IsLoopbackHost) so the "https, or http to loopback" rule lives in one place.
[NIT] The presigned happy path is not testable — internal/api/fs/upload.go:958
transferClient is a package-level var … SecureTransferClient(nil), so it always uses DefaultTransport, and a test cannot swap in a trusted TLS transport for httptest.NewTLSServer. The refusal tests are good, but nothing pins that an https presigned upload still sends the body/checksum headers and returns the ETag, which is the most likely way this change could regress real transfers. A Client field or a small injectable hook would let you cover it.
[NIT] The loopback exception is compiled into release builds — internal/api/transport/secure.go:29
So a server-controlled redirect to http://127.0.0.1:<port>/… is followed in production, which is a (low-impact) local SSRF path: bytes only land in the output file and update artifacts are checksum-verified against a manifest from the same origin. If you prefer belt and braces, gating loopback http behind version.Info.TestControlsAllowed would match how every other hidden test control in this repo is scoped; the current shape is defensible as is, so this is only a question of taste.
One question
ti fs data-plane traffic (including mounts and pack/unpack) runs through the bundled ti-drive9 companion, which has its own transfer code. This PR closes the ti-side surface; if the companion follows presigned URLs the same way, the same flag would apply there. Is that already tracked separately, or worth a follow-up issue so it does not get lost?
On the behavior change
The http-to-non-loopback refusal and the release-notes line in the PR description are enough for a user-facing note. One extra case worth mentioning if you write it: a self-hosted dev stack whose object store is reachable by a container/service hostname (e.g. http://minio:9000, which is not loopback) is refused too — the loopback exception only covers localhost/127.0.0.0/8/::1.
- DoSecure wraps any client with SecureTransferClient, so a plain client still gets the redirect rule. - NewSecureRequest checks a transfer URL before a request exists and reports a malformed one without the URL; all seven transfer sites use it. - net/http's error for an unparseable redirect Location repeats the raw value; DoSecure and internal/api requestError drop it. - A redirect from https to http is refused even to loopback. - fs downloads and ti update report a redirect loop as its own code. - Every TELEMETRY_ENVIRONMENT value counts as production except development, local, test and telemetry-e2e, so a typo fails closed. - update's test-only base URL override uses CheckSecureURL. - Tests for the https presigned happy path, malformed URLs and Locations, redirect loops, and environment typos; docs and AGENTS.md updated.
|
Thanks for the thorough review! Everything is addressed in cb3bca5: MINOR 1, MINOR 2, URL in request-build errors. New MINOR 3, redirect loop. The fs download path returns MINOR 4, MINOR 5, docs. NIT 1, duplicate loopback helper. Removed. NIT 2, presigned happy path. NIT 3, loopback in release builds. I kept it for URLs the server returns directly, but your point is better than my first reasoning: for downloads it only writes bytes to the output file, but for uploads ti would PUT the user's file, with server-chosen headers, to a service on the user's machine, which is a small local SSRF. To narrow it, a redirect from https to http is now refused even when the target is loopback, so a remote https server can't bounce a transfer to a local service. drive9 question. Yes, the companion has the same pattern: it sends presigned PUTs through its normal client with no scheme check, and its redirect policy strips auth headers on cross-origin hops but follows https to http. That code is outside this repo, so we'll track it separately. Behavior change. The PR description now also mentions the |
srstack
left a comment
There was a problem hiding this comment.
Second pass on cb3bca53 — earlier comments verified, two nits left
I re-reviewed the current head after the two new commits (69903bfc, cb3bca53). The first-pass diff (713 additions, 14 files) is now 1204 additions across 18 files, and every point from my previous review is addressed:
| Earlier comment | Status on cb3bca53 |
|---|---|
DoSecure alone leaves redirects unchecked (rule split across two functions) |
Fixed — DoSecure now wraps the client itself, and TestDoSecureHoldsAnyClientToTheRedirectRule pins nil / plain / http.DefaultClient |
| Presigned URL leaks through pre-request build errors | Fixed — NewSecureRequest + ErrMalformedURL, plus res.Location()'s cause dropped and the requestError branch |
| Redirect loop reported as a connectivity error | Fixed — api.redirect_loop / update.redirect_loop, both tested |
A misspelled TELEMETRY_ENVIRONMENT fails open |
Fixed — nonProductionEnvironments fails closed (tests cover prod, prduction, staging, qa-east) |
| Docs/AGENTS not updated | Fixed — docs/telemetry-backend-design.md and the new AGENTS transfer paragraph |
Duplicate isLoopbackHost |
Fixed — testReleaseAPIBaseURL uses CheckSecureURL; local helper deleted |
| Presigned https happy path untestable | Fixed — TestPresignedTransfersOverHTTPS pins body, server headers, Host stripping, both checksum headers, ETag and bytes |
Loopback http reachable from a remote server (SSRF-ish) |
Better than I asked — an https→http hop is now refused even to loopback, before the local service is hit |
I verified locally on this head: go build ./..., go vet ./internal/... ./cmd/..., go test ./... -count=1, go test -race -count=2 ./internal/api/fs/ ./internal/api/transport/, gofmt -l cmd internal e2e — all clean, and CI test (which includes make e2e) is green on this SHA.
Two independent probes worth sharing, because both were holes in the previous head and I wanted to see them closed rather than trust the diff:
- A
Locationheader that does not parse used to leak the raw value: net/http puts it in the message (failed to parse Location header "…?X-Amz-Signature=…"), whichDoSecure'surlErr.URLredaction did not touch. Now:refused a malformed transfer URL: a redirect had an invalid Location header, signature absent. Same for the300path whereReadFileparsesLocationitself. DoSecure(nil, req)no longer panics and still enforces the redirect rule.
Still open (non-blocking)
[MINOR] The install scripts are the one bulk-transfer path left without the redirect rule — scripts/install.sh:88-97, scripts/install.ps1:215-218
ti update now refuses a download that leaves https, but the bootstrap scripts that install the same bytes still follow any redirect. I checked the behaviour rather than assume it: a local https server that redirects to a plain-http server is followed by curl -k -sSL (final_url=http://…, target reached), so curl -fSL does not protect the archive, ti_checksums.txt, or the ti-drive9 companion. Real-world risk is low because the first hop is an authenticated https GitHub/releases origin, and the checksum comes over the same channel — but that is exactly the argument that also applied to ti update. curl --proto-redir =https (plus --proto =https), wget --https-only, and for PowerShell a final-scheme check on BaseResponse.RequestMessage.RequestUri would close it. Extend this PR or file a follow-up, as you prefer — I would not block on it.
[NIT] ErrMalformedURL throws away the one part of the parse error that is safe — internal/api/transport/secure.go:95-105
url.Parse returns *url.Error, and only its URL field repeats the raw URL; the wrapped Err is the parser's own message and never contains it (missing ']' in host, invalid control character in URL, invalid URL escape "%zz", invalid port). So fmt.Errorf("%w: %v", ErrMalformedURL, urlErr.Err) would keep the diagnosis while preserving the invariant. Today a malformed presigned URL reports only "refused a malformed transfer URL", which is hard to act on when a server hands back something broken. The tests that assert no leak would still pass, and they would also catch a future Go version that changes this.
[NIT] The policy is now wrapped three times — internal/update/update.go:332-336, internal/db/product/starter/export.go:342-344, internal/api/fs/upload.go:958
newClient, fileHTTPClient and transferClient are each already SecureTransferClient(...), and DoSecure wraps again on every call. It is harmless (the wrapper is idempotent) and defensible as defence in depth, but since AGENTS now tells future authors to build with NewSecureRequest and send with DoSecure, the constructor-level wrapping reads as if it were required too. Either drop the outer layers or say in the comment that they exist as a second line of defence.
One deployment check before merge
staging now counts as production for TIDB_DSN. There is no staging deployment in this repo, and the documented .env uses production with tls=true, so nothing in-tree breaks. If an ingestion deployment outside this repo runs TELEMETRY_ENVIRONMENT=staging against a DSN without verified TLS, it will start refusing at the next migrate. Worth confirming before merge; the fail-closed choice itself is right.
Also: the PR "Tests" section still lists only the first commit's files — the malformed-Location, redirect-loop, https-happy-path and environment-allowlist tests added later are the more interesting ones now.
- install.sh downloads with curl --proto '=https' --proto-redir '=https'. The wget fallback is removed: GNU wget's --https-only applies only to recursive downloads, so it can't refuse an https-to-http redirect. - install.ps1 sends every download through Save-HttpsDownload, which checks the final URL after redirects is https (Windows PowerShell 5.1 and PowerShell 7) and deletes the file otherwise. - NewSecureRequest keeps the URL parser's reason, which never repeats the URL. - Comments say the constructor-level SecureTransferClient wrapping is a second line of defense; DoSecure applies the policy again. - e2e runs install.sh's real download() against local https and http servers.
|
Thanks for the second pass and for probing the Location leak yourself! Addressed in 83f0f4b: [MINOR] Install scripts. Fixed in this PR.
[NIT] Parser reason. [NIT] Triple wrapping. I kept the constructor-level wrapping and said in each comment ( Deployment check. Agreed. Before the next telemetry redeploy I'll confirm the live PR description. The Tests section now lists everything, including the later tests. |
srstack
left a comment
There was a problem hiding this comment.
Approved — verified on 83f0f4bd
Third and final pass on the current head (22 files, +1422/−62). Everything raised in my two earlier reviews is resolved, including the installer commit:
scripts/install.shnow pinscurl --proto '=https' --proto-redir '=https'and drops the wget path, with the reason in a comment.scripts/install.ps1routes all four downloads throughSave-HttpsDownload, checks the final URI and deletes the file on a non-https landing, covering both 5.1 (ResponseUri) and 7 (RequestMessage.RequestUri).NewSecureRequestkeeps the parser's reason (invalid URL escape "%zz") while dropping the raw URL, and the test now asserts both halves of that.- The wrapping in
newClient/fileHTTPClient/transferClientis documented as a deliberate second line of defence, which settles my last nit.
Verification I ran on this SHA, not just read from the diff:
go build ./...,go vet ./...,go test ./... -count=1,gofmt -l— clean;go test ./e2e -run Installerpasses, and CItest(which includesmake e2e) is green.- I reproduced the installer attack directly: a TLS server redirecting to a plain-http server is refused by the new flags —
curl: (1) Protocol "http" disabled (in redirect)— and the http target is never reached. Extracting the realdownload()frominstall.shand driving it from the e2e test is the right call for a shell installer; a static grep would not have caught a mistyped flag. - Re-checked the PowerShell helper by reading, since this box has no pwsh: the property probing is correct for both hosts, and the failure path is fail-closed (file removed, clear message). Not a blocker.
Three notes, none of which block merge:
- Removing the wget fallback is a user-visible change for minimal systems without curl. The error message is actionable, so a release-notes line is enough.
install.ps1has no executable coverage in CI (Ubuntu-only plusGOOS=windowsvet). Please smoke-test bothWindows PowerShell 5.1andpwsh 7once before the release that ships this, or add a Windows runner if the release workflow has one.- Worth a one-line confirmation that no ingestion deployment outside this repo runs
TELEMETRY_ENVIRONMENT=stagingagainst a DSN without verified TLS —stagingnow counts as production, which is the correct fail-closed behaviour but will surface at the next migrate.
Nice execution on this one: every round of feedback came back with tests that pin the invariant rather than just the example, and the installer work ended up covering the last cleartext path in the product.
Why
A security scan flagged that bulk transfers and the telemetry migrator can move sensitive data in cleartext (information-protection-best-practices, High):
http.DefaultClientand no scheme check.ti-telemetry-migrateopenedTIDB_DSNas is, while the API server already refuses a production DSN without verified TLS.ti update(archive, checksums, ti-drive9 companion) had the same download pattern, so it is covered too, and so are the install scripts that download the same files.What changed
internal/api/transport/secure.go:CheckSecureURL,SecureTransferClientandDoSecure.CheckRedirectdrops it), then defer to a caller's own policy.DoSecureapplies the redirect rule to whatever client it is given, so it can't be paired with the wrong client.NewSecureRequestparses and checks a transfer URL before a request exists; a URL that does not parse is reported asErrMalformedURLwithout the URL.scheme://host, a malformed one is not repeated, net/http's error for a redirectLocationthat does not parse (on any hop) is dropped because it repeats the raw value, and any other failure (timeout, refused connection) keeps the URL without user info, query, or fragment. (Unchanged by this PR: an object-store error response body, up to 8 KiB, is still included in upload and download errors.)internal/api/fs/upload.go,internal/api/fs/client.go: presigned transfers and the download redirect go throughNewSecureRequestandDoSecure. A refused download redirect returnsapi.insecure_redirect, and a redirect loop returnsapi.redirect_loop.internal/api/client.go: when net/http fails on a redirect whoseLocationdoes not parse, its error repeats the rawLocation(presigned signature included).requestErrornow returnsapi.invalid_redirectwithout that cause.internal/db/product/starter/export.go: export file downloads go throughNewSecureRequestandDoSecure; an injectedHTTPClientkeeps its own policy under the https rule.internal/update/update.go: all downloads go throughNewSecureRequestandDoSecure; a refused URL returnsupdate.insecure_url, a redirect loopupdate.redirect_loop. The test-only release base URL override usesCheckSecureURL(one loopback rule instead of two).internal/telemetrybackend/config.go: the DSN rule moves toValidateTiDBDSN, shared by the API andcmd/ti-telemetry-migrate. EveryTELEMETRY_ENVIRONMENTvalue counts as production (unset included) exceptdevelopment,local,testandtelemetry-e2e, so a typo such asprodfails closed.scripts/install.sh: downloads use curl with--proto '=https' --proto-redir '=https'. The wget fallback is removed, because GNU wget's--https-onlyapplies only to recursive downloads and can't refuse an https-to-http redirect.scripts/install.ps1: every download goes throughSave-HttpsDownload, which checks that the final URL (after redirects) is https and deletes the file otherwise. PowerShell 7.4+ already refuses that redirect; the check also covers Windows PowerShell 5.1.docs/telemetry-backend-design.md(the rule covers the migrator, and which names skip it) and a newAGENTS.mdparagraph for server-chosen transfer URLs.e2e/telemetry_test.go: the migrator getsTELEMETRY_ENVIRONMENT=telemetry-e2e, like the API in the same test.Behavior change
A server-returned transfer URL or redirect that is plain http to a non-loopback host is now refused. drive9's S3 and Alibaba OSS presigned URLs are https, so normal use is unaffected. A self-hosted drive9 whose object store is on plain http on a LAN would stop working, and so would a dev stack that reaches its object store by a service hostname such as
http://minio:9000(not loopback); worth a line in the release notes. A redirect from https to http is refused even when it points at loopback.install.shnow needs curl to download (the documentedcurl ... | shinstall already has it); a system with only wget gets a clear "curl is required" error.Production telemetry is unaffected as long as its DSN uses verified TLS, which the running API already requires; a
TELEMETRY_ENVIRONMENTvalue outside the four local/test names now also gets the strict rule.Tests
New test files:
internal/api/transport/secure_test.go,internal/api/fs/secure_transfer_test.go,internal/db/product/starter/export_secure_test.go,internal/update/secure_download_test.go,cmd/ti-telemetry-migrate/main_test.go,e2e/installer_https_test.go; extended:TestValidateTiDBDSN,TestReleaseAPIBaseURLOverrideNeedsLocalBuildAndOptIn.Covered:
TestDoSecureRefusesHTTPSRedirectToLoopbackHTTP);DoSecureholds a nil, plain or default client to the redirect rule without changinghttp.DefaultClient; a relativeLocationstill works; the caller's own redirect policy is kept.TestPresignedTransfersOverHTTPSruns all four presigned functions against a TLS server and checks the body, server-chosen headers (neverHost), the sha256 and crc32c checksum headers, and the ETag or bytes.Locationon the first hop (internal/api), on a later hop (DoSecure), and on a 300 response thatReadFileparses itself never shows the raw value; the same at the export, fs and update entry points.api.redirect_loop(fs) andupdate.redirect_loop.prod,prduction),stagingor unknown environment is held to verified TLS, for the API config and the migrator; no DSN password in logs; the migrator refuses before migrating.install.sh's realdownload()runs against local servers: https works, and both a plain http URL and an https-to-http redirect fail without reaching the http server; without curl it fails with "curl is required". Source checks pin the curl flags, no wget, and thatinstall.ps1sends all four downloads through the final-URL check.Ran locally:
make test,make e2e,make build,GOOS=windows/darwin go vet ./...,make build-telemetry-backend build-telemetry-migrator, gofmt,go mod tidyandsh -n scripts/install.sh.Not run:
make telemetry-e2e(needs a test TiDB DSN), andinstall.ps1itself (no PowerShell here; it needs one real run on Windows before release).