Skip to content

Add Atlas copy and synthetic data commands - #287

Open
ottonomy wants to merge 2 commits into
atlas-read-only-scanfrom
atlas-copy-and-synthetic
Open

ottonomy wants to merge 2 commits into
atlas-read-only-scanfrom
atlas-copy-and-synthetic

Conversation

@ottonomy

Copy link
Copy Markdown
Contributor

Stacked on #286 — review and merge that first.

Adds two commands to scripts/atlas/atlas.ts, beside scan, for moving shared credentials from Atlas to the DynamoDB table the app uses on Lambda. Both are run by hand; neither is part of the app or its deployment.

  • copy reads the shared, unexpired credentials from SOURCE_URI and writes them through the app's own dynamodb store (put, create-only-if-absent), keeping id and createdAt and setting expiresAt from the TTL. It checks the AWS account against --expect-account before anything else, is a dry run unless --write is given, never writes to the source, skips and counts unshared, expired, malformed and over-limit records, counts records an earlier run copied as "already present" (so it can run before and after the DNS switch), and prints counts only. --sample-links prints a few /credentials/<id> paths, for the staging account only.
  • synthetic writes a generated data set in Atlas's 2.x record shape, with a TTL index on createdAt, to a MongoDB on localhost or 127.0.0.1 only, and prints what copy should report for it. A rehearsal never needs real data.

Both take the size limit (MAX_SHARED_VP_BYTES) and the TTL from the app's own modules, and the over-limit fixture is sized from that limit. scan now imports the same constants instead of repeating them. copy and synthetic read SOURCE_* rather than ATLAS_* because their source is Atlas only in the production run; the README explains this and gives the rehearsal and production procedures.

The TTL index synthetic creates (365 days) is assumed: confirm it against scan's output for production Atlas.

New dev dependency: @aws-sdk/client-sts, for the account check. vitest now also runs scripts/**/*.test.ts.

Validation (local)

  • npm test: 164 passed, 33 skipped. npx tsc --noEmit: clean.
  • docker compose -f docker-compose.test.yml up -d then npm run test:stores: 197 passed. This includes an end-to-end test that runs synthetic into MongoDB and copy --write into DynamoDB Local, reads copied records back through the credential service, and runs the copy a second time to see every record counted as already present.
  • By hand against the same containers: synthetic, then copy as a dry run, with --write, and with --write again; scan on the same data; and each refusal and error path. Nothing was run against AWS or Atlas.

🤖 Generated with Claude Code

scripts/atlas/atlas.ts gains two commands beside scan. copy reads the
shared, unexpired credentials from SOURCE_URI and writes them to a
DynamoDB table through the app's own dynamodb store, keeping id and
createdAt and setting expiresAt from the TTL. It checks the AWS account
against --expect-account before anything else, is a dry run unless
--write is given, never writes to the source, skips and counts unshared,
expired, malformed and over-limit records, counts records an earlier run
copied as already present, and prints counts only.

synthetic writes a generated data set in Atlas's 2.x shape, with a TTL
index on createdAt, to a MongoDB on localhost or 127.0.0.1 only, and
prints what copy should report for it, so a rehearsal never needs real
data. Both commands take the app's size limit and TTL from the app, and
the oversize fixture is sized from that limit.

Tests cover selection, mapping, the account check and the generator, and,
under TEST_STORES with mongodb and dynamodb, synthetic then copy into
DynamoDB Local, read back through the credential service, run twice.
vitest now includes scripts/**. @aws-sdk/client-sts is a dev dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ottonomy
ottonomy added this pull request to stack #282 November 11, 2026 06:09
DynamoDB Local without -sharedDb, as CI runs it, keeps a separate set of
tables per access key, so copy --expect-account local could not see the
table the rehearsal test created with key test. copy now signs with test,
exported as LOCAL_CREDENTIALS, and the test uses the same constant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant