Conversation
- pull workspace state from the TFC API instead of per-workspace init
- write migration-summary.{md,json}; replace workspace.tmpl with summary.tmpl
- map TFC vcs settings to SG VCSTriggers (push, PR speculative, file filters)
- add SGDefault*/workspaceOverrides for vcs auth, source kind and triggers
Validate generated payloads against schema/sg-payload.schema.json via yajsv.
- sg-migrate.sh + Dockerfile run the full pipeline inside a container - scripts/migrate.sh: init/apply/enrich/convert/validate/import/triggers/all - merge TFC variable sets, parallel convert/import, register VCS triggers - move convert_hcl_to_json.sh into scripts/
sg-cli exits 0 even when individual workflows fail, so the importer now parses its output. Workflows rejected because their pinned Terraform version is above SG's managed ceiling (1.5.7, last MPL/FOSS release) are re-imported with SGDefaultTerraformVersion, the payload is patched in place, and each case is logged to export/terraform-version-fallbacks.log with a printed notice. Any other per-workflow failure now fails the run. Also: sg_retry captured the wrong exit status (always 0), definitive HTTP 4xx responses are no longer retried, the trigger pass skips workflows that were never created, and failure output no longer echoes the API token.
'sg-migrate.sh completion bash|zsh' prints a completion script for the current shell session; 'init' prints the source line for the user's shell. Runs natively like 'clean'.
Running sg-migrate.sh without a command used to start the whole pipeline; it now prints the help menu, locally, without touching Docker. Usage and hints show the name the user invoked (./sg-migrate.sh) instead of the in-container script path.
Without TFE_TOKEN or a terraform login credentials file, apply used to warn and then fail deep inside terraform with "Invalid provider configuration". Both the host entrypoint and migrate.sh now stop with a clear message before anything runs. Add an explicit provider "tfe" block so tfHostname also applies to the provider, not just the state export.
An expired 'terraform login' session (or a bad TFE_TOKEN) used to fail deep inside terraform. Resolve the token the tfe provider will use, in its own precedence, and check it against /api/v2/account/details first; stop with a clear message on 401/403 or when the host is unreachable.
Every workflow was hardcoded to shared runners unless overridden per workspace. Add SGDefaultRunnerConstraints (default shared) so all workflows can be put behind a private runner group in one place; workspaceOverrides[name].RunnerConstraints still wins per workspace. Validated: type is shared|private, and private requires names.
Move the TFC/SG API helpers out of migrate.sh into scripts/lib/ and add the building blocks the upcoming init wizard, preflight and checklist need: interactive prompt helpers (tty or scripted answers), cached tfvars reads, paginated TFC listing (orgs/projects/workspaces) and SG lookups for integrations, runner groups, workflows and secrets. The enrich script now shares the same token resolution as the provider. Also stop turning curl connection failures into "000000" status codes.
'init' now walks through the configuration instead of copying the example file: it lists the TFC organisations, projects and workspaces the token can see, the SG VCS and cloud connectors and runner groups, and derives the source kind and repo prefix from the chosen connector. Every step falls back to free text when a token is missing or an API call fails, and non-interactive runs keep the template behaviour. The generated terraform.tfvars keeps the example's order and comments; re-running the wizard prefills the current values and keeps a .bak.
Verify up front what used to fail minutes later inside terraform or as API 400s: TFC token and org, that the workspace selection matches anything, SG token and org, every connector / secret / runner group referenced in terraform.tfvars (defaults and workspaceOverrides), SGDefaultSourceConfigDestKind and SGDefaultTerraformVersion format, the exportPath vs export-dir mismatch, and override keys that match no workspace. Runs automatically for apply, import and all; also available as 'preflight'; --skip-preflight bypasses it.
'all' now records each phase's inputs in .sg/state.json and skips phases whose inputs have not changed, so a failed run resumes instead of re-running terraform apply. Import records per payload file which workflows landed and which failed; unchanged, fully imported files are skipped and files with failures are re-imported (sg-cli updates existing workflows in place). --fresh discards the state. --project and --workspace restrict every phase to a subset; apply maps --workspace to the workspacenames variable.
apply now prints a condensed migration summary (workflows per project, sensitive variables skipped, unpinned Terraform versions, non-remote execution modes, renames, failed state exports) instead of leaving it in a file nobody opens. import shows a per-workflow table before the confirmation prompt: create/update, Terraform version with the 1.5.7 fallback marked, runner, triggers, variable and secret counts. 'import --dry-run' stops after the table without touching anything.
Map the API messages users actually hit (unknown connector, missing runner group, repository not reachable, bad approvers, invalid names or VCS kind, rejected token, Terraform ceiling) to a one-line hint naming the terraform.tfvars field to fix. Shown once per distinct hint, both for direct API calls and for sg-cli's per-workflow failures; unknown messages still show the raw response.
After import (or via 'checklist'), write export/post-import-checklist.md
listing everything the migration could not do by itself: secrets to
fill in, workflows that failed to import, Terraform version fallbacks
to verify, VCS triggers that failed, state that could not be exported,
non-remote execution modes and renamed workflows, with deep links into
the SG UI (SG_UI_URL). For every sensitive variable TFC would not
expose, create an SG secret with the value CHANGE_ME, reference it from
the workflow (env var or IaC input, ${secret::<name>}) and patch the
payload to match; --no-secret-stubs opts out. Trigger registration
results are now recorded in the run state.
The integrations list has no ResourceType; the kind (GITHUB_COM, AWS_RBAC, ...) is Settings.kind. Map it from there, make the type filters null-safe, and only treat msg/data as the item list when it is actually an array.
_w_csv_json emitted "[]" twice when the list was empty (grep -v on empty input trips the fallback under pipefail), so the generated file was not valid HCL and preflight reported every field as missing. Validate the file right after writing it, report an unparsable tfvars as such in preflight, remember the SG org / API host chosen in the wizard so a new shell does not need SG_ORG, pretty-print objects in the generated file and separate menu values from their descriptions.
The hint used $SHELL, which is the login shell and was wrong for a bash login shell running zsh, so the bash script got sourced into zsh and broke on the first Tab. sg-migrate.sh now detects the parent shell and 'completion' with no argument prints the matching script; each script also hands off to the other shell's version when sourced by mistake.
The per-workflow plan passed whatever sg_list_workflows returned to jq --argjson and crashed when the response was not a plain array. List helpers now always yield an array (also unwrapping data.Workflows) and the plan guards the value. The wizard no longer asks for the profile name (not required), the repo URL prefix, approvers or the fallback Terraform version — they take sensible defaults and are edited in terraform.tfvars. sg-migrate.sh says when it runs in Docker.
When terraform.tfvars is missing, all runs the wizard and then asks whether to continue right away or edit the file first; only the non-interactive template copy still stops, since it holds placeholders.
Wrap argument parsing and dispatch in main so bash reads the whole script up front. The repo is bind-mounted into the container, and a file edited during a long run was read half-old, half-new, ending in "unexpected EOF while looking for matching quote" after the import had already succeeded.
TFC_* / TFE_* variables (TFC_WORKSPACE_NAME, the TFC_AWS_* dynamic credential settings, ...) only mean something inside Terraform Cloud. A new ignoreVarPatterns input (default ["^TFC_", "^TFE_"]) drops matching terraform and env variables from the payloads, from workspaces and from variable sets, lists them in the migration summary and keeps them out of the sensitive-variable list so no secret stubs are created for them. The init wizard asks whether to strip them. Also fix the secrets link in the checklist (orgs/<org>?tab=secrets).
The "^(AWS|AZURE|GCP)_" filter also matched AZURE_DEVOPS, so an Azure DevOps VCS connector could be chosen as the cloud connector and only schema validation caught it. Both pickers now use the exact kind lists (VCS also accepts AZURE_DEVOPS_SP and GITLAB_OAUTH_SSH and maps them to the source kind), are sorted by name, and preflight rejects a DeploymentPlatformConfig kind outside the schema enum. The validator no longer prints each failure twice.
With more than one TFC project selected, the wizard's new step 3 asks whether the global connectors and the tfc-<project> groups apply to all projects. If not, it asks per project for the cloud connector, the VCS connector (kind and repo prefix follow, with the usual mismatch warning) and the workflow group — the default tfc-<project>, one of the org's existing groups (GET wfgrps/listall/) or a typed name — and writes the answers as projectOverrides. "same as the default" removes the key so the global value flows through; fields the wizard does not manage (Approvers, RunnerConstraints, ...) are kept. A re-run with existing entries defaults to reviewing them and pre-selects the previous picks. terraform.tfvars is rendered wholesale from the wizard's answers, so a hand-written workspaceOverrides (or projectOverrides) block used to be deleted by the next init. Both maps are now read before the write and re-rendered through a JSON-shaped HCL map renderer (_tfvars_map) that hcl2json round-trips; inner comments are the only thing not preserved. The policy step also asks about stripCloudAuthVars, and the review shows one row per project plus what was kept from the file.
Connector ids, private runner groups and DeploymentPlatformConfig kinds inside projectOverrides are checked like the global and per-workspace ones; a project's VCS connector is compared with the VCS kind that project ends up with; projectOverrides keys that match no TFC project are flagged as a typo (the real project names are listed). The import context states the workflow-group policy (reuse when it exists, create otherwise, or must-exist with --no-create-groups) and warns early when a project's group changed since its last import, which the plan would refuse as a move. One line reports whether cloud credential variables are stripped.
…update semantics README: how init picks connectors and the group per project, the projectOverrides precedence, that hand-written override blocks survive init, the workflow-group rules (reuse/create, never moved, no name collisions across projects sharing a group, mapping file deprecated), the cloud credential variables that are stripped per connector kind, and what a re-run updates (payload change -> update, unchanged -> skip, triggers only when changed, secrets never overwritten). CLAUDE.md follows the code: plan_groups, group_for, the trigger sha, _tfvars_map, wizard_projects, the preflight collectors and the state layout.
… workspace After the real projectOverrides / workspaceOverrides maps, terraform.tfvars now carries one commented entry for every selected project and workspace that has no entry yet, pre-filled with what it gets today: the picked connectors, runners and approvers (a workspace inherits its project's override when one exists), the default tfc-<project> group, and for a workspace the Terraform version it runs in TFC. Fine-tuning becomes "move the entry up and change a value" instead of typing field names; each entry is annotated with its workspace count or project and version. _tfvars_map aligns attribute names and takes per-key notes.
apply already passed the patterns to terraform, but the plan, the import subset, the probe, the triggers and the secret stubs compared exact names, so 'import --workspace team-*' imported nothing. One matcher (bash case / jq ws_selected) now serves all of them; '*' alone means no filter so the unchanged-file skip still applies, and a filter matching nothing is an error instead of an empty success.
…orkspace) The provider only excludes by tag, so the module filters the selected workspaces itself (local.selectedWorkspaces) and everything downstream, state export included, follows. terraform.tfvars keeps the permanent list, --exclude-workspace adds to it for one run, and the run scope helpers move to lib/scope.sh so apply, preflight, the plan, the import subset, the probe and the triggers use one definition. Excluded workspaces are listed in the migration summary; init keeps the list on a re-run and the review shows it.
Until now --project only picked payload files after a full export and had to be the file segment. The module gets tfProjects (names or slugs, [] = all), applied after the name filters: tfe_workspace is read for the name-filtered set because the project of a workspace is only known from there. The orchestrator passes --project as tfProjects, matches payload files by slug so 'My First Project' and my-first-project are the same, and preflight refuses a --project that names no project (a tfProjects typo is a warning) and previews the count the apply will export.
--tag replaces tfWorkspaceTags, --exclude-tag adds to tfWorkspaceIgnoreTags, both are handed to terraform apply and previewed by preflight. Tags are not in the payload, so they shape the export only.
The orchestrator read SG_TFVARS but terraform apply still loaded terraform.tfvars from the module dir, the enrich script fell back to the default path and the Docker wrapper did not forward the variable. The path is now resolved once (absolute), passed to -var-file and to enrich, and the wrapper translates a host path into the container: under /app when the file is in the checkout, mounted read-only otherwise. clean --all only removes the module's own file, and apply warns when both files exist because terraform auto-loads terraform.tfvars on top of -var-file.
… set -u sg_create_workflow tested SG_HTTP_CODE after a $(...) call that never sets it in the calling shell, so with set -u the 409 fallback died before the PATCH and the workflow was reported as failed with an empty message. The '<code>: <body>' text the helper prints already carries the status.
import (and all) write export/run-result.json and run-summary.md: outcome (planned, blocked, success, failed), the scope flags, the group table, the plan problems and one row per workflow with plan, result, Terraform version, state and trigger status; the markdown is made for a CI job summary. Import results and state gain an 'updated' list so a PATCHed workflow is told from a created one (the probe workflow counts as created). --dry-run is documented for 'all' too: the local export runs, nothing is created in StackGuardian.
…uns keep unmanaged keys A file written by an older version keeps working (missing settings take their defaults), but users had no way to see what is new short of diffing the example. init --upgrade appends every setting the file lacks with the comment and default from terraform.tfvars.example under a dated header and touches nothing else, so hand comments survive and it runs without a terminal; preflight names the missing settings until then. The wizard re-run now also carries over settings it does not ask about (tfProjects, cloudAuthVarPatterns, hand-added keys) instead of dropping them.
…e's uncommented ones The first version derived the list from terraform.tfvars.example, so the override maps, cloudAuthVarPatterns and tfHostname were never appended. The list now comes from variables.tf; a setting the example only shows commented out is appended that way (plus 'key = <default>' when the default fits on one line), a mention in the file counts as present so a second run adds nothing, the result is parsed and rolled back if broken, and init writes tfHostname so a fresh file is complete.
…--vcs-connector, --runner-group, --workflow-group, --set) A pipeline can now pass the connectors with the trigger instead of editing them into terraform.tfvars. Only the connector id is typed: its kind is looked up in the org, and the VCS kind and repo URL prefix follow the connector. With --project the values become that project's projectOverrides entry, so every workspace of the project inherits them; without it they set the SGDefault* values. --set KEY=VALUE covers any other variable. The flags form an overlay merged over the tfvars for the run, so preflight, the plan, enrich, the import and the run result all see the same values and apply receives them as -var; nothing is written to the file, a later run without the flags PATCHes the workflows back. init, clean and completion refuse the flags, a standalone import warns that they only reach the workflows through the export. Also fixes tfvars_get/tfvars_get_json dropping a false value (jq's // treats false like a missing key), which read stripCloudAuthVars = false as unset.
The API host was only reachable through the undocumented SG_BASE_URL and the checklist's UI links always pointed at app.stackguardian.io. --region (or SG_REGION) sets both: eu = api.app/app.stackguardian.io (default), us = api.us/us.stackguardian.io. An explicit SG_BASE_URL or SG_UI_URL still wins, init remembers the region in .sg/state.json (older files with only sg_base_url keep working), the wrapper forwards SG_REGION, and the run result records region, apiUrl and uiUrl.
…jects The project comes before the workspaces: with more than one project the wizard lists them with their workspace counts and asks for all of them or some by name or slug (unknown names are re-asked, a previous choice is the default), the workspace questions then apply within those projects, and tfProjects is written to the file ([] = every project) instead of only surviving as a carried-over key. Also fixes a stray 'jq: invalid JSON text passed to --argjson' from the paginated workflow list when the API answers with an empty or non-JSON body for a group that does not exist yet.
…--upgrade leave it alone Every user saw a warning that their file predates cloudAuthVarPatterns, a list nobody needs to edit for a normal migration. It stays documented in variables.tf and the example, --set and a hand edit still work, but it is no longer reported as missing or appended.
…ate first The plan already knew a workflow existed (ACTION update), yet the import still sent it through sg-cli's create and waited for the 409 to switch to PATCH, printing a failure line on every re-run. import_bulk now lists the group once, updates the known workflows straight away (state re-uploaded) and only creates the rest; the 409 handling stays as the safety net for a race between plan and import. Update failures print 'Failed to update' and are parsed like create failures. Also drops the empty 'run scope:' line when only configuration flags are given.
… 3.2, so a native enrich merged nothing
…riables on one line each; details with -v
…tead of sg-cli's raw output; -v keeps it
…ort and upload share one line when clean
…e left to the execution preset
…ist sections, per-file ✓s, the group table when every group is reused
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns the migrator from a hand-driven
terraform applyplus manual editing into one guided pipeline:./sg-migrate.sh allconfigures, preflights, extracts and transforms the workspaces, merges Variable Sets, converts and validates the payloads, and bulk-imports them into per-project workflow groups. The transformer gains per-project output, a migration summary, API-based state export, VCS trigger mapping and execution-preset support.Suggested review order:
transformer/terraform-cloud/locals.tf(the TFC to SG mapping, where payload shape is decided), thenscripts/migrate.sh(phase orchestration, import parsing, resume state), thenscripts/lib/(wizard, preflight, checklist). The rest is glue and docs.Changes
sg-payload.<project>.jsonper TFC project withWorkflowGroup.name = tfc-<project>;migration-summary.{md,json}; state pulled from the TFC API instead of a per-workspaceterraform init; TFC VCS settings mapped toVCSTriggers; sensitive variables skipped and reported;ignoreVarPatternsstripsTFC_*/TFE_*variables.SGDefault*inputs plus typedworkspaceOverrides.SGTerraformVersionSourceand nullableSGDefaultTerraformVersion/SGDefaultRunnerConstraintsleave keys out of the payload so the SG API fills them from the org's execution preset.sg-migrate.shrunsscripts/migrate.shin the Docker image (native fallback) withinit|preflight|apply|enrich|convert|validate|import|triggers|checklist|all|clean|completion. Phases resume via.sg/state.json, filter with--project/--workspace;import --dry-runstops at the plan.tfc-<project>groups, parses sg-cli output (it exits 0 on per-workflow failures), retries a pin above SG's 1.5.7 Terraform ceiling with the fallback or preset, maps known API errors to a hint naming the tfvars field, and writesexport/post-import-checklist.mdwithCHANGE_MEsecret stubs for sensitive variables.schema/sg-payload.schema.jsonwithscripts/validate_payload.sh; README rewritten for the orchestrated flow.Testing
./sg-migrate.sh allagainst a multi-project TFC org: wizard, preflight, one payload per project, groups created, workflows imported, checklist written.allafter a failed import: unchanged phases are skipped, only files with failures are re-imported.export/terraform-version-fallbacks.log; withSGDefaultTerraformVersion = nullthe preset applies.terraform fmt -checkon the transformer and./scripts/validate_payload.sh export/sg-payload.*.jsonpass.Risks
sg-payload.jsonbecame one file per project andWorkflowGroup.nameistfc-<project>rather than the TFC project ID.--no-create-groupsand--no-secret-stubsopt out.terraformVersionorRunnerConstraintsomitted depend on the org's execution preset being set as intended.Deployment Notes
SG_API_TOKEN,SG_ORG, optionalSG_BASE_URL, andTFE_TOKEN(preferred overterraform login); tuning knobs are in the README.--buildto rebuild); nothing is pushed to a registry.