Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions .challengeai/challenge-ac.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# ChallengeAC

Agile artifacts: whether what was asked for can be traced to what was built.

## Covers

Acceptance criteria, requirement traceability, ambiguity in what was requested,
and audit readiness of the delivery record.

## The requirement

SP 800-53 Rev. 5 System and Services Acquisition (SA-3, system development life
cycle) and Configuration Management (CM-3, change control). An assessor asks how
the team knows it built what was asked for, and how a change was decided.

## Traceability without ceremony

The requirement is that a capability can be traced to its implementation and its
test. It is not that a particular artifact exists before work starts.

A traceability record written as the work lands describes the system, so it
survives an assessment as written. A backlog written ahead of the work describes
intent, which drifts from what shipped and has to be reconciled against reality
before an assessor can use it. Either can satisfy the requirement; only one is
accurate by construction.

Where the record claims a capability is complete, that claim is checkable. A
partial state written honestly, with a sentence on the shortfall, reads better in
assessment than a completion that an assessor disproves.

## Change control

An assessor asking how a change was controlled is asking for a durable record
that shows what changed, who approved it, and how it was verified. A pull
request carrying one concern, with review and conversation resolution required,
answers that. A pull request carrying several makes the record of why any one of
them landed ambiguous.

## In this repository

FedSpeak has no formal traceability record — no requirements table, no linked
issue tracker mapping capability to implementation to test. What it has
instead is lightweight but real: `.github/ISSUE_TEMPLATE/add_acronym.md`
structures every acronym-addition request into a fixed shape (acronym, full
name, description, agency, category, source), and `.github/pull_request_template.md`
requires the PR itself to check off `lint`, `typecheck`, `test:run`, and
`build` before merge — that's the closest thing to a change-verification
record this repo has, and it's checked by the contributor, not enforced by
CI status alone (CI does run these same four independently, so a checked box
that doesn't match the CI result is visible). There's no traceability record
mapping any of the three named test files (`decoder.test.ts`,
`encoder.test.ts`, `truncate.test.ts`) to a specific capability beyond the
tests' own names and structure.

Whether one PR carries one concern, review approval is required, or
conversation resolution is enforced before merge to `main` is not verified —
see `profile.yml`'s `gates` section for what's actually confirmed versus what
`CONTRIBUTING.md` describe as the intended workflow.

## Evidence

- A traceability record mapping each capability to its implementation and test,
with honest states rather than aspirational ones.
- Change history, one concern per change, with the verification recorded.
- Review approval and conversation resolution enforced rather than optional.

FedSpeak's evidence here is thin by the standard above: the PR template's
checklist and the issue template's fixed schema are the whole of it. Test
files exist and are real, but nothing maps a specific test to a specific
capability by name.

## Review checklist

- Does the capability that just landed appear in the traceability record,
naming its implementation and its test?
- Does the record claim completion for something only partly built?
- Does the change record say how it was verified?
- Is more than one concern bundled here, making the record of why it landed
ambiguous?
- Was something ambiguous resolved by asking, or by guessing and documenting the
guess?
- **FedSpeak-specific:** does a new acronym entry follow
`add_acronym.md`'s fixed schema, and is its source cited in the issue or PR
description? An entry added without a source is unverifiable content in a
reference tool whose entire value is accuracy.
111 changes: 111 additions & 0 deletions .challengeai/challenge-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# ChallengeAPI

The API: its contract, what it accepts, and what it discloses.

## Covers

API contracts, input validation, interoperability, versioning, rate limiting,
and federal API governance.

## The requirement

- **SP 800-53 Rev. 5** System and Communications Protection (SC) and System and
Information Integrity (SI-10, input validation).
- **The Federal Source Code Policy and API guidance** expect government APIs to
be documented, versioned, and stable for the people who build against them.
- **OMB guidance on open data** expects machine-readable access where the data
is public.

## The contract

An API is a promise to people who cannot be consulted before it changes.

- **Versioned in the path**, moving independently of the product release, so
adding a version does not by itself require a major release of the system.
- **Published as a machine-readable document**, so a client can generate against
it rather than read prose.
- **Validated at the edge of the process.** Every parameter is parsed and
bounded before it reaches a query. Unbounded pagination and unbounded result
sizes are both denial-of-service vectors and cost vectors.
- **Health reported against dependencies**, not just process liveness. A process
that is up and cannot reach its data is not healthy.
- **Rate limited at the edge**, so a burst is refused rather than queued.

## Errors

An error names what was wrong with the request without describing the internals
of the system. A stack trace, a driver message, or a query fragment in a
response body is an information disclosure finding.

## In this repository

FedSpeak's is a genuinely public API — two routes, `POST/GET /api/decode` and
`POST/GET /api/encode` (`netlify/functions/decode.ts`, `encode.ts`, redirected
from `/api/*` to `/.netlify/functions/*` in `netlify.toml`), each accepting a
single lookup (`acronym`/`name`) or a text-scan request (`text`), backed by
the same static `src/shared/data/acronyms.json` the website and npm package
read directly.

- **Versioning:** none in the path — `/api/decode` and `/api/encode` carry no
version segment. The API contract's own version (`public/openapi.json`'s
`info.version`) and the product's npm version (`package.json`,
`cli-package/package.json`) are two independent numbers by design — a
package release doesn't have to imply a contract change, and vice versa —
so the two currently reading differently (`"0.1.0"` vs `"1.0.0"`) isn't
itself a defect. What's real is that `info.version` has never moved since
it was introduced: `git log -p -- public/openapi.json` shows only two
commits touching this file, one adding the spec at `0.1.0` and a later one
bumping the *product* to `v1.0.0` without touching the spec at all — so the
contract version isn't being tracked against contract changes either, which
is the actual gap.
- **Published as a machine-readable document:** yes, genuinely —
`public/openapi.json` is a real OpenAPI 3.1 spec with both routes, request
schemas, and examples, served as a static file.
- **Validated at the edge:** partially. Both handlers correctly reject a
missing `acronym`/`name`/`text` with a 400 and a usage example. Neither
bounds the *length* of an incoming `text` parameter before it reaches
`scanText`/`scanTextForNames` (`src/shared/decoder.ts`,
`src/shared/encoder.ts`) — `scanTextForNames` in particular sorts and
substring-searches the full acronym set (1,119 entries) against the
supplied text on every call. An arbitrarily large `text` body is accepted
and processed with no length cap found in either function or either
handler — an unbounded-input-size gap in the sense this tool's "requirement"
section describes, even though the low-value target likely makes it low
severity in practice.
- **Health:** no health endpoint of any kind was found — no `/api/health`,
no dependency check. There's also no external dependency to check against;
the data is a file bundled into the function's own deployment, not a
network call, so "health" here would only ever be process liveness, which
Netlify's own platform already reports.
- **Rate limiting:** none found — no rate-limit logic in either handler, no
Netlify-level configuration in `netlify.toml`. `SECURITY.md` names "no
authentication" as a deliberate design choice but doesn't address rate
limiting or abuse at all.

## Evidence

- The published contract document.
- Route tests exercising the documented behaviour, including failure cases.

`public/openapi.json` is real, versioned-if-stale evidence of the contract.
Route-level test coverage is indirect: `tests/decoder.test.ts` and
`tests/encoder.test.ts` exercise the shared `decode`/`encode`/`scanText`
functions the handlers call, including edge cases, but nothing in `tests/`
invokes `netlify/functions/decode.ts`/`encode.ts` directly — the handler's
own request-parsing, CORS headers, and error-status logic (405, 400, 500)
aren't covered by anything in the test suite as of this writing.

## Review checklist

- Is every new parameter validated and bounded?
- Does a new route appear in the contract document in the same pull request?
- Does an error response leak an internal detail?
- Does a change alter the shape of an existing response? That is a breaking
change to a published contract, and it belongs behind a new version.
- Is anything personal placed in a URL, where it reaches logs and history?
- **FedSpeak-specific:** does a PR that changes the actual API contract — a
route added, a parameter added or removed, a response shape changed —
bump `public/openapi.json`'s `info.version`? That's the trigger that
should move it, independent of any `package.json` version bump; as of
this writing nothing enforces it, and the spec's version has never moved
since it was introduced.
90 changes: 90 additions & 0 deletions .challengeai/challenge-ato.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# ChallengeATO

Authorization readiness: whether this system could be assessed today and whether
the evidence would hold.

## Covers

Control traceability, evidence sufficiency, the security package, the privacy
assessment, and bounded penetration testing against a running instance.

## The requirement

- **FISMA** requires federal systems to implement an information security
program and to be authorized before operating.
- **NIST SP 800-37 Rev. 2** defines the Risk Management Framework.
- **FIPS 199** categorizes the system as low, moderate, or high impact. The
categorization drives everything downstream, and it is recorded in
`profile.yml`.
- **NIST SP 800-53 Rev. 5** provides the control baseline the categorization
selects.
- **NIST SP 800-53A Rev. 5** provides the procedures an assessor uses.
- **The E-Government Act** requires a Privacy Impact Assessment where a
system handles information **in identifiable form** — not merely
information "about" individuals in aggregate — or initiates a qualifying
electronic collection from ten or more non-federal persons.

## The package

A security package is a set, numbered so it can be handed over as one. The
System Security Plan describes the system, its boundary, and how each control is
met. Around it sit the assessment plan and report, the plan of action and
milestones, the risk assessment, the contingency and incident response plans,
configuration management, access control policy, continuous monitoring, and the
privacy impact assessment.

The boundary described in the SSP is the boundary the system actually
provisions. Anything outside it is inherited from the provider's own
authorization and is named as inherited rather than claimed.

## In this repository

FedSpeak has none of this, deliberately rather than by oversight — no
`docs/security/`, no SSP, no FIPS-199 categorization, no POA&M. It is not a
federal information system: no agency operates it, it carries no data on an
agency's behalf, and MetaPhase has not pursued an ATO for it. `profile.yml`
records `control_baseline: null` and `impact_level: null` to say exactly
that — "not pursued," not "unassessed and pending."

What FedSpeak has instead is `SECURITY.md`, a standard open-source
vulnerability-disclosure policy (report via a GitHub Issue tagged `security`,
or GitHub's private vulnerability reporting; 48-hour acknowledgment target).
`SECURITY.md`'s own "Scope" section is an honest, narrow self-assessment —
"public, read-only reference tool... no authentication... no user accounts...
no database" — and it holds up against the actual code: no auth anywhere in
`netlify/functions/`, no persistence beyond the committed
`src/shared/data/acronyms.json`, no environment variables or secrets required
to run it (confirmed: no `.env` file, no secret references in the Netlify
functions or `netlify.toml`).

## Evidence

- Pipeline output produced on every run: scan results, test reports, and a
component inventory.
- A traceability record mapping each capability to its implementation and test.
- A reviewable record of the deployed configuration, so the boundary can be
checked rather than taken on description.

FedSpeak's CI (`ci.yml`) produces none of the first category — no security
scan runs on a pull request or push at all; `dependency-hygiene.yml`'s
weekly `npm audit` is the only scan of any kind, and it's advisory
(`continue-on-error: true`) and never runs against a PR. Test output exists
(`npm run test:run` in CI) but isn't retained as an artifact. The "reviewable
record of deployed configuration" is effectively `netlify.toml` plus
Netlify's own dashboard — there's no infrastructure-as-code to inspect (see
ChallengeIaC).

## Review checklist

- Does the SSP describe the system as it is now, or as it was at the last
release?
- Is every control claim backed by something a reader can open?
- Does the boundary in the SSP match what is actually provisioned?
- Has the data model started handling personal information the PIA does not
mention?
- **A weakness, POA&M item, or vulnerability goes into a security document only
with explicit approval.** Cataloguing theoretical gaps manufactures a record
of insecurity. State what is implemented; independent assessment produces the
authoritative findings. FedSpeak has no security document to add one to —
a real vulnerability report belongs in `SECURITY.md`'s own disclosure
process (a GitHub Security Advisory), not fabricated here.
108 changes: 108 additions & 0 deletions .challengeai/challenge-cd.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# ChallengeCD

Deployment: whether a release can be made safely and undone quickly.

## Covers

Rollout safety, rollback readiness, secrets handling, and the operational
controls around putting a change in front of users.

## The requirement

SP 800-53 Rev. 5 Configuration Management (CM-3 change control, CM-5 access
restrictions for change) and Contingency Planning (CP-10 system recovery). An
authorized system must be able to show that changes are controlled and that a
bad change can be reversed.

## Know where the risk is actually taken

Every deployment model takes the risk somewhere. What matters is that everyone
knows where, because a branch treated as a safety gate that does not gate
anything is worse than no gate: it produces confidence without protection.

Where a promotion redeploys code that is already live, the promotion names a
version rather than de-risking anything, and the risk was taken earlier.

## Rollout and rollback

- **Health-gated rollout**, so a release that never passes its health check
reverses itself rather than waiting to be noticed.
- **Rollback documented and rehearsed.** A procedure that has never been run is
a hypothesis.
- **Credentials from short-lived federated identity**, so no long-lived key
exists to leak or rotate.

A change that cannot be rolled back without also reversing a data migration is
a different class of change, and saying so at review time is the point.

## In this repository

FedSpeak has two genuinely separate deployment paths that don't share the
same rigor, and this section names both honestly.

**The website + API** deploy via Netlify's own GitHub App integration — a
push to `main` (production) or `dev` (per `README.md`'s branching section,
though `netlify.toml` doesn't distinguish branch contexts) triggers a
Netlify build and deploy directly, entirely outside GitHub Actions. There is
no health-gated rollout: Netlify's default deploy model replaces the live
site once the build succeeds, with no automated check of the *deployed*
result before it goes live to all traffic. Rollback is whatever Netlify's
own dashboard provides (redeploying a previous deploy from its deploy
history) — a real capability, but not documented anywhere in this
repository as a runbook, and not something this repository's CI performs or
verifies.

**The npm packages** (`fedspeak` and `@metaphase-tech/fedspeak`, both
confirmed live on the registry at `1.0.0`) are not published by CI at all —
no workflow in `.github/workflows/` runs `npm publish` or references an npm
token. Publishing is a manual, undocumented-in-repo process:
`scripts/sync-cli-package.sh` copies `src/shared/*` into `cli-package/`, and
someone presumably runs `npm publish` from there by hand afterward — this
step, and who's authorized to run it, isn't written down anywhere in
`CONTRIBUTING.md` or `README.md`.

## Secrets

Secrets reach the running system at start from a secret store, never from the
repository and never from a build artifact. The repository is scanned on every
push and on a schedule, and a verified finding fails the build.

A value inlined at build time is baked into whatever ships and is readable by
anyone holding the artifact. That can be an acceptable trade, but it is a
decision to make deliberately rather than discover.

FedSpeak needs none of this machinery because it holds no secrets at all —
confirmed directly: no `.env` file or `.env.example`, no environment
variable referenced in `netlify/functions/`, `src/`, or `netlify.toml`, and
`README.md`/`SECURITY.md` both state this explicitly ("No secrets or env
vars needed"). The npm publish step above would need an npm auth token
somewhere on whoever's machine runs it, but that credential lives outside
this repository entirely — not in CI, not in a GitHub secret.

## Evidence

- Deploy runs recorded per change and retained.
- A release runbook carrying the verification commands and the rollback
procedure.
- Whatever the platform keeps as the previous good revision is the rollback
record.

Netlify retains its own deploy history (visible in Netlify's dashboard, not
in this repository), which functions as the rollback record by platform
default. There is no release runbook in this repository for either the
website/API path or the npm-publish path.

## Review checklist

- Does the deploy watch its own run with a failure exit status? A watch that
returns success whichever way the run ended will let a failed deploy be tagged
as a release.
- Is the run identified by commit rather than by branch? A branch filter asked
seconds after a merge returns the previous run, which is already green.
- Can this change be rolled back without a data migration being reversed? If
not, say so in the pull request.
- Does anything new read a secret at build time rather than at run time?
- **FedSpeak-specific:** if a PR changes `src/shared/`, does the same PR (or
a documented follow-up) run `scripts/sync-cli-package.sh` before the next
npm publish? A merged fix that never gets synced into `cli-package/` ships
to the website and API but not to either published npm package.
Loading
Loading