Skip to content

IOC check: match IaC, container and dependency sources against threat feeds (#110) - #162

Open
ms280690 wants to merge 2 commits into
issue-109-threat-feedsfrom
issue-110-ioc-check
Open

ms280690 wants to merge 2 commits into
issue-109-threat-feedsfrom
issue-110-ioc-check

Conversation

@ms280690

Copy link
Copy Markdown
Collaborator

Closes #110. Stacked on #153 (issue-109-threat-feeds); retarget to main once that merges.

What

  • .github/actions/ioc-check: stdlib-only ioc_check.py extracts IPv4 addresses, CIDRs, hostnames and URLs from selected files and matches them against threatfox.json, feodo.json and urlhaus.json. Categories: c2-ip (Feodo and ThreatFox ip), c2-domain (ThreatFox domain, subdomains included), malware-url (ThreatFox and URLhaus url). A source CIDR that contains a feed address counts. RFC 1918, loopback, link-local, RFC 5737, 0.0.0.0/8, CGNAT, multicast and 0.0.0.0/0 never match on either side. Malware URLs with a path are redacted to <feed>#<id> in annotations, summary, report and findings.json.
  • Files come from git ls-files, or git diff --name-only <base-sha> HEAD on PRs, filtered by include and exclude globs (defaults cover Terraform, CDK, Dockerfiles, YAML/JSON/TOML and lockfiles; skip node_modules, .terraform, vendor, tests). block-on picks which categories fail; empty is advisory. Outputs match-count, blocking-count, report-path.
  • ioc-check.yml reusable workflow: snapshot via the threat-feeds composite (feeds-ref), composite on changed files for PRs, ioc-check artefact, and one PR comment found by its <!-- ioc-check --> marker and edited in place (a clean run rewrites it to "no indicators matched"). Skipped on fork PRs. Caller grants pull-requests: write for the comment.
  • normalize.py: ThreatFox entries keep the IOC id in meta.id so the redaction can cite it (URLhaus already had one).
  • CI: new IOC Check job runs the fixture suite, scans the whole repo against the fixture feeds (must be clean), then the live snapshot in advisory mode once threat-feeds-latest exists (same skip pattern as the Threat Feeds job).
  • README rows and "IOC Check (threat feeds)" section; docs/egress-allowlist.md row.

Tests

tests/ioc-check/run.sh with fixture feeds (class E addresses and .example domains) and one source fixture per extractor row of the issue: Terraform security-group CIDR and DNS record, Dockerfile FROM and curl, Helm hostAliases, package.json git URL and malware URL, go.mod replace, plus a negatives file with private, documentation, loopback and 0.0.0.0/0 values that the fixture feeds also list. Asserts the exact match set with file:line, CIDR containment, redaction in every output, advisory and subset block-on, include and exclude globs, --files-from, a clean tree and exit codes.

Deferred

🤖 Generated with Claude Code

https://claude.ai/code/session_01NPH9145e4gFo6bmXvMgBoF

… matching

Match IPs, CIDRs, domains and URLs found in IaC, container and dependency
sources against the ThreatFox, Feodo Tracker and URLhaus feeds from the
threat-feeds snapshot. Categories are c2-ip, c2-domain and malware-url; a
source CIDR that contains a feed address counts, and private, loopback,
link-local, documentation and catch-all ranges never match on either side.
Malware URLs with a path are redacted to <feed>#<id> in every output.

The composite selects tracked files (or files changed against base-sha on
pull requests), filters them by include and exclude globs, and writes
inline annotations, findings.json and a markdown report. The reusable
workflow downloads the snapshot, runs the composite on changed files, and
keeps one PR comment edited in place. ThreatFox entries now carry their
IOC id in meta so the redaction has something to cite.

CI runs the fixture suite, scans the whole repo against the fixture feeds
expecting no matches, and runs the live snapshot in advisory mode once the
first mirror release exists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NPH9145e4gFo6bmXvMgBoF
@ms280690 ms280690 self-assigned this Oct 12, 2026
dependency-review-action asks api.deps.dev for the repository of any
added dependency GitHub has no source URL for, and that fetch has no
error handling, so the harden-runner block made the job fail with
"fetch failed" once the ioc-check fixtures added a go.mod and a
package.json with placeholder hosts. Allow the two hosts the action's
scorecard column uses and record why in the egress allowlist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NPH9145e4gFo6bmXvMgBoF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant