Skip to content

ci: port IntelOwl CI upgrades (jest exit code, uv, shared caches, DB snapshot, parallel tests) - #245

Open
0ssigeno wants to merge 5 commits into
developfrom
ci/intelowl-upgrades
Open

0ssigeno wants to merge 5 commits into
developfrom
ci/intelowl-upgrades

Conversation

@0ssigeno

@0ssigeno 0ssigeno commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Goal

Bring the CI improvements developed in IntelOwlPrivate back into the shared workflows so every Certego repo can use them.

Why

IntelOwlPrivate cut its GitHub Actions minutes by ~40% and its python test run by ~half, and found a Jest bug that turns failing tests green. Other repos still run the old template.

How

  1. _node.yml: Jest step keeps the test exit code when coverage is uploaded (bug: failing tests used to pass); output stays in the log via tee.
  2. _python.yml: license check moved to its own job, gated at job level (a Docker action's image is built even when its step is skipped).
  3. actions/services: postgres healthcheck 1s × 60 retries instead of 10s × 5.
  4. _python.yml: run_tests input (default true); false only warms caches.
  5. _python.yml: venv installed with uv; disk space freed only on a venv cache miss and with opt-in free_disk_space.
  6. _python.yml: test_parallelism input → manage.py test --parallel (coverage combine from CI: support parallel coverage in reusable python workflow #240 kept).
  7. _python.yml: opt-in use_db_snapshot — cached pg_dump keyed on migration files, --keepdb, cold migrate when the snapshot holds migrations no longer on disk.
  8. _python.yml: cache_key input; set (e.g. ci) to share venv/APT caches warmed by a default-branch push.
  9. _release_and_tag.yml: optional per-image docker_mapping[].context, falling back to working_directory.
  10. workflows/README.md + CHANGELOG: new inputs and recommended caller patterns (path filters, draft gate + ready_for_review, concurrency, cache warm-up push, release-title check).
  11. Re-synced .github/workflows/_release_and_tag.yml with its root copy — it had drifted on develop.

Plan and decisions: .claude/plans/ci_public/plan.md

Behaviour changes with default inputs: the license check is now a separate check (Check requirements licenses) that runs on every run, not only on a venv miss — add it to required checks; the venv is created by uv and has no pip — custom commands must call uv pip.

Out of scope

  • _detect_changes.yml and create_*_cache.yaml are kept, even though IntelOwl replaced them with path-filtered callers.
  • use_db_snapshot / test_parallelism are not wired into the non-coverage (unittest discover) path; documented instead.
  • .github/workflows/release.yml and requirements-linters.txt copies that already differ from root on develop are left as they are.

🤖 Generated with Claude Code

0ssigeno and others added 2 commits October 21, 2026 09:15
…snapshot, parallel tests)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ores .venv, dropdb --force, pip in uv venv

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@0ssigeno

Copy link
Copy Markdown
Collaborator Author

CI green on the self-test fixtures (python, node, licenses). Reviewed before push; fixes in 3ff5b31. Companion IntelOwl sync: certego/IntelOwlPrivate#1373.

@0ssigeno
0ssigeno marked this pull request as ready for review October 21, 2026 07:22
0ssigeno and others added 2 commits October 21, 2026 12:43
Only projects whose requirements fill the runner disk need the 30-70s cleanup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The template never calls pip; callers' custom commands use uv pip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread .github/workflows/_python.yml Outdated
Comment on lines +574 to +576
docker exec postgres dropdb -U "$PG_USER" --if-exists "test_${PG_DB}"
docker exec postgres createdb -U "$PG_USER" "test_${PG_DB}"
docker exec -i postgres psql -v ON_ERROR_STOP=1 -U "$PG_USER" -d "test_${PG_DB}" < db-snapshot.sql

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perché non usare pg_restore per questi 3 step?

Suggested change
docker exec postgres dropdb -U "$PG_USER" --if-exists "test_${PG_DB}"
docker exec postgres createdb -U "$PG_USER" "test_${PG_DB}"
docker exec -i postgres psql -v ON_ERROR_STOP=1 -U "$PG_USER" -d "test_${PG_DB}" < db-snapshot.sql
docker exec postgres pg_restore -U "$PG_USER" --if-exists --clean --create -d "test_${PG_DB}" < db-snapshot.sql

Oppure usare l'opzione template di createdb (che usa il DB indicato come base da copiare)

Suggested change
docker exec postgres dropdb -U "$PG_USER" --if-exists "test_${PG_DB}"
docker exec postgres createdb -U "$PG_USER" "test_${PG_DB}"
docker exec -i postgres psql -v ON_ERROR_STOP=1 -U "$PG_USER" -d "test_${PG_DB}" < db-snapshot.sql
docker exec postgres dropdb -U "$PG_USER" --if-exists "test_${PG_DB}"
docker exec postgres createdb -U "$PG_USER" -T ${PG_DB} "test_${PG_DB}"

Comment thread .github/workflows/_python.yml Outdated
Comment on lines +542 to +552
# A migration applied in the snapshot but gone from disk (renamed, squashed): migrate from zero instead.
find . -path ./.venv -prune -o -path '*/migrations/0*.py' -print | awk -F/ '{sub(/\.py$/, "", $NF); print $(NF-2) "/" $NF}' | sort > "$RUNNER_TEMP/on_disk"
apps=$(cut -d/ -f1 "$RUNNER_TEMP/on_disk" | sort -u | paste -sd'|')
stale=$(docker exec postgres psql -U "$PG_USER" -d "$PG_DB" -Atc "SELECT app || '/' || name FROM django_migrations" \
| awk -v re="^($apps)/" '$0 ~ re' | sort | comm -23 - "$RUNNER_TEMP/on_disk")
if [[ -n "$stale" ]]; then
echo "::notice::DB snapshot holds migrations missing on disk, migrating from zero: $stale"
docker exec postgres dropdb --force -U "$PG_USER" "$PG_DB"
docker exec postgres createdb -U "$PG_USER" "$PG_DB"
exit 0
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ho un dubbio: se la snapshot è cachata in base all'hash dei file delle migrazioni, come è possibile che facciamo una cache-hit e nella snapshot ci siano migrazioni cancellate?


- name: Restore Database from Cache
# An exact or restore-keys match both leave the snapshot on disk.
if: inputs.use_db_snapshot && hashFiles('db-snapshot.sql') != ''

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perché qui hai messo hashFiles('db-snapshot.sql') != '' invece di steps.db-cache.outputs.cache-hit == 'true'?

In base al commento non capisco bene il senso di usare hashFiles.

…ore comments

Server-side copy instead of reloading the SQL dump; cache-hit is false on a
restore-keys match, hence hashFiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants