Skip to content

feat(service): auto clean records from the DB #263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LironEr merged 4 commits into from
Oct 6, 2026
Merged

feat(service): auto clean records from the DB #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2718da4
feat(service): auto clean records from the DB
LironEr Oct 5, 2026
817cd43
fix max duration & run cron every 6 hours to check for tasks
LironEr Oct 5, 2026
9a3a212
upgrade packages to fix security issues
LironEr Oct 5, 2026
7ab72d9
fix
LironEr Oct 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/build-web.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ on:

jobs:
build:
runs-on: ubuntu-22.04
runs-on: ubuntu-26.04
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
node-version: '24'
cache: 'yarn'

- name: Install dependencies
Expand Down
20 changes: 10 additions & 10 deletions .github/workflows/checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ on:

jobs:
packages:
runs-on: ubuntu-22.04
runs-on: ubuntu-26.04

strategy:
fail-fast: false
matrix:
node-version: [20.x, 22.x, 24.x]
node-version: [22.x, 24.x]

steps:
- uses: actions/checkout@v4
Expand All @@ -35,16 +35,16 @@ jobs:
run: yarn test-packages

- uses: codecov/codecov-action@v1
if: ${{ matrix.node-version == '22.x' }} # run this step only one time
if: ${{ matrix.node-version == '24.x' }} # run this step only one time
with:
file: ./coverage/coverage-final.json

- name: Lint
if: ${{ matrix.node-version == '22.x' }} # run this step only one time
if: ${{ matrix.node-version == '24.x' }} # run this step only one time
run: yarn lint-packages

website:
runs-on: ubuntu-22.04
runs-on: ubuntu-26.04
defaults:
run:
working-directory: ./apps/website
Expand All @@ -55,7 +55,7 @@ jobs:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
node-version: '24'
cache: 'yarn'

- name: Install dependencies
Expand All @@ -68,7 +68,7 @@ jobs:
run: yarn build

service:
runs-on: ubuntu-22.04
runs-on: ubuntu-26.04
defaults:
run:
working-directory: ./apps/service
Expand All @@ -82,7 +82,7 @@ jobs:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
node-version: '24'
cache: 'yarn'

- name: Install dependencies
Expand All @@ -106,7 +106,7 @@ jobs:
run: yarn stop:mock-services

platform:
runs-on: ubuntu-22.04
runs-on: ubuntu-26.04
defaults:
run:
working-directory: ./apps/platform
Expand All @@ -120,7 +120,7 @@ jobs:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
node-version: '24'
cache: 'yarn'

- name: Install dependencies
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/publish-docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ concurrency:

jobs:
publish-docker:
runs-on: ubuntu-22.04
runs-on: ubuntu-26.04
permissions:
contents: read
packages: write
Expand Down Expand Up @@ -69,7 +69,7 @@ jobs:
- name: Install Node
uses: actions/setup-node@v4
with:
node-version: 22
node-version: 24
registry-url: https://registry.npmjs.org/
cache: yarn

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/publish-packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
jobs:
test:
name: publish-packages
runs-on: ubuntu-22.04
runs-on: ubuntu-26.04
permissions:
contents: read
id-token: write # needed for provenance data generation
Expand Down Expand Up @@ -38,7 +38,7 @@ jobs:
- name: Install Node
uses: actions/setup-node@v4
with:
node-version: 22
node-version: 24
registry-url: https://registry.npmjs.org/
cache: yarn

Expand Down
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ on:

jobs:
build:
runs-on: ubuntu-22.04
runs-on: ubuntu-26.04
steps:
- uses: actions/checkout@v4
- name: Setup Node
Expand Down Expand Up @@ -259,9 +259,9 @@ For details about what data BundleMon collects and stores, see the [privacy docu

Limitations of the free hosted service:

- Records created by a PR will be deleted after 30 days.
- Records created by a PR will be deleted after 14 days.
- Records in branches without activity (new commits) will be deleted after 180 days.
- After 90 days only the latest record per day will be kept.
- After 60 days only the latest record per day will be kept.

More limitations may be added in the future as more and more projects use the free hosted service.

Expand Down
53 changes: 38 additions & 15 deletions apps/service/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,23 @@

### Environment variables

| Name | Description | Default |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| MONGO_URL | **Required**<br/>MongoDB connection URL | `-` |
| MONGO_DB_NAME | MongoDB database name | `bundlemon` |
| MONGO_DB_USER | MongoDB username | `-` |
| MONGO_DB_PASSWORD | MongoDB password | `-` |
| HTTP_SCHEMA | HTTP schema (`http` or `https`) | `https` |
| PORT | Port number for the service | `8080` |
| ROOT_DOMAIN | Root domain for the service | `bundlemon.dev` |
| APP_DOMAIN | Application domain, defaults to ROOT_DOMAIN | same as `ROOT_DOMAIN` |
| API_PATH_PREFIX | API path prefix | `/api` |
| SHOULD_SERVE_WEBSITE | Flag to determine if the website should be served | `true` |
| SECRET_SESSION_KEY | This key will be used for securely signing session cookies.<br />Auto generated each time the service starts, Prefer to set a key. | Auto generated |
| MAX_SESSION_AGE_SECONDS | Maximum session age in seconds | `21600` (6 hours) |
| MAX_BODY_SIZE_BYTES | Max body size in bytes | `1048576` (1 MB) |
| Name | Description | Default |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| MONGO_URL | **Required**<br/>MongoDB connection URL | `-` |
| MONGO_DB_NAME | MongoDB database name | `bundlemon` |
| MONGO_DB_USER | MongoDB username | `-` |
| MONGO_DB_PASSWORD | MongoDB password | `-` |
| HTTP_SCHEMA | HTTP schema (`http` or `https`) | `https` |
| PORT | Port number for the service | `8080` |
| ROOT_DOMAIN | Root domain for the service | `bundlemon.dev` |
| APP_DOMAIN | Application domain, defaults to ROOT_DOMAIN | same as `ROOT_DOMAIN` |
| API_PATH_PREFIX | API path prefix | `/api` |
| SHOULD_SERVE_WEBSITE | Flag to determine if the website should be served | `true` |
| SECRET_SESSION_KEY | This key will be used for securely signing session cookies.<br />Auto generated each time the service starts, Prefer to set a key. | Auto generated |
| MAX_SESSION_AGE_SECONDS | Maximum session age in seconds | `21600` (6 hours) |
| MAX_BODY_SIZE_BYTES | Max body size in bytes | `1048576` (1 MB) |
| DELETE_STALE_BRANCHES_SCHEDULE | Cron expression (UTC, e.g. `0 3 * * 0`) to automatically delete branches without activity for 180 days. Not set = never runs | `-` |
| DELETE_OLD_RECORDS_SCHEDULE | Cron expression (UTC) to automatically keep one record per day for records older than 60 days. Not set = never runs | `-` |

<details>
<summary>Generate secret session key</summary>
Expand All @@ -30,6 +32,27 @@ node apps/service/scripts/generateSecretKey.js

</details>

### Scheduled tasks

Maintenance tasks can run automatically, each one has its own schedule, set by an environment variable (see the table above).
A task without a schedule never runs, and when no task has a schedule the service doesn't start a scheduler.

- A task runs when it is due: its last successful run is older than the most recent scheduled time (so a run missed while the server was down is made up when it starts).
- Safe with multiple instances, a lease in MongoDB (`taskState` collection) makes sure only one instance runs a task at a time.
- Every task is limited to 2 minutes, a task that didn't finish continues in the next run. A failed or unfinished task is retried after 15 minutes, up to 3 attempts for each scheduled time (then it waits for the next scheduled time). When the server shuts down, the running task is aborted and continues in the next run (right away, without the delay).
- The history of the runs (status, deleted records, errors) is saved in the `taskRuns` collection for a year.
- The tasks can also be run manually, with a confirmation prompt: `scripts/deleteStaleBranches.ts` and `scripts/deleteOldRecords.ts`.

#### Vercel

There is no long living server, Vercel Crons (`crons` in `vercel.json`) call `/cron/tasks` every 6 hours instead of the scheduler ticks. The schedules are the same environment variables as on a regular server, set them in the Vercel project.
Each request runs at most one due task (the one whose scheduled time is the oldest), so tasks that run one after the other can't exceed the function time limit, the next due task runs in the next cron request.
Set the `CRON_SECRET` environment variable in the Vercel project, Vercel sends it as `Authorization: Bearer <CRON_SECRET>` and the `/cron/tasks` endpoint rejects any request without it.
The Hobby plan limits each cron to once a day, so there are 4 daily crons (00:00, 06:00, 12:00, 18:00 UTC) that call the same endpoint. The 2 minutes limit requires Fluid compute (Project Settings → Functions).

On Vercel the DB indexes are not created on startup. `yarn vercel-deploy [--prod]` deploys and then calls `POST /internal/init-db` on the new deployment (protected by `CRON_SECRET` as well).
The script reads `CRON_SECRET` from the Vercel project env vars of the deployed environment (`vercel env run`), if it's a "Sensitive" env var it can't be read back, set it in your local env instead. If the deployment is protected (Deployment Protection), also set `VERCEL_AUTOMATION_BYPASS_SECRET`.

### GitHub integration (optional)

If you want your self hosted BundleMon service to interact with GitHub, you will need to create GitHub App.
Expand Down
5 changes: 3 additions & 2 deletions apps/service/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"lint": "yarn --cwd ../../ nx lint service --verbose",
"generate-schemas": "node ./scripts/generateSchemas.js",
"prevercel-deploy": "yarn --cwd ../../ nx build service -c vercel --verbose",
"vercel-deploy": "vercel deploy",
"vercel-deploy": "./scripts/vercelDeploy.sh",
"start:mock-services": "docker compose -f docker-compose.test.yml up --remove-orphans",
"stop:mock-services": "docker compose -f docker-compose.test.yml down",
"gen-local-data": "node -r @swc-node/register -r dotenv/config ./scripts/generateLocalData.ts dotenv_config_path=.development.env"
Expand All @@ -24,6 +24,7 @@
"@octokit/rest": "^20.0.1",
"bundlemon-markdown-output": "^2.0.1",
"bundlemon-utils": "^2.0.1",
"cron-parser": "^4.9.0",
"env-var": "^7.3.1",
"fastify": "^5.12.5",
"mongodb": "^6.18.0"
Expand All @@ -33,6 +34,6 @@
"dotenv": "^16.3.1",
"ts-json-schema-generator": "^1.3.0",
"typescript": "^5.1.6",
"vercel": "^62.1.0"
"vercel": "^62.4.0"
}
}
10 changes: 8 additions & 2 deletions apps/service/project.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"defaultConfiguration": "production",
"options": {
"platform": "node",
"target": "node18",
"target": "node22",
"outputPath": "{projectRoot}/dist",
"format": ["cjs"],
"bundle": true,
Expand Down Expand Up @@ -76,7 +76,13 @@
"main": "{projectRoot}/vercel/serverless.ts",
"thirdParty": true,
"minify": true,
"assets": []
"assets": [
{
"glob": "tasks.js",
"input": "{projectRoot}/vercel",
"output": "."
}
]
}
}
},
Expand Down
97 changes: 11 additions & 86 deletions apps/service/scripts/deleteOldRecords.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,93 +1,20 @@
// Keep the latest record per day after AGGREGATE_RECORDS_OLDER_THAN_DAYS days
// Keep the latest record per day after x days
// Manual run with confirmation, the same logic runs automatically as a scheduled task (see src/tasks)

import pino from 'pino';
import { closeMongoClient } from '@/framework/mongo/client';
import { getCommitRecordsCollection } from '@/framework/mongo/commitRecords';
import { ObjectId } from 'mongodb';
import { deleteOldRecordsTask } from '@/tasks/definitions/deleteOldRecords';

const AGGREGATE_RECORDS_OLDER_THAN_DAYS = 90;
const logger = pino();
// no time limit, the signal is never aborted
const signal = new AbortController().signal;

(async () => {
try {
console.log('Fetch records...');
const commitRecordsCollection = await getCommitRecordsCollection();
const agg = await commitRecordsCollection
.aggregate<{
projectId: string;
subProject?: string;
branch: string;
aggDate: string;
idsToDelete: string[];
}>([
{
$sort: {
creationDate: -1,
},
},
{
$match: {
creationDate: {
$lt: new Date(new Date().setDate(new Date().getDate() - AGGREGATE_RECORDS_OLDER_THAN_DAYS)),
},
},
},
{
$group: {
_id: {
projectId: '$projectId',
subProject: '$subProject',
branch: '$branch',
aggDate: {
$dateToString: {
format: '%Y-%m-%d',
date: '$creationDate',
},
},
},
records: {
$push: '$_id',
},
},
},
{
// check that the records has at least 2 records
$match: {
'records.1': {
$exists: true,
},
},
},
{
$project: {
projectId: '$_id.projectId',
subProject: '$_id.subProject',
branch: '$_id.branch',
aggDate: '$_id.aggDate',
// the first id is the last record per that period, so we keep it
idsToDelete: {
$slice: [
'$records',
1,
{
$subtract: [
{
$size: '$records',
},
1,
],
},
],
},
},
},
{
$unset: '_id',
},
])
.toArray();
const { details: dryRunDetails } = await deleteOldRecordsTask.run({ dryRun: true, signal, logger });

const idsToDelete = agg.map((x) => x.idsToDelete).flat();

console.log(`Total records to delete: ${idsToDelete.length}`);
console.log(`Total records to delete: ${dryRunDetails?.totalRecords}`);

console.log('Are you sure you want to delete all these records? (y/n)');
const answer = await new Promise<string>((resolve) => {
Expand All @@ -103,11 +30,9 @@ const AGGREGATE_RECORDS_OLDER_THAN_DAYS = 90;

console.log('Deleting records...');

const result = await commitRecordsCollection.deleteMany({
_id: { $in: idsToDelete.map((id) => new ObjectId(id)) },
});
const { details } = await deleteOldRecordsTask.run({ dryRun: false, signal, logger });

console.log(`Deleted records: ${result.deletedCount}`);
console.log(`Deleted records: ${details?.deletedRecords}`);

process.exit(0);
} finally {
Expand Down
Loading
Loading