feat: active/archive split, security hardening, telemetry removal, and core reliability fixes by tinchoz49 · Pull Request #3 · geut/duron

tinchoz49 · 2026-04-27T19:39:16Z

Summary

A major overhaul of the Duron core library spanning architecture, security, performance, reliability, and developer experience. Originally scoped as an Active/Archive table split, this branch grew to include a comprehensive audit-driven cleanup addressing ~50 findings across the codebase.

169 files changed: +4,874 / -9,929 lines (net reduction of ~5,000 lines)

Architecture Changes

Active/Archive Table Split

Split jobs → jobs_active + jobs_archive, job_steps → job_steps_active + job_steps_archive
Status-based query routing: created/active → active table, completed/failed/cancelled → archive
New archive management: _pruneArchive (advisory-locked scheduled cleanup), _truncateArchive, _getArchiveStats
REST API endpoints for archive operations
Single spans table (no active/archive split) with no FK constraints to avoid cascade issues

Instance-Level Heartbeat (replaces ping/pong)

Replaced per-job ping/pong heartbeat with an instance-level heartbeat system
New options: heartbeatInterval (default 5s) and heartbeatTimeout (default 15s, 3× interval)
Workers periodically upsert a liveness record; dead instances detected after 3 missed beats
AbortSignal added to recovery loop to prevent blocking shutdown

DB Column Rename

Renamed branch column to parallel for consistency

Security Fixes

SQL Injection Patches

GroupKey filter: Replaced raw string interpolation with parameterized sql.join / inArray
JSONB filters: Patched injection vectors in JSONB query parameters
Added test/sql-injection.test.ts to prevent regressions

Bulk DELETE Safety

Prevented DELETE /api/jobs from destroying active jobs (only targets archive)
Added test/delete-jobs-safety.test.ts with comprehensive guard tests

Advisory Lock Key

Fixed advisory lock key to prevent hash collisions between concurrent operations

Performance Improvements

Removed 14 unused indexes to reduce write overhead
Single UNION ALL query in _getJobSteps to avoid two round-trips
Removed redundant verify_concurrency CTE from _fetch
Excluded output from _getJobSteps result transfer

Telemetry Removal

ADR: Remove Local OpenTelemetry Span Storage

Removed telemetry: { local: true } option that stored OTel spans in a PostgreSQL spans table
Deleted LocalSpanExporter, spans table schema, recursive CTE queries, orphan cleanup logic
Removed SpansPanel dashboard component, spans-context, use-job-spans hook
Removed 4 @opentelemetry/* runtime dependencies (kept built-in OTel support via @opentelemetry/sdk-node)
Updated all documentation to reflect removal
See docs/adr/0001-remove-local-span-storage.md for full rationale

Reliability & Bug Fixes

Concurrency & Race Conditions

Fixed waitForJob TOCTOU race on fast-completing jobs
Added FOR UPDATE lock to _retryJob to prevent duplicate retries
Handle concurrent _timeTravelJob with ON CONFLICT DO NOTHING
Reordered _cancelJob to check job existence before updating steps
Handle expired active jobs blocking concurrency limits

Resource Leaks & Cleanup

Removed abort listener in waitForAbort release() to prevent memory leak
Removed event listeners on stop() to prevent duplication on restart
Cleaned up orphan spans after _truncateArchive and bulk DELETE
Awaited in-flight start() before proceeding with stop()
Added AbortSignal to recovery loop to prevent blocking shutdown

Data Integrity

Preserved Zod-parsed input in ActionContext constructor
Fixed serializeError to preserve custom enumerable properties
Used spread syntax for _timeTravelJob restore to prevent field drift
Replaced sql.raw(timeoutMs) with parameterized interval arithmetic
Fixed step span leak on recovery of already-completed steps
Removed dead step notification code that was never listened to

Adapter Cleanup

Made PostgresBaseAdapter and _initDb abstract
Removed base.ts.backup file, added *.backup to .gitignore

Developer Experience

Updated AGENTS.md with pre-commit verification instructions, branch workflow, frontend patterns
Added .env.example documenting required environment variables
Added repository/bugs/homepage fields to package.json
Removed dead spansCount remnants from otelia removal
Migrated linting from Biome to Oxlint/Oxfmt
Deleted stale CLAUDE.md (replaced by PI project instructions)

Dashboard Updates

Removed spans-related components (SpansPanel, spans-context, use-job-spans)
UI component updates (shadcn upgrades, import cleanup)
Filter/sort/pagination improvements in data table components

Testing

163+ tests passing
New: test/delete-jobs-safety.test.ts — bulk DELETE guards
New: test/sql-injection.test.ts — SQL injection regression tests
New: test/otel-integration.test.ts — OTel integration tests
Updated: test/client.test.ts, test/server.test.ts, test/time-travel.test.ts
Moved process-order test from shared-actions to duron package

Breaking Changes

Schema: New migration required with active/archive table split
Telemetry: telemetry: { local: true } no longer supported — use external OTel exporters via telemetry: { traceExporter }
Heartbeat: Replaced processTimeout with heartbeatInterval (5s) + heartbeatTimeout (15s)
DB Column: branch renamed to parallel
API: Bulk DELETE now only operates on archive table

…y routing for getJobs/getJobById/retry

- Split jobs/steps into active and archive tables - Jobs move from active to archive on complete/fail/cancel - Single spans table without FK constraints - SQL-native archive operations with USING joins - Prune with orphan span cleanup - Time travel restores archived jobs to active - Dashboard archive page and filter toggles - Add archive-specific tests

The test was importing from 'duron' package which failed in CI because workspace resolution doesn't work during test execution. Moving it to the duron package allows using relative imports.

- Exclude expired active jobs from verify_concurrency CTE count - _recoverJobs now archives expired jobs as failed (not reset to created) - Non-expired jobs from unresponsive owners still reset to created - Add recoverJobsInterval option (default 60s) for periodic recovery - Recovery runs before fetch when interval has elapsed - Add test: expired jobs no longer block new job selection - Update docs: client-api.mdx and multi-worker.mdx

…ecovery test - Change processTimeout default from 5000ms to 500ms for faster orphan detection - Update docs (client-api, examples, multi-worker) with new default - Rename and improve test: 'should recover expired orphan jobs from dead processes' - Add multiProcessMode + recoverJobsInterval + fake client_id to simulate crash - Add detailed comment explaining the multi-process orphan recovery scenario - Keep fetch() recovery only in multiProcessMode to avoid single-process overhead

- Replace sql.raw interpolation with parameterized inArray for groupKey array filter - Replace sql.raw(JSON.stringify) with sql.json for inputFilter/outputFilter - Add comprehensive SQL injection tests for both attack vectors Fixes #1 and #2 from FINDINGS.md

- Add status guard to _deleteJobs to always exclude active jobs - Fix sql.json() → JSON.stringify() for drizzle-orm 1.0.0-rc.4 compatibility - Add comprehensive tests for bulk delete safety Fixes #3 from FINDINGS.md

- Add FOR UPDATE to locked_source CTE to lock the source job row - Prevents concurrent retry calls from creating duplicate jobs Fixes #4 from FINDINGS.md

- Register wait in #pendingJobWaits BEFORE checking job status - If job is already terminal, remove wait and resolve immediately - Prevents NOTIFY from being missed during the status check await gap Fixes #5 from FINDINGS.md

- Remove redundant assignment that overwrote parsed input with raw input - Now input is properly validated by Zod schema before reaching action handlers - Invalid input will throw validation error instead of silently passing through Fixes #6 from FINDINGS.md

- End span and delete from #stepSpans before early return - Prevents memory leak and orphaned 'started but never ended' spans in telemetry Fixes #7 from FINDINGS.md

- Store listener references in fields for proper cleanup - Remove listeners in stop() before database.stop() to prevent use-after-close - Add guard to #setupPushListener to prevent duplicate registration - Reset listener setup flags on stop() for clean restart Fixes #8 from FINDINGS.md

- Add check for #starting at top of stop() - Prevents tearing down incomplete initialization when stop() is called during start() Fixes #9 from FINDINGS.md

- Add orphan span cleanup query after _deleteJobs - Removes spans whose job no longer exists in active or archive tables - Prevents orphan span accumulation after bulk deletion Fixes #10 from FINDINGS.md

- Add orphan span cleanup query at end of _truncateArchive - Removes spans whose job no longer exists in active or archive tables - Prevents permanent orphan spans after archive truncation Fixes #11 from FINDINGS.md

…utdown - Add optional signal parameter to RecoverJobsOptions - Check signal.aborted inside the ping loop to allow early termination - Prevents slow shutdown when stop() is called during recovery Fixes #12 from FINDINGS.md

- Replace sql.raw(timeoutMs.toString()) with parameterized ( * interval '1 millisecond') - Prevents potential SQL injection through timeout values - Improves code auditability Fixes #13 from FINDINGS.md

- Replace explicit field-by-field copy with { ...job } and { ...s } - Prevents new columns from being silently dropped during time-travel restore - More resilient to schema changes Fixes #14 from FINDINGS.md

- These dependencies were not imported anywhere in the codebase - Reduces install size and attack surface Fixes #16 from FINDINGS.md

@default

- Update @default from 5000 to 500 to match schema default - Prevents user confusion about actual timeout behavior Fixes #19 from FINDINGS.md

- Spread enumerable properties from Error instances into result - Preserves custom properties like 'code' from Node.js errors (ENOENT, ECONNREFUSED) - Improves debugging context in logs and persisted errors Fixes #20 from FINDINGS.md

- Add repository URL pointing to GitHub - Add bugs URL for issue tracking - Add homepage URL for documentation - Improves npm discoverability and user experience Fixes #32 from FINDINGS.md

- Make PostgresBaseAdapter class abstract to prevent direct instantiation - Make _initDb method abstract to enforce implementation in subclasses - Prevents runtime errors from missing implementations Fixes #33 from FINDINGS.md

Remove indexes that are never used in WHERE clauses or are redundant: jobs_active (15 → 8): - Remove description, started_at (dashboard-only) - Remove concurrency_limit, concurrency_step_limit (NEVER queried) - Remove action_status (redundant with single-column) - Remove input_fts, output_fts (output is NULL for active jobs) job_steps_active (8 → 5): - Remove job_status, job_name (redundant composites) - Remove output_fts (expensive GIN, rarely used) spans (11 → 7): - Remove kind, status_code (rarely queried) - Remove attributes, events (expensive GIN, dashboard-only) Total: 34 → 20 indexes (~40% reduction in write overhead) Fixes #34 from FINDINGS.md

- Add onConflictDoNothing() to job and step restore inserts - Prevents unhandled PK violations on concurrent time-travel of same job - Gracefully handles race condition as no-op Fixes #15 from FINDINGS.md

- Remove _notify calls from completeJobStep, failJobStep, delayJobStep, cancelJobStep - Remove step-status-changed and step-delayed event type definitions - Eliminates wasted DB work from unhandled NOTIFYs Fixes #17 from FINDINGS.md

- Add JSDoc explaining that bulk delete only affects active (non-running) jobs - Archive jobs are not affected by bulk delete - Document that active jobs are always excluded for safety Fixes #18 from FINDINGS.md

- Move job existence check before step update - Prevents wasted DB work when cancelling non-existent jobs Fixes #26 from FINDINGS.md

- Replace djb2 hash with fixed key (42) - Simpler and avoids potential lock contention from hash collisions Fixes #24 from FINDINGS.md

- Rename column in schema.ts for both active and archive step tables - Update all raw SQL references in base.ts - TypeScript field was already 'parallel', now DB column matches Fixes #25 from FINDINGS.md

- Mark Finding #30 as already fixed in FINDINGS.md - Regenerate migrations for branch→parallel rename and index removals - Add improvement plans directory

- Remove double-check of concurrency limit after acquiring lock - Use count from eligible_groups instead of re-counting - Eliminates correlated subquery per candidate job Fixes #23 from FINDINGS.md

…rips - Query both active and archive tables in single query - Add NULL as jobFinishedAt placeholder for active table - Return camelCase column names for Zod validation Fixes #27 from FINDINGS.md

…y leak - Store abort listener reference - Remove listener when release() is called - Prevents listener from staying attached after resolution Fixes #28 from FINDINGS.md

- Add .catch() to getJobStatus/getJobById chain in waitForJob - Settle wait as null on error instead of hanging forever - Clean up pending wait entry on failure - Remove unused registeredResolve variable Found in post-fix re-audit

- Use explicit column list in outer SELECT instead of SELECT * - output stays in subquery only for the search filter - Postgres no longer transfers output jsonb to the client - Drop unnecessary NULL jobFinishedAt placeholder Found in post-fix re-audit

…tance recovery

…rter support - Remove telemetry.local option and LocalSpanExporter - Remove spans table from schema - Remove /jobs/:id/spans and /steps/:id/spans API endpoints - Remove dashboard SpansPanel, SpansContext, useJobSpans/useStepSpans - Remove spansEnabled getter, getJobSpans(), getStepSpans(), flushTelemetry() - Move @opentelemetry/sdk-trace-base and sdk-trace-node to optional peer deps - Add traceExporter array support (SpanExporter | SpanExporter[]) - Add batchDelayMs option for configuring export interval - Add otel-integration.test.ts with in-memory exporter tests - Add ADR 0001 documenting the decision - Add CONTEXT.md with domain glossary

- Update telemetry.mdx to focus on external exporters - Remove spans table documentation from adapters.mdx - Remove getSpans, spansEnabled, flushTelemetry from client-api.mdx - Remove spansEnabled and spans endpoints from server-api.mdx - Remove spans visualization from dashboard.mdx

- Fix trailing comma in packages/duron/package.json exports (root cause of TS1295 errors) - Remove dead spans-related code from dashboard (SpansProvider, spans-panel, spans-context refs) - Remove unused imports/variables left from otelia removal (adapter.ts, schema.ts, base.ts, test files) - Run oxfmt on affected files

client.ts statically imports @opentelemetry/resources, @opentelemetry/semantic-conventions, sdk-trace-base, and sdk-trace-node. These must be in dependencies — the module loader resolves all static imports on first import of 'duron/client', regardless of whether telemetry is configured. Removes the optional peerDependencies since they're not optional.

- telemetry.mdx: update installation to only require exporter package - client-api.mdx: remove stale local/spanProcessors options, show current API

- Fix stale comment in client.ts referencing telemetry.local - Remove spansCount from ArchiveStatsResponse type (server no longer returns it) - Remove dead 'Archived Spans' card from archive page

tinchoz49 added 30 commits April 18, 2026 17:25

feat: add active/archive schema definitions and abstract adapter methods

c050789

feat: rename tables to active/archive schema, all tests pass

08a6b4c

feat: implement archive move logic for complete/fail/cancel, add quer…

52e8bee

…y routing for getJobs/getJobById/retry

feat: implement step and span query routing for active/archive tables

365acfa

docs: update AGENTS.md - do not use git worktrees

2eb1cf5

fix: move process-order test from shared-actions to duron package

2bc71fa

The test was importing from 'duron' package which failed in CI because workspace resolution doesn't work during test execution. Moving it to the duron package allows using relative imports.

docs: add pre-commit verification instructions to AGENTS.md

282e70c

fix: type error and import order in process-order test

1df6561

fix kimi bad code

2400c9f

more changes, created FINDINGS.md to solve race condition issues

bf333d2

fix lint

e974aed

fix: prevent bulk DELETE from destroying active jobs

978ddb0

- Add status guard to _deleteJobs to always exclude active jobs - Fix sql.json() → JSON.stringify() for drizzle-orm 1.0.0-rc.4 compatibility - Add comprehensive tests for bulk delete safety Fixes #3 from FINDINGS.md

fix: add FOR UPDATE lock to _retryJob to prevent duplicate retries

bc27d1d

- Add FOR UPDATE to locked_source CTE to lock the source job row - Prevents concurrent retry calls from creating duplicate jobs Fixes #4 from FINDINGS.md

fix: prevent waitForJob TOCTOU race on fast-completing jobs

714bf44

- Register wait in #pendingJobWaits BEFORE checking job status - If job is already terminal, remove wait and resolve immediately - Prevents NOTIFY from being missed during the status check await gap Fixes #5 from FINDINGS.md

fix: prevent step span leak on recovery of already-completed steps

823c53c

- End span and delete from #stepSpans before early return - Prevents memory leak and orphaned 'started but never ended' spans in telemetry Fixes #7 from FINDINGS.md

fix: await in-flight start() before proceeding with stop()

dd52470

- Add check for #starting at top of stop() - Prevents tearing down incomplete initialization when stop() is called during start() Fixes #9 from FINDINGS.md

fix: clean up orphan spans after bulk DELETE /api/jobs

071e4a5

- Add orphan span cleanup query after _deleteJobs - Removes spans whose job no longer exists in active or archive tables - Prevents orphan span accumulation after bulk deletion Fixes #10 from FINDINGS.md

fix: clean up orphan spans after _truncateArchive

bc91a9e

- Add orphan span cleanup query at end of _truncateArchive - Removes spans whose job no longer exists in active or archive tables - Prevents permanent orphan spans after archive truncation Fixes #11 from FINDINGS.md

fix: replace sql.raw(timeoutMs) with parameterized interval arithmetic

0a9c619

- Replace sql.raw(timeoutMs.toString()) with parameterized ( * interval '1 millisecond') - Prevents potential SQL injection through timeout values - Improves code auditability Fixes #13 from FINDINGS.md

fix: use spread syntax for _timeTravelJob restore to prevent field drift

ffa9e75

- Replace explicit field-by-field copy with { ...job } and { ...s } - Prevents new columns from being silently dropped during time-travel restore - More resilient to schema changes Fixes #14 from FINDINGS.md

chore: remove unused dependencies p-all and p-timeout

af26789

- These dependencies were not imported anywhere in the codebase - Reduces install size and attack surface Fixes #16 from FINDINGS.md

fix: correct processTimeout JSDoc to match actual default of 500ms

a126312

- Update @default from 5000 to 500 to match schema default - Prevents user confusion about actual timeout behavior Fixes #19 from FINDINGS.md

tinchoz49 added 24 commits July 20, 2026 10:24

docs: add repository, bugs, and homepage fields to package.json

86d5de4

- Add repository URL pointing to GitHub - Add bugs URL for issue tracking - Add homepage URL for documentation - Improves npm discoverability and user experience Fixes #32 from FINDINGS.md

fix: make PostgresBaseAdapter and _initDb abstract

c641e40

- Make PostgresBaseAdapter class abstract to prevent direct instantiation - Make _initDb method abstract to enforce implementation in subclasses - Prevents runtime errors from missing implementations Fixes #33 from FINDINGS.md

fix: handle concurrent _timeTravelJob with ON CONFLICT DO NOTHING

b6d71d5

- Add onConflictDoNothing() to job and step restore inserts - Prevents unhandled PK violations on concurrent time-travel of same job - Gracefully handles race condition as no-op Fixes #15 from FINDINGS.md

docs: document _deleteJobs limitation to only operate on active table

013a67d

- Add JSDoc explaining that bulk delete only affects active (non-running) jobs - Archive jobs are not affected by bulk delete - Document that active jobs are always excluded for safety Fixes #18 from FINDINGS.md

fix: reorder _cancelJob to check job existence before updating steps

16372a5

- Move job existence check before step update - Prevents wasted DB work when cancelling non-existent jobs Fixes #26 from FINDINGS.md

fix: use fixed advisory lock key to prevent hash collisions

828c560

- Replace djb2 hash with fixed key (42) - Simpler and avoids potential lock contention from hash collisions Fixes #24 from FINDINGS.md

refactor: rename DB column from 'branch' to 'parallel' for consistency

4c574ff

- Rename column in schema.ts for both active and archive step tables - Update all raw SQL references in base.ts - TypeScript field was already 'parallel', now DB column matches Fixes #25 from FINDINGS.md

chore: update findings status and regenerate migrations

77a277f

- Mark Finding #30 as already fixed in FINDINGS.md - Regenerate migrations for branch→parallel rename and index removals - Add improvement plans directory

perf: remove redundant verify_concurrency CTE from _fetch

c334c92

- Remove double-check of concurrency limit after acquiring lock - Use count from eligible_groups instead of re-counting - Eliminates correlated subquery per candidate job Fixes #23 from FINDINGS.md

perf: use single UNION ALL query in _getJobSteps to avoid two round-t…

b0bdbd4

…rips - Query both active and archive tables in single query - Add NULL as jobFinishedAt placeholder for active table - Return camelCase column names for Zod validation Fixes #27 from FINDINGS.md

fix: remove abort listener in waitForAbort release() to prevent memor…

94224f6

…y leak - Store abort listener reference - Remove listener when release() is called - Prevents listener from staying attached after resolution Fixes #28 from FINDINGS.md

feat: replace ping/pong with instance-level heartbeat for crashed-ins…

5742996

…tance recovery

docs: fix telemetry docs for built-in OTel dependencies

4dbd0e3

- telemetry.mdx: update installation to only require exporter package - client-api.mdx: remove stale local/spanProcessors options, show current API

remove unused files

26aa59a

remove unused files

c33acf5

chore: remove dead spansCount remnants from otelia removal

ead5ee2

- Fix stale comment in client.ts referencing telemetry.local - Remove spansCount from ArchiveStatsResponse type (server no longer returns it) - Remove dead 'Archived Spans' card from archive page

tinchoz49 changed the title ~~Feature: active archive table~~ feat: active/archive split, security hardening, telemetry removal, and core reliability fixes Aug 29, 2026

tinchoz49 added 4 commits August 29, 2026 16:13

chore: update jobs-table view and tsconfig

6ec5f5f

chore: remove CLAUDE.md and clean up stale local telemetry references

2603889

chore: deduplicate telemetry entry in AGENTS.md

ee750f7

docs: add frontend patterns section to AGENTS.md

85aca58

tinchoz49 merged commit d170f6c into main Aug 29, 2026
2 checks passed

Navigation Menu

Uh oh!

feat: active/archive split, security hardening, telemetry removal, and core reliability fixes - #3

feat: active/archive split, security hardening, telemetry removal, and core reliability fixes#3
tinchoz49 merged 60 commits into
mainfrom
feature/active-archive-and-time-travel

tinchoz49 commented Apr 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

tinchoz49 commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Architecture Changes

Active/Archive Table Split

Instance-Level Heartbeat (replaces ping/pong)

DB Column Rename

Security Fixes

SQL Injection Patches

Bulk DELETE Safety

Advisory Lock Key

Performance Improvements

Telemetry Removal

ADR: Remove Local OpenTelemetry Span Storage

Reliability & Bug Fixes

Concurrency & Race Conditions

Resource Leaks & Cleanup

Data Integrity

Adapter Cleanup

Developer Experience

Dashboard Updates

Testing

Breaking Changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

tinchoz49 commented Apr 27, 2026 •

edited

Loading