CI: cap the RustFS blocking-thread pool on the CAS-S3 lanes and collect its log

[filimonov]

CI only, one file (ci/jobs/scripts/clickhouse_proc.py, start_rustfs): environment knobs for the RustFS instance that backs the CAS-S3 stateless lanes, plus routing its log into the existing rustfs.log artifact. No server code, no test changes.

Why RustFS memory counts as ClickHouse memory. The stateless job runs the server, MinIO, azurite, RustFS and the test workers in one docker container, i.e. one cgroup, with no --memory flag. The server's MemoryWorker uses the cgroup source, which sums the whole cgroup's anon from memory.stat, and max_server_memory_usage is derived from the host (0.9 × 30.6 GiB). So every byte RustFS allocates is charged against the server's limit: it inflates MemoryTracking, drives the memory-capped max_threads, and under ASan produced the Code: 241 storms.

Why RustFS grows. RustFS 1.0.0-rc.3 runs every disk I/O through tokio's spawn_blocking: under the default strict durability a PUT costs four or more blocking dispatches (write, fsync chain, rename). The blocking pool grows one thread per concurrent dispatch, an idle thread lives 60 s, and each thread pins about 12 MB of retained mimalloc arena plus a 1 MiB stack. Under a parallel test suite the pool only ratchets up: on a three-hour MSan shard it reached 1074 threads and 14 GB of anonymous memory (r = 0.95 between thread count and RSS). The same mechanism also made rename_data cost 37–69 ms per call.

Knobs

Variable	Value	What it does	Expected effect on the shard
RUSTFS_RUNTIME_MAX_BLOCKING_THREADS	64	Hard cap on the tokio blocking pool (default unbounded up to 512+)	Bounds the thread-owned memory to ~130 threads / ~2 GB instead of 1074 / 14 GB
RUSTFS_RUNTIME_THREAD_KEEP_ALIVE	5	Idle blocking threads exit after 5 s instead of 60 s	The pool shrinks between test bursts instead of ratcheting
RUSTFS_ALLOCATOR_RECLAIM_ENABLED	true	Periodic mimalloc arena return to the OS (default off)	Freed arenas leave RSS instead of staying retained
RUSTFS_ALLOCATOR_RECLAIM_INTERVAL_SECS	30	Reclaim period	Keeps the reclaim cheap and regular
RUSTFS_DURABILITY_MODE	relaxed	No fsync chain on commit; 2 blocking dispatches per PUT instead of 4+	Removes the 37–69 ms rename_data and halves thread demand. Acceptable for CI: the pool lives under ci/tmp and is wiped per run
RUSTFS_OBS_USE_STDOUT	true	RustFS writes its log (error level by default) to stdout	The already-uploaded rustfs.log artifact gets content; today RustFS writes to logs/ under its cwd and the artifact is nearly empty

Local measurement (same six-pass workload of the MSan CAS-S3 shard, CI MSan binary, two RustFS instances; knobs applied to the CAS instance only, the other one as control):

	without knobs	with knobs
wall time, six passes	205 min	143 min
RustFS threads, max	1074	130
RustFS anonymous memory, max	15.2 GB	2.55 GB
rename_data per call	44–69 ms	0.5–0.8 ms
Code: 241 in test output	12	4
test median / p90, first → last sixth	2.90/17.0 → 3.77/23.2 s	1.85/9.7 → 2.61/12.9 s

The five memory knobs were measured together, so their individual contributions are not separated; the durability knob is the one with a semantic cost and is intended for CI only. The RustFS log volume at error level was 1.5 MB over 143 minutes.

Related: #2298

Changelog category (leave one):

• CI Fix or improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

CI: cap the RustFS blocking-thread pool and enable allocator reclaim on the CAS-S3 stateless lanes, so the object store no longer inflates the server's cgroup memory; collect the RustFS log as an artifact.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

🤖 Generated with Claude Code

https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx

[github-actions]

Workflow [PR], commit [e283ead]