Skip to content

fix: frame streaming UploadPartCopy to bound memory (OOM under copy flood) - #108

Merged
ServerSideHannes merged 2 commits into
mainfrom
fix/copy-oom-frame-streaming
Jul 5, 2026
Merged

ServerSideHannes merged 2 commits into
mainfrom
fix/copy-oom-frame-streaming

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

Problem

The streaming UploadPartCopy path OOMKilled s3proxy-python under ScyllaDB's snapshot-rename copy flood (backups returning 502/503; every host accumulating ~TiB of failed uploads).

Root cause: _pump_copy_chunks buffered a whole 64 MB internal part and crypto.encrypt()'d it in one shot, with 2 uploads in flight, so a single copy peaked at ~192 MB — while copy_pipeline_peak() reserved only 4 * MAX_BUFFER_SIZE (32 MB). The memory governor therefore admitted ~6× too many concurrent copies and the pod exceeded its 1 GiB limit.

tracemalloc from a dying pod confirmed it: crypto.py:354 (whole-part encrypt) + multipart/copy.py:393/398 at 192 MB ×3, governed_active_mb=162 vs rss=922.

Fix — converge the copy writer onto the framed UploadPart path

  • _pump_copy_chunks rewritten to mirror _stream_and_upload_framed: reads FRAME_PLAINTEXT_SIZE plaintext frames via _PlaintextReader, encrypts each with encrypt_frame, accumulates one internal part, uploads it, then moves on — one part at a time, framed. Peak is now O(one part ciphertext + one frame); never holds a whole part's plaintext+ciphertext together, no parallel whole-part encrypts.
  • Internal parts sized with memory_bounded_part_size (same as the upload path). This also caps parts at MAX_INTERNAL_PARTS_PER_CLIENT, fixing a latent range collision: the old calculate_optimal_part_size (64 MB) overflowed the per-client 20-part range for parts >1.28 GB.
  • copy_pipeline_peak == streaming_upload_peak + 2 * MAX_BUFFER_SIZE (framed per-part peak plus the copy-only source-read pipeline) — reservation now covers the real peak.
  • Gate the previously ungoverned _simple_copy_part behind reserve_memory.

Measured (tracemalloc, real handler)

copy size before after
64 MB ~192 MB (32 MB reserved) 32 MB
512 MB scales with parts 73.6 MB

Peak is now bounded and independent of object size, so the governor bounds concurrency without clamping copies to run alone.

Backwards compatibility

Writer-only change emitting the framed format the upload path already writes — no new format. The reader is untouched and still decodes legacy single-seal parts (overhead == 1 frame). Verified: legacy 3 MB and 40 MB single-seal parts still decode; new multi-frame parts round-trip. No migration, no re-encryption.

The CopyObject (misc) path was already framed-safe (8 MB sequential parts) and governed — not the culprit; it now reserves conservatively.

Tests

  • New test_copy_reservation_vs_real.py — drives the real handler under tracemalloc, asserts reserved >= real_peak (fails on old code).
  • New multi-frame copy round-trip guard in test_streaming_copy.py (old writer only produced single-seal parts, so this was never exercised).
  • test_copy_memory_governing.py updated to pin the convergence.
  • Full unit suite green (568 passed), ruff clean.

…lood)

The streaming UploadPartCopy path buffered a whole 64MB internal part and
encrypted it in one shot, with 2 uploads in flight, so a single copy peaked
at ~192MB while copy_pipeline_peak() reserved only 4*MAX_BUFFER_SIZE (32MB).
The memory governor therefore admitted ~6x too many concurrent copies and the
pod was OOMKilled under ScyllaDB's snapshot-rename copy flood, returning 502/503
to callers.

Converge the copy writer onto the framed UploadPart path:
- _pump_copy_chunks now reads FRAME_PLAINTEXT_SIZE plaintext frames via
  _PlaintextReader, encrypts each with encrypt_frame, accumulates one internal
  part, uploads it, then moves on -- one part at a time. Peak is O(one part
  ciphertext + one frame); it never holds a whole part's plaintext and
  ciphertext together and no longer runs parallel whole-part encrypts.
- Size internal parts with memory_bounded_part_size (same as the upload path),
  which also caps parts at MAX_INTERNAL_PARTS_PER_CLIENT so a large copy can no
  longer overflow the per-client internal-part range (latent collision with
  the old calculate_optimal_part_size for parts >1.28GB).
- copy_pipeline_peak == streaming_upload_peak + 2*MAX_BUFFER_SIZE (the framed
  per-part peak plus the copy-only source-read pipeline), so the reservation
  now covers the real peak. Measured (tracemalloc): 64MB copy 192MB -> 32MB,
  512MB copy -> 73.6MB, independent of object size.
- Gate the previously ungoverned _simple_copy_part behind reserve_memory.

Backwards compatible: writer-only change emitting the framed format the upload
path already writes; the reader is untouched and still decodes legacy
single-seal parts (overhead == 1 frame). No migration or re-encryption.

Tests: new test_copy_reservation_vs_real.py drives the real handler under
tracemalloc (reserved >= real peak; fails on the old code); new multi-frame
copy round-trip guard; test_copy_memory_governing updated to pin the
convergence. Full unit suite green (568), ruff clean.
@ServerSideHannes
ServerSideHannes merged commit be8f489 into main Jul 5, 2026
4 checks passed
@ServerSideHannes
ServerSideHannes deleted the fix/copy-oom-frame-streaming branch July 5, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant