Repository navigation
fix: frame streaming UploadPartCopy to bound memory (OOM under copy flood) - #108
Merged
Merged
Conversation
…lood) The streaming UploadPartCopy path buffered a whole 64MB internal part and encrypted it in one shot, with 2 uploads in flight, so a single copy peaked at ~192MB while copy_pipeline_peak() reserved only 4*MAX_BUFFER_SIZE (32MB). The memory governor therefore admitted ~6x too many concurrent copies and the pod was OOMKilled under ScyllaDB's snapshot-rename copy flood, returning 502/503 to callers. Converge the copy writer onto the framed UploadPart path: - _pump_copy_chunks now reads FRAME_PLAINTEXT_SIZE plaintext frames via _PlaintextReader, encrypts each with encrypt_frame, accumulates one internal part, uploads it, then moves on -- one part at a time. Peak is O(one part ciphertext + one frame); it never holds a whole part's plaintext and ciphertext together and no longer runs parallel whole-part encrypts. - Size internal parts with memory_bounded_part_size (same as the upload path), which also caps parts at MAX_INTERNAL_PARTS_PER_CLIENT so a large copy can no longer overflow the per-client internal-part range (latent collision with the old calculate_optimal_part_size for parts >1.28GB). - copy_pipeline_peak == streaming_upload_peak + 2*MAX_BUFFER_SIZE (the framed per-part peak plus the copy-only source-read pipeline), so the reservation now covers the real peak. Measured (tracemalloc): 64MB copy 192MB -> 32MB, 512MB copy -> 73.6MB, independent of object size. - Gate the previously ungoverned _simple_copy_part behind reserve_memory. Backwards compatible: writer-only change emitting the framed format the upload path already writes; the reader is untouched and still decodes legacy single-seal parts (overhead == 1 frame). No migration or re-encryption. Tests: new test_copy_reservation_vs_real.py drives the real handler under tracemalloc (reserved >= real peak; fails on the old code); new multi-frame copy round-trip guard; test_copy_memory_governing updated to pin the convergence. Full unit suite green (568), ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The streaming UploadPartCopy path OOMKilled
s3proxy-pythonunder ScyllaDB's snapshot-rename copy flood (backups returning 502/503; every host accumulating ~TiB of failed uploads).Root cause:
_pump_copy_chunksbuffered a whole 64 MB internal part andcrypto.encrypt()'d it in one shot, with 2 uploads in flight, so a single copy peaked at ~192 MB — whilecopy_pipeline_peak()reserved only4 * MAX_BUFFER_SIZE(32 MB). The memory governor therefore admitted ~6× too many concurrent copies and the pod exceeded its 1 GiB limit.tracemalloc from a dying pod confirmed it:
crypto.py:354(whole-part encrypt) +multipart/copy.py:393/398at 192 MB ×3,governed_active_mb=162vsrss=922.Fix — converge the copy writer onto the framed UploadPart path
_pump_copy_chunksrewritten to mirror_stream_and_upload_framed: readsFRAME_PLAINTEXT_SIZEplaintext frames via_PlaintextReader, encrypts each withencrypt_frame, accumulates one internal part, uploads it, then moves on — one part at a time, framed. Peak is nowO(one part ciphertext + one frame); never holds a whole part's plaintext+ciphertext together, no parallel whole-part encrypts.memory_bounded_part_size(same as the upload path). This also caps parts atMAX_INTERNAL_PARTS_PER_CLIENT, fixing a latent range collision: the oldcalculate_optimal_part_size(64 MB) overflowed the per-client 20-part range for parts >1.28 GB.copy_pipeline_peak == streaming_upload_peak + 2 * MAX_BUFFER_SIZE(framed per-part peak plus the copy-only source-read pipeline) — reservation now covers the real peak._simple_copy_partbehindreserve_memory.Measured (tracemalloc, real handler)
Peak is now bounded and independent of object size, so the governor bounds concurrency without clamping copies to run alone.
Backwards compatibility
Writer-only change emitting the framed format the upload path already writes — no new format. The reader is untouched and still decodes legacy single-seal parts (overhead == 1 frame). Verified: legacy 3 MB and 40 MB single-seal parts still decode; new multi-frame parts round-trip. No migration, no re-encryption.
The CopyObject (misc) path was already framed-safe (8 MB sequential parts) and governed — not the culprit; it now reserves conservatively.
Tests
test_copy_reservation_vs_real.py— drives the real handler under tracemalloc, assertsreserved >= real_peak(fails on old code).test_streaming_copy.py(old writer only produced single-seal parts, so this was never exercised).test_copy_memory_governing.pyupdated to pin the convergence.ruffclean.