Repository navigation
Assessment batch: build the request JSONL in temporary storage instead of RAM #1332
Description
Is your feature request related to a problem?
Submitting a BATCH assessment builds the provider request JSONL in memory, which can lead to out-of-memory errors for large submissions due to holding the entire batch in RAM.
Describe the solution you'd like
- Write the request JSONL to a temporary file line by line.
- Upload from that file to the provider.
- Delete the file after batch submission, even if it fails.
- Use a context manager for file operations to handle creation, writing, reading, and deletion seamlessly.
Original issue
Describe the bug
When a BATCH assessment is submitted, the provider request JSONL is built as a Python list in memory (build_openai_jsonl / build_google_jsonl / build_anthropic_jsonl in backend/app/crud/assessment/batch.py, jsonl_data) and then handed to the provider batch client. For big submissions, with long prompts and attachments, this holds the whole batch in RAM and can run the worker out of memory.
Describe the solution you'd like
- Write the request JSONL to a temporary file line by line instead of collecting it in a list.
- Upload from that file to the provider.
- Delete the file once the batch is submitted, including when submission fails.
- Put create, write, read and delete behind one context manager, so no caller handles the file or its cleanup directly.
Activity
[-]Assessment batch: build the request JSONL in temporary storage instead of RAM[/-][+]Batch Processing: Streamline JSONL submission[/+][-]Batch Processing: Streamline JSONL submission[/-][+]Assessment batch: build the request JSONL in temporary storage instead of RAM[/+]