🚀 Feature
Provide an additional mount point backed by a physical device (ext4), separate from /kaggle/working, to use as a high-throughput I/O cache (e.g., /tmp).
Motivation
- The container root (
/) appears to be provided via an overlay filesystem; write/metadata performance on overlay can be slower or less predictable than on a host-backed volume such as /kaggle/working.
- Some applications default to
/tmp for I/O-intensive caches; on overlay this default can become an I/O bottleneck.
/kaggle/working is relatively fast and can be an alternatives to /tmp, but its capacity is limited to ~20 GB per session, which is sometimes insufficient to host checkpoint of LLMs (~100GB).- An additional fast endpoint for caching has the potential to better utilize the instance’s compute resources (e.g., T4, P100) that are often bottlenecked by disk I/O.
Additional context
I benchmarked sequential I/O and observed both writes and reads to be slower on /tmp (overlay) than on /kaggle/working (host-backed). Write throughput on /tmp is especially unstable; it sometimes gets ~5–10× lower in my tests.
🚀 Feature
Provide an additional mount point backed by a physical device (
ext4), separate from/kaggle/working, to use as a high-throughput I/O cache (e.g.,/tmp).Motivation
/) appears to be provided via an overlay filesystem; write/metadata performance on overlay can be slower or less predictable than on a host-backed volume such as/kaggle/working./tmpfor I/O-intensive caches; on overlay this default can become an I/O bottleneck./kaggle/workingis relatively fast and can be an alternatives to/tmp, but its capacity is limited to ~20 GB per session, which is sometimes insufficient to host checkpoint of LLMs (~100GB).Additional context
I benchmarked sequential I/O and observed both writes and reads to be slower on
/tmp(overlay) than on/kaggle/working(host-backed). Write throughput on/tmpis especially unstable; it sometimes gets ~5–10× lower in my tests.