[christopher5106]

Changes in this pull request

Closes #348.

Stream serves every read, seek and write of the native library through a Python ctypes callback. That is tens of thousands of callbacks per asset, each retaking the GIL, so signing from several threads is slower than from one, and a busy Python thread stalls a signature for tens of seconds. Measurements are in #348.

When the wrapped object is a regular file on a 64-bit POSIX platform, Stream now passes c2pa_create_stream libc read / lseek / write instead, with a private os.dup() of the file descriptor as the stream context. The native call never re-enters Python, so the GIL stays released for the whole call.

• Position: the duplicate starts at the Python object's position, after flushing its buffer. close() moves the Python object to where the native library stopped, so callers see the same positions as before. In the tests, source and destination end at the same place in both modes.
• Lifetime: the duplicate is owned and closed by the Stream. If the Python file is closed while the Stream is still alive, its descriptor number can be reused by another file, but the native side keeps reading the right one.
• Flush: a no-op on the descriptor, since the Python buffer is flushed when the stream is created and isn't used meanwhile.
• Fallback: in-memory streams, pipes and sockets (anything fstat doesn't report as a regular file), append-mode files, Windows and 32-bit builds keep the Python callbacks unchanged.
• Opt-out: Stream.native_file_io = False. Stream.uses_native_file_io tells which path a stream took.

Same benchmark as #348, macOS arm64, 14 cores, in signatures per second:

Asset	Mode	1 thread	8 threads	1 signature next to a busy thread
4K PNG, 4 KB chunks	before	6.6/s	2.2/s	68 s
	after	7.6/s	19.2/s	0.22 s
4K PNG, 64 KB chunks	before	11.3/s	13.0/s	36 s
	after	11.7/s	71.5/s	0.16 s
MP4, 72.7 MB	before	10.6/s	5.8/s	98 s
	after	17.4/s	36.3/s	0.18 s

New tests in TestNativeFileStreams:

• Only regular files use native I/O; BytesIO and append-mode files don't.
• The opt-out works.
• Signing file to file makes no Python read or write call at all: a counting io.FileIO subclass records zero calls.
• Native and Python streams produce signed files of the same size and validation state.
• A stream starts at the Python position and gives it back on close.
• A stream survives its Python file being closed first.

tests/test_unit_tests.py (469 tests) and tests/test_unit_tests_threaded.py (88) pass, and make check-format is clean.

Checklist

• This PR represents a single feature, fix, or change.
• All applicable changes have been documented. The behavior is transparent to callers. I can add a note to docs/usage.md about multithreaded use and Stream.native_file_io if you'd like one.
• Any TO DO items (or similar) have been entered as GitHub issues and the link to that issue has been included in a comment.

[tmathern]

Is there another solution for this without using libc?

[christopher5106]

Good point. Doing it without libc means the native side has to read the file itself, so I opened contentauth/c2pa-rs#2916: it adds c2pa_create_stream_from_fd(fd) to the C API, a stream backed by a native File on a duplicate of the descriptor, on Windows too. Measured from this binding through that constructor, it gives the same results as this PR (8 threads: 2.2 → 16.3 signatures/s on a 4K PNG, 5.8 → 34.4 on a 73 MB MP4; 0.14–0.21 s instead of 36–95 s next to a busy thread). Once it's released, I'll rework this PR to call it instead of libc: no libc, and no POSIX-only restriction.

[christopher5106]

I updated contentauth/c2pa-rs#2916 (contentauth/c2pa-rs#2916) so it covers files held in memory too. It now adds native file and memory streams to the C API.

[tmathern]

Is there a solution without adding a direct libc dependency?