[python] Use conditional OSS puts for atomic metadata writes

[wangzhigang1999]

Purpose

Fixes #9714.

Concurrent OSS writers can overwrite the same snapshot and lose committed data. Add OssFileIO with conditional PUTs (x-oss-forbid-overwrite=true), following Java #8228, while reusing existing PyArrow/Jindo file operations. Route OSS atomic writes through it and support Java-aligned SSE options via the optional oss2 dependency. Scope REST FileIO caches to each instance to isolate bucket and catalog settings.

Conditional creation requires a bucket that has never enabled versioning. Versioned buckets or denied versioning queries fall back to legacy writes with a warning; that fallback does not protect concurrent commits.

Tests

• 223 related tests passed. Extended cache lifecycle checks and 50 repeated concurrency runs also passed locally; Flake8 and diff checks passed.
• Real OSS: 64 processes × 300 rounds produced one successful creator per round, with no overwrites. Concurrent append testing recovered all 320,000 rows; the baseline lost 88,000 rows under the same settings. Details are in [Bug][python] Concurrent OSS snapshot commits can overwrite successful writes #9714.
• Seven real OSS SSE scenarios passed; local protocol tests cover conflicts, lost responses, URI handling and compatibility fallback.

[Copilot]

🟡 Changes recommended

OssFileIO.try_to_write_atomic derives the OSS object key incorrectly for credential-in-URI OSS forms, which can write metadata to the wrong object path.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes concurrent snapshot/metadata commit overwrites for OSS in PyPaimon by introducing an OSS-specific atomic-create path that uses the OSS SDK’s conditional PutObject (x-oss-forbid-overwrite=true) instead of the existing temp-file-and-rename publication flow.

Changes:

• Add OssFileIO (subclassing PyArrowFileIO) to implement conditional metadata writes via oss2, with bucket-versioning detection and a legacy fallback.
• Route OSS selection and REST token refresh / ResolvingFileIO atomic writes through the OSS implementation; update internal OSS call sites accordingly.
• Add OSS SSE option forwarding for conditional PUTs, extend optional extras to include oss2, document configuration, and add protocol-level tests.

File summaries

File	Description
paimon-python/setup.py	Adds oss2 to oss and jindo extras for OSS atomic metadata writes.
paimon-python/README.md	Documents OSS atomic metadata commits, versioning fallback semantics, and SSE option behavior.
paimon-python/pypaimon/tests/py36/ao_simple_test.py	Switches OSS initialization tests to use OssFileIO.
paimon-python/pypaimon/tests/oss_legacy_mode_test.py	Updates legacy-mode tests to target OssFileIO while keeping legacy assertions.
paimon-python/pypaimon/tests/oss_file_io_test.py	Updates OSS integration-style tests to instantiate OssFileIO.
paimon-python/pypaimon/tests/oss_atomic_write_test.py	New protocol tests exercising conditional PUT behavior, SSE headers, failures, and fallback.
paimon-python/pypaimon/tests/lance_utils_test.py	Uses FileIO.get(...) for OSS so the correct implementation is selected.
paimon-python/pypaimon/tests/file_io_test.py	Updates OSS-related unit tests to use OssFileIO.
paimon-python/pypaimon/sample/rest_catalog_blob_as_descriptor_sample.py	Uses FileIO.get(...) for external OSS IO selection.
paimon-python/pypaimon/filesystem/resolving_file_io.py	Forwards try_to_write_atomic to the scheme-resolved underlying FileIO.
paimon-python/pypaimon/filesystem/oss_file_io.py	Implements OSS conditional atomic creation and SSE header resolution.
paimon-python/pypaimon/common/options/config.py	Adds OSS SSE-related config options for atomic metadata PUTs.
paimon-python/pypaimon/common/file_io.py	Selects OssFileIO for oss:// URIs in FileIO.get(...).
paimon-python/pypaimon/catalog/rest/rest_token_file_io.py	Uses FileIO.get(...) so OSS paths pick up the OSS atomic-write implementation.

Review details

• Files reviewed: 14/14 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Credential-style fallback paths and cross-bucket REST FileIO caching can target or bind to the wrong OSS bucket.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 14/14 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

The critical conditional-write test module is skipped in all configured Python CI lanes because oss2 is not installed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 15/15 changed files
• Comments generated: 1
• Review effort level: Balanced