fix: apply LightGBM dataset parameters during creation

[Lenin Mookiah (leninworld)]

This PR fixes issue #2769.

Related Issues/PRs

Fix #2769

What changes are proposed in this pull request?

• Forward explicit maxBinByFeature, minDataPerBin, isEnableSparse, useMissing, and zeroAsMissing settings when LightGBM datasets are created.
• Emit the native min_data_in_bin key and remove the later duplicate invalid min_data_per_bin entry.
• Validate the shape and values of maxBinByFeature before invoking native LightGBM.
• Add focused bulk and streaming regression coverage for parameter forwarding and invalid inputs.

How is this patch tested?

• Tests were added to verify the changes.
• Local testing passed.

Validation on Linux amd64 Docker with Java 11.0.32, Spark 3.5.0, Scala 2.12, and Python 3.11.8:

• Exact-base fail-before: 0/5 focused tests passed for the intended missing behavior.
• Pass-after: 32/32 focused and adjacent tests passed.
• Main and test Scalastyle: zero errors and zero warnings.
• A branch-built Python/JVM end-to-end run matched native LightGBM 3.3.5 controls in bulk and streaming modes.
• git diff --check passed.

The manual Fabric multi-executor regression script was not run locally. Dataset parameters supplied only through passThroughArgs remain outside this change.

Does this PR change dependencies?

• No. You can skip this section.
• Yes. Please specify what changes are proposed.

Does this PR add a new feature? If so, how is this feature tested?

• No. You can skip this section.
• Yes. Please describe how the feature is tested.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Lenin Mookiah (@leninworld) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Copilot]

🟢 Approval recommended

No blocking issues remain, and focused regression coverage addresses both transfer modes; tests were not independently rerun.

0 open findings

What changed in this PR

Fixes #2769 by applying LightGBM dataset settings during dataset creation, before training.

Changes:

• Forwards binning, sparsity, and missing-value settings.
• Corrects min_data_in_bin serialization and validates maxBinByFeature.
• Adds bulk and streaming regression tests.

File	Description
lightgbm/​src/​test/​scala/​com/​microsoft/​azure/​synapse/​ml/​lightgbm/​split1/​VerifyLightGBMDatasetParams.scala	Tests forwarding, invalid inputs, and training behavior.
lightgbm/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​lightgbm/​params/​BaseTrainParams.scala	Corrects the native parameter key and removes duplication.
lightgbm/​src/​main/​scala/​com/​microsoft/​azure/​synapse/​ml/​lightgbm/​LightGBMBase.scala	Forwards dataset settings and validates per-feature bins.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.91%. Comparing base (eca68f8) to head (7a7253f).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2774      +/-   ##
==========================================
+ Coverage   87.88%   87.91%   +0.02%     
==========================================
  Files         346      346              
  Lines       21438    21444       +6     
  Branches     2226     2233       +7     
==========================================
+ Hits        18841    18852      +11     
+ Misses       2597     2592       -5     

Files with missing lines	Coverage Δ
...osoft/azure/synapse/ml/lightgbm/LightGBMBase.scala	97.10% <100.00%> (+0.05%)	⬆️
...e/synapse/ml/lightgbm/params/BaseTrainParams.scala	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Rana Singh (ranadeepsingh)]

Thanks Lenin Mookiah (@leninworld)! This fixes a real issue: explicit LightGBM binning and missing-value settings were reaching training too late to affect dataset creation. I kept your implementation and tests unchanged and independently validated 7a7253fbbb7cbd8245eaa356a89cd689655ea1eb.

• Before/after: the original production code reproduces the reported 229 thresholds instead of 1. All five new regression tests fail against that baseline and pass with your fix locally, including bulk and streaming training.
• Full CI: Azure build 241234907 is green, with all 65 final job results successful. It was triggered by /azp run (reason=pullRequest) against PR merge 73e870813e5d5e2b4a29f1875fb53effe90c2aab, whose parents match this head and current master. The first deep-learning-HF job lost its hosted agent during setup, before tests ran; retrying only that failed job preserved the other 64 successes, and all four HF tests then passed.
• Regression evidence: published test run 1173585851 confirms all five added tests passed. All six LightGBM Scala splits and the LightGBM Python/R jobs passed. The existing local-only performance utility remains explicitly ignored.
• Review/checks: all GitHub workflows passed, Codecov reports all modified coverable lines tested, and the current-head automated review has no unresolved or suppressed findings.

Engineering validation is complete for this head. The remaining merge gate is one maintainer approval; this comment is validation evidence, not that approval. No source changes were pushed.