From eb747c18d4c345265a40e3d083acbb4bdbf86ed8 Mon Sep 17 00:00:00 2001 From: wsad1 Date: Tue, 27 Oct 2026 20:55:52 +0000 Subject: [PATCH 1/8] Add fine-tuning guide Explain full fine-tuning of tabular foundation models through the ICL interface, summarize the fine-tuning benchmark results, and add measured guidance on context size, query size, learning rate, and inference context reduction. --- docs/source/finetuning.md | 308 +++++ .../source/images/finetuning_context_dark.svg | 614 +++++++++ .../images/finetuning_context_light.svg | 614 +++++++++ docs/source/images/finetuning_dark.svg | 63 + docs/source/images/finetuning_light.svg | 63 + docs/source/images/finetuning_sweep_dark.svg | 1110 +++++++++++++++++ docs/source/images/finetuning_sweep_light.svg | 1110 +++++++++++++++++ docs/source/index.md | 1 + 8 files changed, 3883 insertions(+) create mode 100644 docs/source/finetuning.md create mode 100644 docs/source/images/finetuning_context_dark.svg create mode 100644 docs/source/images/finetuning_context_light.svg create mode 100644 docs/source/images/finetuning_dark.svg create mode 100644 docs/source/images/finetuning_light.svg create mode 100644 docs/source/images/finetuning_sweep_dark.svg create mode 100644 docs/source/images/finetuning_sweep_light.svg diff --git a/docs/source/finetuning.md b/docs/source/finetuning.md new file mode 100644 index 0000000000..00e7680822 --- /dev/null +++ b/docs/source/finetuning.md @@ -0,0 +1,308 @@ +# Fine-Tuning + +Tabular foundation models (TFMs) such as {py:class}`~sdm.models.TabICLv2` and {py:class}`~sdm.models.KumoTabular` are pretrained on a broad distribution of tasks and solve a new task through [in-context learning](icl): they read labeled context rows and predict query rows without updating their weights. +This works well out of the box, but the pretrained weights encode a general prior rather than the specifics of your dataset. + +**Full fine-tuning** updates every parameter of a pretrained model on a single dataset, so the model adapts to that dataset's features and targets. +The fine-tuned model is still an in-context learner: it keeps the same interface and predicts query rows from labeled context rows. +No dedicated training API is needed either. +Calling {py:meth}`~sdm.models.ICLModel.forward` in train mode (`model.train()`) tracks gradients through the recipe and the model, so a standard PyTorch training loop is all it takes. + +## How Full Fine-Tuning Works + +Fine-tuning keeps the objective a TFM is pretrained with: predict held-out rows from labeled examples. +What changes is where the tasks come from. +Instead of drawing them from a synthetic prior, each training step draws one from the training split of your dataset: + +```{figure} images/finetuning_light.svg +:figclass: light-only +:width: 100% +``` + +```{figure} images/finetuning_dark.svg +:figclass: dark-only +:width: 100% +``` + +1. Sample rows from the training split and divide them into a **context** of *C* labeled rows and a **query** of *Q* rows. +2. Run the model on the labeled context and the query features. +3. Compute the loss of the query predictions against the held-out query labels. +4. Backpropagate and update every weight of the model. + +Because the context changes from step to step, the model is trained to read its context, just as it does at inference, instead of fitting a fixed set of rows. +After fine-tuning, inference is unchanged: the training split becomes the context, and the model predicts new rows through {py:meth}`~sdm.models.ICLModel.forward` or {py:meth}`~sdm.models.ICLModel.fit` and {py:meth}`~sdm.models.ICLModel.predict`. + +```{note} +The rest of this guide walks through [`examples/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/examples/tabular/finetune.py), which fine-tunes {py:class}`~sdm.models.TabICLv2` or {py:class}`~sdm.models.KumoTabular` on a classification or regression dataset. +``` + +## Data Splits + +Fine-tuning learns from labels, so it needs three disjoint splits: + +- **Training** rows are sampled into context and query rows during fine-tuning, and serve as the context when the model is evaluated. +- **Validation** rows select the best checkpoint and never appear in a training step. +- **Test** rows are used once, for the final evaluation. + +```python +import torch +from sklearn.datasets import load_digits + +import sdm + +df = load_digits(as_frame=True).frame + +table = sdm.TableTensor.from_pandas( + df=df, + stypes=sdm.infer_stypes(df, overrides={"target": "categorical"}), + device="cuda", +) + +# Split rows into training/validation/test (60/20/20): +index = torch.randperm(table.size(0), device="cuda") +num_train, num_val = int(0.6 * index.numel()), int(0.2 * index.numel()) +train_table = table[index[:num_train]] +val_table = table[index[num_train:num_train + num_val]] +test_table = table[index[num_train + num_val:]] +``` + +## A Training Recipe + +Every model ships a default {py:class}`~sdm.processing.recipe.Recipe` (see [here](processing) for the accompanying tutorial) built for inference: its output pipeline averages the outputs of all estimators and, for classification, turns logits into probabilities. +For training, we want the raw model outputs instead, so the loss can be computed directly on logits and quantiles: + +```python +import sdm.processing as sp + +model = sdm.models.TabICLv2(device="cuda") + +train_recipe = model.default_recipe() +train_recipe.target = sp.StypeDispatch( # No target flipping. + categorical=[ + sp.AlignCategories(), + sp.ShuffleCategories(method="shift"), + ], + numerical=sp.Standardize(), +) +train_recipe.output = sp.Identity() # No post-processing. +``` + +- With {py:class}`~sdm.processing.common.Identity` as the output pipeline, the model returns its raw outputs stacked along a leading estimator dimension. + Training steps run a single estimator, so `out[0]` selects its output. +- The target pipeline leaves out random sign flipping of regression targets ({py:class}`~sdm.processing.numerical.FlipSign`, part of the default recipe of {py:class}`~sdm.models.KumoTabular`). + A flipped target reverses the order of the predicted quantiles, which the default output pipeline restores via {py:class}`~sdm.processing.output.SortQuantiles`, but the identity output does not. +- Feature preprocessing stays as is. + Stochastic processors such as {py:class}`~sdm.processing.common.ShuffleColumns` and {py:class}`~sdm.processing.categorical.ShuffleCategories` draw a new column or class order at every step, which acts as data augmentation. + +## The Training Loop + +Each step samples a fresh context and query from the training rows and runs the model in train mode with the training recipe: + +```python +import torch.nn.functional as F + +optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5) + +model.train() +for step in range(num_steps): + index = torch.randperm(train_table.size(0), device="cuda") + context = train_table[index[:context_size]] + query = train_table[index[context_size:context_size + query_size]] + + out = model( + x_context=context.drop_columns("target"), + y_context=context["target"], + x_query=query.drop_columns("target"), + recipe=train_recipe, + )[0] + + logits, y = sdm.evaluation.to_class_indices( + pred=out, + target=query["target"], + missing_score=-torch.inf, + ) + loss = F.cross_entropy(logits, y) + + optimizer.zero_grad() + loss.backward() + optimizer.step() +``` + +For classification, the output has one logit column per class that appears in the sampled context. +{py:func}`~sdm.evaluation.to_class_indices` aligns these columns with the classes of the target column and converts the query targets to class indices. +Setting `missing_score=-torch.inf` gives classes that are absent from the context zero probability, so the loss is a standard cross-entropy over the aligned logits. + +For regression, {py:class}`~sdm.models.TabICLv2` predicts 999 quantiles, named `"q001"` through `"q999"`. +The pinball (quantile) loss trains each output column toward its quantile level, and with it the full predictive distribution rather than only a point estimate: + +```python +diff = query["target"].numerical - out.numerical # [Q, 999] +level = torch.linspace(0.001, 0.999, 999, device="cuda") +loss = torch.maximum(level * diff, (level - 1) * diff).mean() +``` + +{py:meth}`~sdm.models.ICLModel.predict` raises if the model is in train mode, so training steps always go through {py:meth}`~sdm.models.ICLModel.forward`. + +## Model Selection + +Fine-tuning can overfit, in particular on small datasets. +We therefore score the model on the validation rows after every epoch and keep the best weights. +Evaluation runs the model exactly as it will be used at inference: in eval mode, with the default recipe, with estimator ensembling, and with the training rows as context: + +```python +import copy + +def evaluate(context: sdm.TableTensor, query: sdm.TableTensor) -> float: + model.eval() + out = model( + x_context=context.drop_columns("target"), + y_context=context["target"], + x_query=query.drop_columns("target"), + num_estimators=8, + ) + prob, y = sdm.evaluation.to_class_indices( + pred=out, + target=query["target"], + missing_score=0.0, + ) + return (prob.argmax(dim=-1) == y).float().mean().item() + +best_acc = evaluate(context=train_table, query=val_table) # Pretrained model. +best_state = copy.deepcopy(model.state_dict()) + +for epoch in range(max_epochs): + model.train() + ... # Training steps from above. + + acc = evaluate(context=train_table, query=val_table) + if acc > best_acc: + best_acc, best_state = acc, copy.deepcopy(model.state_dict()) + +model.load_state_dict(best_state) +test_acc = evaluate(context=train_table, query=test_table) +``` + +Since the first evaluation scores the pretrained model, selection never returns weights that score below the zero-shot model on the validation rows. +For large test sets, replace the one-shot {py:meth}`~sdm.models.ICLModel.forward` call with {py:meth}`~sdm.models.ICLModel.fit` and batched {py:meth}`~sdm.models.ICLModel.predict` calls, optionally under mixed-precision autocasting (see [here](icl)). + +## Benchmark Results + +The benchmark harness in [`benchmark/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/benchmark/tabular/finetune.py) implements this procedure for TALENT, TabArena, BeyondArena, and ScoringBench. +By default, it trains with AdamW at a learning rate of `1e-6` for 75 epochs of 10 steps, samples up to 10,000 rows per step and splits them 80/20 into context and query rows, and holds out 20% of each dataset's training split for model selection. + +Each row below counts the datasets on which the fine-tuned model scores better (win), worse (loss), or the same (tie) as the same model without fine-tuning, according to each benchmark's own metric: + +| Benchmark | Model | Tasks | Wins | Losses | Ties | +| ------------ | ------------------------------------------- | ------------------------- | ---- | ------ | ---- | +| TALENT | {py:class}`~sdm.models.TabICLv2` | Classification | 73 | 51 | 31 | +| TALENT | {py:class}`~sdm.models.KumoTabular` (small) | Classification | 51 | 39 | 38 | +| ScoringBench | {py:class}`~sdm.models.TabICLv2` | Probabilistic regression | 50 | 26 | 10 | +| TabArena | {py:class}`~sdm.models.KumoTabular` (large) | Classification/regression | 35 | 12 | 0 | + +Fine-tuning improves results on more datasets than it degrades on every benchmark. +On TALENT, the gains of {py:class}`~sdm.models.TabICLv2` are larger on multiclass datasets (36 wins, 18 losses, 15 ties) than on binary ones (37 wins, 33 losses, 16 ties). + +To reproduce these results, add `--finetune` to the TALENT and ScoringBench runners, or pick a fine-tuned `-ft` model variant on TabArena and BeyondArena: + +```bash +python -m benchmark.tabular.talent.main \ + --model tabiclv2 \ + --dataset-path /path/to/talent/data \ + --finetune + +python -m benchmark.tabular.tabarena.main --model kumo-tabular-large-ft +``` + +See the [tabular benchmark instructions](https://github.com/NVIDIA/structured-data-models/tree/main/benchmark/tabular) for setup and all fine-tuning flags. + +## Choosing Context Size, Query Size, and Learning Rate + +Three settings of the training loop shape the result: + +- **Learning rate:** How far each step moves the pretrained weights. +- **Context size *C*:** How many labeled rows the model reads per step. +- **Query size *Q*:** How many rows contribute to the loss per step, *i.e.*, the batch size. + +To measure their effect, we fine-tuned {py:class}`~sdm.models.TabICLv2` on two of the larger TALENT datasets, `customer_satisfaction_in_airline` (*Airline* in the figures; binary classification, 83,123 training rows) and `Diamonds` (regression, 34,521 training rows). +Each run uses the benchmark harness configuration from above with a fixed budget of 750 steps and changes one setting at a time, starting from the defaults (learning rate `1e-6`, *C* = 8,000, *Q* = 2,000). +The figure shows test accuracy (higher is better) and RMSE (lower is better) as mean and standard deviation over 3 seeds, using the full training split as context; the dashed line is the zero-shot model: + +```{figure} images/finetuning_sweep_light.svg +:figclass: light-only +:width: 100% +``` + +```{figure} images/finetuning_sweep_dark.svg +:figclass: dark-only +:width: 100% +``` + +- **Context size has the clearest effect.** With *C* = 500 or 2,000, fine-tuning does not improve on the zero-shot model on either dataset. Gains start at *C* = 8,000 and are largest at *C* = 16,000, the largest size we tried (96.53% *vs.* 96.42% accuracy and 489.3 *vs.* 495.1 RMSE). At inference, the model reads the whole training split, so larger training contexts are closer to the setting it is evaluated in. +- **Learning rates between `1e-6` and `1e-5` work best.** With `1e-7` or `3e-7`, 750 steps leave accuracy on `customer_satisfaction_in_airline` at its zero-shot level and recover only part of the gain on `Diamonds`. On `customer_satisfaction_in_airline`, `3e-6` and `1e-5` improve further over `1e-6`; on `Diamonds`, `1e-6` and `1e-5` perform alike. We did not observe degradation up to `1e-5`, the largest rate we tried. +- **Query size matters less.** Accuracy on `customer_satisfaction_in_airline` rises steadily from *Q* = 128 to *Q* = 8,000, while on `Diamonds`, *Q* = 2,000 scores best and *Q* = 8,000 is within the variation across seeds. + +Memory and time grow with *C* + *Q*. +On an H100 GPU, fine-tuning on `customer_satisfaction_in_airline` peaks at 6 GB of memory for *C* = 500, 13 GB for *C* = 8,000, and 23 GB for *C* = 16,000 (all with *Q* = 2,000), and takes 46 to 60 minutes per run. + +In practice, start from the harness defaults, increase *C* as far as memory allows, and tune the learning rate between `1e-6` and `1e-5` on a validation split. +Note that the absolute gains on these two datasets are small, since {py:class}`~sdm.models.TabICLv2` is already strong on them zero-shot: the best settings, selected on validation, improve test accuracy from 96.42% to 96.54% (learning rate `1e-5`) and test RMSE from 495.1 to 489.3 (*C* = 16,000). + +The TALENT runner exposes all three settings: `--finetune-lr`, `--finetune-train-size` (*C* + *Q*), and `--finetune-context-frac` (*C* / (*C* + *Q*)). +For example, the best `Diamonds` run uses: + +```bash +python -m benchmark.tabular.talent.main \ + --model tabiclv2 \ + --dataset-path /path/to/talent/data \ + --dataset Diamonds \ + --finetune \ + --finetune-train-size 18000 \ + --finetune-context-frac 0.8889 +``` + +## Smaller Contexts After Fine-Tuning + +A zero-shot TFM knows about your dataset only what its context shows it. +Its accuracy therefore typically keeps improving as more training rows are passed as context, which makes the full training split the default choice. +Inference cost grows with the context as well: {py:meth}`~sdm.models.ICLModel.forward` processes all context rows on every call, and {py:meth}`~sdm.models.ICLModel.fit` keeps per-row key/value state in memory for subsequent {py:meth}`~sdm.models.ICLModel.predict` calls. + +Fine-tuning moves part of that dataset knowledge into the model weights. +A fine-tuned model can therefore reach comparable accuracy with fewer context rows, which reduces inference memory and latency, *e.g.*, when a model serves predictions for a large training split. +To use a smaller context, fit the fine-tuned model on a random subset of the training rows: + +```python +index = torch.randperm(train_table.size(0), device="cuda")[:4096] +context = train_table[index] + +model.fit(x=context.drop_columns("target"), y=context["target"]) +out = model.predict(test_table.drop_columns("target")) +``` + +To measure the effect, we took the best fine-tuned model per dataset from the previous section, selected on validation (learning rate `1e-5` for `customer_satisfaction_in_airline` and *C* = 16,000 for `Diamonds`), and fitted it and the zero-shot model on the same random subsets of 256 to 16,384 training rows, as well as on all of them. +The figure and table show test accuracy (*Airline*) and RMSE (*Diamonds*) as the mean over 3 seeds, with the standard deviation as error bars; the dashed line is the zero-shot model with all training rows as context: + +```{figure} images/finetuning_context_light.svg +:figclass: light-only +:width: 100% +``` + +```{figure} images/finetuning_context_dark.svg +:figclass: dark-only +:width: 100% +``` + +| Context rows | Airline, zero-shot | Airline, fine-tuned | Diamonds, zero-shot | Diamonds, fine-tuned | +| ------------ | ------------------ | ------------------- | ------------------- | -------------------- | +| 256 | 89.08% | 89.88% | 930.4 | 908.5 | +| 1,024 | 92.67% | 93.82% | 630.2 | 621.5 | +| 4,096 | 94.61% | 95.62% | 548.3 | 528.3 | +| 16,384 | 95.87% | 96.30% | 505.0 | 496.0 | +| All | 96.42% | 96.57% | 495.1 | 488.8 | + +- **The fine-tuned model is better at every context size.** The gap is largest between 1,024 and 4,096 rows, *e.g.*, 93.82% *vs.* 92.67% accuracy at 1,024 rows and 528.3 *vs.* 548.3 RMSE at 4,096 rows. At 256 rows, it is within the variation across seeds. +- **With 16,384 rows, the fine-tuned model comes close to the zero-shot model with all rows**, using 20% of the `customer_satisfaction_in_airline` training split (96.30% *vs.* 96.42%) and 47% of the `Diamonds` training split (496.0 *vs.* 495.1). +- **More context still helps.** Both models improve with every increase in context size, and the fine-tuned model with all rows scores best on both datasets. + +The full-context results of the fine-tuned models differ slightly from the previous section because estimator ensembles are randomized at inference. + +Estimator ensembling complements smaller contexts: each estimator can receive a different subset of the training rows, so an ensemble covers more of the training split while each estimator keeps a small context (see the ensembling section [here](icl)). diff --git a/docs/source/images/finetuning_context_dark.svg b/docs/source/images/finetuning_context_dark.svg new file mode 100644 index 0000000000..c0c392d2a0 --- /dev/null +++ b/docs/source/images/finetuning_context_dark.svg @@ -0,0 +1,614 @@ + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + 256 + + + + + + + + + + 1k + + + + + + + + + + 4k + + + + + + + + + + 16k + + + + + + + + + + all + (83k) + + + + Inference context rows + + + + + + + + + + 88 + + + + + + + + + 90 + + + + + + + + + 92 + + + + + + + + + 94 + + + + + + + + + 96 + + + + Airline accuracy (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + zero-shot, all rows + + + + + + + + + + + + + + + 256 + + + + + + + + + + 1k + + + + + + + + + + 4k + + + + + + + + + + 16k + + + + + + + + + + all + (35k) + + + + Inference context rows + + + + + + + + + + 500 + + + + + + + + + 600 + + + + + + + + + 700 + + + + + + + + + 800 + + + + + + + + + 1000 + + + + Diamonds RMSE + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + zero-shot, all rows + + + + + + + + + + + + + + + + + + + + + + + + + + Zero-shot + + + + + + + + + + + + + + + + + + + + + + + + Fine-tuned + + + + + + + + + + + + diff --git a/docs/source/images/finetuning_context_light.svg b/docs/source/images/finetuning_context_light.svg new file mode 100644 index 0000000000..c130c8c591 --- /dev/null +++ b/docs/source/images/finetuning_context_light.svg @@ -0,0 +1,614 @@ + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + 256 + + + + + + + + + + 1k + + + + + + + + + + 4k + + + + + + + + + + 16k + + + + + + + + + + all + (83k) + + + + Inference context rows + + + + + + + + + + 88 + + + + + + + + + 90 + + + + + + + + + 92 + + + + + + + + + 94 + + + + + + + + + 96 + + + + Airline accuracy (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + zero-shot, all rows + + + + + + + + + + + + + + + 256 + + + + + + + + + + 1k + + + + + + + + + + 4k + + + + + + + + + + 16k + + + + + + + + + + all + (35k) + + + + Inference context rows + + + + + + + + + + 500 + + + + + + + + + 600 + + + + + + + + + 700 + + + + + + + + + 800 + + + + + + + + + 1000 + + + + Diamonds RMSE + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + zero-shot, all rows + + + + + + + + + + + + + + + + + + + + + + + + + + Zero-shot + + + + + + + + + + + + + + + + + + + + + + + + Fine-tuned + + + + + + + + + + + + diff --git a/docs/source/images/finetuning_dark.svg b/docs/source/images/finetuning_dark.svg new file mode 100644 index 0000000000..228e7e4639 --- /dev/null +++ b/docs/source/images/finetuning_dark.svg @@ -0,0 +1,63 @@ + + + + + + + + + + + + Training split + + + + + + + + + + + x₁ + x₂ + x₃ + y + + + sample + + + + + + Context + C rows with labels + + Query + Q rows, labels hidden + + + + + + TFM + all weights + trainable + + + + predictions + + Loss + on query rows only + + + + query labels + + + + backpropagate and update every weight + diff --git a/docs/source/images/finetuning_light.svg b/docs/source/images/finetuning_light.svg new file mode 100644 index 0000000000..cdf5ae3634 --- /dev/null +++ b/docs/source/images/finetuning_light.svg @@ -0,0 +1,63 @@ + + + + + + + + + + + + Training split + + + + + + + + + + + x₁ + x₂ + x₃ + y + + + sample + + + + + + Context + C rows with labels + + Query + Q rows, labels hidden + + + + + + TFM + all weights + trainable + + + + predictions + + Loss + on query rows only + + + + query labels + + + + backpropagate and update every weight + diff --git a/docs/source/images/finetuning_sweep_dark.svg b/docs/source/images/finetuning_sweep_dark.svg new file mode 100644 index 0000000000..7b1b3c1734 --- /dev/null +++ b/docs/source/images/finetuning_sweep_dark.svg @@ -0,0 +1,1110 @@ + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + 1e-7 + + + + + + + + + + 3e-7 + + + + + + + + + + 1e-6 + + + + + + + + + + 3e-6 + + + + + + + + + + 1e-5 + + + + + + + + + + + 96.40 + + + + + + + + + 96.45 + + + + + + + + + 96.50 + + + + + + + + + 96.55 + + + + Airline accuracy (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Learning rate + + + + + + + + + + + + + + + 500 + + + + + + + + + + 2k + + + + + + + + + + 8k + + + + + + + + + + 16k + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + C + o + n + t + e + x + t +   + s + i + z + e +   + C + + + + + + + + + + + + + + + + + 128 + + + + + + + + + + 512 + + + + + + + + + + 2k + + + + + + + + + + 8k + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Q + u + e + r + y +   + s + i + z + e +   + Q + + + + + + + + + + + + + + + + + 1e-7 + + + + + + + + + + 3e-7 + + + + + + + + + + 1e-6 + + + + + + + + + + 3e-6 + + + + + + + + + + 1e-5 + + + + + + + + + + + 488 + + + + + + + + + 490 + + + + + + + + + 492 + + + + + + + + + 494 + + + + + + + + + 496 + + + + + + + + + 498 + + + + Diamonds RMSE + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 500 + + + + + + + + + + 2k + + + + + + + + + + 8k + + + + + + + + + + 16k + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 128 + + + + + + + + + + 512 + + + + + + + + + + 2k + + + + + + + + + + 8k + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Fine-tuned + + + + + + Zero-shot + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/source/images/finetuning_sweep_light.svg b/docs/source/images/finetuning_sweep_light.svg new file mode 100644 index 0000000000..8fbe295a23 --- /dev/null +++ b/docs/source/images/finetuning_sweep_light.svg @@ -0,0 +1,1110 @@ + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + 1e-7 + + + + + + + + + + 3e-7 + + + + + + + + + + 1e-6 + + + + + + + + + + 3e-6 + + + + + + + + + + 1e-5 + + + + + + + + + + + 96.40 + + + + + + + + + 96.45 + + + + + + + + + 96.50 + + + + + + + + + 96.55 + + + + Airline accuracy (%) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Learning rate + + + + + + + + + + + + + + + 500 + + + + + + + + + + 2k + + + + + + + + + + 8k + + + + + + + + + + 16k + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + C + o + n + t + e + x + t +   + s + i + z + e +   + C + + + + + + + + + + + + + + + + + 128 + + + + + + + + + + 512 + + + + + + + + + + 2k + + + + + + + + + + 8k + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Q + u + e + r + y +   + s + i + z + e +   + Q + + + + + + + + + + + + + + + + + 1e-7 + + + + + + + + + + 3e-7 + + + + + + + + + + 1e-6 + + + + + + + + + + 3e-6 + + + + + + + + + + 1e-5 + + + + + + + + + + + 488 + + + + + + + + + 490 + + + + + + + + + 492 + + + + + + + + + 494 + + + + + + + + + 496 + + + + + + + + + 498 + + + + Diamonds RMSE + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 500 + + + + + + + + + + 2k + + + + + + + + + + 8k + + + + + + + + + + 16k + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 128 + + + + + + + + + + 512 + + + + + + + + + + 2k + + + + + + + + + + 8k + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Fine-tuned + + + + + + Zero-shot + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/source/index.md b/docs/source/index.md index 61e3ef40b7..cdc176b177 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -32,6 +32,7 @@ tensor icl forecasting processing +finetuning nanotabicl nim ``` From f814d1de708596b8af6d1ec15be0e2c9b8a9561b Mon Sep 17 00:00:00 2001 From: wsad1 Date: Tue, 27 Oct 2026 21:32:21 +0000 Subject: [PATCH 2/8] Revise fine-tuning guide Fix the training recipe and code snippets, state how the benchmark harness differs from the example, scope the hyperparameter guidance to the measured datasets, and link to other guides instead of restating them. --- docs/source/finetuning.md | 170 ++++++++++++++++++++++---------------- 1 file changed, 97 insertions(+), 73 deletions(-) diff --git a/docs/source/finetuning.md b/docs/source/finetuning.md index 00e7680822..e21eed5395 100644 --- a/docs/source/finetuning.md +++ b/docs/source/finetuning.md @@ -1,18 +1,22 @@ # Fine-Tuning -Tabular foundation models (TFMs) such as {py:class}`~sdm.models.TabICLv2` and {py:class}`~sdm.models.KumoTabular` are pretrained on a broad distribution of tasks and solve a new task through [in-context learning](icl): they read labeled context rows and predict query rows without updating their weights. -This works well out of the box, but the pretrained weights encode a general prior rather than the specifics of your dataset. +Tabular foundation models (TFMs) such as {py:class}`~sdm.models.TabICLv2` and {py:class}`~sdm.models.KumoTabular` solve a new task through [in-context learning](icl): they read labeled context rows and predict query rows without updating their weights. +This works well out of the box, but the pretrained weights know nothing specific about your dataset. -**Full fine-tuning** updates every parameter of a pretrained model on a single dataset, so the model adapts to that dataset's features and targets. -The fine-tuned model is still an in-context learner: it keeps the same interface and predicts query rows from labeled context rows. -No dedicated training API is needed either. -Calling {py:meth}`~sdm.models.ICLModel.forward` in train mode (`model.train()`) tracks gradients through the recipe and the model, so a standard PyTorch training loop is all it takes. +**Full fine-tuning** updates every weight of a pretrained model on one dataset. +The result is still an in-context learner with the same interface, and it trains with an ordinary PyTorch loop, since calling {py:meth}`~sdm.models.ICLModel.forward` in train mode enables gradients (see [In-Context Learning](icl)). +Below, the *pretrained model* is the model before fine-tuning, labeled *zero-shot* in the figures. + +Fine-tuning does not always pay off. +On standard benchmarks, it helps on more datasets than it hurts, but not on all of them, and gains tend to be small when the pretrained model is already strong. +On datasets with tens of thousands of rows, a run took between 10 and 60 minutes on a single H100 GPU in our experiments. +The procedure below therefore keeps the pretrained weights whenever fine-tuning does not improve the validation score. ## How Full Fine-Tuning Works Fine-tuning keeps the objective a TFM is pretrained with: predict held-out rows from labeled examples. What changes is where the tasks come from. -Instead of drawing them from a synthetic prior, each training step draws one from the training split of your dataset: +Instead of its pretraining data, each training step draws a task from the training split of your dataset: ```{figure} images/finetuning_light.svg :figclass: light-only @@ -29,21 +33,23 @@ Instead of drawing them from a synthetic prior, each training step draws one fro 3. Compute the loss of the query predictions against the held-out query labels. 4. Backpropagate and update every weight of the model. -Because the context changes from step to step, the model is trained to read its context, just as it does at inference, instead of fitting a fixed set of rows. +Because the context changes from step to step, the model learns to read its context, as it does at inference, instead of memorizing a fixed set of rows. After fine-tuning, inference is unchanged: the training split becomes the context, and the model predicts new rows through {py:meth}`~sdm.models.ICLModel.forward` or {py:meth}`~sdm.models.ICLModel.fit` and {py:meth}`~sdm.models.ICLModel.predict`. ```{note} -The rest of this guide walks through [`examples/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/examples/tabular/finetune.py), which fine-tunes {py:class}`~sdm.models.TabICLv2` or {py:class}`~sdm.models.KumoTabular` on a classification or regression dataset. +The next four sections walk through [`examples/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/examples/tabular/finetune.py), which fine-tunes {py:class}`~sdm.models.TabICLv2` or {py:class}`~sdm.models.KumoTabular` on a classification or regression dataset. ``` ## Data Splits -Fine-tuning learns from labels, so it needs three disjoint splits: +Fine-tuning updates the weights on the training rows, so you need separate validation rows to pick the weights and test rows to report results: -- **Training** rows are sampled into context and query rows during fine-tuning, and serve as the context when the model is evaluated. -- **Validation** rows select the best checkpoint and never appear in a training step. +- **Training** rows are sampled into context and query rows during fine-tuning, and also serve as the context when the model is evaluated. +- **Validation** rows pick the best weights and never appear in a training step. - **Test** rows are used once, for the final evaluation. +The snippet below builds a {py:class}`~sdm.tensor.TableTensor` (see [Table Semantics](tensor)) and splits its rows: + ```python import torch from sklearn.datasets import load_digits @@ -68,43 +74,46 @@ test_table = table[index[num_train + num_val:]] ## A Training Recipe -Every model ships a default {py:class}`~sdm.processing.recipe.Recipe` (see [here](processing) for the accompanying tutorial) built for inference: its output pipeline averages the outputs of all estimators and, for classification, turns logits into probabilities. -For training, we want the raw model outputs instead, so the loss can be computed directly on logits and quantiles: +Every model ships a default {py:class}`~sdm.processing.recipe.Recipe` (see [Data Processing](processing)) whose output pipeline is built for inference: it combines the outputs of all estimators and turns classification logits into probabilities. +For training, we replace it with {py:class}`~sdm.processing.common.Identity`, so the loss is computed directly on the model outputs: ```python import sdm.processing as sp -model = sdm.models.TabICLv2(device="cuda") +model = sdm.models.TabICLv2(task="classification", device="cuda") train_recipe = model.default_recipe() -train_recipe.target = sp.StypeDispatch( # No target flipping. +train_recipe.output = sp.Identity() # No post-processing. +``` + +The rest of the recipe stays as is. +Its stochastic processors, *e.g.*, {py:class}`~sdm.processing.common.ShuffleColumns` in the features and {py:class}`~sdm.processing.categorical.ShuffleCategories` in the target, draw a new order at every step, which acts as data augmentation. + +For regression with {py:class}`~sdm.models.KumoTabular`, also remove random sign flipping ({py:class}`~sdm.processing.numerical.FlipSign`) from the target pipeline. +A flipped target reverses the order of the predicted quantiles, which the default output pipeline restores via {py:class}`~sdm.processing.output.SortQuantiles`, but the identity output does not: + +```python +train_recipe.target = sp.StypeDispatch( # Default target pipeline without FlipSign. categorical=[ - sp.AlignCategories(), + sp.AlignCategories(sort_by="value"), sp.ShuffleCategories(method="shift"), ], numerical=sp.Standardize(), ) -train_recipe.output = sp.Identity() # No post-processing. ``` -- With {py:class}`~sdm.processing.common.Identity` as the output pipeline, the model returns its raw outputs stacked along a leading estimator dimension. - Training steps run a single estimator, so `out[0]` selects its output. -- The target pipeline leaves out random sign flipping of regression targets ({py:class}`~sdm.processing.numerical.FlipSign`, part of the default recipe of {py:class}`~sdm.models.KumoTabular`). - A flipped target reverses the order of the predicted quantiles, which the default output pipeline restores via {py:class}`~sdm.processing.output.SortQuantiles`, but the identity output does not. -- Feature preprocessing stays as is. - Stochastic processors such as {py:class}`~sdm.processing.common.ShuffleColumns` and {py:class}`~sdm.processing.categorical.ShuffleCategories` draw a new column or class order at every step, which acts as data augmentation. - ## The Training Loop -Each step samples a fresh context and query from the training rows and runs the model in train mode with the training recipe: +Each step samples a fresh context and query from the training rows and runs a single estimator in train mode with the training recipe: ```python import torch.nn.functional as F +context_size, query_size, steps_per_epoch = 256, 128, 30 optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5) model.train() -for step in range(num_steps): +for step in range(steps_per_epoch): index = torch.randperm(train_table.size(0), device="cuda") context = train_table[index[:context_size]] query = train_table[index[context_size:context_size + query_size]] @@ -113,6 +122,7 @@ for step in range(num_steps): x_context=context.drop_columns("target"), y_context=context["target"], x_query=query.drop_columns("target"), + num_estimators=1, recipe=train_recipe, )[0] @@ -128,11 +138,14 @@ for step in range(num_steps): optimizer.step() ``` +With the identity output, the model returns its outputs stacked along a leading estimator dimension, so `[0]` selects the output of the single estimator. + For classification, the output has one logit column per class that appears in the sampled context. -{py:func}`~sdm.evaluation.to_class_indices` aligns these columns with the classes of the target column and converts the query targets to class indices. -Setting `missing_score=-torch.inf` gives classes that are absent from the context zero probability, so the loss is a standard cross-entropy over the aligned logits. +{py:func}`~sdm.evaluation.to_class_indices` aligns these columns with the classes of the query targets and converts the targets to class indices. +Setting `missing_score=-torch.inf` gives classes that are absent from the context zero probability. +If the class of a query row is absent from the context, its loss is infinite, so make sure that every class appears in the context, or compute the loss on probabilities with a finite `missing_score`, as the benchmark harness below does. -For regression, {py:class}`~sdm.models.TabICLv2` predicts 999 quantiles, named `"q001"` through `"q999"`. +For regression, both models predict 999 quantiles (see [In-Context Learning](icl)). The pinball (quantile) loss trains each output column toward its quantile level, and with it the full predictive distribution rather than only a point estimate: ```python @@ -141,13 +154,11 @@ level = torch.linspace(0.001, 0.999, 999, device="cuda") loss = torch.maximum(level * diff, (level - 1) * diff).mean() ``` -{py:meth}`~sdm.models.ICLModel.predict` raises if the model is in train mode, so training steps always go through {py:meth}`~sdm.models.ICLModel.forward`. - ## Model Selection Fine-tuning can overfit, in particular on small datasets. -We therefore score the model on the validation rows after every epoch and keep the best weights. -Evaluation runs the model exactly as it will be used at inference: in eval mode, with the default recipe, with estimator ensembling, and with the training rows as context: +We therefore score the model on the validation rows after every *epoch*, here a fixed number of training steps rather than a full pass over the data, and keep the best weights. +Evaluation runs the model as it will be used at inference: in eval mode, with the default recipe, with [estimator ensembling](icl), and with the training rows as context: ```python import copy @@ -170,6 +181,7 @@ def evaluate(context: sdm.TableTensor, query: sdm.TableTensor) -> float: best_acc = evaluate(context=train_table, query=val_table) # Pretrained model. best_state = copy.deepcopy(model.state_dict()) +max_epochs = 10 for epoch in range(max_epochs): model.train() ... # Training steps from above. @@ -182,15 +194,24 @@ model.load_state_dict(best_state) test_acc = evaluate(context=train_table, query=test_table) ``` -Since the first evaluation scores the pretrained model, selection never returns weights that score below the zero-shot model on the validation rows. -For large test sets, replace the one-shot {py:meth}`~sdm.models.ICLModel.forward` call with {py:meth}`~sdm.models.ICLModel.fit` and batched {py:meth}`~sdm.models.ICLModel.predict` calls, optionally under mixed-precision autocasting (see [here](icl)). +Since the first evaluation scores the pretrained model, selection never returns weights that score below it on the validation rows. +For large test sets, replace the one-shot {py:meth}`~sdm.models.ICLModel.forward` call with {py:meth}`~sdm.models.ICLModel.fit` and batched {py:meth}`~sdm.models.ICLModel.predict` calls, optionally with mixed-precision autocasting (see [In-Context Learning](icl)). ## Benchmark Results -The benchmark harness in [`benchmark/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/benchmark/tabular/finetune.py) implements this procedure for TALENT, TabArena, BeyondArena, and ScoringBench. -By default, it trains with AdamW at a learning rate of `1e-6` for 75 epochs of 10 steps, samples up to 10,000 rows per step and splits them 80/20 into context and query rows, and holds out 20% of each dataset's training split for model selection. +The benchmark harness in [`benchmark/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/benchmark/tabular/finetune.py) runs this procedure on [TALENT](https://github.com/LAMDA-Tabular/TALENT), [TabArena](https://github.com/autogluon/tabarena), and [ScoringBench](https://scoringbench.com). +It differs from the example in one respect: it keeps the default recipe during training, so the loss is computed on class probabilities and sorted quantiles instead of raw outputs. +Its default settings are tuned for larger datasets than the example: -Each row below counts the datasets on which the fine-tuned model scores better (win), worse (loss), or the same (tie) as the same model without fine-tuning, according to each benchmark's own metric: +| Setting | Example | Harness | +| ---------------- | --------------- | ------------------------- | +| Learning rate | `1e-5` | `1e-6` | +| Context size *C* | 256 | up to 8,000 | +| Query size *Q* | 128 | up to 2,000 | +| Epochs × steps | 10 × 30 | 75 × 10 | +| Validation rows | 20% of all rows | 20% of the training split | + +Each row below counts the datasets on which the fine-tuned model scores better (win), worse (loss), or the same (tie) as the pretrained model, according to each benchmark's own metric: | Benchmark | Model | Tasks | Wins | Losses | Ties | | ------------ | ------------------------------------------- | ------------------------- | ---- | ------ | ---- | @@ -199,10 +220,12 @@ Each row below counts the datasets on which the fine-tuned model scores better ( | ScoringBench | {py:class}`~sdm.models.TabICLv2` | Probabilistic regression | 50 | 26 | 10 | | TabArena | {py:class}`~sdm.models.KumoTabular` (large) | Classification/regression | 35 | 12 | 0 | -Fine-tuning improves results on more datasets than it degrades on every benchmark. -On TALENT, the gains of {py:class}`~sdm.models.TabICLv2` are larger on multiclass datasets (36 wins, 18 losses, 15 ties) than on binary ones (37 wins, 33 losses, 16 ties). +On every benchmark, fine-tuning helps on more datasets than it hurts. +On TALENT, fine-tuning {py:class}`~sdm.models.TabICLv2` helps on twice as many multiclass datasets as it hurts (36 *vs.* 18), but on binary datasets about as often as it hurts (37 *vs.* 33). +The {py:class}`~sdm.models.KumoTabular` row on TALENT covers fewer datasets because its fine-tuning variant supports at most 10 classes. +On binary tasks, it also fine-tunes for 50 instead of 75 epochs. -To reproduce these results, add `--finetune` to the TALENT and ScoringBench runners, or pick a fine-tuned `-ft` model variant on TabArena and BeyondArena: +To reproduce these results, add `--finetune` to the TALENT and ScoringBench runners (with `--model kumo-tabular-small-ft` for {py:class}`~sdm.models.KumoTabular` on TALENT), or pick a fine-tuned `-ft` model on TabArena: ```bash python -m benchmark.tabular.talent.main \ @@ -215,17 +238,15 @@ python -m benchmark.tabular.tabarena.main --model kumo-tabular-large-ft See the [tabular benchmark instructions](https://github.com/NVIDIA/structured-data-models/tree/main/benchmark/tabular) for setup and all fine-tuning flags. -## Choosing Context Size, Query Size, and Learning Rate - -Three settings of the training loop shape the result: +## Choosing Hyperparameters -- **Learning rate:** How far each step moves the pretrained weights. -- **Context size *C*:** How many labeled rows the model reads per step. -- **Query size *Q*:** How many rows contribute to the loss per step, *i.e.*, the batch size. +On the two datasets we tested, the context size *C* mattered most, learning rates between `1e-6` and `1e-5` worked best, and the query size *Q* mattered least. +In practice, start from the harness defaults, make *C* as large as memory allows, and tune the learning rate on a validation split. +The absolute gains were small, since the pretrained model is already strong on both datasets. -To measure their effect, we fine-tuned {py:class}`~sdm.models.TabICLv2` on two of the larger TALENT datasets, `customer_satisfaction_in_airline` (*Airline* in the figures; binary classification, 83,123 training rows) and `Diamonds` (regression, 34,521 training rows). -Each run uses the benchmark harness configuration from above with a fixed budget of 750 steps and changes one setting at a time, starting from the defaults (learning rate `1e-6`, *C* = 8,000, *Q* = 2,000). -The figure shows test accuracy (higher is better) and RMSE (lower is better) as mean and standard deviation over 3 seeds, using the full training split as context; the dashed line is the zero-shot model: +We fine-tuned {py:class}`~sdm.models.TabICLv2` with the benchmark harness on two of the larger TALENT datasets: Airline (`customer_satisfaction_in_airline`; binary classification, 83,123 training rows) and `Diamonds` (regression, 34,521 training rows). +Each run took 750 steps and changed one setting at a time from the harness defaults (learning rate `1e-6`, *C* = 8,000, *Q* = 2,000). +The figure shows test accuracy (higher is better) and RMSE (lower is better), averaged over 3 seeds with the full training split as context; error bars show the standard deviation, and the dashed line is the pretrained model: ```{figure} images/finetuning_sweep_light.svg :figclass: light-only @@ -237,18 +258,15 @@ The figure shows test accuracy (higher is better) and RMSE (lower is better) as :width: 100% ``` -- **Context size has the clearest effect.** With *C* = 500 or 2,000, fine-tuning does not improve on the zero-shot model on either dataset. Gains start at *C* = 8,000 and are largest at *C* = 16,000, the largest size we tried (96.53% *vs.* 96.42% accuracy and 489.3 *vs.* 495.1 RMSE). At inference, the model reads the whole training split, so larger training contexts are closer to the setting it is evaluated in. -- **Learning rates between `1e-6` and `1e-5` work best.** With `1e-7` or `3e-7`, 750 steps leave accuracy on `customer_satisfaction_in_airline` at its zero-shot level and recover only part of the gain on `Diamonds`. On `customer_satisfaction_in_airline`, `3e-6` and `1e-5` improve further over `1e-6`; on `Diamonds`, `1e-6` and `1e-5` perform alike. We did not observe degradation up to `1e-5`, the largest rate we tried. -- **Query size matters less.** Accuracy on `customer_satisfaction_in_airline` rises steadily from *Q* = 128 to *Q* = 8,000, while on `Diamonds`, *Q* = 2,000 scores best and *Q* = 8,000 is within the variation across seeds. +- **Context size mattered most.** With *C* = 500 or 2,000, fine-tuning did not improve on the pretrained model on either dataset, while *C* = 16,000, the largest size we tried, did best on both. A likely reason is that the model reads the whole training split at inference, so larger training contexts are closer to how it is used. These sizes are specific to datasets of this scale: the example uses *C* = 256 because its dataset has only about 1,000 training rows. +- **Learning rates between `1e-6` and `1e-5` worked best** within the range we tried (`1e-7` to `1e-5`). Smaller rates left Airline at its pretrained accuracy and recovered only part of the gain on Diamonds. On Airline, `3e-6` and `1e-5` improved over `1e-6`; on Diamonds, `1e-6` and `1e-5` performed alike. +- **Query size mattered least.** Airline accuracy rose steadily from *Q* = 128 to *Q* = 8,000, while on Diamonds, *Q* = 2,000 scored best and *Q* = 8,000 was within the variation across seeds. -Memory and time grow with *C* + *Q*. -On an H100 GPU, fine-tuning on `customer_satisfaction_in_airline` peaks at 6 GB of memory for *C* = 500, 13 GB for *C* = 8,000, and 23 GB for *C* = 16,000 (all with *Q* = 2,000), and takes 46 to 60 minutes per run. - -In practice, start from the harness defaults, increase *C* as far as memory allows, and tune the learning rate between `1e-6` and `1e-5` on a validation split. -Note that the absolute gains on these two datasets are small, since {py:class}`~sdm.models.TabICLv2` is already strong on them zero-shot: the best settings, selected on validation, improve test accuracy from 96.42% to 96.54% (learning rate `1e-5`) and test RMSE from 495.1 to 489.3 (*C* = 16,000). +Selected on validation, the best settings improved test accuracy on Airline from 96.42% to 96.54% (learning rate `1e-5`) and test RMSE on Diamonds from 495.1 to 489.3 (*C* = 16,000). +Memory and time grow with *C* + *Q*: on Airline, fine-tuning peaked at 6 GB of GPU memory for *C* = 500, 13 GB for *C* = 8,000, and 23 GB for *C* = 16,000 (all with *Q* = 2,000), and a run took 46 to 60 minutes on an H100 GPU. The TALENT runner exposes all three settings: `--finetune-lr`, `--finetune-train-size` (*C* + *Q*), and `--finetune-context-frac` (*C* / (*C* + *Q*)). -For example, the best `Diamonds` run uses: +For example, the best Diamonds run uses *C* = 16,000 and *Q* = 2,000: ```bash python -m benchmark.tabular.talent.main \ @@ -262,24 +280,31 @@ python -m benchmark.tabular.talent.main \ ## Smaller Contexts After Fine-Tuning -A zero-shot TFM knows about your dataset only what its context shows it. +A pretrained TFM knows about your dataset only what its context shows it. Its accuracy therefore typically keeps improving as more training rows are passed as context, which makes the full training split the default choice. -Inference cost grows with the context as well: {py:meth}`~sdm.models.ICLModel.forward` processes all context rows on every call, and {py:meth}`~sdm.models.ICLModel.fit` keeps per-row key/value state in memory for subsequent {py:meth}`~sdm.models.ICLModel.predict` calls. +Inference cost grows with the context as well, both for one-shot calls and for {py:meth}`~sdm.models.ICLModel.fit` and {py:meth}`~sdm.models.ICLModel.predict` (see [In-Context Learning](icl)). -Fine-tuning moves part of that dataset knowledge into the model weights. -A fine-tuned model can therefore reach comparable accuracy with fewer context rows, which reduces inference memory and latency, *e.g.*, when a model serves predictions for a large training split. +Fine-tuning moves part of that dataset knowledge into the weights. +A fine-tuned model can therefore reach comparable accuracy with fewer context rows, which reduces inference memory and latency, *e.g.*, when serving predictions with a large training split as context. To use a smaller context, fit the fine-tuned model on a random subset of the training rows: ```python +model.eval() + index = torch.randperm(train_table.size(0), device="cuda")[:4096] context = train_table[index] -model.fit(x=context.drop_columns("target"), y=context["target"]) +model.fit( + x=context.drop_columns("target"), + y=context["target"], + num_estimators=8, +) out = model.predict(test_table.drop_columns("target")) ``` -To measure the effect, we took the best fine-tuned model per dataset from the previous section, selected on validation (learning rate `1e-5` for `customer_satisfaction_in_airline` and *C* = 16,000 for `Diamonds`), and fitted it and the zero-shot model on the same random subsets of 256 to 16,384 training rows, as well as on all of them. -The figure and table show test accuracy (*Airline*) and RMSE (*Diamonds*) as the mean over 3 seeds, with the standard deviation as error bars; the dashed line is the zero-shot model with all training rows as context: +To measure the effect, we took the best fine-tuned model per dataset from the previous section (learning rate `1e-5` for Airline, *C* = 16,000 for Diamonds). +We fitted it and the pretrained model on the same random subsets of 256 to 16,384 training rows, and on the full training split. +The figure and table show test accuracy (Airline) and RMSE (Diamonds), averaged over 3 seeds; the dashed line is the pretrained model with the full training split as context: ```{figure} images/finetuning_context_light.svg :figclass: light-only @@ -299,10 +324,9 @@ The figure and table show test accuracy (*Airline*) and RMSE (*Diamonds*) as the | 16,384 | 95.87% | 96.30% | 505.0 | 496.0 | | All | 96.42% | 96.57% | 495.1 | 488.8 | -- **The fine-tuned model is better at every context size.** The gap is largest between 1,024 and 4,096 rows, *e.g.*, 93.82% *vs.* 92.67% accuracy at 1,024 rows and 528.3 *vs.* 548.3 RMSE at 4,096 rows. At 256 rows, it is within the variation across seeds. -- **With 16,384 rows, the fine-tuned model comes close to the zero-shot model with all rows**, using 20% of the `customer_satisfaction_in_airline` training split (96.30% *vs.* 96.42%) and 47% of the `Diamonds` training split (496.0 *vs.* 495.1). -- **More context still helps.** Both models improve with every increase in context size, and the fine-tuned model with all rows scores best on both datasets. - -The full-context results of the fine-tuned models differ slightly from the previous section because estimator ensembles are randomized at inference. +- **The fine-tuned model scored better at every context size**, but at the smallest contexts (256 rows on both datasets and 1,024 rows on Diamonds), the difference was within the variation across seeds. The largest clear gaps were at 1,024 rows on Airline (93.82% *vs.* 92.67%) and at 4,096 rows on Diamonds (528.3 *vs.* 548.3 RMSE). +- **With 16,384 rows, the fine-tuned model came close to the pretrained model with the full training split**, using 20% of the Airline training split (96.30% *vs.* 96.42%) and 47% of the Diamonds training split (496.0 *vs.* 495.1). +- **More context still helped.** Both models improved with every increase in context size, and the fine-tuned model with the full training split scored best on both datasets. -Estimator ensembling complements smaller contexts: each estimator can receive a different subset of the training rows, so an ensemble covers more of the training split while each estimator keeps a small context (see the ensembling section [here](icl)). +The full-context results differ slightly from the previous section because estimator ensembles are randomized at inference. +[Estimator ensembling](icl) also complements smaller contexts by giving each estimator a different subset of the training rows. From 87b5176b55ddf10cc80ffbd02dd7c65c9a3baf6f Mon Sep 17 00:00:00 2001 From: wsad1 Date: Tue, 27 Oct 2026 22:20:48 +0000 Subject: [PATCH 3/8] Tighten fine-tuning guide State each fact once, drop details that the figures and tables already show, and shorten lead-ins. --- docs/source/finetuning.md | 116 +++++++++++++------------------------- 1 file changed, 38 insertions(+), 78 deletions(-) diff --git a/docs/source/finetuning.md b/docs/source/finetuning.md index e21eed5395..73f4afec00 100644 --- a/docs/source/finetuning.md +++ b/docs/source/finetuning.md @@ -5,18 +5,12 @@ This works well out of the box, but the pretrained weights know nothing specific **Full fine-tuning** updates every weight of a pretrained model on one dataset. The result is still an in-context learner with the same interface, and it trains with an ordinary PyTorch loop, since calling {py:meth}`~sdm.models.ICLModel.forward` in train mode enables gradients (see [In-Context Learning](icl)). -Below, the *pretrained model* is the model before fine-tuning, labeled *zero-shot* in the figures. - -Fine-tuning does not always pay off. -On standard benchmarks, it helps on more datasets than it hurts, but not on all of them, and gains tend to be small when the pretrained model is already strong. -On datasets with tens of thousands of rows, a run took between 10 and 60 minutes on a single H100 GPU in our experiments. -The procedure below therefore keeps the pretrained weights whenever fine-tuning does not improve the validation score. +Below, the *pretrained model* is the model before fine-tuning, labeled *zero-shot* in the figures and tables. +Fine-tuning does not always pay off: on standard benchmarks, it helps on more datasets than it hurts, but not on all of them. ## How Full Fine-Tuning Works -Fine-tuning keeps the objective a TFM is pretrained with: predict held-out rows from labeled examples. -What changes is where the tasks come from. -Instead of its pretraining data, each training step draws a task from the training split of your dataset: +Fine-tuning keeps the pretraining objective, predicting held-out rows from labeled examples, but draws each task from the training split of your dataset: ```{figure} images/finetuning_light.svg :figclass: light-only @@ -33,8 +27,7 @@ Instead of its pretraining data, each training step draws a task from the traini 3. Compute the loss of the query predictions against the held-out query labels. 4. Backpropagate and update every weight of the model. -Because the context changes from step to step, the model learns to read its context, as it does at inference, instead of memorizing a fixed set of rows. -After fine-tuning, inference is unchanged: the training split becomes the context, and the model predicts new rows through {py:meth}`~sdm.models.ICLModel.forward` or {py:meth}`~sdm.models.ICLModel.fit` and {py:meth}`~sdm.models.ICLModel.predict`. +Because the context changes at every step, the model learns to read its context, as it does at inference, instead of memorizing fixed rows. ```{note} The next four sections walk through [`examples/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/examples/tabular/finetune.py), which fine-tunes {py:class}`~sdm.models.TabICLv2` or {py:class}`~sdm.models.KumoTabular` on a classification or regression dataset. @@ -42,14 +35,12 @@ The next four sections walk through [`examples/tabular/finetune.py`](https://git ## Data Splits -Fine-tuning updates the weights on the training rows, so you need separate validation rows to pick the weights and test rows to report results: +Fine-tuning splits the rows of a {py:class}`~sdm.tensor.TableTensor` (see [Table Semantics](tensor)) three ways: -- **Training** rows are sampled into context and query rows during fine-tuning, and also serve as the context when the model is evaluated. +- **Training** rows are sampled into context and query rows during fine-tuning, and serve as the context at inference. - **Validation** rows pick the best weights and never appear in a training step. - **Test** rows are used once, for the final evaluation. -The snippet below builds a {py:class}`~sdm.tensor.TableTensor` (see [Table Semantics](tensor)) and splits its rows: - ```python import torch from sklearn.datasets import load_digits @@ -74,8 +65,8 @@ test_table = table[index[num_train + num_val:]] ## A Training Recipe -Every model ships a default {py:class}`~sdm.processing.recipe.Recipe` (see [Data Processing](processing)) whose output pipeline is built for inference: it combines the outputs of all estimators and turns classification logits into probabilities. -For training, we replace it with {py:class}`~sdm.processing.common.Identity`, so the loss is computed directly on the model outputs: +Every model ships a default {py:class}`~sdm.processing.recipe.Recipe` (see [Data Processing](processing)) whose output pipeline is built for inference: it combines the estimator outputs and turns classification logits into probabilities. +For training, replace it with {py:class}`~sdm.processing.common.Identity` to compute the loss directly on the model outputs: ```python import sdm.processing as sp @@ -86,11 +77,10 @@ train_recipe = model.default_recipe() train_recipe.output = sp.Identity() # No post-processing. ``` -The rest of the recipe stays as is. -Its stochastic processors, *e.g.*, {py:class}`~sdm.processing.common.ShuffleColumns` in the features and {py:class}`~sdm.processing.categorical.ShuffleCategories` in the target, draw a new order at every step, which acts as data augmentation. +The rest of the recipe stays, and its stochastic processors, *e.g.*, {py:class}`~sdm.processing.common.ShuffleColumns` and {py:class}`~sdm.processing.categorical.ShuffleCategories`, draw a new order at every step, which acts as data augmentation. -For regression with {py:class}`~sdm.models.KumoTabular`, also remove random sign flipping ({py:class}`~sdm.processing.numerical.FlipSign`) from the target pipeline. -A flipped target reverses the order of the predicted quantiles, which the default output pipeline restores via {py:class}`~sdm.processing.output.SortQuantiles`, but the identity output does not: +For regression with {py:class}`~sdm.models.KumoTabular`, also remove sign flipping ({py:class}`~sdm.processing.numerical.FlipSign`) from the target pipeline. +A flipped target reverses the predicted quantiles, which the default output pipeline re-sorts via {py:class}`~sdm.processing.output.SortQuantiles`, but the identity output does not: ```python train_recipe.target = sp.StypeDispatch( # Default target pipeline without FlipSign. @@ -104,7 +94,7 @@ train_recipe.target = sp.StypeDispatch( # Default target pipeline without FlipS ## The Training Loop -Each step samples a fresh context and query from the training rows and runs a single estimator in train mode with the training recipe: +Each step runs a single estimator in train mode with the training recipe: ```python import torch.nn.functional as F @@ -138,15 +128,12 @@ for step in range(steps_per_epoch): optimizer.step() ``` -With the identity output, the model returns its outputs stacked along a leading estimator dimension, so `[0]` selects the output of the single estimator. +With the identity output, outputs are stacked along a leading estimator dimension, so `[0]` selects the single estimator. -For classification, the output has one logit column per class that appears in the sampled context. -{py:func}`~sdm.evaluation.to_class_indices` aligns these columns with the classes of the query targets and converts the targets to class indices. -Setting `missing_score=-torch.inf` gives classes that are absent from the context zero probability. -If the class of a query row is absent from the context, its loss is infinite, so make sure that every class appears in the context, or compute the loss on probabilities with a finite `missing_score`, as the benchmark harness below does. +For classification, the output has one logit column per class in the sampled context, which {py:func}`~sdm.evaluation.to_class_indices` aligns with the query targets. +`missing_score=-torch.inf` gives classes absent from the context zero probability, so a query row whose own class is absent gets infinite loss: make sure that every class appears in the context, or compute the loss on probabilities with a finite `missing_score`. -For regression, both models predict 999 quantiles (see [In-Context Learning](icl)). -The pinball (quantile) loss trains each output column toward its quantile level, and with it the full predictive distribution rather than only a point estimate: +For regression, both models predict 999 quantiles (see [In-Context Learning](icl)), and the pinball (quantile) loss trains each output column toward its quantile level: ```python diff = query["target"].numerical - out.numerical # [Q, 999] @@ -156,9 +143,8 @@ loss = torch.maximum(level * diff, (level - 1) * diff).mean() ## Model Selection -Fine-tuning can overfit, in particular on small datasets. -We therefore score the model on the validation rows after every *epoch*, here a fixed number of training steps rather than a full pass over the data, and keep the best weights. -Evaluation runs the model as it will be used at inference: in eval mode, with the default recipe, with [estimator ensembling](icl), and with the training rows as context: +Fine-tuning can overfit, especially on small datasets, so we score the model on the validation rows after every *epoch* (here a fixed number of training steps) and keep the best weights. +Evaluation matches inference: eval mode, the default recipe, and [estimator ensembling](icl): ```python import copy @@ -195,13 +181,12 @@ test_acc = evaluate(context=train_table, query=test_table) ``` Since the first evaluation scores the pretrained model, selection never returns weights that score below it on the validation rows. -For large test sets, replace the one-shot {py:meth}`~sdm.models.ICLModel.forward` call with {py:meth}`~sdm.models.ICLModel.fit` and batched {py:meth}`~sdm.models.ICLModel.predict` calls, optionally with mixed-precision autocasting (see [In-Context Learning](icl)). +For large test sets, use {py:meth}`~sdm.models.ICLModel.fit` and batched {py:meth}`~sdm.models.ICLModel.predict` calls instead (see [In-Context Learning](icl)). ## Benchmark Results The benchmark harness in [`benchmark/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/benchmark/tabular/finetune.py) runs this procedure on [TALENT](https://github.com/LAMDA-Tabular/TALENT), [TabArena](https://github.com/autogluon/tabarena), and [ScoringBench](https://scoringbench.com). -It differs from the example in one respect: it keeps the default recipe during training, so the loss is computed on class probabilities and sorted quantiles instead of raw outputs. -Its default settings are tuned for larger datasets than the example: +Unlike the example, it keeps the default recipe during training, so the loss is computed on class probabilities and sorted quantiles, and its defaults target larger datasets: | Setting | Example | Harness | | ---------------- | --------------- | ------------------------- | @@ -211,7 +196,7 @@ Its default settings are tuned for larger datasets than the example: | Epochs × steps | 10 × 30 | 75 × 10 | | Validation rows | 20% of all rows | 20% of the training split | -Each row below counts the datasets on which the fine-tuned model scores better (win), worse (loss), or the same (tie) as the pretrained model, according to each benchmark's own metric: +Wins, losses, and ties count the datasets on which fine-tuning improved, worsened, or matched the pretrained model's score on each benchmark's own metric: | Benchmark | Model | Tasks | Wins | Losses | Ties | | ------------ | ------------------------------------------- | ------------------------- | ---- | ------ | ---- | @@ -220,10 +205,8 @@ Each row below counts the datasets on which the fine-tuned model scores better ( | ScoringBench | {py:class}`~sdm.models.TabICLv2` | Probabilistic regression | 50 | 26 | 10 | | TabArena | {py:class}`~sdm.models.KumoTabular` (large) | Classification/regression | 35 | 12 | 0 | -On every benchmark, fine-tuning helps on more datasets than it hurts. -On TALENT, fine-tuning {py:class}`~sdm.models.TabICLv2` helps on twice as many multiclass datasets as it hurts (36 *vs.* 18), but on binary datasets about as often as it hurts (37 *vs.* 33). -The {py:class}`~sdm.models.KumoTabular` row on TALENT covers fewer datasets because its fine-tuning variant supports at most 10 classes. -On binary tasks, it also fine-tunes for 50 instead of 75 epochs. +On TALENT, fine-tuning {py:class}`~sdm.models.TabICLv2` wins twice as often as it loses on multiclass datasets (36 *vs.* 18), but about as often on binary ones (37 *vs.* 33). +{py:class}`~sdm.models.KumoTabular` covers fewer TALENT datasets because its fine-tuning variant supports at most 10 classes. To reproduce these results, add `--finetune` to the TALENT and ScoringBench runners (with `--model kumo-tabular-small-ft` for {py:class}`~sdm.models.KumoTabular` on TALENT), or pick a fine-tuned `-ft` model on TabArena: @@ -240,13 +223,8 @@ See the [tabular benchmark instructions](https://github.com/NVIDIA/structured-da ## Choosing Hyperparameters -On the two datasets we tested, the context size *C* mattered most, learning rates between `1e-6` and `1e-5` worked best, and the query size *Q* mattered least. -In practice, start from the harness defaults, make *C* as large as memory allows, and tune the learning rate on a validation split. -The absolute gains were small, since the pretrained model is already strong on both datasets. - -We fine-tuned {py:class}`~sdm.models.TabICLv2` with the benchmark harness on two of the larger TALENT datasets: Airline (`customer_satisfaction_in_airline`; binary classification, 83,123 training rows) and `Diamonds` (regression, 34,521 training rows). -Each run took 750 steps and changed one setting at a time from the harness defaults (learning rate `1e-6`, *C* = 8,000, *Q* = 2,000). -The figure shows test accuracy (higher is better) and RMSE (lower is better), averaged over 3 seeds with the full training split as context; error bars show the standard deviation, and the dashed line is the pretrained model: +We fine-tuned {py:class}`~sdm.models.TabICLv2` with the benchmark harness on two of the larger TALENT datasets, Airline (`customer_satisfaction_in_airline`; binary classification, 83,123 training rows) and Diamonds (regression, 34,521 training rows), changing one setting at a time from the harness defaults. +The figure shows test accuracy and RMSE with the full training split as context, averaged over 3 seeds; error bars show the standard deviation: ```{figure} images/finetuning_sweep_light.svg :figclass: light-only @@ -258,35 +236,20 @@ The figure shows test accuracy (higher is better) and RMSE (lower is better), av :width: 100% ``` -- **Context size mattered most.** With *C* = 500 or 2,000, fine-tuning did not improve on the pretrained model on either dataset, while *C* = 16,000, the largest size we tried, did best on both. A likely reason is that the model reads the whole training split at inference, so larger training contexts are closer to how it is used. These sizes are specific to datasets of this scale: the example uses *C* = 256 because its dataset has only about 1,000 training rows. -- **Learning rates between `1e-6` and `1e-5` worked best** within the range we tried (`1e-7` to `1e-5`). Smaller rates left Airline at its pretrained accuracy and recovered only part of the gain on Diamonds. On Airline, `3e-6` and `1e-5` improved over `1e-6`; on Diamonds, `1e-6` and `1e-5` performed alike. -- **Query size mattered least.** Airline accuracy rose steadily from *Q* = 128 to *Q* = 8,000, while on Diamonds, *Q* = 2,000 scored best and *Q* = 8,000 was within the variation across seeds. +- **Context size mattered most.** *C* = 500 or 2,000 did not improve on the pretrained model, while *C* = 16,000, the largest size we tried, did best on both datasets, likely because larger training contexts are closer to inference, where the model reads the whole training split. These sizes are specific to datasets of this scale; the example uses *C* = 256 because its dataset has only about 1,000 training rows. +- **Learning rates between `1e-6` and `1e-5` worked best** within the range we tried (`1e-7` to `1e-5`); smaller rates helped less or not at all. +- **Query size mattered least.** Airline accuracy rose steadily from *Q* = 128 to *Q* = 8,000, while on Diamonds, *Q* = 2,000 scored best. -Selected on validation, the best settings improved test accuracy on Airline from 96.42% to 96.54% (learning rate `1e-5`) and test RMSE on Diamonds from 495.1 to 489.3 (*C* = 16,000). -Memory and time grow with *C* + *Q*: on Airline, fine-tuning peaked at 6 GB of GPU memory for *C* = 500, 13 GB for *C* = 8,000, and 23 GB for *C* = 16,000 (all with *Q* = 2,000), and a run took 46 to 60 minutes on an H100 GPU. +The gains were small, since the pretrained model is already strong on both datasets: selected on validation, the best settings improved test accuracy on Airline from 96.42% to 96.54% (learning rate `1e-5`) and test RMSE on Diamonds from 495.1 to 489.3 (*C* = 16,000). -The TALENT runner exposes all three settings: `--finetune-lr`, `--finetune-train-size` (*C* + *Q*), and `--finetune-context-frac` (*C* / (*C* + *Q*)). -For example, the best Diamonds run uses *C* = 16,000 and *Q* = 2,000: - -```bash -python -m benchmark.tabular.talent.main \ - --model tabiclv2 \ - --dataset-path /path/to/talent/data \ - --dataset Diamonds \ - --finetune \ - --finetune-train-size 18000 \ - --finetune-context-frac 0.8889 -``` +For datasets of this size, start from the harness defaults, make *C* as large as memory allows, and tune the learning rate on a validation split. +With *Q* = 2,000 on Airline, fine-tuning peaked at 13 GB of GPU memory for *C* = 8,000 and 23 GB for *C* = 16,000, and a run took 46 to 60 minutes on an H100 GPU. +The TALENT runner exposes the three settings as `--finetune-lr`, `--finetune-train-size` (*C* + *Q*), and `--finetune-context-frac` (*C* / (*C* + *Q*)). ## Smaller Contexts After Fine-Tuning -A pretrained TFM knows about your dataset only what its context shows it. -Its accuracy therefore typically keeps improving as more training rows are passed as context, which makes the full training split the default choice. -Inference cost grows with the context as well, both for one-shot calls and for {py:meth}`~sdm.models.ICLModel.fit` and {py:meth}`~sdm.models.ICLModel.predict` (see [In-Context Learning](icl)). - -Fine-tuning moves part of that dataset knowledge into the weights. -A fine-tuned model can therefore reach comparable accuracy with fewer context rows, which reduces inference memory and latency, *e.g.*, when serving predictions with a large training split as context. -To use a smaller context, fit the fine-tuned model on a random subset of the training rows: +A pretrained TFM knows your dataset only through its context, so more context rows typically improve accuracy but also raise inference cost (see [In-Context Learning](icl)). +Fine-tuning moves part of that knowledge into the weights, so a fine-tuned model can reach comparable accuracy with a smaller context, *e.g.*, a random subset of the training rows: ```python model.eval() @@ -302,9 +265,7 @@ model.fit( out = model.predict(test_table.drop_columns("target")) ``` -To measure the effect, we took the best fine-tuned model per dataset from the previous section (learning rate `1e-5` for Airline, *C* = 16,000 for Diamonds). -We fitted it and the pretrained model on the same random subsets of 256 to 16,384 training rows, and on the full training split. -The figure and table show test accuracy (Airline) and RMSE (Diamonds), averaged over 3 seeds; the dashed line is the pretrained model with the full training split as context: +We fitted the best fine-tuned model per dataset from the previous section and the pretrained model on the same random subsets of training rows, averaging test scores over 3 seeds: ```{figure} images/finetuning_context_light.svg :figclass: light-only @@ -324,9 +285,8 @@ The figure and table show test accuracy (Airline) and RMSE (Diamonds), averaged | 16,384 | 95.87% | 96.30% | 505.0 | 496.0 | | All | 96.42% | 96.57% | 495.1 | 488.8 | -- **The fine-tuned model scored better at every context size**, but at the smallest contexts (256 rows on both datasets and 1,024 rows on Diamonds), the difference was within the variation across seeds. The largest clear gaps were at 1,024 rows on Airline (93.82% *vs.* 92.67%) and at 4,096 rows on Diamonds (528.3 *vs.* 548.3 RMSE). -- **With 16,384 rows, the fine-tuned model came close to the pretrained model with the full training split**, using 20% of the Airline training split (96.30% *vs.* 96.42%) and 47% of the Diamonds training split (496.0 *vs.* 495.1). -- **More context still helped.** Both models improved with every increase in context size, and the fine-tuned model with the full training split scored best on both datasets. +- **The fine-tuned model scored better at every context size**, but at the smallest contexts (256 rows on both datasets and 1,024 rows on Diamonds), the difference was within the variation across seeds. +- **With 16,384 rows, the fine-tuned model came close to the pretrained model with the full training split**, using only 20% (Airline) and 47% (Diamonds) of the training split. +- **More context still helped.** Both models improved with every increase in context size, and the fine-tuned model scored best with the full training split. The full-context results differ slightly from the previous section because estimator ensembles are randomized at inference. -[Estimator ensembling](icl) also complements smaller contexts by giving each estimator a different subset of the training rows. From 90ccf23560ecad05c6eff1b9e4c7f7d4cfdd46e6 Mon Sep 17 00:00:00 2001 From: Jinu Sunil Date: Tue, 27 Oct 2026 18:09:50 -0700 Subject: [PATCH 4/8] Apply suggestion from @wsad1 --- docs/source/finetuning.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/finetuning.md b/docs/source/finetuning.md index 73f4afec00..eb4162db35 100644 --- a/docs/source/finetuning.md +++ b/docs/source/finetuning.md @@ -27,7 +27,6 @@ Fine-tuning keeps the pretraining objective, predicting held-out rows from label 3. Compute the loss of the query predictions against the held-out query labels. 4. Backpropagate and update every weight of the model. -Because the context changes at every step, the model learns to read its context, as it does at inference, instead of memorizing fixed rows. ```{note} The next four sections walk through [`examples/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/examples/tabular/finetune.py), which fine-tunes {py:class}`~sdm.models.TabICLv2` or {py:class}`~sdm.models.KumoTabular` on a classification or regression dataset. From 04f12bda296890de6d48c6ec6fa33b28bc2e63db Mon Sep 17 00:00:00 2001 From: Ard <88337265+aw471@users.noreply.github.com> Date: Wed, 28 Oct 2026 08:03:45 -0700 Subject: [PATCH 5/8] docs: add regression validation to fine-tuning guide --- docs/source/finetuning.md | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/docs/source/finetuning.md b/docs/source/finetuning.md index eb4162db35..872cf1739a 100644 --- a/docs/source/finetuning.md +++ b/docs/source/finetuning.md @@ -70,7 +70,8 @@ For training, replace it with {py:class}`~sdm.processing.common.Identity` to com ```python import sdm.processing as sp -model = sdm.models.TabICLv2(task="classification", device="cuda") +task = "classification" # Use "regression" for a numerical target. +model = sdm.models.TabICLv2(task=task, device="cuda") train_recipe = model.default_recipe() train_recipe.output = sp.Identity() # No post-processing. @@ -143,7 +144,8 @@ loss = torch.maximum(level * diff, (level - 1) * diff).mean() ## Model Selection Fine-tuning can overfit, especially on small datasets, so we score the model on the validation rows after every *epoch* (here a fixed number of training steps) and keep the best weights. -Evaluation matches inference: eval mode, the default recipe, and [estimator ensembling](icl): +Evaluation matches inference: eval mode, the default recipe, and [estimator ensembling](icl). +For classification, maximize accuracy; for regression, average the predicted quantiles into a point prediction and minimize root mean squared error (RMSE): ```python import copy @@ -156,14 +158,20 @@ def evaluate(context: sdm.TableTensor, query: sdm.TableTensor) -> float: x_query=query.drop_columns("target"), num_estimators=8, ) - prob, y = sdm.evaluation.to_class_indices( - pred=out, - target=query["target"], - missing_score=0.0, - ) - return (prob.argmax(dim=-1) == y).float().mean().item() - -best_acc = evaluate(context=train_table, query=val_table) # Pretrained model. + if task == "classification": + prob, y = sdm.evaluation.to_class_indices( + pred=out, + target=query["target"], + missing_score=0.0, + ) + return (prob.argmax(dim=-1) == y).float().mean().item() + + pred = out.numerical.mean(dim=-1) + y = query["target"].numerical.squeeze(-1) + return (pred - y).pow(2).mean().sqrt().item() + +higher_is_better = task == "classification" +best_metric = evaluate(context=train_table, query=val_table) # Pretrained model. best_state = copy.deepcopy(model.state_dict()) max_epochs = 10 @@ -171,12 +179,12 @@ for epoch in range(max_epochs): model.train() ... # Training steps from above. - acc = evaluate(context=train_table, query=val_table) - if acc > best_acc: - best_acc, best_state = acc, copy.deepcopy(model.state_dict()) + metric = evaluate(context=train_table, query=val_table) + if metric > best_metric if higher_is_better else metric < best_metric: + best_metric, best_state = metric, copy.deepcopy(model.state_dict()) model.load_state_dict(best_state) -test_acc = evaluate(context=train_table, query=test_table) +test_metric = evaluate(context=train_table, query=test_table) ``` Since the first evaluation scores the pretrained model, selection never returns weights that score below it on the validation rows. From 47a7c8e78c6d6567346f5a6441b526edf11ba2d5 Mon Sep 17 00:00:00 2001 From: Ard <88337265+aw471@users.noreply.github.com> Date: Wed, 28 Oct 2026 08:07:56 -0700 Subject: [PATCH 6/8] docs: clarify regression validation and fix Markdown formatting --- docs/source/finetuning.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/finetuning.md b/docs/source/finetuning.md index 872cf1739a..3537f61291 100644 --- a/docs/source/finetuning.md +++ b/docs/source/finetuning.md @@ -27,7 +27,6 @@ Fine-tuning keeps the pretraining objective, predicting held-out rows from label 3. Compute the loss of the query predictions against the held-out query labels. 4. Backpropagate and update every weight of the model. - ```{note} The next four sections walk through [`examples/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/examples/tabular/finetune.py), which fine-tunes {py:class}`~sdm.models.TabICLv2` or {py:class}`~sdm.models.KumoTabular` on a classification or regression dataset. ``` @@ -166,6 +165,7 @@ def evaluate(context: sdm.TableTensor, query: sdm.TableTensor) -> float: ) return (prob.argmax(dim=-1) == y).float().mean().item() + # Regression: average the predicted quantiles and compute RMSE. pred = out.numerical.mean(dim=-1) y = query["target"].numerical.squeeze(-1) return (pred - y).pow(2).mean().sqrt().item() From 6f8cd0ba24444be8f578f8c64757a295a076975e Mon Sep 17 00:00:00 2001 From: wsad1 Date: Thu, 29 Oct 2026 18:53:55 +0000 Subject: [PATCH 7/8] Explain why the training recipe keeps random processors --- docs/source/finetuning.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/finetuning.md b/docs/source/finetuning.md index 3537f61291..fd1f3a9155 100644 --- a/docs/source/finetuning.md +++ b/docs/source/finetuning.md @@ -76,7 +76,9 @@ train_recipe = model.default_recipe() train_recipe.output = sp.Identity() # No post-processing. ``` -The rest of the recipe stays, and its stochastic processors, *e.g.*, {py:class}`~sdm.processing.common.ShuffleColumns` and {py:class}`~sdm.processing.categorical.ShuffleCategories`, draw a new order at every step, which acts as data augmentation. +The feature and target pipelines stay unchanged: they turn raw columns into the inputs that the pretrained model expects, and evaluation and inference use them too. +Some of their processors are random: {py:class}`~sdm.processing.common.ShuffleColumns` reorders the feature columns, and {py:class}`~sdm.processing.categorical.ShuffleCategories` relabels the classes, which gives each estimator a different view of the same task at inference (see Ensembling in [In-Context Learning](icl)). +During training, each step draws a new view, so the model cannot rely on a column's position or a class's label; this acts as data augmentation, like random crops and flips for images. For regression with {py:class}`~sdm.models.KumoTabular`, also remove sign flipping ({py:class}`~sdm.processing.numerical.FlipSign`) from the target pipeline. A flipped target reverses the predicted quantiles, which the default output pipeline re-sorts via {py:class}`~sdm.processing.output.SortQuantiles`, but the identity output does not: From 941fc600d66acef7b0a6ecce493eeae2fd378369 Mon Sep 17 00:00:00 2001 From: wsad1 Date: Thu, 29 Oct 2026 20:16:17 +0000 Subject: [PATCH 8/8] Restructure fine-tuning guide --- docs/source/finetuning.md | 130 ++++++++++++++++++++------------------ 1 file changed, 68 insertions(+), 62 deletions(-) diff --git a/docs/source/finetuning.md b/docs/source/finetuning.md index fd1f3a9155..bed5e7e63e 100644 --- a/docs/source/finetuning.md +++ b/docs/source/finetuning.md @@ -3,14 +3,14 @@ Tabular foundation models (TFMs) such as {py:class}`~sdm.models.TabICLv2` and {py:class}`~sdm.models.KumoTabular` solve a new task through [in-context learning](icl): they read labeled context rows and predict query rows without updating their weights. This works well out of the box, but the pretrained weights know nothing specific about your dataset. -**Full fine-tuning** updates every weight of a pretrained model on one dataset. -The result is still an in-context learner with the same interface, and it trains with an ordinary PyTorch loop, since calling {py:meth}`~sdm.models.ICLModel.forward` in train mode enables gradients (see [In-Context Learning](icl)). -Below, the *pretrained model* is the model before fine-tuning, labeled *zero-shot* in the figures and tables. -Fine-tuning does not always pay off: on standard benchmarks, it helps on more datasets than it hurts, but not on all of them. +**Full fine-tuning** trains every weight of the pretrained model on your dataset, and the result is still an in-context learner with the same interface. +On standard benchmarks, it helps on more datasets than it hurts: on TALENT, for example, it improved {py:class}`~sdm.models.TabICLv2` on 73 datasets and worsened it on 51. +A fine-tuned model can also use a smaller, cheaper inference context: on two large datasets, it came close to the pretrained model's full-context score with 20% to 47% of the training rows. +In exchange, each dataset needs its own GPU training run, its own copy of the weights, and a validation split to catch overfitting. ## How Full Fine-Tuning Works -Fine-tuning keeps the pretraining objective, predicting held-out rows from labeled examples, but draws each task from the training split of your dataset: +Fine-tuning keeps the pretraining task, predicting query labels from labeled context rows, but samples each task from the training split of your dataset: ```{figure} images/finetuning_light.svg :figclass: light-only @@ -28,7 +28,7 @@ Fine-tuning keeps the pretraining objective, predicting held-out rows from label 4. Backpropagate and update every weight of the model. ```{note} -The next four sections walk through [`examples/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/examples/tabular/finetune.py), which fine-tunes {py:class}`~sdm.models.TabICLv2` or {py:class}`~sdm.models.KumoTabular` on a classification or regression dataset. +The code below for splitting, training, and keeping the best weights is condensed from [`examples/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/examples/tabular/finetune.py), which fine-tunes {py:class}`~sdm.models.TabICLv2` or {py:class}`~sdm.models.KumoTabular` on a classification or regression dataset. ``` ## Data Splits @@ -36,7 +36,7 @@ The next four sections walk through [`examples/tabular/finetune.py`](https://git Fine-tuning splits the rows of a {py:class}`~sdm.tensor.TableTensor` (see [Table Semantics](tensor)) three ways: - **Training** rows are sampled into context and query rows during fine-tuning, and serve as the context at inference. -- **Validation** rows pick the best weights and never appear in a training step. +- **Validation** rows are used to choose the weights and settings, and never appear in a training step. - **Test** rows are used once, for the final evaluation. ```python @@ -61,9 +61,10 @@ val_table = table[index[num_train:num_train + num_val]] test_table = table[index[num_train + num_val:]] ``` -## A Training Recipe +## Training -Every model ships a default {py:class}`~sdm.processing.recipe.Recipe` (see [Data Processing](processing)) whose output pipeline is built for inference: it combines the estimator outputs and turns classification logits into probabilities. +Since {py:meth}`~sdm.models.ICLModel.forward` enables gradients in train mode (see [In-Context Learning](icl)), fine-tuning is an ordinary PyTorch loop that runs the model with a training recipe. +Every model ships a default {py:class}`~sdm.processing.recipe.Recipe` (see [Data Processing](processing)) whose output pipeline is built for inference: it combines the outputs of the ensemble members (*estimators*) and turns classification logits into probabilities. For training, replace it with {py:class}`~sdm.processing.common.Identity` to compute the loss directly on the model outputs: ```python @@ -77,10 +78,11 @@ train_recipe.output = sp.Identity() # No post-processing. ``` The feature and target pipelines stay unchanged: they turn raw columns into the inputs that the pretrained model expects, and evaluation and inference use them too. -Some of their processors are random: {py:class}`~sdm.processing.common.ShuffleColumns` reorders the feature columns, and {py:class}`~sdm.processing.categorical.ShuffleCategories` relabels the classes, which gives each estimator a different view of the same task at inference (see Ensembling in [In-Context Learning](icl)). +Some of their processors are random: {py:class}`~sdm.processing.common.ShuffleColumns` reorders the feature columns, and {py:class}`~sdm.processing.categorical.ShuffleCategories` relabels the classes. +At inference, this gives each estimator a different view of the same task (see Ensembling in [In-Context Learning](icl)). During training, each step draws a new view, so the model cannot rely on a column's position or a class's label; this acts as data augmentation, like random crops and flips for images. -For regression with {py:class}`~sdm.models.KumoTabular`, also remove sign flipping ({py:class}`~sdm.processing.numerical.FlipSign`) from the target pipeline. +For regression with {py:class}`~sdm.models.KumoTabular`, remove the random sign flip ({py:class}`~sdm.processing.numerical.FlipSign`) from the target pipeline. A flipped target reverses the predicted quantiles, which the default output pipeline re-sorts via {py:class}`~sdm.processing.output.SortQuantiles`, but the identity output does not: ```python @@ -93,9 +95,7 @@ train_recipe.target = sp.StypeDispatch( # Default target pipeline without FlipS ) ``` -## The Training Loop - -Each step runs a single estimator in train mode with the training recipe: +Each step runs a single estimator on a newly sampled context and query, and a fixed number of steps forms an *epoch*: ```python import torch.nn.functional as F @@ -129,10 +129,10 @@ for step in range(steps_per_epoch): optimizer.step() ``` -With the identity output, outputs are stacked along a leading estimator dimension, so `[0]` selects the single estimator. - +The identity output keeps a leading estimator dimension, so `[0]` selects the single estimator. For classification, the output has one logit column per class in the sampled context, which {py:func}`~sdm.evaluation.to_class_indices` aligns with the query targets. -`missing_score=-torch.inf` gives classes absent from the context zero probability, so a query row whose own class is absent gets infinite loss: make sure that every class appears in the context, or compute the loss on probabilities with a finite `missing_score`. +`missing_score=-torch.inf` gives classes absent from the context zero probability, so a query row of an absent class gets infinite loss. +Make sure that every class appears in the context, or compute the loss on probabilities with a finite `missing_score`, as the benchmark harness does. For regression, both models predict 999 quantiles (see [In-Context Learning](icl)), and the pinball (quantile) loss trains each output column toward its quantile level: @@ -142,10 +142,14 @@ level = torch.linspace(0.001, 0.999, 999, device="cuda") loss = torch.maximum(level * diff, (level - 1) * diff).mean() ``` -## Model Selection +## Choosing Hyperparameters + +Validation scores decide both which weights to keep and which settings to train with. -Fine-tuning can overfit, especially on small datasets, so we score the model on the validation rows after every *epoch* (here a fixed number of training steps) and keep the best weights. -Evaluation matches inference: eval mode, the default recipe, and [estimator ensembling](icl). +### Keeping the Best Weights + +Fine-tuning can overfit, especially on small datasets, so the example scores the model on the validation rows after every epoch and keeps the best weights, which picks the number of epochs automatically. +Evaluation matches inference: eval mode, the default recipe, and estimator ensembling. For classification, maximize accuracy; for regression, average the predicted quantiles into a point prediction and minimize root mean squared error (RMSE): ```python @@ -190,12 +194,11 @@ test_metric = evaluate(context=train_table, query=test_table) ``` Since the first evaluation scores the pretrained model, selection never returns weights that score below it on the validation rows. -For large test sets, use {py:meth}`~sdm.models.ICLModel.fit` and batched {py:meth}`~sdm.models.ICLModel.predict` calls instead (see [In-Context Learning](icl)). -## Benchmark Results +### Settings for Larger Datasets -The benchmark harness in [`benchmark/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/benchmark/tabular/finetune.py) runs this procedure on [TALENT](https://github.com/LAMDA-Tabular/TALENT), [TabArena](https://github.com/autogluon/tabarena), and [ScoringBench](https://scoringbench.com). -Unlike the example, it keeps the default recipe during training, so the loss is computed on class probabilities and sorted quantiles, and its defaults target larger datasets: +The example's settings suit its dataset of about 1,000 training rows. +The benchmark harness in [`benchmark/tabular/finetune.py`](https://github.com/NVIDIA/structured-data-models/blob/main/benchmark/tabular/finetune.py) keeps the default recipe during training, and its defaults target larger datasets: | Setting | Example | Harness | | ---------------- | --------------- | ------------------------- | @@ -205,35 +208,9 @@ Unlike the example, it keeps the default recipe during training, so the loss is | Epochs × steps | 10 × 30 | 75 × 10 | | Validation rows | 20% of all rows | 20% of the training split | -Wins, losses, and ties count the datasets on which fine-tuning improved, worsened, or matched the pretrained model's score on each benchmark's own metric: - -| Benchmark | Model | Tasks | Wins | Losses | Ties | -| ------------ | ------------------------------------------- | ------------------------- | ---- | ------ | ---- | -| TALENT | {py:class}`~sdm.models.TabICLv2` | Classification | 73 | 51 | 31 | -| TALENT | {py:class}`~sdm.models.KumoTabular` (small) | Classification | 51 | 39 | 38 | -| ScoringBench | {py:class}`~sdm.models.TabICLv2` | Probabilistic regression | 50 | 26 | 10 | -| TabArena | {py:class}`~sdm.models.KumoTabular` (large) | Classification/regression | 35 | 12 | 0 | - -On TALENT, fine-tuning {py:class}`~sdm.models.TabICLv2` wins twice as often as it loses on multiclass datasets (36 *vs.* 18), but about as often on binary ones (37 *vs.* 33). -{py:class}`~sdm.models.KumoTabular` covers fewer TALENT datasets because its fine-tuning variant supports at most 10 classes. - -To reproduce these results, add `--finetune` to the TALENT and ScoringBench runners (with `--model kumo-tabular-small-ft` for {py:class}`~sdm.models.KumoTabular` on TALENT), or pick a fine-tuned `-ft` model on TabArena: - -```bash -python -m benchmark.tabular.talent.main \ - --model tabiclv2 \ - --dataset-path /path/to/talent/data \ - --finetune - -python -m benchmark.tabular.tabarena.main --model kumo-tabular-large-ft -``` - -See the [tabular benchmark instructions](https://github.com/NVIDIA/structured-data-models/tree/main/benchmark/tabular) for setup and all fine-tuning flags. - -## Choosing Hyperparameters - -We fine-tuned {py:class}`~sdm.models.TabICLv2` with the benchmark harness on two of the larger TALENT datasets, Airline (`customer_satisfaction_in_airline`; binary classification, 83,123 training rows) and Diamonds (regression, 34,521 training rows), changing one setting at a time from the harness defaults. -The figure shows test accuracy and RMSE with the full training split as context, averaged over 3 seeds; error bars show the standard deviation: +Starting from the harness defaults, we fine-tuned {py:class}`~sdm.models.TabICLv2` on two of the larger TALENT datasets and changed one setting at a time. +Airline (`customer_satisfaction_in_airline`) is a binary classification task with 83,123 training rows, and Diamonds is a regression task with 34,521 training rows. +The figure shows test accuracy and RMSE with the full training split as the inference context, averaged over 3 seeds; error bars show the standard deviation, and *zero-shot* marks the pretrained model: ```{figure} images/finetuning_sweep_light.svg :figclass: light-only @@ -245,19 +222,19 @@ The figure shows test accuracy and RMSE with the full training split as context, :width: 100% ``` -- **Context size mattered most.** *C* = 500 or 2,000 did not improve on the pretrained model, while *C* = 16,000, the largest size we tried, did best on both datasets, likely because larger training contexts are closer to inference, where the model reads the whole training split. These sizes are specific to datasets of this scale; the example uses *C* = 256 because its dataset has only about 1,000 training rows. -- **Learning rates between `1e-6` and `1e-5` worked best** within the range we tried (`1e-7` to `1e-5`); smaller rates helped less or not at all. -- **Query size mattered least.** Airline accuracy rose steadily from *Q* = 128 to *Q* = 8,000, while on Diamonds, *Q* = 2,000 scored best. +The gains were small, since the pretrained model is already strong on both datasets. +Picked by validation score, the best settings improved test accuracy on Airline from 96.42% to 96.54% (learning rate `1e-5`) and test RMSE on Diamonds from 495.1 to 489.3 (*C* = 16,000). -The gains were small, since the pretrained model is already strong on both datasets: selected on validation, the best settings improved test accuracy on Airline from 96.42% to 96.54% (learning rate `1e-5`) and test RMSE on Diamonds from 495.1 to 489.3 (*C* = 16,000). +- **Context size mattered most on Diamonds**, while on Airline, the three settings mattered about equally. *C* = 500 or 2,000 did not improve on the pretrained model, while *C* = 16,000, the largest size we tried, did best on both datasets, likely because larger training contexts are closer to inference, where the model reads the whole training split. +- **Learning rates between `1e-6` and `1e-5` worked best** within the range we tried (`1e-7` to `1e-5`); smaller rates helped less or not at all. +- **Query size had no consistent best value.** Airline accuracy rose steadily from *Q* = 128 to *Q* = 8,000, while on Diamonds, *Q* = 2,000 scored best. For datasets of this size, start from the harness defaults, make *C* as large as memory allows, and tune the learning rate on a validation split. -With *Q* = 2,000 on Airline, fine-tuning peaked at 13 GB of GPU memory for *C* = 8,000 and 23 GB for *C* = 16,000, and a run took 46 to 60 minutes on an H100 GPU. -The TALENT runner exposes the three settings as `--finetune-lr`, `--finetune-train-size` (*C* + *Q*), and `--finetune-context-frac` (*C* / (*C* + *Q*)). +With *Q* = 2,000 on Airline, fine-tuning peaked at 13 GB of GPU memory for *C* = 8,000 and 23 GB for *C* = 16,000, and each run took 46 to 60 minutes on an H100 GPU. ## Smaller Contexts After Fine-Tuning -A pretrained TFM knows your dataset only through its context, so more context rows typically improve accuracy but also raise inference cost (see [In-Context Learning](icl)). +A pretrained TFM knows your dataset only through its context, so more context rows typically improve accuracy but also raise inference cost. Fine-tuning moves part of that knowledge into the weights, so a fine-tuned model can reach comparable accuracy with a smaller context, *e.g.*, a random subset of the training rows: ```python @@ -274,7 +251,7 @@ model.fit( out = model.predict(test_table.drop_columns("target")) ``` -We fitted the best fine-tuned model per dataset from the previous section and the pretrained model on the same random subsets of training rows, averaging test scores over 3 seeds: +We compared the best fine-tuned model per dataset from the previous section with the pretrained model, giving both the same random subsets of training rows as context and averaging test scores over 3 seeds: ```{figure} images/finetuning_context_light.svg :figclass: light-only @@ -296,6 +273,35 @@ We fitted the best fine-tuned model per dataset from the previous section and th - **The fine-tuned model scored better at every context size**, but at the smallest contexts (256 rows on both datasets and 1,024 rows on Diamonds), the difference was within the variation across seeds. - **With 16,384 rows, the fine-tuned model came close to the pretrained model with the full training split**, using only 20% (Airline) and 47% (Diamonds) of the training split. -- **More context still helped.** Both models improved with every increase in context size, and the fine-tuned model scored best with the full training split. +- **More context still helped.** Both models improved with every increase in context size. The full-context results differ slightly from the previous section because estimator ensembles are randomized at inference. + +## Benchmark Results + +Beyond these two datasets, the benchmark harness runs this procedure on [TALENT](https://github.com/LAMDA-Tabular/TALENT), [TabArena](https://github.com/autogluon/tabarena), and [ScoringBench](https://scoringbench.com). +Wins, losses, and ties count the datasets on which fine-tuning improved, worsened, or matched the pretrained model's score on each benchmark's own metric: + +| Benchmark | Model | Tasks | Wins | Losses | Ties | +| ------------ | ------------------------------------------- | ------------------------- | ---- | ------ | ---- | +| TALENT | {py:class}`~sdm.models.TabICLv2` | Classification | 73 | 51 | 31 | +| TALENT | {py:class}`~sdm.models.KumoTabular` (small) | Classification | 51 | 39 | 38 | +| ScoringBench | {py:class}`~sdm.models.TabICLv2` | Probabilistic regression | 50 | 26 | 10 | +| TabArena | {py:class}`~sdm.models.KumoTabular` (large) | Classification/regression | 35 | 12 | 0 | + +On TALENT, fine-tuning {py:class}`~sdm.models.TabICLv2` won twice as often as it lost on multiclass datasets (36 *vs.* 18), but only slightly more often on binary ones (37 *vs.* 33). +{py:class}`~sdm.models.KumoTabular` covers fewer TALENT datasets because its fine-tuning variant supports at most 10 classes. + +To reproduce these results, add `--finetune` to the TALENT and ScoringBench runners (with `--model kumo-tabular-small-ft` for {py:class}`~sdm.models.KumoTabular` on TALENT), or pick a fine-tuned `-ft` model on TabArena: + +```bash +python -m benchmark.tabular.talent.main \ + --model tabiclv2 \ + --dataset-path /path/to/talent/data \ + --finetune + +python -m benchmark.tabular.tabarena.main --model kumo-tabular-large-ft +``` + +The TALENT and ScoringBench runners expose the learning rate, *C*, and *Q* as `--finetune-lr`, `--finetune-train-size` (*C* + *Q*), and `--finetune-context-frac` (*C* / (*C* + *Q*)). +See the [tabular benchmark instructions](https://github.com/NVIDIA/structured-data-models/tree/main/benchmark/tabular) for setup and all fine-tuning flags.