From 6ae33f17027b0166c759a089f93a4d77b46f8c45 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 10:17:12 +0000 Subject: [PATCH 01/13] update --- README.md | 4 +- THIRD_PARTY_LICENSES.md | 9 + benchmark/timeseries/chronos2/README.md | 133 +++++++++++ benchmark/timeseries/chronos2/norm.py | 133 +++++++++++ benchmark/timeseries/chronos2/parity.py | 283 ++++++++++++++++++++++++ examples/timeseries/forecast.py | 2 +- pyproject.toml | 1 + sdm/models/__init__.py | 3 + sdm/models/chronos2/LICENSE | 179 +++++++++++++++ test/models/chronos2/test_model.py | 2 +- 10 files changed, 746 insertions(+), 3 deletions(-) create mode 100644 benchmark/timeseries/chronos2/README.md create mode 100644 benchmark/timeseries/chronos2/norm.py create mode 100644 benchmark/timeseries/chronos2/parity.py create mode 100644 sdm/models/chronos2/LICENSE diff --git a/README.md b/README.md index 9cb2a48b89..11734fe229 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ **A GPU-native library of foundation models, tensor subclasses, and data processors for structured data.** -- **Models:** Reference implementations of structured data foundation models, including tabular models ([`TabICLv2`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.TabICLv2), [`KumoTabular`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.KumoTabular), *etc*), relational models ([`KumoRelational`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.KumoRelational)), and time series models ([`KumoForecast`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.KumoForecast), [`TimesFM3`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.TimesFM3), [`KumoAnomaly`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.KumoAnomaly), *etc*), built on a unified interface with room for future model families. +- **Models:** Reference implementations of structured data foundation models, including tabular models ([`TabICLv2`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.TabICLv2), [`KumoTabular`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.KumoTabular), *etc*), relational models ([`KumoRelational`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.KumoRelational)), and time series models ([`Chronos2`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.Chronos2), [`KumoForecast`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.KumoForecast), [`TimesFM3`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.TimesFM3), [`KumoAnomaly`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.KumoAnomaly), *etc*), built on a unified interface with room for future model families. - **Tensor semantics:** PyTorch-compatible tensor types for numerical, categorical, datetime, text, and relational data. - **Data processing:** Composable, extensible, and GPU-accelerated preprocessing and postprocessing for structured data workflows. @@ -53,6 +53,7 @@ pip install git+https://github.com/NVIDIA/structured-data-models.git **Times Series Forecasting Foundation Models:** +- **[`Chronos2`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.Chronos2)** from Ansari *et al.*: [Chronos-2: From Univariate to Universal Forecasting](https://arxiv.org/abs/2510.15821) (CoRR '25) - **[`KumoForecast`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.KumoForecast)** from Gautam *et al.*: [Memory-Augmented Forecasting: Scalability and Generalization Across Temporal Domains](https://ieeexplore.ieee.org/abstract/document/11402442) (IEEE BigData '25) - **[`TimesFM3`](https://nvidia.github.io/structured-data-models/api/generated/sdm.models.TimesFM3)** from Jain *et al.*: [TimesFM-3: A Zero-Shot Foundation Model for Multivariate Forecasting](https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting) ('26) @@ -116,6 +117,7 @@ Third-party software and separately distributed model assets are documented in [ - [**`sdm/models/tabiclv2/`**](sdm/models/tabiclv2/) contains code derived from [`TabICLv2`](https://github.com/soda-inria/tabicl) under the BSD 3-Clause License; its terms are distributed in [`sdm/models/tabiclv2/LICENSE`](sdm/models/tabiclv2/LICENSE). - [**`sdm/models/tabfm/`**](sdm/models/tabfm/) contains code derived from [`TabFM`](https://github.com/google-research/tabfm) under the Apache License 2.0; its terms are distributed in [`sdm/models/tabfm/LICENSE`](sdm/models/tabfm/LICENSE). +- [**`sdm/models/chronos2/`**](sdm/models/chronos2/) contains code derived from [`Chronos2`](https://github.com/amazon-science/chronos-forecasting) under the Apache License 2.0; its terms are distributed in [`sdm/models/chronos2/LICENSE`](sdm/models/chronos2/LICENSE). - [**`sdm/models/timesfm3/`**](sdm/models/timesfm3/) contains code derived from [`TimesFM`](https://github.com/google-research/timesfm) under the Apache License 2.0; its terms are distributed in [`sdm/models/timesfm3/LICENSE`](sdm/models/timesfm3/LICENSE). - [**`sdm/models/timestral/`**](sdm/models/timestral/) contains code derived from [`Timestral`](https://github.com/mistralai/timestral) under the Apache License 2.0; its terms are distributed in [`sdm/models/timestral/LICENSE`](sdm/models/timestral/LICENSE). - [**`sdm/models/kumo/relational/NOTICE`**](sdm/models/kumo/relational/NOTICE) documents [`KumoRelational`](sdm/models/kumo/relational/)'s reuse of [`TabICLv2`](sdm/models/tabiclv2/)-derived components. diff --git a/THIRD_PARTY_LICENSES.md b/THIRD_PARTY_LICENSES.md index 7dac166b03..e0be653a50 100644 --- a/THIRD_PARTY_LICENSES.md +++ b/THIRD_PARTY_LICENSES.md @@ -32,6 +32,15 @@ The [`sdm/models/tabiclv2/`](sdm/models/tabiclv2/) and [`sdm/models/kumo/relatio The [`sdm/models/tabfm/`](sdm/models/tabfm/) implementation contains code derived from [`TabFM`](https://github.com/google-research/tabfm). Its weights are not bundled with this project and may be downloaded only after the user accepts their separate license. +## Chronos-2 + +- Source: https://github.com/amazon-science/chronos-forecasting +- License: Apache License 2.0 +- License terms: [`sdm/models/chronos2/LICENSE`](sdm/models/chronos2/LICENSE) +- Optional pretrained weights: https://huggingface.co/amazon/chronos-2 + +The [`sdm/models/chronos2/`](sdm/models/chronos2/) implementation contains code derived from [`Chronos2`](https://github.com/amazon-science/chronos-forecasting). + ## TimesFM 3.0 - Source: https://github.com/google-research/timesfm diff --git a/benchmark/timeseries/chronos2/README.md b/benchmark/timeseries/chronos2/README.md new file mode 100644 index 0000000000..6768d54493 --- /dev/null +++ b/benchmark/timeseries/chronos2/README.md @@ -0,0 +1,133 @@ +# Chronos-2 parity and runtime + +This port targets the released `amazon/chronos-2` checkpoint. It supports numerical univariate and multivariate targets, past-only and known-future numerical covariates, NaN observations, context truncation, and forecasts longer than 1,024 steps. It deliberately excludes categorical preprocessing, fine-tuning, and upstream's optional cross-learning between independent tasks. No upstream runtime dependency is needed by SDM. + +- Upstream source: `amazon-science/chronos-forecasting@1224fafa91e1ecd6674f5204066b159e40b5e4a1`. +- Checkpoint: `amazon/chronos-2@29ec3766d36d6f73f0696f85560a422f50e8498c`. +- Source and weights: Apache-2.0; source attribution is in `sdm/models/chronos2/NOTICE`. + +## Use + +```python +import torch + +from sdm.models.chronos2 import Chronos2 + +model = Chronos2(device="cuda") +t = torch.arange(256, device="cuda").float() +y = (t / 10).sin().unsqueeze(-1) +out = model( + x_context=y.new_empty(256, 0), + y_context=y, + x_query=y.new_empty(48, 0), +) +# [48, 21], ordered from q1 to q99; float32 for these float32 inputs. +``` + +The SDM recipe uses `Standardize()` followed by the invertible `Asinh()` independently for features and targets, then averages estimator outputs. Context is capped before recipe fitting in both `forward` and `fit`; future query rows are preserved. The recipe performs initial standardization and the asinh transform; target inversion applies sinh followed by inverse standardization. The model recomputes normalization only after extending the context during long-horizon rollout. Rollout statistics and quantile mixtures are computed on the linear scale, then outputs are returned in asinh space for recipe inversion. `Standardize.inverse_transform` broadcasts fitted target statistics across consecutive quantile outputs. For batched benchmark inputs, an explicit leading estimator axis separates independent batch entries from estimators. Both `Chronos2` and `_Chronos2` live in `model.py`; the package is intentionally not exported from `sdm.models`. + +## Reproduce + +Install the optional reference library into an existing GPU-compatible SDM environment: + +```bash +uv pip install --python .venv/bin/python \ + "chronos-forecasting @ git+https://github.com/amazon-science/chronos-forecasting.git@1224fafa91e1ecd6674f5204066b159e40b5e4a1" \ + "transformers==5.14.1" +.venv/bin/python -m benchmark.timeseries.chronos2.parity \ + --output /tmp/chronos2-results.json --warmup 5 --repeats 15 +``` + +The standalone benchmark creates deterministic sinusoidal inputs without seeds. It compares upstream `Chronos2Pipeline.predict` with SDM's public `forward`, including checkpoint loading and output column order. It separately compares CUDA-resident numerical execution. Public upstream prediction includes CPU dictionary preparation and copying results to CPU; SDM accepts tensors already on CUDA and returns CUDA tensors. Core timings remove that boundary difference and include standardization, asinh, and their inverses for the SDM numerical call. Core timings cover one chunk even when public forecasts require multiple chunks. + +All calls run without gradients, using float32 or float16 parameters with float32 scaling and forecast restoration. Models are constructed outside `inference_mode`; only prediction runs inside it. Constructing parameters inside inference mode selected slower execution paths in this environment and is not representative of ordinary model initialization. Float32 matmul precision is `highest` (TF32 disabled). Five warmups precede fifteen alternating paired repetitions, with CUDA events and explicit synchronization around wall timing. JSON output records median latency and wall p10/p90. No compilation or CUDA graphs are used. + +## Measurements + +The full matrix below was measured after moving standardization into the recipe, before extracting Asinh. The current Asinh recipe follow-up is reported separately below. + +Measured on 2026-10-29: NVIDIA GB10, driver 580.178.04, PyTorch 2.14.0+cu130, Python 3.14.7, Transformers 5.14.1, Chronos reference 2.3.3.dev0 at the source revision above. These are latency measurements on synthetic inputs, not forecasting quality benchmark scores. + +These results include the recipe standardization. Its extra fitting and inversion add overhead to small public calls, so the initial port's general speedup does not apply uniformly. + +Every timing is **upstream → SDM**, in milliseconds, using median synchronized wall time. + +| Case | Float32 public | Float32 core | Float16 public | Float16 core | +| ----------------- | -------------: | ------------: | -------------: | ------------: | +| univariate | 7.12 → 7.20 | 6.60 → 6.13 | 7.45 → 7.58 | 7.16 → 6.73 | +| batch univariate | 22.26 → 19.61 | 21.93 → 19.07 | 10.03 → 9.57 | 9.52 → 9.00 | +| multivariate | 12.26 → 11.54 | 11.96 → 11.01 | 7.85 → 7.68 | 7.47 → 6.77 | +| covariates | 12.48 → 12.54 | 11.89 → 11.02 | 8.08 → 8.58 | 7.55 → 6.82 | +| missing | 9.60 → 10.32 | 9.02 → 8.78 | 8.00 → 8.46 | 7.48 → 6.77 | +| all missing | 7.29 → 7.08 | 6.79 → 6.10 | 7.64 → 7.49 | 7.27 → 6.65 | +| constant | 8.44 → 8.30 | 8.03 → 7.76 | 7.75 → 7.56 | 7.43 → 6.76 | +| one step | 7.07 → 7.09 | 6.61 → 6.15 | 7.37 → 7.53 | 7.10 → 6.66 | +| truncated context | 33.14 → 32.07 | 32.17 → 30.88 | 14.38 → 14.80 | 13.96 → 14.05 | +| long horizon | 70.35 → 68.91 | 11.92 → 11.09 | 33.33 → 32.85 | 7.48 → 6.81 | +| three chunks | 81.38 → 78.47 | 8.69 → 8.45 | 43.59 → 39.73 | 7.79 → 7.11 | + +The cases include a sixteen-item univariate batch; two independent three-target groups; mixed target/past/future covariates; irregular patch lengths; internal and leading missing observations; an entirely missing target; entirely missing histories; constant and one-point histories; truncation of 8,301 context steps; and horizons of 1,041 and 2,051 steps. The case shapes are recorded in `parity.py` and the result JSON. + +## Asinh recipe follow-up + +Moving asinh and its inverse into a processor was checked on twelve comparisons: six shapes in float32 and float16, with five warmups and fifteen alternating paired repetitions. Short-horizon prediction differences were unchanged; long-horizon differences changed only by small rounding effects. The accepted constant-history limitation remains. The command above can reproduce this subset with `--cases univariate covariates constant one_step long_horizon three_chunks`. + +Every value is upstream → SDM public synchronized wall latency in milliseconds. + +| Case | Float32 | Float16 | +| ------------ | ------------: | ------------: | +| univariate | 6.60 → 6.83 | 8.17 → 8.29 | +| covariates | 12.62 → 13.08 | 8.95 → 9.85 | +| constant | 8.49 → 8.48 | 8.57 → 8.41 | +| one step | 7.14 → 7.38 | 8.15 → 8.30 | +| long horizon | 70.71 → 69.10 | 34.01 → 34.56 | +| three chunks | 81.09 → 77.69 | 43.73 → 40.90 | + +## Plain SDM RMSNorm comparison + +Plain `sdm.nn.RMSNorm` is the chosen implementation to keep normalization shared across models. Relative RMSE below `1e-3` is accepted; float32 and ordinary float16 cases satisfy this threshold. The missing-history half-precision cases above that threshold are retained as documented exceptions, without a model-specific normalization fallback. + +On 2026-10-30, the local T5-style LayerNorm was replaced with `sdm.nn.RMSNorm(eps=1e-6)` throughout the attention blocks, feed-forward layers, and final normalization. The comparison holds weights and the current Standardize/Asinh recipe fixed, isolating the normalization change from the already accepted constant-scale differences. The baseline substitutes the pinned upstream `Chronos2LayerNorm`, whose arithmetic matches the removed custom class, and preserves shared query/key normalization objects. + +```bash +.venv/bin/python -m benchmark.timeseries.chronos2.norm \ + --output /tmp/chronos2-norm-results.json --warmup 3 --repeats 15 +``` + +This covers eleven cases in each of three modes: float32 parameters, float16 parameters, and float32 parameters with CUDA float16 autocast. The last mode exercises SDM's optimized RMSNorm kernel. Models are constructed outside inference mode. All 33 public-API comparisons produced finite outputs. Wall and CUDA event timing use warmups, alternating paired repetitions, and explicit synchronization on the same GB10. + +Differences below compare **plain RMSNorm against the previous custom normalization**, with identical weights and preprocessing. Speedup is the median percentage latency reduction across the eleven cases, not a forecasting accuracy metric. + +| Mode | Largest relative RMSE | Case with largest difference | Median latency reduction | +| ---------------- | --------------------: | ---------------------------- | -----------------------: | +| Float32 | 1.71e-06 | missing | 5.1% | +| Float16 weights | 1.454% | all missing | 13.3% | +| Float16 autocast | 0.466% | missing | 12.4% | + +For ordinary univariate, batched, multivariate, covariate, and long-context inputs, float16-weight relative RMSE is 0.033–0.087%, and autocast relative RMSE is 0.026–0.053%. Entirely missing histories with float16 weights have the largest change: 1.45% relative RMSE and maximum absolute difference 694,210 because forecasts are already in the millions. Strict half-precision parity is not claimed. Float32 changes remain at floating-point rounding scale; the maximum relative RMSE is 1.71e-6. + +The JSON also records old-versus-upstream and new-versus-upstream errors separately. The previously accepted Standardize constant-history and one-observation discrepancies remain; they must not be attributed to this norm swap. + +## MLP ownership comparison + +`Chronos2TransformerBlock` now accepts `mlp: torch.nn.Module | None`. Time blocks receive `None`; group-attention blocks own the normalized feed-forward network, as in TimesFM-3. The separate MLP list is removed, and checkpoint remapping targets `icl_block.group_blocks..mlp`. + +The 2026-10-30 comparison held SDM RMSNorm, weights, and preprocessing fixed and restored the saved pre-change ICL forward for the baseline. It covered eleven shapes in each of float32, float16-weight, and float16-autocast modes, with three warmups and ten alternating paired repetitions. Float32 results were bit-for-bit identical in all eleven cases. Ordinary half-precision cases, including long context and multi-chunk horizons, were also bit-for-bit identical. + +Differences appeared only in missing-history and constant-history cases. Stored float16 weights had maximum aggregate relative RMSE 3.94e-5 (constant case, maximum absolute difference 0.0346). Float16 autocast reached 0.00382 relative RMSE on the mixed missing-history case. Constant-case aggregate errors are dominated by the 100-valued target: individual near-zero quantile outputs had larger relative errors, up to 15.6% with float16 weights and 8.92% under autocast. These are documented degenerate-input exceptions, not a claim that every half-precision output meets the 1e-3 tolerance. + +Latency was effectively unchanged: median changes across cases were -0.49% for float32, -0.36% for float16 weights, and -0.75% for float16 autocast. All outputs remained finite. The standalone comparison, saved before-change source, and JSON results are in `/tmp/sdm-chronos2-mlp-20261030/` for this run. + +## Numerical agreement + +CPU float32 checks cover univariate, covariates, constant, and missing data. Cached `fit`/`predict` uses the same capped context and fitted recipe as direct prediction; empty horizons preserve batch and output column dimensions. + +Public output shape and target/quantile column ordering matched in the original 22 comparisons and the twelve Asinh follow-up comparisons. The full-matrix precision measurements below refer to the Standardize-only recipe; the Asinh follow-up retained those precision characteristics. For nonconstant histories and entirely missing histories, float32 relative RMSE (`RMSE(reference - SDM) / RMS(reference)`) is at most **1.83e-6**. Ordinary multivariate and covariate cases have maximum absolute errors below **1.5e-6**. The 2,051-step rollout reaches **2.06e-4** maximum absolute error, with relative RMSE **8.07e-7**. + +This is numerical agreement, not bitwise identity. Entirely missing histories can generate forecasts in the millions upstream: float32 differences reach **60** in those cases despite relative RMSE below **1.83e-6**. Fused QKV projections and different tensor layouts change floating-point rounding; inverse `sinh` amplifies it. These cases must not be judged using absolute error alone. + +Float16 is less exact: ordinary batched/multivariate/covariate cases have relative RMSE **0.050–0.058%** and maximum absolute error below **0.0048**. Relative RMSE reaches **0.44%** across three rollout chunks and **0.83%** in the mixed missing-target case. The latter has **244,213** maximum absolute difference because of extremely large upstream forecasts. All measured outputs remain finite, but strict float16 parity is not claimed for these unstable inputs; use float32 when matching the reference is important. + +**Accepted parity limitation:** the recipe deliberately uses unchanged `Standardize()` fitting semantics. For constant histories it uses scale 1 rather than upstream's 1e-5. Removing the previous duplicate normalization exposes this difference. The constant-series case has relative RMSE **1.32%** (float32) or **1.33%** (float16), with maximum absolute difference **5.98** or **6.01**. With only one context observation, output scale differs by roughly **100,000×**; the maximum absolute difference is **11.7 million**. These cases do not have upstream parity. The choice to retain Standardize's default scale was made explicitly after measuring these differences. They are prediction disagreements on synthetic inputs, not measured changes in forecasting quality on real datasets. + +The core uses float32 NaN reductions instead of `RevIN.stats`' float64 reductions. The recipe retains SDM Standardize's finite-value and constant-column rules. Affine normalization reuses `RevIN`. The implementation now uses plain SDM RMSNorm; its measured precision tradeoffs relative to the previous T5-style normalization are recorded above. Attention uses scale 1.0 and upstream's finite additive masks, including its fully masked-row behavior. Long forecasts use nine quantile paths and the weighted empirical CDF rather than median-only feedback. diff --git a/benchmark/timeseries/chronos2/norm.py b/benchmark/timeseries/chronos2/norm.py new file mode 100644 index 0000000000..dbe5474d35 --- /dev/null +++ b/benchmark/timeseries/chronos2/norm.py @@ -0,0 +1,133 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Isolate the prediction and latency effects of using SDM RMSNorm.""" + +import argparse +import copy +import json +from contextlib import nullcontext +from pathlib import Path +from typing import Any + +import torch +from chronos import Chronos2Pipeline +from chronos.chronos2.layers import Chronos2LayerNorm + +from benchmark.timeseries.chronos2.parity import ( + CASES, + REVISION, + difference, + inputs, + measure, +) +from sdm.models.chronos2 import Chronos2 +from sdm.models.chronos2.ckpt import remap_ckpt +from sdm.nn import RMSNorm + + +def original_norm(model: Chronos2) -> Chronos2: + model = copy.deepcopy(model) + replacements: dict[int, Chronos2LayerNorm] = {} + # Preserve the shared query/key normalization objects. The pinned upstream + # class implements the same arithmetic as the removed custom LayerNorm. + for name, module in list(model.named_modules(remove_duplicate=False)): + if isinstance(module, RMSNorm): + if id(module) not in replacements: + norm = Chronos2LayerNorm(module.weight.numel(), eps=1e-6) + norm.weight = module.weight + replacements[id(module)] = norm + model.set_submodule(name, replacements[id(module)]) + return model.eval() + + +@torch.inference_mode() +def compare( + baseline: Chronos2, + plain: Chronos2, + upstream: Chronos2Pipeline, + case: str, + mode: str, + args: argparse.Namespace, +) -> dict[str, Any]: + dictionaries, tables, *_ = inputs(case, "cuda") + horizon = CASES[case][2] + + def autocast() -> Any: + if mode == "autocast_float16": + return torch.autocast("cuda", dtype=torch.float16) + return nullcontext() + + def previous() -> torch.Tensor: + with autocast(): + return baseline(*tables).numerical + + def candidate() -> torch.Tensor: + with autocast(): + return plain(*tables).numerical + + with autocast(): + reference = ( + torch.stack( + upstream.predict( + dictionaries, prediction_length=horizon, batch_size=10000 + ) + ) + .permute(0, 3, 1, 2) + .flatten(-2) + ) + a, b = previous(), candidate() + latency = measure((previous, candidate), args.warmup, args.repeats) + return { + "mode": mode, + "case": case, + "shape": CASES[case], + "plain_vs_previous": difference(a, b), + "previous_vs_upstream": difference(reference, a), + "plain_vs_upstream": difference(reference, b), + "previous_latency": latency[0], + "plain_latency": latency[1], + } + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--cases", nargs="+", default=list(CASES)) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--repeats", type=int, default=15) + args = parser.parse_args() + torch.set_float32_matmul_precision("highest") + result: dict[str, Any] = { + "torch": torch.__version__, + "gpu": torch.cuda.get_device_name(), + "checkpoint": REVISION, + "source": "1224fafa91e1ecd6674f5204066b159e40b5e4a1", + "warmup": args.warmup, + "repeats": args.repeats, + "cases": [], + } + for mode in ("float32", "float16", "autocast_float16"): + dtype = torch.float16 if mode == "float16" else torch.float32 + upstream = Chronos2Pipeline.from_pretrained( + "amazon/chronos-2", + revision=REVISION, + device_map="cuda", + dtype=dtype, + ) + plain = Chronos2(pretrained=False, device="meta") + plain.model.load_state_dict( + remap_ckpt(upstream.model.state_dict()), strict=True, assign=True + ) + baseline = original_norm(plain) + for case in args.cases: + row = compare(baseline, plain, upstream, case, mode, args) + print(json.dumps(row), flush=True) + result["cases"].append(row) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(result, indent=2) + "\n") + del upstream, plain, baseline + + +if __name__ == "__main__": + main() diff --git a/benchmark/timeseries/chronos2/parity.py b/benchmark/timeseries/chronos2/parity.py new file mode 100644 index 0000000000..003295c315 --- /dev/null +++ b/benchmark/timeseries/chronos2/parity.py @@ -0,0 +1,283 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Compare the pinned public Chronos-2 pipeline and SDM on identical inputs.""" + +import argparse +import json +import platform +import statistics +import time +from collections.abc import Callable +from importlib.metadata import version +from pathlib import Path +from typing import Any + +import torch +from chronos import Chronos2Pipeline + +from sdm import Stype, TableTensor +from sdm.models.chronos2 import Chronos2 +from sdm.models.chronos2.ckpt import remap_ckpt +from sdm.processing import Standardize + +REVISION = "29ec3766d36d6f73f0696f85560a422f50e8498c" + +CASES = { + "univariate": (1, 95, 31, 1, 0, 0), + "batch_univariate": (16, 512, 96, 1, 0, 0), + "multivariate": (2, 512, 96, 3, 0, 0), + "covariates": (2, 255, 65, 2, 1, 2), + "missing": (2, 95, 33, 2, 1, 1), + "all_missing": (2, 47, 17, 1, 0, 0), + "constant": (1, 127, 32, 2, 0, 0), + "one_step": (1, 1, 1, 1, 0, 0), + "truncated_context": (1, 8301, 17, 1, 0, 1), + "long_horizon": (1, 95, 1041, 1, 1, 1), + "three_chunks": (1, 47, 2051, 1, 0, 0), +} + + +def inputs(name: str, device: str) -> tuple: + batch, length, horizon, targets, past, known = CASES[name] + total = targets + past + known + index = torch.arange(batch * total * (length + horizon)).float() + values = (index * 0.07).sin() + (index * 0.013).cos() * 0.5 + values = values.reshape(batch, total, length + horizon) + context = values[..., :length].clone() + future = values[..., length:].clone() + future[:, : targets + past] = torch.nan + if name == "missing": + context[..., :21] = torch.nan + context[..., 25::7] = torch.nan + context[0, 0] = torch.nan + future[..., ::5] = torch.nan + elif name == "all_missing": + context[:] = torch.nan + elif name == "constant": + context[:, 0] = 0 + context[:, 1] = 100 + elif name == "truncated_context": + context[..., :30] = 1e6 + dictionaries = [] + for row in range(batch): + dictionaries.append( + { + "target": context[row, :targets].numpy(), + "past_covariates": { + f"past{i}": context[row, targets + i].numpy() + for i in range(past) + } + | { + f"known{i}": context[row, targets + past + i].numpy() + for i in range(known) + }, + "future_covariates": { + f"known{i}": future[row, targets + past + i].numpy() + for i in range(known) + }, + } + ) + features = context[:, targets:].transpose(-1, -2).to(device) + query = future[:, targets + past :].transpose(-1, -2).to(device) + names = [f"past{i}" for i in range(past)] + [ + f"known{i}" for i in range(known) + ] + x = TableTensor.from_tensor(features.unsqueeze(0), columns=names) + y = TableTensor.from_tensor( + context[:, :targets].transpose(-1, -2).to(device).unsqueeze(0), + columns=[f"target{i}" for i in range(targets)], + ) + q = TableTensor.from_tensor(query.unsqueeze(0), columns=names[past:]) + groups = torch.arange(batch, device=device).repeat_interleave(total) + return ( + dictionaries, + (x, y, q), + context.flatten(0, 1).to(device), + future.flatten(0, 1).to(device), + groups, + ) + + +def difference(a: torch.Tensor, b: torch.Tensor) -> dict[str, float]: + a, b = a.float().cpu(), b.float().cpu() + if a.shape != b.shape: + raise RuntimeError(f"Output shapes differ: {a.shape} vs {b.shape}") + delta = (a - b).abs() + scale = a.square().mean().sqrt().clamp_min(1e-6) + return { + "max_abs": delta.max().item(), + "rmse": delta.square().mean().sqrt().item(), + "relative_rmse": (delta.square().mean().sqrt() / scale).item(), + "max_series_relative_rmse": ( + delta.square().mean(dim=-2).sqrt() + / a.square().mean(dim=-2).sqrt().clamp_min(1e-6) + ) + .max() + .item(), + "finite": bool(a.isfinite().all() and b.isfinite().all()), + } + + +def measure( + calls: tuple[Callable, Callable], warmup: int, repeats: int +) -> list[dict[str, float]]: + for _ in range(warmup): + for call in calls: + call() + walls: list[list[float]] = [[], []] + events: list[list[float]] = [[], []] + for repeat in range(repeats): + for index in range(2) if repeat % 2 == 0 else reversed(range(2)): + start, end = torch.cuda.Event(True), torch.cuda.Event(True) + torch.cuda.synchronize() + tick = time.perf_counter() + start.record() + calls[index]() + end.record() + torch.cuda.synchronize() + walls[index].append((time.perf_counter() - tick) * 1000) + events[index].append(start.elapsed_time(end)) + return [ + { + "wall_ms": statistics.median(wall), + "cuda_ms": statistics.median(event), + "wall_p10_ms": sorted(wall)[len(wall) // 10], + "wall_p90_ms": sorted(wall)[ + min(len(wall) - 1, 9 * len(wall) // 10) + ], + } + for wall, event in zip(walls, events) + ] + + +@torch.inference_mode() +def run_case( + upstream: Chronos2Pipeline, + model: Chronos2, + case: str, + args: argparse.Namespace, +) -> dict[str, Any]: + dictionaries, tables, context, future, groups = inputs(case, "cuda") + _, _, horizon, targets, _, _ = CASES[case] + + def reference() -> torch.Tensor: + out = upstream.predict( + dictionaries, + prediction_length=horizon, + batch_size=10000, + ) + return torch.stack(out).permute(0, 3, 1, 2).flatten(-2) + + def candidate() -> torch.Tensor: + return model(*tables).numerical + + a, b = reference(), candidate() + public_diff = difference(a, b) + expected_names = [ + f"target{i}__q{round(q * 100)}" + for i in range(targets) + for q in upstream.quantiles + ] + if list(model(*tables).columns[Stype.numerical]) != expected_names: + raise RuntimeError("Quantile column order differs") + public_time = measure((reference, candidate), args.warmup, args.repeats) + # The public upstream API includes CPU input preparation and a CPU + # result copy. Compare numerical execution separately on CUDA. + patches = min((horizon + 15) // 16, 64) + + def reference_core() -> torch.Tensor: + return upstream._predict_step( + context=context, + group_ids=groups, + future_covariates=future, + num_output_patches=patches, + ) + + def candidate_core() -> torch.Tensor: + processor = Standardize() + retained = context[..., -model.model.context_length :] + table = TableTensor.from_tensor(retained.T) + x = processor.fit_transform(table).numerical.T.asinh() + f = processor.transform( + TableTensor.from_tensor(future.T) + ).numerical.T.asinh() + batch, _, _, targets, past, known = CASES[case] + variates = targets + past + known + out = ( + model.model._predict( + x.unflatten(0, (batch, variates)), + f.unflatten(0, (batch, variates)), + patches, + ) + .flatten(0, 1) + .sinh() + ) + # Put the quantile axis first so fitted [1, S] statistics broadcast. + table = TableTensor.from_tensor(out.permute(1, 2, 0)) + return processor.inverse_transform(table).numerical.permute(2, 0, 1) + + core_diff = difference(reference_core(), candidate_core()) + core_time = measure( + (reference_core, candidate_core), args.warmup, args.repeats + ) + return { + "dtype": str(next(model.parameters()).dtype), + "case": case, + "shape": CASES[case], + "public_difference": public_diff, + "core_difference": core_diff, + "public_upstream": public_time[0], + "public_sdm": public_time[1], + "core_upstream": core_time[0], + "core_sdm": core_time[1], + } + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--dtypes", nargs="+", default=["float32", "float16"]) + parser.add_argument("--cases", nargs="+", default=list(CASES)) + parser.add_argument("--warmup", type=int, default=5) + parser.add_argument("--repeats", type=int, default=15) + args = parser.parse_args() + torch.set_float32_matmul_precision("highest") + results: dict[str, Any] = { + "torch": torch.__version__, + "cuda": torch.version.cuda, + "transformers": version("transformers"), + "chronos_forecasting": version("chronos-forecasting"), + "matmul_precision": torch.get_float32_matmul_precision(), + "recipe": repr(Chronos2.default_recipe()), + "gpu": torch.cuda.get_device_name(), + "python": platform.python_version(), + "checkpoint": REVISION, + "source": "1224fafa91e1ecd6674f5204066b159e40b5e4a1", + "warmup": args.warmup, + "repeats": args.repeats, + "cases": [], + } + for name in args.dtypes: + dtype = getattr(torch, name) + upstream = Chronos2Pipeline.from_pretrained( + "amazon/chronos-2", + revision=REVISION, + device_map="cuda", + dtype=dtype, + ) + model = Chronos2(device="cuda") + if dtype != torch.float32: + state = remap_ckpt(upstream.model.state_dict()) + model.model.load_state_dict(state, strict=True, assign=True) + for case in args.cases: + row = run_case(upstream, model, case, args) + print(json.dumps(row), flush=True) + results["cases"].append(row) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(results, indent=2) + "\n") + del upstream, model + + +if __name__ == "__main__": + main() diff --git a/examples/timeseries/forecast.py b/examples/timeseries/forecast.py index 4a3e7bb60f..42893be8ec 100644 --- a/examples/timeseries/forecast.py +++ b/examples/timeseries/forecast.py @@ -8,7 +8,7 @@ url = "https://autogluon.s3.amazonaws.com/datasets/timeseries" device = torch.device("cuda" if torch.cuda.is_available() else "cpu") -model = sdm.models.TimesFM3(device=device) +model = sdm.models.Chronos2(device=device) print("# Univariate Forecasting") ############################################ # 414 items ("item_id"), each with up to 960 observations: diff --git a/pyproject.toml b/pyproject.toml index ee24ad0381..b41913d789 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -12,6 +12,7 @@ license = "Apache-2.0" license-files = [ "LICENSE", "THIRD_PARTY_LICENSES.md", + "sdm/models/chronos2/LICENSE", "sdm/models/tabiclv2/LICENSE", "sdm/models/tabfm/LICENSE", "sdm/models/timesfm3/LICENSE", diff --git a/sdm/models/__init__.py b/sdm/models/__init__.py index 30e914f36c..3791affdb5 100644 --- a/sdm/models/__init__.py +++ b/sdm/models/__init__.py @@ -7,6 +7,7 @@ "sdm.models.tabiclv2", "sdm.models.tabfm", "sdm.models.kumo", + "sdm.models.chronos2", "sdm.models.timesfm3", } @@ -19,6 +20,7 @@ KumoForecast, KumoAnomaly, ) +from sdm.models.chronos2 import Chronos2 from sdm.models.timesfm3 import TimesFM3 @@ -33,5 +35,6 @@ "KumoRelational", "KumoForecast", "KumoAnomaly", + "Chronos2", "TimesFM3", ] diff --git a/sdm/models/chronos2/LICENSE b/sdm/models/chronos2/LICENSE new file mode 100644 index 0000000000..2ee8ece92b --- /dev/null +++ b/sdm/models/chronos2/LICENSE @@ -0,0 +1,179 @@ +This package includes code derived from the Chronos-2 implementation and may +load pretrained Chronos-2 weights: +https://github.com/amazon-science/chronos-forecasting +https://huggingface.co/amazon/chronos-2 + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. diff --git a/test/models/chronos2/test_model.py b/test/models/chronos2/test_model.py index fed35d69d1..9d05b549ac 100644 --- a/test/models/chronos2/test_model.py +++ b/test/models/chronos2/test_model.py @@ -4,7 +4,7 @@ import torch from sdm import TableTensor -from sdm.models.chronos2 import Chronos2 +from sdm.models import Chronos2 def test_forward() -> None: From 0616dba8a2dcc2cfc746e973d8a4c46cc126fde1 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 10:17:21 +0000 Subject: [PATCH 02/13] update --- benchmark/timeseries/chronos2/README.md | 133 ----------- benchmark/timeseries/chronos2/norm.py | 133 ----------- benchmark/timeseries/chronos2/parity.py | 283 ------------------------ 3 files changed, 549 deletions(-) delete mode 100644 benchmark/timeseries/chronos2/README.md delete mode 100644 benchmark/timeseries/chronos2/norm.py delete mode 100644 benchmark/timeseries/chronos2/parity.py diff --git a/benchmark/timeseries/chronos2/README.md b/benchmark/timeseries/chronos2/README.md deleted file mode 100644 index 6768d54493..0000000000 --- a/benchmark/timeseries/chronos2/README.md +++ /dev/null @@ -1,133 +0,0 @@ -# Chronos-2 parity and runtime - -This port targets the released `amazon/chronos-2` checkpoint. It supports numerical univariate and multivariate targets, past-only and known-future numerical covariates, NaN observations, context truncation, and forecasts longer than 1,024 steps. It deliberately excludes categorical preprocessing, fine-tuning, and upstream's optional cross-learning between independent tasks. No upstream runtime dependency is needed by SDM. - -- Upstream source: `amazon-science/chronos-forecasting@1224fafa91e1ecd6674f5204066b159e40b5e4a1`. -- Checkpoint: `amazon/chronos-2@29ec3766d36d6f73f0696f85560a422f50e8498c`. -- Source and weights: Apache-2.0; source attribution is in `sdm/models/chronos2/NOTICE`. - -## Use - -```python -import torch - -from sdm.models.chronos2 import Chronos2 - -model = Chronos2(device="cuda") -t = torch.arange(256, device="cuda").float() -y = (t / 10).sin().unsqueeze(-1) -out = model( - x_context=y.new_empty(256, 0), - y_context=y, - x_query=y.new_empty(48, 0), -) -# [48, 21], ordered from q1 to q99; float32 for these float32 inputs. -``` - -The SDM recipe uses `Standardize()` followed by the invertible `Asinh()` independently for features and targets, then averages estimator outputs. Context is capped before recipe fitting in both `forward` and `fit`; future query rows are preserved. The recipe performs initial standardization and the asinh transform; target inversion applies sinh followed by inverse standardization. The model recomputes normalization only after extending the context during long-horizon rollout. Rollout statistics and quantile mixtures are computed on the linear scale, then outputs are returned in asinh space for recipe inversion. `Standardize.inverse_transform` broadcasts fitted target statistics across consecutive quantile outputs. For batched benchmark inputs, an explicit leading estimator axis separates independent batch entries from estimators. Both `Chronos2` and `_Chronos2` live in `model.py`; the package is intentionally not exported from `sdm.models`. - -## Reproduce - -Install the optional reference library into an existing GPU-compatible SDM environment: - -```bash -uv pip install --python .venv/bin/python \ - "chronos-forecasting @ git+https://github.com/amazon-science/chronos-forecasting.git@1224fafa91e1ecd6674f5204066b159e40b5e4a1" \ - "transformers==5.14.1" -.venv/bin/python -m benchmark.timeseries.chronos2.parity \ - --output /tmp/chronos2-results.json --warmup 5 --repeats 15 -``` - -The standalone benchmark creates deterministic sinusoidal inputs without seeds. It compares upstream `Chronos2Pipeline.predict` with SDM's public `forward`, including checkpoint loading and output column order. It separately compares CUDA-resident numerical execution. Public upstream prediction includes CPU dictionary preparation and copying results to CPU; SDM accepts tensors already on CUDA and returns CUDA tensors. Core timings remove that boundary difference and include standardization, asinh, and their inverses for the SDM numerical call. Core timings cover one chunk even when public forecasts require multiple chunks. - -All calls run without gradients, using float32 or float16 parameters with float32 scaling and forecast restoration. Models are constructed outside `inference_mode`; only prediction runs inside it. Constructing parameters inside inference mode selected slower execution paths in this environment and is not representative of ordinary model initialization. Float32 matmul precision is `highest` (TF32 disabled). Five warmups precede fifteen alternating paired repetitions, with CUDA events and explicit synchronization around wall timing. JSON output records median latency and wall p10/p90. No compilation or CUDA graphs are used. - -## Measurements - -The full matrix below was measured after moving standardization into the recipe, before extracting Asinh. The current Asinh recipe follow-up is reported separately below. - -Measured on 2026-10-29: NVIDIA GB10, driver 580.178.04, PyTorch 2.14.0+cu130, Python 3.14.7, Transformers 5.14.1, Chronos reference 2.3.3.dev0 at the source revision above. These are latency measurements on synthetic inputs, not forecasting quality benchmark scores. - -These results include the recipe standardization. Its extra fitting and inversion add overhead to small public calls, so the initial port's general speedup does not apply uniformly. - -Every timing is **upstream → SDM**, in milliseconds, using median synchronized wall time. - -| Case | Float32 public | Float32 core | Float16 public | Float16 core | -| ----------------- | -------------: | ------------: | -------------: | ------------: | -| univariate | 7.12 → 7.20 | 6.60 → 6.13 | 7.45 → 7.58 | 7.16 → 6.73 | -| batch univariate | 22.26 → 19.61 | 21.93 → 19.07 | 10.03 → 9.57 | 9.52 → 9.00 | -| multivariate | 12.26 → 11.54 | 11.96 → 11.01 | 7.85 → 7.68 | 7.47 → 6.77 | -| covariates | 12.48 → 12.54 | 11.89 → 11.02 | 8.08 → 8.58 | 7.55 → 6.82 | -| missing | 9.60 → 10.32 | 9.02 → 8.78 | 8.00 → 8.46 | 7.48 → 6.77 | -| all missing | 7.29 → 7.08 | 6.79 → 6.10 | 7.64 → 7.49 | 7.27 → 6.65 | -| constant | 8.44 → 8.30 | 8.03 → 7.76 | 7.75 → 7.56 | 7.43 → 6.76 | -| one step | 7.07 → 7.09 | 6.61 → 6.15 | 7.37 → 7.53 | 7.10 → 6.66 | -| truncated context | 33.14 → 32.07 | 32.17 → 30.88 | 14.38 → 14.80 | 13.96 → 14.05 | -| long horizon | 70.35 → 68.91 | 11.92 → 11.09 | 33.33 → 32.85 | 7.48 → 6.81 | -| three chunks | 81.38 → 78.47 | 8.69 → 8.45 | 43.59 → 39.73 | 7.79 → 7.11 | - -The cases include a sixteen-item univariate batch; two independent three-target groups; mixed target/past/future covariates; irregular patch lengths; internal and leading missing observations; an entirely missing target; entirely missing histories; constant and one-point histories; truncation of 8,301 context steps; and horizons of 1,041 and 2,051 steps. The case shapes are recorded in `parity.py` and the result JSON. - -## Asinh recipe follow-up - -Moving asinh and its inverse into a processor was checked on twelve comparisons: six shapes in float32 and float16, with five warmups and fifteen alternating paired repetitions. Short-horizon prediction differences were unchanged; long-horizon differences changed only by small rounding effects. The accepted constant-history limitation remains. The command above can reproduce this subset with `--cases univariate covariates constant one_step long_horizon three_chunks`. - -Every value is upstream → SDM public synchronized wall latency in milliseconds. - -| Case | Float32 | Float16 | -| ------------ | ------------: | ------------: | -| univariate | 6.60 → 6.83 | 8.17 → 8.29 | -| covariates | 12.62 → 13.08 | 8.95 → 9.85 | -| constant | 8.49 → 8.48 | 8.57 → 8.41 | -| one step | 7.14 → 7.38 | 8.15 → 8.30 | -| long horizon | 70.71 → 69.10 | 34.01 → 34.56 | -| three chunks | 81.09 → 77.69 | 43.73 → 40.90 | - -## Plain SDM RMSNorm comparison - -Plain `sdm.nn.RMSNorm` is the chosen implementation to keep normalization shared across models. Relative RMSE below `1e-3` is accepted; float32 and ordinary float16 cases satisfy this threshold. The missing-history half-precision cases above that threshold are retained as documented exceptions, without a model-specific normalization fallback. - -On 2026-10-30, the local T5-style LayerNorm was replaced with `sdm.nn.RMSNorm(eps=1e-6)` throughout the attention blocks, feed-forward layers, and final normalization. The comparison holds weights and the current Standardize/Asinh recipe fixed, isolating the normalization change from the already accepted constant-scale differences. The baseline substitutes the pinned upstream `Chronos2LayerNorm`, whose arithmetic matches the removed custom class, and preserves shared query/key normalization objects. - -```bash -.venv/bin/python -m benchmark.timeseries.chronos2.norm \ - --output /tmp/chronos2-norm-results.json --warmup 3 --repeats 15 -``` - -This covers eleven cases in each of three modes: float32 parameters, float16 parameters, and float32 parameters with CUDA float16 autocast. The last mode exercises SDM's optimized RMSNorm kernel. Models are constructed outside inference mode. All 33 public-API comparisons produced finite outputs. Wall and CUDA event timing use warmups, alternating paired repetitions, and explicit synchronization on the same GB10. - -Differences below compare **plain RMSNorm against the previous custom normalization**, with identical weights and preprocessing. Speedup is the median percentage latency reduction across the eleven cases, not a forecasting accuracy metric. - -| Mode | Largest relative RMSE | Case with largest difference | Median latency reduction | -| ---------------- | --------------------: | ---------------------------- | -----------------------: | -| Float32 | 1.71e-06 | missing | 5.1% | -| Float16 weights | 1.454% | all missing | 13.3% | -| Float16 autocast | 0.466% | missing | 12.4% | - -For ordinary univariate, batched, multivariate, covariate, and long-context inputs, float16-weight relative RMSE is 0.033–0.087%, and autocast relative RMSE is 0.026–0.053%. Entirely missing histories with float16 weights have the largest change: 1.45% relative RMSE and maximum absolute difference 694,210 because forecasts are already in the millions. Strict half-precision parity is not claimed. Float32 changes remain at floating-point rounding scale; the maximum relative RMSE is 1.71e-6. - -The JSON also records old-versus-upstream and new-versus-upstream errors separately. The previously accepted Standardize constant-history and one-observation discrepancies remain; they must not be attributed to this norm swap. - -## MLP ownership comparison - -`Chronos2TransformerBlock` now accepts `mlp: torch.nn.Module | None`. Time blocks receive `None`; group-attention blocks own the normalized feed-forward network, as in TimesFM-3. The separate MLP list is removed, and checkpoint remapping targets `icl_block.group_blocks..mlp`. - -The 2026-10-30 comparison held SDM RMSNorm, weights, and preprocessing fixed and restored the saved pre-change ICL forward for the baseline. It covered eleven shapes in each of float32, float16-weight, and float16-autocast modes, with three warmups and ten alternating paired repetitions. Float32 results were bit-for-bit identical in all eleven cases. Ordinary half-precision cases, including long context and multi-chunk horizons, were also bit-for-bit identical. - -Differences appeared only in missing-history and constant-history cases. Stored float16 weights had maximum aggregate relative RMSE 3.94e-5 (constant case, maximum absolute difference 0.0346). Float16 autocast reached 0.00382 relative RMSE on the mixed missing-history case. Constant-case aggregate errors are dominated by the 100-valued target: individual near-zero quantile outputs had larger relative errors, up to 15.6% with float16 weights and 8.92% under autocast. These are documented degenerate-input exceptions, not a claim that every half-precision output meets the 1e-3 tolerance. - -Latency was effectively unchanged: median changes across cases were -0.49% for float32, -0.36% for float16 weights, and -0.75% for float16 autocast. All outputs remained finite. The standalone comparison, saved before-change source, and JSON results are in `/tmp/sdm-chronos2-mlp-20261030/` for this run. - -## Numerical agreement - -CPU float32 checks cover univariate, covariates, constant, and missing data. Cached `fit`/`predict` uses the same capped context and fitted recipe as direct prediction; empty horizons preserve batch and output column dimensions. - -Public output shape and target/quantile column ordering matched in the original 22 comparisons and the twelve Asinh follow-up comparisons. The full-matrix precision measurements below refer to the Standardize-only recipe; the Asinh follow-up retained those precision characteristics. For nonconstant histories and entirely missing histories, float32 relative RMSE (`RMSE(reference - SDM) / RMS(reference)`) is at most **1.83e-6**. Ordinary multivariate and covariate cases have maximum absolute errors below **1.5e-6**. The 2,051-step rollout reaches **2.06e-4** maximum absolute error, with relative RMSE **8.07e-7**. - -This is numerical agreement, not bitwise identity. Entirely missing histories can generate forecasts in the millions upstream: float32 differences reach **60** in those cases despite relative RMSE below **1.83e-6**. Fused QKV projections and different tensor layouts change floating-point rounding; inverse `sinh` amplifies it. These cases must not be judged using absolute error alone. - -Float16 is less exact: ordinary batched/multivariate/covariate cases have relative RMSE **0.050–0.058%** and maximum absolute error below **0.0048**. Relative RMSE reaches **0.44%** across three rollout chunks and **0.83%** in the mixed missing-target case. The latter has **244,213** maximum absolute difference because of extremely large upstream forecasts. All measured outputs remain finite, but strict float16 parity is not claimed for these unstable inputs; use float32 when matching the reference is important. - -**Accepted parity limitation:** the recipe deliberately uses unchanged `Standardize()` fitting semantics. For constant histories it uses scale 1 rather than upstream's 1e-5. Removing the previous duplicate normalization exposes this difference. The constant-series case has relative RMSE **1.32%** (float32) or **1.33%** (float16), with maximum absolute difference **5.98** or **6.01**. With only one context observation, output scale differs by roughly **100,000×**; the maximum absolute difference is **11.7 million**. These cases do not have upstream parity. The choice to retain Standardize's default scale was made explicitly after measuring these differences. They are prediction disagreements on synthetic inputs, not measured changes in forecasting quality on real datasets. - -The core uses float32 NaN reductions instead of `RevIN.stats`' float64 reductions. The recipe retains SDM Standardize's finite-value and constant-column rules. Affine normalization reuses `RevIN`. The implementation now uses plain SDM RMSNorm; its measured precision tradeoffs relative to the previous T5-style normalization are recorded above. Attention uses scale 1.0 and upstream's finite additive masks, including its fully masked-row behavior. Long forecasts use nine quantile paths and the weighted empirical CDF rather than median-only feedback. diff --git a/benchmark/timeseries/chronos2/norm.py b/benchmark/timeseries/chronos2/norm.py deleted file mode 100644 index dbe5474d35..0000000000 --- a/benchmark/timeseries/chronos2/norm.py +++ /dev/null @@ -1,133 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Isolate the prediction and latency effects of using SDM RMSNorm.""" - -import argparse -import copy -import json -from contextlib import nullcontext -from pathlib import Path -from typing import Any - -import torch -from chronos import Chronos2Pipeline -from chronos.chronos2.layers import Chronos2LayerNorm - -from benchmark.timeseries.chronos2.parity import ( - CASES, - REVISION, - difference, - inputs, - measure, -) -from sdm.models.chronos2 import Chronos2 -from sdm.models.chronos2.ckpt import remap_ckpt -from sdm.nn import RMSNorm - - -def original_norm(model: Chronos2) -> Chronos2: - model = copy.deepcopy(model) - replacements: dict[int, Chronos2LayerNorm] = {} - # Preserve the shared query/key normalization objects. The pinned upstream - # class implements the same arithmetic as the removed custom LayerNorm. - for name, module in list(model.named_modules(remove_duplicate=False)): - if isinstance(module, RMSNorm): - if id(module) not in replacements: - norm = Chronos2LayerNorm(module.weight.numel(), eps=1e-6) - norm.weight = module.weight - replacements[id(module)] = norm - model.set_submodule(name, replacements[id(module)]) - return model.eval() - - -@torch.inference_mode() -def compare( - baseline: Chronos2, - plain: Chronos2, - upstream: Chronos2Pipeline, - case: str, - mode: str, - args: argparse.Namespace, -) -> dict[str, Any]: - dictionaries, tables, *_ = inputs(case, "cuda") - horizon = CASES[case][2] - - def autocast() -> Any: - if mode == "autocast_float16": - return torch.autocast("cuda", dtype=torch.float16) - return nullcontext() - - def previous() -> torch.Tensor: - with autocast(): - return baseline(*tables).numerical - - def candidate() -> torch.Tensor: - with autocast(): - return plain(*tables).numerical - - with autocast(): - reference = ( - torch.stack( - upstream.predict( - dictionaries, prediction_length=horizon, batch_size=10000 - ) - ) - .permute(0, 3, 1, 2) - .flatten(-2) - ) - a, b = previous(), candidate() - latency = measure((previous, candidate), args.warmup, args.repeats) - return { - "mode": mode, - "case": case, - "shape": CASES[case], - "plain_vs_previous": difference(a, b), - "previous_vs_upstream": difference(reference, a), - "plain_vs_upstream": difference(reference, b), - "previous_latency": latency[0], - "plain_latency": latency[1], - } - - -def main() -> None: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--output", type=Path, required=True) - parser.add_argument("--cases", nargs="+", default=list(CASES)) - parser.add_argument("--warmup", type=int, default=3) - parser.add_argument("--repeats", type=int, default=15) - args = parser.parse_args() - torch.set_float32_matmul_precision("highest") - result: dict[str, Any] = { - "torch": torch.__version__, - "gpu": torch.cuda.get_device_name(), - "checkpoint": REVISION, - "source": "1224fafa91e1ecd6674f5204066b159e40b5e4a1", - "warmup": args.warmup, - "repeats": args.repeats, - "cases": [], - } - for mode in ("float32", "float16", "autocast_float16"): - dtype = torch.float16 if mode == "float16" else torch.float32 - upstream = Chronos2Pipeline.from_pretrained( - "amazon/chronos-2", - revision=REVISION, - device_map="cuda", - dtype=dtype, - ) - plain = Chronos2(pretrained=False, device="meta") - plain.model.load_state_dict( - remap_ckpt(upstream.model.state_dict()), strict=True, assign=True - ) - baseline = original_norm(plain) - for case in args.cases: - row = compare(baseline, plain, upstream, case, mode, args) - print(json.dumps(row), flush=True) - result["cases"].append(row) - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(result, indent=2) + "\n") - del upstream, plain, baseline - - -if __name__ == "__main__": - main() diff --git a/benchmark/timeseries/chronos2/parity.py b/benchmark/timeseries/chronos2/parity.py deleted file mode 100644 index 003295c315..0000000000 --- a/benchmark/timeseries/chronos2/parity.py +++ /dev/null @@ -1,283 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Compare the pinned public Chronos-2 pipeline and SDM on identical inputs.""" - -import argparse -import json -import platform -import statistics -import time -from collections.abc import Callable -from importlib.metadata import version -from pathlib import Path -from typing import Any - -import torch -from chronos import Chronos2Pipeline - -from sdm import Stype, TableTensor -from sdm.models.chronos2 import Chronos2 -from sdm.models.chronos2.ckpt import remap_ckpt -from sdm.processing import Standardize - -REVISION = "29ec3766d36d6f73f0696f85560a422f50e8498c" - -CASES = { - "univariate": (1, 95, 31, 1, 0, 0), - "batch_univariate": (16, 512, 96, 1, 0, 0), - "multivariate": (2, 512, 96, 3, 0, 0), - "covariates": (2, 255, 65, 2, 1, 2), - "missing": (2, 95, 33, 2, 1, 1), - "all_missing": (2, 47, 17, 1, 0, 0), - "constant": (1, 127, 32, 2, 0, 0), - "one_step": (1, 1, 1, 1, 0, 0), - "truncated_context": (1, 8301, 17, 1, 0, 1), - "long_horizon": (1, 95, 1041, 1, 1, 1), - "three_chunks": (1, 47, 2051, 1, 0, 0), -} - - -def inputs(name: str, device: str) -> tuple: - batch, length, horizon, targets, past, known = CASES[name] - total = targets + past + known - index = torch.arange(batch * total * (length + horizon)).float() - values = (index * 0.07).sin() + (index * 0.013).cos() * 0.5 - values = values.reshape(batch, total, length + horizon) - context = values[..., :length].clone() - future = values[..., length:].clone() - future[:, : targets + past] = torch.nan - if name == "missing": - context[..., :21] = torch.nan - context[..., 25::7] = torch.nan - context[0, 0] = torch.nan - future[..., ::5] = torch.nan - elif name == "all_missing": - context[:] = torch.nan - elif name == "constant": - context[:, 0] = 0 - context[:, 1] = 100 - elif name == "truncated_context": - context[..., :30] = 1e6 - dictionaries = [] - for row in range(batch): - dictionaries.append( - { - "target": context[row, :targets].numpy(), - "past_covariates": { - f"past{i}": context[row, targets + i].numpy() - for i in range(past) - } - | { - f"known{i}": context[row, targets + past + i].numpy() - for i in range(known) - }, - "future_covariates": { - f"known{i}": future[row, targets + past + i].numpy() - for i in range(known) - }, - } - ) - features = context[:, targets:].transpose(-1, -2).to(device) - query = future[:, targets + past :].transpose(-1, -2).to(device) - names = [f"past{i}" for i in range(past)] + [ - f"known{i}" for i in range(known) - ] - x = TableTensor.from_tensor(features.unsqueeze(0), columns=names) - y = TableTensor.from_tensor( - context[:, :targets].transpose(-1, -2).to(device).unsqueeze(0), - columns=[f"target{i}" for i in range(targets)], - ) - q = TableTensor.from_tensor(query.unsqueeze(0), columns=names[past:]) - groups = torch.arange(batch, device=device).repeat_interleave(total) - return ( - dictionaries, - (x, y, q), - context.flatten(0, 1).to(device), - future.flatten(0, 1).to(device), - groups, - ) - - -def difference(a: torch.Tensor, b: torch.Tensor) -> dict[str, float]: - a, b = a.float().cpu(), b.float().cpu() - if a.shape != b.shape: - raise RuntimeError(f"Output shapes differ: {a.shape} vs {b.shape}") - delta = (a - b).abs() - scale = a.square().mean().sqrt().clamp_min(1e-6) - return { - "max_abs": delta.max().item(), - "rmse": delta.square().mean().sqrt().item(), - "relative_rmse": (delta.square().mean().sqrt() / scale).item(), - "max_series_relative_rmse": ( - delta.square().mean(dim=-2).sqrt() - / a.square().mean(dim=-2).sqrt().clamp_min(1e-6) - ) - .max() - .item(), - "finite": bool(a.isfinite().all() and b.isfinite().all()), - } - - -def measure( - calls: tuple[Callable, Callable], warmup: int, repeats: int -) -> list[dict[str, float]]: - for _ in range(warmup): - for call in calls: - call() - walls: list[list[float]] = [[], []] - events: list[list[float]] = [[], []] - for repeat in range(repeats): - for index in range(2) if repeat % 2 == 0 else reversed(range(2)): - start, end = torch.cuda.Event(True), torch.cuda.Event(True) - torch.cuda.synchronize() - tick = time.perf_counter() - start.record() - calls[index]() - end.record() - torch.cuda.synchronize() - walls[index].append((time.perf_counter() - tick) * 1000) - events[index].append(start.elapsed_time(end)) - return [ - { - "wall_ms": statistics.median(wall), - "cuda_ms": statistics.median(event), - "wall_p10_ms": sorted(wall)[len(wall) // 10], - "wall_p90_ms": sorted(wall)[ - min(len(wall) - 1, 9 * len(wall) // 10) - ], - } - for wall, event in zip(walls, events) - ] - - -@torch.inference_mode() -def run_case( - upstream: Chronos2Pipeline, - model: Chronos2, - case: str, - args: argparse.Namespace, -) -> dict[str, Any]: - dictionaries, tables, context, future, groups = inputs(case, "cuda") - _, _, horizon, targets, _, _ = CASES[case] - - def reference() -> torch.Tensor: - out = upstream.predict( - dictionaries, - prediction_length=horizon, - batch_size=10000, - ) - return torch.stack(out).permute(0, 3, 1, 2).flatten(-2) - - def candidate() -> torch.Tensor: - return model(*tables).numerical - - a, b = reference(), candidate() - public_diff = difference(a, b) - expected_names = [ - f"target{i}__q{round(q * 100)}" - for i in range(targets) - for q in upstream.quantiles - ] - if list(model(*tables).columns[Stype.numerical]) != expected_names: - raise RuntimeError("Quantile column order differs") - public_time = measure((reference, candidate), args.warmup, args.repeats) - # The public upstream API includes CPU input preparation and a CPU - # result copy. Compare numerical execution separately on CUDA. - patches = min((horizon + 15) // 16, 64) - - def reference_core() -> torch.Tensor: - return upstream._predict_step( - context=context, - group_ids=groups, - future_covariates=future, - num_output_patches=patches, - ) - - def candidate_core() -> torch.Tensor: - processor = Standardize() - retained = context[..., -model.model.context_length :] - table = TableTensor.from_tensor(retained.T) - x = processor.fit_transform(table).numerical.T.asinh() - f = processor.transform( - TableTensor.from_tensor(future.T) - ).numerical.T.asinh() - batch, _, _, targets, past, known = CASES[case] - variates = targets + past + known - out = ( - model.model._predict( - x.unflatten(0, (batch, variates)), - f.unflatten(0, (batch, variates)), - patches, - ) - .flatten(0, 1) - .sinh() - ) - # Put the quantile axis first so fitted [1, S] statistics broadcast. - table = TableTensor.from_tensor(out.permute(1, 2, 0)) - return processor.inverse_transform(table).numerical.permute(2, 0, 1) - - core_diff = difference(reference_core(), candidate_core()) - core_time = measure( - (reference_core, candidate_core), args.warmup, args.repeats - ) - return { - "dtype": str(next(model.parameters()).dtype), - "case": case, - "shape": CASES[case], - "public_difference": public_diff, - "core_difference": core_diff, - "public_upstream": public_time[0], - "public_sdm": public_time[1], - "core_upstream": core_time[0], - "core_sdm": core_time[1], - } - - -def main() -> None: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--output", type=Path, required=True) - parser.add_argument("--dtypes", nargs="+", default=["float32", "float16"]) - parser.add_argument("--cases", nargs="+", default=list(CASES)) - parser.add_argument("--warmup", type=int, default=5) - parser.add_argument("--repeats", type=int, default=15) - args = parser.parse_args() - torch.set_float32_matmul_precision("highest") - results: dict[str, Any] = { - "torch": torch.__version__, - "cuda": torch.version.cuda, - "transformers": version("transformers"), - "chronos_forecasting": version("chronos-forecasting"), - "matmul_precision": torch.get_float32_matmul_precision(), - "recipe": repr(Chronos2.default_recipe()), - "gpu": torch.cuda.get_device_name(), - "python": platform.python_version(), - "checkpoint": REVISION, - "source": "1224fafa91e1ecd6674f5204066b159e40b5e4a1", - "warmup": args.warmup, - "repeats": args.repeats, - "cases": [], - } - for name in args.dtypes: - dtype = getattr(torch, name) - upstream = Chronos2Pipeline.from_pretrained( - "amazon/chronos-2", - revision=REVISION, - device_map="cuda", - dtype=dtype, - ) - model = Chronos2(device="cuda") - if dtype != torch.float32: - state = remap_ckpt(upstream.model.state_dict()) - model.model.load_state_dict(state, strict=True, assign=True) - for case in args.cases: - row = run_case(upstream, model, case, args) - print(json.dumps(row), flush=True) - results["cases"].append(row) - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(results, indent=2) + "\n") - del upstream, model - - -if __name__ == "__main__": - main() From 1cfe87be7ec62cfb4f74e1cedab1bd5feb728141 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 10:18:44 +0000 Subject: [PATCH 03/13] update --- docs/source/forecasting.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/forecasting.md b/docs/source/forecasting.md index bdde01ed69..61fabd8a4d 100644 --- a/docs/source/forecasting.md +++ b/docs/source/forecasting.md @@ -52,10 +52,10 @@ history = sdm.TableTensor.from_columns( ``` We can forecast with any time series forecasting model in [`sdm.models`](api/models). -Here, we use {py:class}`~sdm.models.TimesFM3`: +Here, we use {py:class}`~sdm.models.Chronos2`: ```python -model = sdm.models.TimesFM3(device="cuda") +model = sdm.models.Chronos2(device="cuda") out = model( x_context=sdm.TableTensor(size=(5, ), device="cuda"), y_context=history, @@ -68,7 +68,7 @@ Since we have none yet, both tables hold zero columns. However, we still need to pass them to {py:meth}`~sdm.models.ICLModel.forward`, in particular because the number of rows in `x_query` defines the **forecast horizon**: here, we request predictions for the next three time steps. The forecast is returned as a {py:class}`~sdm.tensor.TableTensor` with one row per future step. -Its output is model-specific, *e.g.*, {py:class}`~sdm.models.TimesFM3` outputs one column per target and quantile, from the 10th to the 90th percentile, named `"__q"`: +Its output is model-specific, *e.g.*, {py:class}`~sdm.models.Chronos2` outputs one column per target and quantile, named `"__q"`: ```python out["ice_cream__q50"].numerical # median forecast, [3, 1] From 83e784aed8c1b8920debfb11273d633d6d705afb Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 12:11:07 +0000 Subject: [PATCH 04/13] update --- benchmark/timeseries/fev_bench/README.md | 9 +++++++++ benchmark/timeseries/fev_bench/main.py | 2 +- benchmark/timeseries/fev_bench/predictor.py | 2 +- benchmark/timeseries/gift_eval/README.md | 9 +++++++++ benchmark/timeseries/gift_eval/main.py | 2 +- benchmark/timeseries/model.py | 8 ++++++++ benchmark/timeseries/predictor.py | 2 +- benchmark/timeseries/time_bench/README.md | 9 +++++++++ benchmark/timeseries/time_bench/main.py | 8 +++++--- 9 files changed, 44 insertions(+), 7 deletions(-) diff --git a/benchmark/timeseries/fev_bench/README.md b/benchmark/timeseries/fev_bench/README.md index 658d9111cb..7a58d42719 100644 --- a/benchmark/timeseries/fev_bench/README.md +++ b/benchmark/timeseries/fev_bench/README.md @@ -12,6 +12,15 @@ pip install . fev==0.10.0 ## Run +- **`Chronos2`**: + + ```bash + python -m benchmark.timeseries.fev_bench.main --model chronos2 + ``` + + `TimesFM3` follows official evaluation with symmetric averaging over two estimators, contexts truncated to 15,360 time steps and up to 32 variates per forward pass. + Unlike the official `float32` evaluation, we use `bfloat16` mixed precision for improved efficiency with negligible differences in aggregate benchmark accuracy. + - **`TimesFM3`**: ```bash diff --git a/benchmark/timeseries/fev_bench/main.py b/benchmark/timeseries/fev_bench/main.py index 828c016da0..b234a2efa6 100644 --- a/benchmark/timeseries/fev_bench/main.py +++ b/benchmark/timeseries/fev_bench/main.py @@ -18,7 +18,7 @@ parser.add_argument( "--model", choices=tuple(MODEL_CONFIGS), - default="timesfm3", + default="chronos2", help="SDM model to benchmark.", ) parser.add_argument( diff --git a/benchmark/timeseries/fev_bench/predictor.py b/benchmark/timeseries/fev_bench/predictor.py index 6d8ba0762a..45d47e00f0 100644 --- a/benchmark/timeseries/fev_bench/predictor.py +++ b/benchmark/timeseries/fev_bench/predictor.py @@ -155,7 +155,7 @@ def _fit_predict(self, task: fev.Task) -> list[datasets.DatasetDict]: pred = torch.cat(preds, dim=-2)[..., : y.size(-1), :] pred = pred.cpu().numpy() # [B, R, C, Q] levels = out.numerical_columns[: pred.shape[-1]] - levels = [float(f"0.{q.split('q')[-1]}") for q in levels] + levels = [int(q.rsplit("q", 1)[-1]) / 100 for q in levels] wanted = [0.5, *task.quantile_levels] index = [levels.index(q) for q in wanted] values.append(pred[..., index]) diff --git a/benchmark/timeseries/gift_eval/README.md b/benchmark/timeseries/gift_eval/README.md index 2ef2833ce3..546989f7f4 100644 --- a/benchmark/timeseries/gift_eval/README.md +++ b/benchmark/timeseries/gift_eval/README.md @@ -13,6 +13,15 @@ pip install --no-deps "git+https://github.com/SalesforceAIResearch/gift-eval.git ## Run +- **`Chronos2`**: + + ```bash + python -m benchmark.timeseries.gift_eval.main --model chronos2 + ``` + + `TimesFM3` follows official evaluation with symmetric averaging over two estimators, contexts truncated to 15,360 time steps and up to 32 variates per forward pass. + Unlike the official `float32` evaluation, we use `bfloat16` mixed precision for improved efficiency with negligible differences in aggregate benchmark accuracy. + - **`TimesFM3`**: ```bash diff --git a/benchmark/timeseries/gift_eval/main.py b/benchmark/timeseries/gift_eval/main.py index 3f2fd8d2fd..f642230223 100644 --- a/benchmark/timeseries/gift_eval/main.py +++ b/benchmark/timeseries/gift_eval/main.py @@ -33,7 +33,7 @@ parser.add_argument( "--model", choices=tuple(MODEL_CONFIGS), - default="timesfm3", + default="chronos2", help="SDM model to benchmark.", ) parser.add_argument( diff --git a/benchmark/timeseries/model.py b/benchmark/timeseries/model.py index d1b87ac5d4..1ea8a012d9 100644 --- a/benchmark/timeseries/model.py +++ b/benchmark/timeseries/model.py @@ -20,6 +20,14 @@ class ModelConfig: MODEL_CONFIGS = { + "chronos2": ModelConfig( + name="Chronos2", + factory=lambda device: sdm.models.Chronos2(device=device), + autocast_dtype=torch.float16, + num_estimators=1, + max_context_size=8_192, + max_variates=None, + ), "timesfm3": ModelConfig( name="TimesFM3", factory=lambda device: sdm.models.TimesFM3( diff --git a/benchmark/timeseries/predictor.py b/benchmark/timeseries/predictor.py index 7c774326b6..d1056c3880 100644 --- a/benchmark/timeseries/predictor.py +++ b/benchmark/timeseries/predictor.py @@ -115,7 +115,7 @@ def predict( pred = torch.cat(preds, dim=-2)[..., : y.size(-1), :] pred = pred.permute(0, 3, 1, 2).cpu().numpy() # [B, Q, R, C] levels = out.numerical_columns[: pred.shape[1]] - levels = [str(float(f"0.{q.split('q')[-1]}")) for q in levels] + levels = [str(int(q.rsplit("q", 1)[-1]) / 100) for q in levels] for entry, pred_i in zip(batch, pred): if entry["target"].ndim == 1: diff --git a/benchmark/timeseries/time_bench/README.md b/benchmark/timeseries/time_bench/README.md index 52bf2a1253..f2dfdb51ad 100644 --- a/benchmark/timeseries/time_bench/README.md +++ b/benchmark/timeseries/time_bench/README.md @@ -13,6 +13,15 @@ pip install --no-deps "git+https://github.com/zqiao11/TIME.git@3ca5c41d71c76f3c7 ## Run +- **`Chronos2`**: + + ```bash + python -m benchmark.timeseries.time_bench.main --model chronos2 + ``` + + `TimesFM3` follows official evaluation with symmetric averaging over two estimators, contexts truncated to 15,360 time steps and up to 32 variates per forward pass. + Unlike the official `float32` evaluation, we use `bfloat16` mixed precision for improved efficiency with negligible differences in aggregate benchmark accuracy. + - **`TimesFM3`**: ```bash diff --git a/benchmark/timeseries/time_bench/main.py b/benchmark/timeseries/time_bench/main.py index 6a80b776d5..e57955aef9 100644 --- a/benchmark/timeseries/time_bench/main.py +++ b/benchmark/timeseries/time_bench/main.py @@ -29,7 +29,7 @@ parser.add_argument( "--model", choices=tuple(MODEL_CONFIGS), - default="timesfm3", + default="chronos2", help="SDM model to benchmark.", ) parser.add_argument( @@ -79,7 +79,9 @@ ) ) - quantiles = np.stack([out.forecast_array for out in outs]) + levels = [i / 10 for i in range(1, 10)] + indices = [outs[0].forecast_keys.index(str(q)) for q in levels] + quantiles = np.stack([out.forecast_array for out in outs])[:, indices] if quantiles.ndim == 4: quantiles = quantiles.swapaxes(-1, -2) # [B, Q, C, R] @@ -90,7 +92,7 @@ ds_config=f"{name}/{term}", output_base_dir=str(path), seasonality=get_seasonality(dataset.freq), - quantile_levels=[float(q) for q in outs[0].forecast_keys], + quantile_levels=levels, ) with np.load(path / name / term / "metrics.npz") as metrics: mase = np.nanmean(metrics["MASE"]) From 636262cae9d2878e39af46de8e0dcdf6d9e8d6b1 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 12:29:36 +0000 Subject: [PATCH 05/13] update --- benchmark/timeseries/gift_eval/README.md | 5 ++-- benchmark/timeseries/gift_eval/evaluate.py | 7 ++--- benchmark/timeseries/gift_eval/main.py | 33 ++++++++++++++++++++-- benchmark/timeseries/model.py | 1 + benchmark/timeseries/predictor.py | 1 + 5 files changed, 37 insertions(+), 10 deletions(-) diff --git a/benchmark/timeseries/gift_eval/README.md b/benchmark/timeseries/gift_eval/README.md index 546989f7f4..3c8a4127fb 100644 --- a/benchmark/timeseries/gift_eval/README.md +++ b/benchmark/timeseries/gift_eval/README.md @@ -19,8 +19,8 @@ pip install --no-deps "git+https://github.com/SalesforceAIResearch/gift-eval.git python -m benchmark.timeseries.gift_eval.main --model chronos2 ``` - `TimesFM3` follows official evaluation with symmetric averaging over two estimators, contexts truncated to 15,360 time steps and up to 32 variates per forward pass. - Unlike the official `float32` evaluation, we use `bfloat16` mixed precision for improved efficiency with negligible differences in aggregate benchmark accuracy. + `Chronos2` uses cross-learning across batches of 64 examples, contexts truncated to 8,192 time steps, and `float16` mixed precision. Historical covariates are included when available. Each evaluation window is predicted separately to prevent leakage between rolling windows. + The published reference uses `float32`, targets only, and a batch budget of 100 series, so these settings can produce different scores. - **`TimesFM3`**: @@ -36,6 +36,7 @@ pip install --no-deps "git+https://github.com/SalesforceAIResearch/gift-eval.git > Review the license before running the `TimesFM3` benchmark, which will download its weights noninteractively. Results will be written to `benchmark/timeseries/gift_eval_out//`, and already completed configurations are skipped. +Chronos-2 cross-learning results use the `Chronos2CrossLearning` directory to keep them separate from previous independent forecasts. Pass `--dataset` to run only selected GIFT-Eval datasets, *e.g.*, `--dataset hospital --dataset ett1/H`. diff --git a/benchmark/timeseries/gift_eval/evaluate.py b/benchmark/timeseries/gift_eval/evaluate.py index b981d19eac..77d255121c 100644 --- a/benchmark/timeseries/gift_eval/evaluate.py +++ b/benchmark/timeseries/gift_eval/evaluate.py @@ -9,7 +9,6 @@ import pandas as pd from benchmark.timeseries.gift_eval import GIFT_EVAL_URL -from benchmark.timeseries.model import MODEL_CONFIGS BASELINE = "Seasonal_Naive" REFERENCES = ("TimesFM-3", "TimesFM-2.5", "Chronos-2", BASELINE) @@ -21,10 +20,8 @@ root = Path(__file__).parent.parent / "gift_eval_out" dfs = [] -for config in MODEL_CONFIGS.values(): - path = root / config.name / "all_results.csv" - if path.exists(): - dfs.append(pd.read_csv(path)) +for path in sorted(root.glob("*/all_results.csv")): + dfs.append(pd.read_csv(path)) datasets = set.intersection(*(set(df["dataset"]) for df in dfs)) for name in REFERENCES: diff --git a/benchmark/timeseries/gift_eval/main.py b/benchmark/timeseries/gift_eval/main.py index f642230223..a9af65f84f 100644 --- a/benchmark/timeseries/gift_eval/main.py +++ b/benchmark/timeseries/gift_eval/main.py @@ -7,7 +7,9 @@ import logging import os import warnings +from dataclasses import replace from functools import partial +from itertools import islice from pathlib import Path from unittest.mock import patch @@ -15,7 +17,7 @@ import torch from gift_eval.data import Dataset from gluonts.ev import metrics -from gluonts.model import evaluate_model +from gluonts.model import evaluate_forecasts from gluonts.time_feature import get_seasonality from huggingface_hub import snapshot_download from tqdm.auto import tqdm @@ -65,6 +67,12 @@ ] config = MODEL_CONFIGS[args.model] +if args.model == "chronos2": + config = replace( + config, + name=f"{config.name}CrossLearning", + cross_learning=True, + ) path = Path(__file__).parent.parent path = path / "gift_eval_out" / config.name / "all_results.csv" path.parent.mkdir(parents=True, exist_ok=True) @@ -95,12 +103,31 @@ dataset = Dataset(name=name, term=term, to_univariate=False) predictor = SDMPredictor(model, config, dataset.prediction_length) + num_windows = dataset.test_data.windows + forecasts = [] + with tqdm(total=len(dataset.test_data), desc=dataset_id) as progress: + for window in range(num_windows): + # Share information only within the same evaluation window. + inputs = islice( + dataset.test_data.input, window, None, num_windows + ) + window_forecasts = [] + for forecast in predictor.predict(inputs): + window_forecasts.append(forecast) + progress.update() + forecasts.append(window_forecasts) + # Restore the series-major order expected by the evaluator. + forecasts = [ + forecast + for window_forecasts in zip(*forecasts, strict=True) + for forecast in window_forecasts + ] with patch( "gluonts.model.evaluation.tqdm", partial(tqdm, total=len(dataset.test_data), desc=dataset_id), ): - result = evaluate_model( - predictor, + result = evaluate_forecasts( + forecasts, test_data=dataset.test_data, metrics=METRICS, seasonality=get_seasonality(dataset.freq), diff --git a/benchmark/timeseries/model.py b/benchmark/timeseries/model.py index 1ea8a012d9..a0ba278ee6 100644 --- a/benchmark/timeseries/model.py +++ b/benchmark/timeseries/model.py @@ -15,6 +15,7 @@ class ModelConfig: factory: Callable[[torch.device], sdm.models.ICLModel] autocast_dtype: torch.dtype num_estimators: int = 1 + cross_learning: bool = False max_context_size: int | None = None max_variates: int | None = None diff --git a/benchmark/timeseries/predictor.py b/benchmark/timeseries/predictor.py index d1056c3880..111f2d9678 100644 --- a/benchmark/timeseries/predictor.py +++ b/benchmark/timeseries/predictor.py @@ -107,6 +107,7 @@ def predict( device=self.device, ), num_estimators=self.config.num_estimators, + cross_learning=self.config.cross_learning, ) pred = out.numerical pred = pred.view(y.size(0), self.prediction_length, size, -1) From f5e6dd4fbd9b8de72804a21cfc3d170b3bf94cce Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 12:32:57 +0000 Subject: [PATCH 06/13] update --- benchmark/timeseries/fev_bench/predictor.py | 2 +- benchmark/timeseries/gift_eval/README.md | 1 - benchmark/timeseries/gift_eval/evaluate.py | 7 +++++-- benchmark/timeseries/gift_eval/main.py | 6 +----- 4 files changed, 7 insertions(+), 9 deletions(-) diff --git a/benchmark/timeseries/fev_bench/predictor.py b/benchmark/timeseries/fev_bench/predictor.py index 45d47e00f0..d2bb231b3e 100644 --- a/benchmark/timeseries/fev_bench/predictor.py +++ b/benchmark/timeseries/fev_bench/predictor.py @@ -155,7 +155,7 @@ def _fit_predict(self, task: fev.Task) -> list[datasets.DatasetDict]: pred = torch.cat(preds, dim=-2)[..., : y.size(-1), :] pred = pred.cpu().numpy() # [B, R, C, Q] levels = out.numerical_columns[: pred.shape[-1]] - levels = [int(q.rsplit("q", 1)[-1]) / 100 for q in levels] + levels = [int(q.split("q")[-1]) / 100 for q in levels] wanted = [0.5, *task.quantile_levels] index = [levels.index(q) for q in wanted] values.append(pred[..., index]) diff --git a/benchmark/timeseries/gift_eval/README.md b/benchmark/timeseries/gift_eval/README.md index 3c8a4127fb..23b743f0cf 100644 --- a/benchmark/timeseries/gift_eval/README.md +++ b/benchmark/timeseries/gift_eval/README.md @@ -36,7 +36,6 @@ pip install --no-deps "git+https://github.com/SalesforceAIResearch/gift-eval.git > Review the license before running the `TimesFM3` benchmark, which will download its weights noninteractively. Results will be written to `benchmark/timeseries/gift_eval_out//`, and already completed configurations are skipped. -Chronos-2 cross-learning results use the `Chronos2CrossLearning` directory to keep them separate from previous independent forecasts. Pass `--dataset` to run only selected GIFT-Eval datasets, *e.g.*, `--dataset hospital --dataset ett1/H`. diff --git a/benchmark/timeseries/gift_eval/evaluate.py b/benchmark/timeseries/gift_eval/evaluate.py index 77d255121c..b981d19eac 100644 --- a/benchmark/timeseries/gift_eval/evaluate.py +++ b/benchmark/timeseries/gift_eval/evaluate.py @@ -9,6 +9,7 @@ import pandas as pd from benchmark.timeseries.gift_eval import GIFT_EVAL_URL +from benchmark.timeseries.model import MODEL_CONFIGS BASELINE = "Seasonal_Naive" REFERENCES = ("TimesFM-3", "TimesFM-2.5", "Chronos-2", BASELINE) @@ -20,8 +21,10 @@ root = Path(__file__).parent.parent / "gift_eval_out" dfs = [] -for path in sorted(root.glob("*/all_results.csv")): - dfs.append(pd.read_csv(path)) +for config in MODEL_CONFIGS.values(): + path = root / config.name / "all_results.csv" + if path.exists(): + dfs.append(pd.read_csv(path)) datasets = set.intersection(*(set(df["dataset"]) for df in dfs)) for name in REFERENCES: diff --git a/benchmark/timeseries/gift_eval/main.py b/benchmark/timeseries/gift_eval/main.py index a9af65f84f..f06f1a1e53 100644 --- a/benchmark/timeseries/gift_eval/main.py +++ b/benchmark/timeseries/gift_eval/main.py @@ -68,11 +68,7 @@ config = MODEL_CONFIGS[args.model] if args.model == "chronos2": - config = replace( - config, - name=f"{config.name}CrossLearning", - cross_learning=True, - ) + config = replace(config, cross_learning=True) path = Path(__file__).parent.parent path = path / "gift_eval_out" / config.name / "all_results.csv" path.parent.mkdir(parents=True, exist_ok=True) From 1b143e97bd40791f0e7e427d72b6a46f132af722 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 12:36:58 +0000 Subject: [PATCH 07/13] update --- benchmark/timeseries/predictor.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmark/timeseries/predictor.py b/benchmark/timeseries/predictor.py index 111f2d9678..b410414c5a 100644 --- a/benchmark/timeseries/predictor.py +++ b/benchmark/timeseries/predictor.py @@ -116,7 +116,7 @@ def predict( pred = torch.cat(preds, dim=-2)[..., : y.size(-1), :] pred = pred.permute(0, 3, 1, 2).cpu().numpy() # [B, Q, R, C] levels = out.numerical_columns[: pred.shape[1]] - levels = [str(int(q.rsplit("q", 1)[-1]) / 100) for q in levels] + levels = [str(int(q.split("q")[-1]) / 100) for q in levels] for entry, pred_i in zip(batch, pred): if entry["target"].ndim == 1: From 3295975a8151cf0ab0fa2653720e5e471a65e2d0 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 12:37:52 +0000 Subject: [PATCH 08/13] update --- benchmark/timeseries/model.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/benchmark/timeseries/model.py b/benchmark/timeseries/model.py index a0ba278ee6..337190b406 100644 --- a/benchmark/timeseries/model.py +++ b/benchmark/timeseries/model.py @@ -15,9 +15,9 @@ class ModelConfig: factory: Callable[[torch.device], sdm.models.ICLModel] autocast_dtype: torch.dtype num_estimators: int = 1 - cross_learning: bool = False max_context_size: int | None = None max_variates: int | None = None + cross_learning: bool = False MODEL_CONFIGS = { @@ -28,6 +28,7 @@ class ModelConfig: num_estimators=1, max_context_size=8_192, max_variates=None, + cross_learning=True, ), "timesfm3": ModelConfig( name="TimesFM3", From 1d916d723b70738ebae91eff5de55e8166b39535 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 12:39:12 +0000 Subject: [PATCH 09/13] update --- benchmark/timeseries/fev_bench/README.md | 3 +-- benchmark/timeseries/fev_bench/main.py | 3 +++ benchmark/timeseries/fev_bench/predictor.py | 1 + 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/benchmark/timeseries/fev_bench/README.md b/benchmark/timeseries/fev_bench/README.md index 7a58d42719..a671198071 100644 --- a/benchmark/timeseries/fev_bench/README.md +++ b/benchmark/timeseries/fev_bench/README.md @@ -18,8 +18,7 @@ pip install . fev==0.10.0 python -m benchmark.timeseries.fev_bench.main --model chronos2 ``` - `TimesFM3` follows official evaluation with symmetric averaging over two estimators, contexts truncated to 15,360 time steps and up to 32 variates per forward pass. - Unlike the official `float32` evaluation, we use `bfloat16` mixed precision for improved efficiency with negligible differences in aggregate benchmark accuracy. + `Chronos2` uses cross-learning across batches of 64 examples, contexts truncated to 8,192 time steps, and `float16` mixed precision. Targets and available covariates are shared within each batch. Evaluation windows are processed separately. - **`TimesFM3`**: diff --git a/benchmark/timeseries/fev_bench/main.py b/benchmark/timeseries/fev_bench/main.py index b234a2efa6..0e9b341a79 100644 --- a/benchmark/timeseries/fev_bench/main.py +++ b/benchmark/timeseries/fev_bench/main.py @@ -4,6 +4,7 @@ r"""Run an SDM time series forecasting model on fev-bench.""" import argparse +from dataclasses import replace from pathlib import Path import fev @@ -34,6 +35,8 @@ args = parser.parse_args() config = MODEL_CONFIGS[args.model] +if args.model == "chronos2": + config = replace(config, cross_learning=True) path = Path(__file__).parent.parent path = path / "fev_bench_out" / config.name / "results.csv" path.parent.mkdir(parents=True, exist_ok=True) diff --git a/benchmark/timeseries/fev_bench/predictor.py b/benchmark/timeseries/fev_bench/predictor.py index d2bb231b3e..55e794bea1 100644 --- a/benchmark/timeseries/fev_bench/predictor.py +++ b/benchmark/timeseries/fev_bench/predictor.py @@ -147,6 +147,7 @@ def _fit_predict(self, task: fev.Task) -> list[datasets.DatasetDict]: y_context=TableTensor.from_tensor(y_i), x_query=TableTensor.from_tensor(x_query), num_estimators=self.config.num_estimators, + cross_learning=self.config.cross_learning, ) pred = out.numerical pred = pred.view(y.size(0), task.horizon, size, -1) From 733eace4e447933a0726c1b62d4ecb213601746a Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 12:40:06 +0000 Subject: [PATCH 10/13] update --- benchmark/timeseries/fev_bench/main.py | 3 --- benchmark/timeseries/gift_eval/main.py | 3 --- benchmark/timeseries/time_bench/main.py | 3 ++- 3 files changed, 2 insertions(+), 7 deletions(-) diff --git a/benchmark/timeseries/fev_bench/main.py b/benchmark/timeseries/fev_bench/main.py index 0e9b341a79..b234a2efa6 100644 --- a/benchmark/timeseries/fev_bench/main.py +++ b/benchmark/timeseries/fev_bench/main.py @@ -4,7 +4,6 @@ r"""Run an SDM time series forecasting model on fev-bench.""" import argparse -from dataclasses import replace from pathlib import Path import fev @@ -35,8 +34,6 @@ args = parser.parse_args() config = MODEL_CONFIGS[args.model] -if args.model == "chronos2": - config = replace(config, cross_learning=True) path = Path(__file__).parent.parent path = path / "fev_bench_out" / config.name / "results.csv" path.parent.mkdir(parents=True, exist_ok=True) diff --git a/benchmark/timeseries/gift_eval/main.py b/benchmark/timeseries/gift_eval/main.py index f06f1a1e53..2761d2859e 100644 --- a/benchmark/timeseries/gift_eval/main.py +++ b/benchmark/timeseries/gift_eval/main.py @@ -7,7 +7,6 @@ import logging import os import warnings -from dataclasses import replace from functools import partial from itertools import islice from pathlib import Path @@ -67,8 +66,6 @@ ] config = MODEL_CONFIGS[args.model] -if args.model == "chronos2": - config = replace(config, cross_learning=True) path = Path(__file__).parent.parent path = path / "gift_eval_out" / config.name / "all_results.csv" path.parent.mkdir(parents=True, exist_ok=True) diff --git a/benchmark/timeseries/time_bench/main.py b/benchmark/timeseries/time_bench/main.py index e57955aef9..d27a10eec9 100644 --- a/benchmark/timeseries/time_bench/main.py +++ b/benchmark/timeseries/time_bench/main.py @@ -7,6 +7,7 @@ import os import warnings from contextlib import redirect_stdout +from dataclasses import replace from pathlib import Path import numpy as np @@ -41,7 +42,7 @@ warnings.filterwarnings("ignore", message=r"'.*' is deprecated.*") -config = MODEL_CONFIGS[args.model] +config = replace(MODEL_CONFIGS[args.model], cross_learning=False) dataset_config = load_dataset_config() path = Path(__file__).parent.parent / "time_bench_out" / config.name From 9426d0ff9232e3e515a7c362883886e121d83754 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 13:27:18 +0000 Subject: [PATCH 11/13] update --- benchmark/timeseries/gift_eval/main.py | 47 +++++++++++++++----------- 1 file changed, 27 insertions(+), 20 deletions(-) diff --git a/benchmark/timeseries/gift_eval/main.py b/benchmark/timeseries/gift_eval/main.py index 2761d2859e..804b0e5a81 100644 --- a/benchmark/timeseries/gift_eval/main.py +++ b/benchmark/timeseries/gift_eval/main.py @@ -96,31 +96,38 @@ dataset = Dataset(name=name, term=term, to_univariate=False) predictor = SDMPredictor(model, config, dataset.prediction_length) - num_windows = dataset.test_data.windows - forecasts = [] - with tqdm(total=len(dataset.test_data), desc=dataset_id) as progress: - for window in range(num_windows): - # Share information only within the same evaluation window. - inputs = islice( - dataset.test_data.input, window, None, num_windows + + if config.cross_learning: # Share information only within same window: + outs = [] + with tqdm(total=len(dataset.test_data), desc=dataset_id) as pbar: + for window in range(dataset.test_data.windows): + inputs = islice( + dataset.test_data.input, + window, + None, + dataset.test_data.windows, + ) + windows = [] + for out in predictor.predict(inputs): + windows.append(out) + pbar.update() + outs.append(windows) + outs = [out for windows in zip(*outs) for out in windows] + else: + outs = list( + tqdm( + predictor.predict(dataset.test_data.input), + total=len(dataset.test_data), + desc=dataset_id, ) - window_forecasts = [] - for forecast in predictor.predict(inputs): - window_forecasts.append(forecast) - progress.update() - forecasts.append(window_forecasts) - # Restore the series-major order expected by the evaluator. - forecasts = [ - forecast - for window_forecasts in zip(*forecasts, strict=True) - for forecast in window_forecasts - ] + ) + with patch( "gluonts.model.evaluation.tqdm", - partial(tqdm, total=len(dataset.test_data), desc=dataset_id), + partial(tqdm, disable=True), ): result = evaluate_forecasts( - forecasts, + outs, test_data=dataset.test_data, metrics=METRICS, seasonality=get_seasonality(dataset.freq), From 8af449f4a7bbc4f74384aed1345801a6b56dea72 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 13:29:32 +0000 Subject: [PATCH 12/13] update --- benchmark/timeseries/fev_bench/README.md | 3 ++- benchmark/timeseries/gift_eval/README.md | 4 ++-- benchmark/timeseries/time_bench/README.md | 4 ++-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/benchmark/timeseries/fev_bench/README.md b/benchmark/timeseries/fev_bench/README.md index a671198071..0f6611b4d5 100644 --- a/benchmark/timeseries/fev_bench/README.md +++ b/benchmark/timeseries/fev_bench/README.md @@ -18,7 +18,8 @@ pip install . fev==0.10.0 python -m benchmark.timeseries.fev_bench.main --model chronos2 ``` - `Chronos2` uses cross-learning across batches of 64 examples, contexts truncated to 8,192 time steps, and `float16` mixed precision. Targets and available covariates are shared within each batch. Evaluation windows are processed separately. + `Chronos2` follows official evaluation with cross-learning across batches, and contexts truncated to 8,192 time steps. + Unlike the official `float32` evaluation, we use `float16` mixed precision for improved efficiency with negligible differences in aggregate benchmark accuracy. - **`TimesFM3`**: diff --git a/benchmark/timeseries/gift_eval/README.md b/benchmark/timeseries/gift_eval/README.md index 23b743f0cf..9a8d2cfcbc 100644 --- a/benchmark/timeseries/gift_eval/README.md +++ b/benchmark/timeseries/gift_eval/README.md @@ -19,8 +19,8 @@ pip install --no-deps "git+https://github.com/SalesforceAIResearch/gift-eval.git python -m benchmark.timeseries.gift_eval.main --model chronos2 ``` - `Chronos2` uses cross-learning across batches of 64 examples, contexts truncated to 8,192 time steps, and `float16` mixed precision. Historical covariates are included when available. Each evaluation window is predicted separately to prevent leakage between rolling windows. - The published reference uses `float32`, targets only, and a batch budget of 100 series, so these settings can produce different scores. + `Chronos2` follows official evaluation with cross-learning across batches, and contexts truncated to 8,192 time steps. + Unlike the official `float32` evaluation, we use `float16` mixed precision for improved efficiency with negligible differences in aggregate benchmark accuracy. - **`TimesFM3`**: diff --git a/benchmark/timeseries/time_bench/README.md b/benchmark/timeseries/time_bench/README.md index f2dfdb51ad..70a257fec2 100644 --- a/benchmark/timeseries/time_bench/README.md +++ b/benchmark/timeseries/time_bench/README.md @@ -19,8 +19,8 @@ pip install --no-deps "git+https://github.com/zqiao11/TIME.git@3ca5c41d71c76f3c7 python -m benchmark.timeseries.time_bench.main --model chronos2 ``` - `TimesFM3` follows official evaluation with symmetric averaging over two estimators, contexts truncated to 15,360 time steps and up to 32 variates per forward pass. - Unlike the official `float32` evaluation, we use `bfloat16` mixed precision for improved efficiency with negligible differences in aggregate benchmark accuracy. + `Chronos2` follows official evaluation with contexts truncated to 8,192 time steps. + Unlike the official `float32` evaluation, we use `float16` mixed precision for improved efficiency with negligible differences in aggregate benchmark accuracy. - **`TimesFM3`**: From 2584bbbab6b13117cda78e04e72b46cea7999b33 Mon Sep 17 00:00:00 2001 From: rusty1s Date: Fri, 30 Oct 2026 13:39:16 +0000 Subject: [PATCH 13/13] update --- docs/source/api/models.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/source/api/models.rst b/docs/source/api/models.rst index 9a7b49bc25..421874aa5d 100644 --- a/docs/source/api/models.rst +++ b/docs/source/api/models.rst @@ -49,6 +49,12 @@ Overview - 2.44M - `Apache-2.0 `__ - `Apache-2.0 `__ + * - | :class:`~sdm.models.Chronos2` + | (`Paper `__) + - 2025-10-17 + - 119.48M + - `Apache-2.0 `__ + - `Apache-2.0 `__ * - | :class:`~sdm.models.TimesFM3` | (`Blog `__) - 2026-08-31