Skip to content

Add Filtergram.ratio_noise() to measure the noise of XRT images from consecutive frames - #6

Merged
roytsmart merged 2 commits into
mainfrom
feature/measured-noise
Oct 12, 2026
Merged

roytsmart merged 2 commits into
mainfrom
feature/measured-noise

Conversation

@roytsmart

@roytsmart roytsmart commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Adds hinode.xrt.Filtergram.ratio_noise(noise_photon, where=True, threshold=10 DN/s, gap=25 s, clip=5, num_iteration=3). It returns the ratio of the variance of the noise measured in a sequence of images to the variance of Filtergram.uncertainty().

This is the measurement from the "Uncertainty of XRT" section of the DEM tutorial, moved into the library. The tutorial now calls it, and the DEM movie of event E for the ESIS jet paper uses it.

What it does

  • Pairs compared: each image and the next, only if the next begins after it and within gap, with the same exposure (to 0.01 s), and with the same plate scale and pointing, so that their first pixels land within a pixel of each other on the Sun. Images out of time order, a file loaded twice, or interleaved programs with another field or binning are not compared.
  • Pixels used: those selected by where in both images (it may differ from image to image), not saturated in either, with a finite z, and whose mean over the two images is fainter than threshold. Selecting on the mean rather than on each image keeps the largest differences near threshold.
  • Statistic: z = (I₂ − I₁) / √(σ₁² + σ₂²). The ratio starts from the median of z² over that of Gaussian noise, which outliers barely move, and is then the mean of z² with z² > clip²·ratio left out, repeated num_iteration times.
  • Axes: it reduces over the time and detector axes only, so other axes, such as those of noise_photon, give one ratio each.
  • Errors: it raises ValueError if there are fewer than two images, or if no pair has a pixel to measure the noise in, instead of returning NaN.

The docstring explains why the ratio is below one for XRT: charge diffusion and the JPEG compression. It also notes that the ratio is measured in faint pixels, and that the errors of the dark and vignetting models are the same in both images and cancel in the difference, so the ratio describes only the noise that changes from image to image.

uncertainty() now takes the square root of a private _variance(), which ratio_noise() uses directly.

Checked against the tutorial

On the tutorial's frames from 18:06:11 to 18:11:01, with the 128-pixel box excluded, it gives 0.1527. The tutorial's old inline code gave 0.1525. The correlations of neighboring pixels in the next cell are unchanged (0.425 inside a block, 0.282 across an edge). All 21 cells of the tutorial run.

Tests

On synthetic flat sequences with noise of a known ratio of 0.2:

  • test_ratio_noise: the ratio comes back to 2%.
  • test_ratio_noise_pairs: a pair with a different exposure and one with a long gap are skipped; a longer gap lets the second in.
  • test_ratio_noise_order: an image that began before the previous one, and a duplicate, are skipped.
  • test_ratio_noise_aligned: an image two pixels off and a binned one are skipped; one half a pixel off is compared.
  • test_ratio_noise_excluded: cosmic rays are clipped, and the median start already ignores them; brighter pixels and a changing region outside where are left out.
  • test_ratio_noise_outliers: a 9σ change in 5% of the pixels of one image is clipped (the old mean start gave 0.30).
  • test_ratio_noise_threshold: pixels at the threshold give 0.207 (selecting on each image gave 0.078).
  • test_ratio_noise_where_time, _saturated, _axes, _timedelta, _nan, _none: a per-image where, saturated pixels, an extra axis, a scalar exposure, a NaN in noise_photon, and the errors.

The whole suite passes (255 passed, 1 skipped). black --check and ruff check pass, and pyright 1.1.414 reports the 4 errors main already has.

🤖 Generated with Claude Code

https://claude.ai/code/session_011VGPHLGJi8acbuLYSfBHTk

…e of images from the difference between each image and the next.

The measurement of the DEM tutorial, as a method, so that the DEM of every
frame of a movie can scale its uncertainty the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011VGPHLGJi8acbuLYSfBHTk
@read-the-docs-community

read-the-docs-community Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 hinode | 🛠️ Build #35097545 | 📁 Comparing 9d04f8b against latest (62b50ea)

  🔍 Preview build  

4 files changed
± genindex.html
± _autosummary/hinode.xrt.Filtergram.html
± tutorials/dem.html
± _modules/hinode/xrt/_filtergrams.html

@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.86%. Comparing base (62b50ea) to head (9d04f8b).

Additional details and impacted files
@@            Coverage Diff             @@
##             main       #6      +/-   ##
==========================================
+ Coverage   99.84%   99.86%   +0.01%     
==========================================
  Files          18       18              
  Lines        1968     2160     +192     
==========================================
+ Hits         1965     2157     +192     
  Misses          3        3              
Flag Coverage Δ
unittests 99.86% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Compare an image with the next only if the next begins after it, so
  images out of time order or loaded twice are not compared, and only if
  both have the same plate scale and point within a pixel of each other.
- Leave out pixels whose `z` is not finite, saturated pixels, and raise a
  `ValueError` when there are fewer than two images or no pixel to measure
  the noise in, instead of returning NaN.
- Select pixels by the mean of the two images instead of requiring both
  below `threshold`, which left out the largest differences: for noise of
  0.2 at the threshold that gave 0.08, and now gives 0.21.
- Start the clipping from the median of z^2, so outliers in many pixels
  cannot raise the first estimate enough to survive it.
- Reduce over the time and detector axes only, giving a ratio for each
  entry of the other axes, and accept a `where` per image and a scalar
  `timedelta`.
- Take the intervals by named indexing, find the variance once with a
  shared `_variance()`, and note in the docstring that the errors of the
  dark and vignetting models cancel in the difference.
- The tutorial calls the method, and its example leaves out the jet.

On event E the ratio is 0.1527 (0.1525 before).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011VGPHLGJi8acbuLYSfBHTk
@roytsmart
roytsmart merged commit 19ab6ff into main Oct 12, 2026
12 checks passed
@roytsmart
roytsmart deleted the feature/measured-noise branch October 12, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant