Skip to content

qmatlib.H: remove a pole that was never really there - #23

Merged
remislp merged 1 commit into
masterfrom
fix-H-removable-pole
Sep 10, 2026
Merged

remislp merged 1 commit into
masterfrom
fix-H-removable-pole

Conversation

@remislp

@remislp remislp commented Sep 10, 2026

Copy link
Copy Markdown
Member

The bug

qmatlib.H evaluates

H(s) = QAA + QAF · X⁻¹ (I − exp(−X·tres)) · QFA,      X = sI − QFF

and computed X⁻¹ directly. That breaks down whenever s reaches an eigenvalue of QFF — and the asymptotic root search that calls H walks s across exactly that range, because the roots of the determinant equation are interlaced with those eigenvalues.

It was never really a pole

X⁻¹ (I − exp(−X·t)) is the integral of exp(−Xu) over [0, t]. The offending mode contributes (1 − e^{−μt})/μ, whose limit as μ → 0 is simply t. The singularity is removable; only the factored form has a pole.

integral_expQt evaluates the integral spectrally, taking that limit explicitly with expm1 for accuracy near zero, and H uses it.

How it was found

In the reproduction of Colquhoun, Hatton & Hawkes (2003), 2 of the first 12 fits of one scenario raised LinAlgError("Singular matrix") out of HJC_shut_time_pdf at 10 µM. The rates looked entirely ordinary — α₂ = 2119 against a true 2000.

The pattern turned out to be that both fits had a rate constant sitting at its upper limit of 1e6, which puts an eigenvalue of QAA exactly at −1e6 for the shut-time problem to walk into. All twelve now work.

The worse failure is the silent one

Whether the old form raised or merely returned nonsense depended on whether the eigenvalue was exactly representable:

  • round-numbered rates — a limit of exactly 1e6 — land on the pole and raise;
  • otherwise the matrix is simply catastrophically ill-conditioned and the answer is quietly wrong.

That second case is why this is a fix rather than a guard: a try/except would have papered over the loud failure and left the silent one.

dW is deliberately untouched

dW contains a similar-looking inv(sI − QFF), and its pole is genuine, not removable — W really does have poles at the eigenvalues, which is why the bisection brackets roots between them by design. Changing it would be wrong.

Tests

19 in tests/test_integral_expQt.py. The integral is checked

  • against inv(M)(exp(Mt) − I) on 8 random matrices — agrees to 1e-8;
  • against a quadrature, so the identity is verified rather than asserted;
  • for continuity as an eigenvalue passes through zero (μ = 1e-3 … 0);
  • for giving exactly t at the zero mode.

and H is checked unchanged away from the pole, and finite and continuous through it.

One test failed honestly while I wrote it: I had asserted the old form raises at any computed eigenvalue, and for CH82 it doesn't — the eigenvalue isn't exactly representable, so inv merely becomes ill-conditioned. The test now asserts the property that actually matters (finiteness and continuity) plus a measured condition number, and the docstrings were corrected to match.

Nothing else moves

  • 635 passed, 1 skipped — the full suite including the slow simulator tests.
  • The asymptotic areas still agree with the 2003 Fortran to 5×10⁻⁷ on all seven time constants and areas, which is that program's print precision.

🤖 Generated with Claude Code

H(s) = QAA + QAF X^-1 (I - exp(-X tres)) QFA, with X = sI - QFF, was computed
by inverting X. That breaks down whenever s reaches an eigenvalue of QFF -- and
the asymptotic root search that calls H walks s across exactly that range,
because the roots of the determinant equation are interlaced with those
eigenvalues.

The quantity itself is fine there. X^-1 (I - exp(-X t)) is the integral of
exp(-Xu) over [0, t], and the offending mode contributes (1 - e^{-mu t})/mu,
whose limit as mu -> 0 is t. The singularity is removable; only the factored
form has a pole. integral_expQt evaluates it spectrally, taking that limit
explicitly, and H uses it.

Found in the reproduction of Colquhoun, Hatton & Hawkes (2003): two of the
first twelve fits of one scenario raised LinAlgError from HJC_shut_time_pdf at
10 uM. Both had a rate constant sitting at its upper limit of 1e6, which puts
an eigenvalue of QAA exactly at -1e6 for the shut-time problem to walk into.
All twelve now work.

Whether the old form raised or merely returned nonsense depended on whether the
eigenvalue was exactly representable. Round-numbered rates -- a limit of 1e6 --
land on the pole exactly and raise; otherwise the matrix is simply
catastrophically ill-conditioned and the answer is quietly wrong. The second is
the worse failure and the reason this is a fix rather than a guard.

dW has the same-looking inversion and is deliberately left alone: its pole is
genuine, not removable, and the bisection brackets roots between poles by
design.

19 tests. The integral is checked against inv(M)(exp(Mt) - I) on random
matrices, against a quadrature, for continuity as an eigenvalue passes through
zero, and for giving exactly t at the zero mode; H is checked unchanged away
from the pole and finite and continuous through it. Nothing else moves: 635
pass, and the asymptotic areas still agree with the 2003 Fortran to 5e-7, which
is its print precision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@remislp
remislp merged commit 4dea660 into master Sep 10, 2026
9 checks passed
@remislp
remislp deleted the fix-H-removable-pole branch September 10, 2026 09:51
@remislp remislp mentioned this pull request Sep 11, 2026
remislp added a commit that referenced this pull request Sep 11, 2026
Seven merged pull requests have been waiting since 1.1.0, and one of them is
now blocking work elsewhere: HJCFIT's fitting layer is duck-typed on a
mechanism and needs Rate.is_free, which exists here on master and not in the
published 1.1.0. Its [fitting] extra pins scalcs>=1.1.0, which is too loose --
the integration tests fail on the released wheel with "'Rate' object has no
attribute 'is_free'".

A minor rather than a patch release, because it corrects computed values as
well as adding API.

  #17  root bracketing for the asymptotic pdf. Counts were not recomputed when
       a search bound moved, widening happened once where it needed to repeat,
       and the upper bound was compared against the wrong state count. det[W(s)]
       is also unusable at large |s|, where it is of order 1e53 and changes sign
       hundreds of times in noise, so roots are now refined by bisecting the
       eigenvalue count where the determinant cannot be trusted. Roots are
       validated before being returned.
  #18  scsim re-exports burst segmentation from dcio; one implementation in the
       stack rather than two.
  #20  the dARSdS sign. exact_mean_open_shut_time was too large by exactly
       twice the dead time and the conditional means by once. This changes
       numbers a caller will have seen.
  #21  impose_resolution checked against RESHJC2.FOR interval for interval.
  #22  the simulator checked against Q-matrix theory for multi-state
       mechanisms, not only the two-state case where a reciprocal rate is the
       answer.
  #23  qmatlib.H inverted sI - QFF directly, which fails where the root search
       walks. The singularity is removable; only the factored form has a pole.
  #24  GPL-2.0-or-later rather than GPL-2.0-only, without which HJCFIT
       (GPL-3.0-or-later) could not depend on this at all.

Both version files are bumped; the versions-agree gate added in #16 is what
makes that hard to get wrong. Verified: the built wheel reports 1.2.0, carries
its three data files, declares GPL-2.0-or-later, and Rate.is_free is present
when installed from it into a clean environment.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant