Skip to content

Re-derive dARSdS and correct the apparent mean open and shut times - #20

Merged
remislp merged 2 commits into
masterfrom
fix-apparent-means
Sep 4, 2026
Merged

remislp merged 2 commits into
masterfrom
fix-apparent-means

Conversation

@remislp

@remislp remislp commented Sep 4, 2026 •

Copy link
Copy Markdown
Member

exact_mean_open_shut_time was too large by exactly twice the dead time, and HJC_adjacent_mean_open_to_shut_time_pdf by once.

The cause is one sign in qmatlib.dARSdS.

The derivation

HJC give the Laplace transform as

AR*(s) = [sI - QAA - QAF M(s) QFA]^-1 ,
M(s)   = (I - exp(-(sI - QFF) tres)) (sI - QFF)^-1

Factorising as CHS96 eqn (3.6) does, pulling out (sI - QAA):

AR*(s) = inv(VA(s)) inv(U(s)) ,  U(s) = sI - QAA ,
VA(s)  = I - inv(U(s)) QAF M(s) QFA

Differentiating the product of two inverses reproduces exactly the DARS formula already in the docstring, so the outer structure was never in question — only dVAds.

With N(s) = sI - QFF, M'(0) = -tres expQFF inv(QFF) - SFF inv(QFF)^2, and therefore

dVAds = -inv(QAA) GAF SFF GFA
        - tres GAF expQFF GFA
        - GAF SFF inv(QFF) GFA

Two of those three terms were right. The dead-time term was coded positive. That made dVAds too large by 2 tres GAF expQFF GFA, and every apparent mean duration built on it too long by 2 tres.

What changes

With the sign fixed, the three call sites add tres rather than subtracting it — which is what the theory says, because the density is written in u = t - tres and integrates to 1 (phi inv(VA) GAF expQFF u = 1, tested).

The numbers do not change. The correction now lives where the error was, instead of being cancelled three times downstream — and anyone calling dARSdS directly gets a correct answer for the first time.

How it was checked

Against references that owe nothing to the implementation:

  • A numerical derivative of AR*(s), formed straight from its definition using nothing else in the library, at three dead times. A flipped sign fails it. This is the test that pins the root cause.
  • Colquhoun & Hawkes (1995b), eqns (121) and (122). For a two-state channel the values reproduce the closed form exactly, at five dead times.
  • The asymptotic HJC pdf. For CH82 and the AChR mechanism the result agrees with tres + sum(area * tau) — a different route through the library.
  • A long simulation, and tres = 0, where the apparent means must become the ideal ones.
  • On a mechanism with one open state the conditional mean open time must be flat against adjacent shut time, and is only with the correction.

Also in this PR

HJC_adjacent_mean_open_to_shut_range_mean — the mean apparent open time for openings whose adjacent shut time falls in a given range. This is what is superimposed on the observed conditional means in Fig. 6D of Colquhoun, Hatton & Hawkes (2003), and is not the continuous relationship, which cannot be compared with data directly because the data must be binned. It needs the apparent shut time density as a weight, so HJC_shut_time_pdf is exposed on its own.

Test changes

The finite-dead-time reference values in test_real_dtype.py move with the correction, and for a second, smaller reason recorded there: popen.EC50 now converges on the concentration rather than stopping as soon as Popen is within 0.001 of half maximal. Nothing about the real-dtype cast those tests guard has changed.

tests/test_apparent_means.py is new and carries the checks above.

Verification

590 passed, 1 skipped.

Found while reproducing Colquhoun, Hatton & Hawkes (2003), where the corrected means are needed for Fig. 6D; that reproduction's own step checks pass against this branch.

🤖 Generated with Claude Code

remislp and others added 2 commits September 4, 2026 13:45
exact_mean_open_shut_time was too large by exactly twice the dead time, and
HJC_adjacent_mean_open_to_shut_time_pdf by once. The algebra in neither was
wrong; the offset assumed for the dARSdS term was. That term evaluates to the
mean apparent duration plus tres, not minus it, so the mean is recovered by
subtracting tres rather than adding it.

Checked against references that owe nothing to the implementation: for a
two-state channel it now reproduces eqns (121) and (122) of Colquhoun & Hawkes
(1995b) exactly, and for CH82 and the AChR mechanism it agrees with the mean
of the asymptotic HJC pdf, tres + sum(area * tau), which reaches the same
quantity by a different route through the library. At tres = 0 the apparent
means become the ideal ones, as they must.

The root cause is the convention of qmatlib.dARSdS, which has not been
re-derived against CHS96 eqn (3.6). The correction is applied at each call
site instead, where what is wanted is unambiguous. Anyone calling dARSdS
directly still inherits the same trap.

Adds HJC_adjacent_mean_open_to_shut_range_mean, the mean apparent open time
for openings whose adjacent shut time falls in a given range -- what is
superimposed on the observed conditional means in Fig. 6D of Colquhoun, Hatton
& Hawkes (2003), and not the same thing as the continuous relationship, which
cannot be compared with data directly because the data must be binned. It
needs the apparent shut time density as a weight, so HJC_shut_time_pdf is
exposed on its own rather than remaining a denominator inside the conditional
mean.

The finite-dead-time reference values in test_real_dtype.py move with the
correction. They also move for a second, smaller reason recorded there:
popen.EC50 was made to converge on the concentration rather than stopping as
soon as Popen was within 0.001 of half maximal. Nothing about the real-dtype
cast those tests guard has changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 2*tres offset patched at the call sites in the previous commit had a
single cause, now found by doing the derivation the docstring only sketched.

HJC give AR*(s) = [sI - QAA - QAF M(s) QFA]^-1 with
M(s) = (I - exp(-(sI - QFF) tres)) (sI - QFF)^-1. Factorising as CHS96 eqn
(3.6) does, AR*(s) = inv(VA(s)) inv(U(s)) with U(s) = sI - QAA and
VA(s) = I - inv(U) QAF M(s) QFA, and differentiating the product of two
inverses gives exactly the DARS formula already in the docstring. So the
outer structure was right and only dVAds was in question.

Differentiating M with N(s) = sI - QFF gives
M'(0) = -tres expQFF inv(QFF) - SFF inv(QFF)^2, and hence

  dVAds = -inv(QAA) GAF SFF GFA - tres GAF expQFF GFA - GAF SFF inv(QFF) GFA

Two of those three terms were coded correctly. The dead-time term was coded
POSITIVE. That made dVAds too large by 2 tres GAF expQFF GFA, and every
apparent mean duration built on it too long by 2 tres.

With the sign fixed, the callers add tres rather than subtracting it, which
is what the theory says: the density is written in u = t - tres and
integrates to 1, since phi inv(VA) GAF expQFF u = 1. The numbers do not
change -- all four independent checks in test_apparent_means.py still hold,
and the 590 tests pass -- but the correction now lives where the error was
instead of being cancelled three times downstream. Anyone calling dARSdS
directly gets a correct answer for the first time.

Three tests added. The one that matters differentiates AR*(s) numerically
straight from its definition, using nothing else in the library, and compares
with dARSdS at three dead times; a flipped sign fails it. The others pin the
normalisation phi inv(VA) GAF expQFF u = 1 and the resulting
mean = DARS term + tres.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@remislp remislp changed the title Correct the apparent mean open and shut times Re-derive dARSdS and correct the apparent mean open and shut times Sep 4, 2026
@remislp
remislp merged commit 0b1c5c6 into master Sep 4, 2026
9 checks passed
@remislp
remislp deleted the fix-apparent-means branch September 4, 2026 13:22
remislp added a commit that referenced this pull request Sep 11, 2026
Seven merged pull requests have been waiting since 1.1.0, and one of them is
now blocking work elsewhere: HJCFIT's fitting layer is duck-typed on a
mechanism and needs Rate.is_free, which exists here on master and not in the
published 1.1.0. Its [fitting] extra pins scalcs>=1.1.0, which is too loose --
the integration tests fail on the released wheel with "'Rate' object has no
attribute 'is_free'".

A minor rather than a patch release, because it corrects computed values as
well as adding API.

  #17  root bracketing for the asymptotic pdf. Counts were not recomputed when
       a search bound moved, widening happened once where it needed to repeat,
       and the upper bound was compared against the wrong state count. det[W(s)]
       is also unusable at large |s|, where it is of order 1e53 and changes sign
       hundreds of times in noise, so roots are now refined by bisecting the
       eigenvalue count where the determinant cannot be trusted. Roots are
       validated before being returned.
  #18  scsim re-exports burst segmentation from dcio; one implementation in the
       stack rather than two.
  #20  the dARSdS sign. exact_mean_open_shut_time was too large by exactly
       twice the dead time and the conditional means by once. This changes
       numbers a caller will have seen.
  #21  impose_resolution checked against RESHJC2.FOR interval for interval.
  #22  the simulator checked against Q-matrix theory for multi-state
       mechanisms, not only the two-state case where a reciprocal rate is the
       answer.
  #23  qmatlib.H inverted sI - QFF directly, which fails where the root search
       walks. The singularity is removable; only the factored form has a pole.
  #24  GPL-2.0-or-later rather than GPL-2.0-only, without which HJCFIT
       (GPL-3.0-or-later) could not depend on this at all.

Both version files are bumped; the versions-agree gate added in #16 is what
makes that hard to get wrong. Verified: the built wheel reports 1.2.0, carries
its three data files, declares GPL-2.0-or-later, and Rate.is_free is present
when installed from it into a clean environment.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant