Add MultiModalSymplecticAutoencoder, multimodal losses, and HRedSys reference shift and VJP

CHANGELOG.md

@@ -150,6 +150,8 @@ breaking release).
 
 - **`BilinearStiefelLayer`, `TransposeBilinearStiefelLayer`, `LowRankStiefelLayer`, `DecodeLowRankStiefelLayer`, `FastDLRATraceLoss`, and `solve!` on `BilinearStiefelLayer` add two-sided velocity-parity Stiefel frontends, a gauge-invariant dynamical low-rank Stiefel frontend and decoder, a fast low-rank trace loss, and Tucker-2 HOSVD initialization.** The five new types are exported. `BilinearStiefelLayer(Nx, Nv, mx, mv; metric_factors = nothing)` (`Nx >= 1`, even `Nv >= 2`, `1 <= mx <= Nx`, `1 <= mv <= Nv ÷ 2`) allocates `(Φx = StiefelManifold(...), Φv = StiefelManifold(...))` with ``\Phi_x \in \mathrm{St}(m_x, N_x)`` and ``\Phi_v \in \mathrm{St}(m_v, N_v / 2)``, splits an input array ``F`` of shape `(Nx, Nv)` or `(Nx, Nv, B)` into velocity-even ``F^+ = \frac{1}{\sqrt{2}}(F_{:, 1:N_v/2} + F_{:, N_v:-1:N_v/2+1})`` and velocity-odd ``F^- = \frac{1}{\sqrt{2}}(F_{:, 1:N_v/2} - F_{:, N_v:-1:N_v/2+1})`` components (or accepts a pre-split `(q, p)` `NamedTuple` of shape `(Nx, Nv ÷ 2)` or `(Nx, Nv ÷ 2, B)`), applies optional Cholesky mass factors `(Rx, Rv)` when `metric_factors` is supplied, and projects ``Q_{\mathrm{red}} = \Phi_x^T F^+ \Phi_v`` and ``P_{\mathrm{red}} = \Phi_x^T F^- \Phi_v`` to return vectorized `(q = vec(Q_red), p = vec(P_red))` of half-dimension ``n = m_x m_v``; `TransposeBilinearStiefelLayer` (`inverse_layer(l)`) is its exact right-inverse. `LowRankStiefelLayer(Nx, Nv, mx, mv; r_out = mx * mv, metric_factors = nothing)` takes a DLRA `NamedTuple` `(X = X, S = S, V = V)` of arbitrary rank ``r_{\mathrm{in}} \ge 1`` and evaluates ``Q_{\mathrm{red}} = (\Phi_x^T X) S ((V^+)^T \Phi_v)`` and ``P_{\mathrm{red}} = (\Phi_x^T X) S ((V^-)^T \Phi_v)`` in ``O(r_{\mathrm{in}}(m_x N_x + m_v N_v))`` operations without forming ``X S V^T``, matching `BilinearStiefelLayer` on ``F = X S V^T`` identically and remaining invariant under ``O(r_{\mathrm{in}}) \times O(r_{\mathrm{in}})`` gauge transformations, while `DecodeLowRankStiefelLayer` (`inverse_layer(l)`) computes the truncated SVD of ``[Q_{\mathrm{red}}\;\; P_{\mathrm{red}}]`` of rank ``r_{\mathrm{eff}} = \min(r_{\mathrm{out}}, m_x, 2 m_v)`` and QR re-orthonormalizes the lifted factors to return `(X = X_hat, S = S_hat, V = V_hat)` on the Stiefel manifold. `FastDLRATraceLoss()` evaluates ``\|X S V^T - \hat{X} \hat{S} \hat{V}^T\|_F^2`` in ``O(r^2(N_x + N_v))`` operations via Gram-matrix traces clamped at `zero(T)`. `solve!(nn, F_snapshots)` and `solve!(l, ps, F_snapshots)` initialize `Φx` and `Φv` on a `BilinearStiefelLayer` from the top-``m_x`` spatial and top-``m_v`` parity-velocity left singular vectors of `F_snapshots`. The tests are `test/layers/bilinear_and_low_rank_stiefel.jl`.
 
+- **`MultiModalSymplecticAutoencoder`, `encode`, `decode`, `translate`, `CharbonnierRelativeLoss`, `IsometricRolloutLoss`, and `CrossModalCycleLoss` add multi-modal symplectic compression and multi-modal training losses, and `HRedSys` gains reference-state shifting, reverse-mode VJP vector fields, and preallocated work buffers.** All seven new names are exported. `MultiModalSymplecticAutoencoder(frontends::NamedTuple, core::SymplecticCompression) <: SymplecticCompression` combines a non-empty `NamedTuple` of modality frontends (individual layers or `Chain`s of symplectic particle layers) whose canonical output dimension matches `core.full_dim` with a shared `SymplecticCompression` core; `encode(nn, modality, x)` (with `modality` a `Symbol` or `Val(modality)`) applies `frontends[modality]` followed by `encoder(core)`, `decode(nn, modality, ξ)` applies `decoder(core)` followed by `inverse_layer(frontends[modality])`, and `translate(nn, from, to, x)` computes `decode(nn, to, encode(nn, from, x))`, raising an `ArgumentError` on an empty or dimension-mismatched `frontends` or an unknown modality symbol. `CharbonnierRelativeLoss(; ε = 1e-3, λ_lat = 1e-2)` evaluates the exact-zero Charbonnier relative reconstruction loss ``(\sqrt{\|\hat{x} - x\|_F^2 + \varepsilon^2} - \varepsilon) / \sqrt{\|x\|_F^2 + \varepsilon^2} + \lambda_{\mathrm{lat}} \|\mathrm{encode}(nn, m, \hat{x}) - \mathrm{encode}(nn, m, x)\|_F^2 / (\|x\|_F^2 + \varepsilon^2)`` (supporting low-rank `(X, S, V)` inputs via `FastDLRATraceLoss`), `IsometricRolloutLoss(integrator; α_lat = 1.0, α_rec = 1.0, α_H = 0.1, H = nothing)` evaluates multi-step latent rollout error, decoded trajectory error, and latent Hamiltonian conservation penalty, and `CrossModalCycleLoss(mod_a, mod_b; β_align = 1.0, β_cycle = 0.1)` evaluates paired latent alignment ``\|z_a - z_b\|_F^2`` plus cycle consistency ``\|\mathrm{encode}(nn, b, \mathrm{decode}(nn, b, z_a)) - z_a\|_F^2``. `IsometricRolloutLoss` raises an `ArgumentError` naming both element types when the data, the output of `integrator` or `H` has another element type than the network's parameters, and `CrossModalCycleLoss` does so for `x_a` or `x_b`. `HRedSys`, `build_v_reduced`, and `build_f_reduced` accept optional reference states `q_ref` and `p_ref` (converted to the state element type, shifting the decoder lift by ``(q_{\mathrm{ref}}, p_{\mathrm{ref}})`` and subtracting ``(q_{\mathrm{ref}}, p_{\mathrm{ref}})`` before `rs.encoder` in `integrate_reduced_system`) and `pullback_mode::Symbol = :jacobian | :vjp`, where `:vjp` evaluates ``(\nabla_\xi \Psi^{\mathrm{dec}})^T \mathbb{J}_{2N}^T [v(z); f(z)]`` in one reverse-mode pullback `Zygote.pullback(decoder, ξ)` and both modes use the preallocated buffers of `cache` (`ξ_buf`, `v_intermediate`, `f_intermediate`, and `jac_result` for `:jacobian`, with a per-element-type `IdDict{DataType, Any}` of caches for `ForwardDiff.Dual` Newton states). The default `cache` of `build_v_reduced` and `build_f_reduced` has the element type of the decoder's parameters, and the default `cache` of `HRedSys` has the element type of `ics.q`; a state of another floating-point element type than the cache's raises an `ArgumentError`. The tests are `test/architectures/multimodal_symplectic_autoencoder.jl` and `test/reduced_system/reference_shift_and_vjp.jl`.
+
 ### Changed
 
 - **`SymplecticAttentionQ` and `SymplecticAttentionP` accept a single sample.** A `(q, p)` `NamedTuple` of vectors, and a plain vector, are reshaped to one column, passed through the matrix method and returned as vectors, so the forward layer accepts the same inputs as its inverse. On `main` a `NamedTuple` of vectors and a plain vector raised `MethodError: no method matching _custom_transpose(::Vector{Float64})`. The test is `test/layers/symplectic_attention.jl`.

KNOWN_ISSUES.md

@@ -62,6 +62,43 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   entry means deciding what of an untrained network is worth testing at all — training the two
   networks first would make the comparison mean something, at a cost the suite does not pay.
 
+### K40 · `HRedSys` carries a cache's buffers and its `pullback_mode` in fields that no code in `src/` reads.
+
+- **location:** `src/reduced_system/reduced_system.jl:93`
+- **kind:** dead code
+- **found:** #370
+- **evidence:**
+  The constructor sets `pullback_mode`, `ξ_buf`, `v_intermediate`, `f_intermediate`, `jac_result`
+  and `dual_cache`, and the type parameter `JR` carries the type of `jac_result`. `v_reduced` and
+  `f_reduced` hold the `ReducedVFCache` and the mode in their closures, and every read of
+  `.ξ_buf`, `.v_intermediate`, `.f_intermediate`, `.jac_result` and `.dual_cache` in `src/` is a read
+  of a `ReducedVFCache`, never of an `HRedSys`. `test/reduced_system/reference_shift_and_vjp.jl`
+  asserts the fields with `hasfield`. The fields stay because code outside this package may need
+  them; removing them changes the type's parameters, its layout and those tests.
+
+### K41 · `CrossModalCycleLoss(nn, x_a, x_b)` with a mixed-component `NamedTuple` raises a `MethodError`, not the documented `ArgumentError`.
+
+- **location:** `src/loss/multimodal_losses.jl:323`
+- **kind:** defect
+- **found:** #370
+- **evidence:**
+  The three `(nn, x_a, x_b)` methods (lines 323, 329 and 335) annotate `x_a` and `x_b` with
+  `ArrayOrNamedTuple`, and AbstractNeuralNetworks defines that as
+  `Union{AbstractArray{T}, NamedTupleOfArrays{T}}`, with one element type `T` for every component.
+  A `NamedTuple` whose `q` and `p` differ in element type matches no `T`, so dispatch fails before
+  `_check_eltype` (line 204) runs. In a fresh `julia --startup-file=no --project=.` process:
+
+  ```julia
+  T = typeof((q = zeros(Float32, 2, 3), p = zeros(Float64, 2, 3)))
+  T <: AbstractNeuralNetworks.ArrayOrNamedTuple   # false
+  ```
+
+  The pair form `CrossModalCycleLoss(nn, (x_a, x_b))` takes `Tuple{Any, Any}`, reaches
+  `_check_eltype`, and is tested. The docstring says that another element type raises an
+  `ArgumentError`. Closing this entry means widening the `x_a` and `x_b` annotations of the three
+  methods, for example to `Union{AbstractArray, NamedTuple}`. That is a design call about what the
+  method accepts, and the pull request does not make it.
+
 ## Found late
 
 ### K17 · `assign_q_and_p` drops the rows from `2N + 1` on without an error.

docs/src/architectures/symplectic_autoencoder.md

@@ -136,6 +136,10 @@ We also see that the intermediate dimension in the decoder is `54`  for the spec
 ```@docs
 SymplecticAutoencoder
 WeightTiedSymplecticAutoencoder
+MultiModalSymplecticAutoencoder
+encode
+decode
+translate
 ```
 
 ```@raw latex

docs/src/reduced_order_modeling/losses.md

@@ -46,6 +46,9 @@ where ``\mathbf{x}^{(t)}`` is the solution of the FOM at point ``t`` and ``\math
 TransformerLoss
 AutoEncoderLoss
 ReducedLoss
+CharbonnierRelativeLoss
+IsometricRolloutLoss
+CrossModalCycleLoss
 GeometricMachineLearning.ParametricLoss
 GeometricMachineLearning.apply_parametric
 projection_error

src/GeometricMachineLearning.jl

@@ -255,6 +255,8 @@ include("architectures/linear_symplectic_transformer.jl")
 include("architectures/symplectic_transformer.jl")
 include("architectures/weight_tied_symplectic_autoencoder.jl")
 include("architectures/symmetric_symplectic_autoencoder.jl")
+include("architectures/multimodal_symplectic_autoencoder.jl")
+include("loss/multimodal_losses.jl")
 
 export LinearSymplecticAttention, LinearSymplecticAttentionQ, LinearSymplecticAttentionP
 export LinearSymplecticTransformer
@@ -273,6 +275,8 @@ export BilinearStiefelLayer, TransposeBilinearStiefelLayer
 export LowRankStiefelLayer, DecodeLowRankStiefelLayer
 export FastDLRATraceLoss
 export SymmetricSymplecticAutoencoder
+export MultiModalSymplecticAutoencoder, encode, decode, translate
+export CharbonnierRelativeLoss, IsometricRolloutLoss, CrossModalCycleLoss
 
 include("map_to_cpu.jl")