Skip to content

audio2vid_ltx2(): long audio-driven video in one call - #85

Draft
TroyHernandez wants to merge 2 commits into
mainfrom
feat/audio2vid-ltx2
Draft

TroyHernandez wants to merge 2 commits into
mainfrom
feat/audio2vid-ltx2

Conversation

@TroyHernandez

Copy link
Copy Markdown
Contributor

audio2vid_ltx2() renders a narration of any length as one call: a chain of LTX windows, the first from the start image, each later window conditioned on the previous window's last carry frames as latents, every window driven by the narration over its own span. It is the loop that cornductor's timeline layout runs today one request at a time through xtx.api and the gpu host, moved into the package that owns the model, so a host can serve a whole track as one job and the latents never leave it.

  • Windows are written as they finish: windowNN.mp4 (carried head included, with its audio), windowNN.latents.pt (tail latents) and windowNN.json (plan, prompt, seed). A second call resumes at the first window missing or out of plan, from the saved latents; once one window renders again, every later one does.
  • on_window reports each window and may stop the run after it, leaving resumable windows: the checkpoint a scheduler needs to cut in between windows.
  • One prompt or one per window; a loaded pipeline or a resident LTX handle; ... goes to txt2vid_ltx2().
  • ltx2_windows() is the default plan, the same layout as cornductor::timeline_windows() (97-frame windows, 25 carried, the last sized to the end on the 8k+1 grid). A caller's own plan (phrase windows) is checked for a valid chain.

Found by the tests: torch_save() of the narrowed tail view writes the start of its storage, so the saved tail is copied first; without that, a resumed chain continued from the window's first latent frames.

Tests: test_audio2vid_ltx2.R, 51 assertions with a fake generator, CPU only. Not yet run against the model on a card; that smoke is next.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant