Repository navigation
audio2vid_ltx2(): long audio-driven video in one call - #85
Draft
TroyHernandez wants to merge 2 commits into
Draft
TroyHernandez wants to merge 2 commits into
TroyHernandez wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
audio2vid_ltx2()renders a narration of any length as one call: a chain of LTX windows, the first from the start image, each later window conditioned on the previous window's lastcarryframes as latents, every window driven by the narration over its own span. It is the loop that cornductor's timeline layout runs today one request at a time through xtx.api and the gpu host, moved into the package that owns the model, so a host can serve a whole track as one job and the latents never leave it.windowNN.mp4(carried head included, with its audio),windowNN.latents.pt(tail latents) andwindowNN.json(plan, prompt, seed). A second call resumes at the first window missing or out of plan, from the saved latents; once one window renders again, every later one does.on_windowreports each window and may stop the run after it, leaving resumable windows: the checkpoint a scheduler needs to cut in between windows....goes totxt2vid_ltx2().ltx2_windows()is the default plan, the same layout ascornductor::timeline_windows()(97-frame windows, 25 carried, the last sized to the end on the 8k+1 grid). A caller's own plan (phrase windows) is checked for a valid chain.Found by the tests:
torch_save()of the narrowed tail view writes the start of its storage, so the saved tail is copied first; without that, a resumed chain continued from the window's first latent frames.Tests:
test_audio2vid_ltx2.R, 51 assertions with a fake generator, CPU only. Not yet run against the model on a card; that smoke is next.