In this talk we walk through a real world Hollywood use case for Open Source AI that can save hundreds of thousands of dollars for a shoot. Open source models such as WAN VACE and LTX-2 represent unified frameworks for video creation and editing, offering powerful capabilities for professional video manipulation. This talk presents a comprehensive workflow for achieving Hollywood-level video edits using this open-source model, addressing the practical challenges content creators face when attempting AI-powered video editing.
We demonstrate an end-to-end pipeline for wardrobe replacement in existing footage, using the specific example of transforming Samuel L. Jackson's attire with Eddie Murphy's recognizable jacket. The workflow begins with masked video preparation, where we identify target regions for editing using tools like After Effects or Segment Anything 2. We then incorporate pose estimation to guide the model and maintain realistic body positioning throughout the edit.
A critical component of our approach involves custom LoRA fine-tuning. While zero-shot prompting produces impressive initial results, we demonstrate that fine-tuned models are essential for achieving production-ready quality. Our process includes downloading reference footage, extracting frames, generating captions with language models, and training custom LoRAs directly through the Oxen.ai platform. The platform handles GPU infrastructure automatically, streamlining what would otherwise be a complex training process.
The ComfyUI workflow integrates multiple inputs: masked videos, pose guidance, custom LoRAs, and acceleration LoRAs for faster iteration. We explain how each component feeds through nodes into the sampler, creating a cohesive editing pipeline.
Through before-and-after comparisons, we highlight common pitfalls in AI video editing, including unrealistic lighting, texture issues, and artificial-looking results, and demonstrate how strategic fine-tuning overcomes these limitations. This workflow has significant implications for film production, enabling cost-effective post-production corrections and reducing the need for expensive reshoots when continuity errors or wardrobe changes are discovered after principal photography.
