Research · University of Florida
Measuring when generated videos fail to follow a prompt.
A video can look plausible and still omit an object, change an attribute, or show the wrong action. My research focuses on detecting these prompt-consistency failures.
My contribution
- Ran video-generation experiments with Wan 2.1 and HunyuanVideo on NVIDIA B200 GPU clusters using benchmark prompts.
- Contributed to dataset development and annotation, distinguishing prompt inconsistency from physical implausibility.
- Worked on prompt-reconstruction signals, VLM baselines, evaluation pipelines, and error analysis for TRACE.
- Evaluated detection with AUROC, AUPRC, macro F1, balanced accuracy, accuracy, and MCC.
How TRACE works
A frozen Qwen3-VL model receives video keyframes and the original prompt. Under teacher forcing, it scores the true prompt tokens. Token ranks provide signals about words the video may not support, which are aggregated into a video-level detection score.
The method also produces a token evidence map to make potentially unsupported content easier to inspect.
Results and limits
TRACE-L reached 0.738 AUROC on T2V-CompBench-Wan and led the six reported metrics on both evaluated T2V-CompBench corpora.
The 0.738 value measures detection performance on a specific research benchmark; it is not an overall accuracy rate. The 15,000+ generation total and the 4,201-pair annotated corpus describe different stages of the work.
Controls using removed or shuffled frames approached chance, and detection improved with violation severity. These experiments support the role of visual evidence within the evaluated setting; they do not establish reliability across every generator or domain.
Current role
Volunteer Research Assistant, Department of Electrical and Computer Engineering, University of Florida. I work on vision-language and generative model evaluation on HiPerGator.
Contact me about the research