-
Notifications
You must be signed in to change notification settings - Fork 307
Add Wan I2V/T2V example. #2483
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
changwangss
wants to merge
16
commits into
main
Choose a base branch
from
wangchang/wan
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Add Wan I2V/T2V example. #2483
Changes from 6 commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
e9b3dc8
add wan example i2v/t2v task
changwangss 9c2ea40
improve benchmark
changwangss a875b79
update readme and add multi-gpus support and shard dataset
changwangss 50ca62f
Merge branch 'master' into wangchang/wan
changwangss 3503937
add ratio for evaluate_i2v.py
changwangss 1469b24
improve mxfp8 oom by row chunk
changwangss ce5fa32
Merge branch 'master' into wangchang/wan
changwangss 630863c
remove requirement
changwangss d020210
add requirements
changwangss bfd62b5
add example readme
changwangss 2433e9d
update VBench install method
changwangss d8d5a2e
add vbench_dir and gdown==4.7.3, imageio-ffepeg
changwangss ff3a209
Merge branch 'main' into wangchang/wan
changwangss c7fa3ec
add setup.sh
chensuyue 5adf041
Update setup.sh
changwangss b2968eb
i2v/t2v use quantized model loading directly
changwangss File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
159 changes: 159 additions & 0 deletions
159
examples/pytorch/diffusion_model/diffusers/wan/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| # Step-by-Step | ||
|
|
||
| This example provides a unified Wan entry for quantization and evaluation, with both t2v and i2v support. | ||
|
|
||
| # Prerequisite | ||
|
|
||
| ## 1. Environment | ||
|
|
||
| ```shell | ||
| pip install -r requirements.txt | ||
| # Use latest dev branch if needed before release | ||
| # INC_PT_ONLY=1 pip install git+https://github.com/intel/neural-compressor.git@master | ||
| # pip install git+https://github.com/intel/auto-round.git@main | ||
| pip install neural-compressor-pt | ||
| pip install auto-round | ||
| ``` | ||
|
|
||
| ## 2. Prepare Model | ||
|
|
||
| Use a local Wan diffusers model path, for example: | ||
|
|
||
| - Wan2.2-T2V-A14B-Diffusers | ||
| - Wan2.2-I2V-A14B-Diffusers | ||
|
|
||
| Download example (from Hugging Face): | ||
|
|
||
| ```bash | ||
| pip install -U "huggingface_hub[cli]" | ||
|
|
||
| # t2v model | ||
| huggingface-cli download Wan-AI/Wan2.2-T2V-A14B-Diffusers \ | ||
| --local-dir /path/to/Wan2.2-T2V-A14B-Diffusers | ||
|
|
||
| # i2v model | ||
| huggingface-cli download Wan-AI/Wan2.2-I2V-A14B-Diffusers \ | ||
| --local-dir /path/to/Wan2.2-I2V-A14B-Diffusers | ||
| ``` | ||
|
|
||
| ## 3. Prepare Dataset | ||
| Clone VBench to prepare the required dataset, then download i2v data: | ||
|
|
||
| ```bash | ||
| # recommended: install VBench from pip | ||
| python3 -m pip install VBench | ||
|
|
||
| # required for dataset preparation | ||
| git clone https://github.com/Vchitect/VBench.git | ||
| cd VBench | ||
| bash vbench2_beta_i2v/download_data.sh | ||
| ``` | ||
|
|
||
| - t2v: pass prompt folder with --prompt_folder, and set --dimension to select `${prompt_folder}/${dimension}.txt` | ||
| - t2v/i2v: pass comma-separated values in `--dimension` to run multiple dimensions in one command (e.g., `subject_consistency,overall_consistency`) | ||
| - t2v: can pass --dimension for evaluation filtering (validated dimensions include `subject_consistency` and `overall_consistency`) | ||
| - i2v: pass --image_folder, --info_json, and --dimension (validated dimensions include `i2v_subject`, `i2v_background`, `subject_consistency`, `background_consistency`, and `motion_smoothness`) | ||
|
|
||
| # Run | ||
|
|
||
| ## Quantization | ||
|
|
||
| ```bash | ||
| # topology supports wan_mxfp8 or wan_fp8 | ||
| bash run_quant.sh \ | ||
| --topology=wan_mxfp8 \ | ||
| --input_model=/path/to/Wan2.2-T2V-A14B-Diffusers \ | ||
| --task=t2v \ | ||
| --output_model=wan_mxfp8_model | ||
| ``` | ||
|
|
||
| ## Inference + Evaluation | ||
|
|
||
| ### t2v bf16 | ||
|
|
||
| ```bash | ||
| bash run_benchmark.sh \ | ||
| --topology=wan_bf16 \ | ||
| --input_model=/path/to/Wan2.2-T2V-A14B-Diffusers \ | ||
| --task=t2v \ | ||
| --dimension=subject_consistency,overall_consistency \ | ||
| --prompt_folder=/path/to/VBench/prompts/prompts_per_dimension/ \ | ||
| --output_video_path=wan_t2v_bf16_video \ | ||
| --accuracy | ||
| ``` | ||
|
|
||
| ### t2v mxfp8/fp8 | ||
|
|
||
| ```bash | ||
| # topology supports wan_mxfp8 or wan_fp8 | ||
| bash run_benchmark.sh \ | ||
| --topology=wan_mxfp8 \ | ||
| --input_model=/path/to/Wan2.2-T2V-A14B-Diffusers \ | ||
| --quantized_model=wan_mxfp8_model \ | ||
| --task=t2v \ | ||
| --dimension=subject_consistency,overall_consistency \ | ||
| --prompt_folder=/path/to/VBench/prompts/prompts_per_dimension/ \ | ||
| --output_video_path=wan_t2v_mxfp8_video \ | ||
| --accuracy | ||
| ``` | ||
|
|
||
| ### i2v bf16 | ||
|
|
||
| ```bash | ||
| bash run_benchmark.sh \ | ||
| --topology=wan_bf16 \ | ||
| --input_model=/path/to/Wan2.2-I2V-A14B-Diffusers \ | ||
| --task=i2v \ | ||
| --dimension=i2v_background,i2v_subject \ | ||
| --image_folder=/path/to/VBench/vbench2_beta_i2v/data/crop/16-9 \ | ||
| --info_json=/path/to/VBench/vbench2_beta_i2v/vbench2_i2v_full_info.json \ | ||
| --output_video_path=wan_i2v_bf16_video \ | ||
| --accuracy | ||
| ``` | ||
|
|
||
| ### i2v mxfp8/fp8 | ||
|
|
||
| ```bash | ||
| # topology supports wan_mxfp8 or wan_fp8 | ||
| bash run_benchmark.sh \ | ||
| --topology=wan_mxfp8 \ | ||
| --input_model=/path/to/Wan2.2-I2V-A14B-Diffusers \ | ||
| --quantized_model=wan_mxfp8_model \ | ||
| --task=i2v \ | ||
| --dimension=i2v_background,i2v_subject \ | ||
| --image_folder=/path/to/VBench/vbench2_beta_i2v/data/crop/16-9 \ | ||
| --info_json=/path/to/VBench/vbench2_beta_i2v/vbench2_i2v_full_info.json \ | ||
| --output_video_path=wan_i2v_mxfp8_video \ | ||
| --accuracy | ||
| ``` | ||
|
|
||
| Note: For sharding and multi-GPU execution, set `--gpu_ids` (for example `--gpu_ids=0,1,2,3`) or set `CUDA_VISIBLE_DEVICES` before running `run_benchmark.sh`. | ||
|
|
||
| ### Standalone Accuracy Evaluation (Optional) | ||
|
|
||
| If you already use `--accuracy` in `run_benchmark.sh`, you can skip this section. | ||
| Use this section when you want to evaluate existing videos without re-running generation. | ||
|
|
||
| ```bash | ||
| # t2v accuracy on generated videos | ||
| cd /path/to/VBench | ||
| python evaluate.py \ | ||
| --dimension subject_consistency motion_smoothness aesthetic_quality overall_consistency imaging_quality \ | ||
| --videos_path /path/to/wan_t2v_bf16_video \ | ||
| --mode vbench_standard | ||
|
|
||
| # i2v accuracy on generated videos | ||
| cd /path/to/VBench | ||
| python evaluate_i2v.py \ | ||
| --dimension i2v_background i2v_subject subject_consistency background_consistency motion_smoothness \ | ||
| --videos_path /path/to/wan_i2v_bf16_video \ | ||
| --ratio 16-9 \ | ||
| --mode vbench_standard | ||
| ``` | ||
|
|
||
| # Notes | ||
|
|
||
| - Quantized weights are saved under: | ||
| - <output_model>/transformer | ||
| - <output_model>/transformer_2 | ||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.