sft

SFT Training

Supervised fine-tuning code for the two synthetic tasks studied in the paper — graph coloring (gc) and Sudoku — covering the standard SFT and the two perception-rebalancing interventions: fixed loss reweighting and tuning-free NGDiff.

Note: we currently release training code for Qwen3-VL only.

Setup

First, install PyTorch matching your CUDA setup (see the official guide):

torch==2.9.1
torchaudio==2.9.1
torchdata==0.11.0
torchvision==0.24.1

Then install the remaining dependencies:

pip install -r requirements.txt

Finally, install LLaMA-Factory pinned to a specific commit. This code builds on LLaMA-Factory and imports its internals directly, so other revisions may break:

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
git checkout ddd7dcc7223e4232f06af417148cd3a161c81e04
pip install -e .

Data

Training uses our distilled SFT data, converted to LLaMA-Factory's ShareGPT format and published at xqwu/asymmetric-VLM__lf (two splits, gc and sudoku). It is registered in data/dataset_info.json, which points LLaMA-Factory straight at the Hub repo — no local download needed.

Training

The commands below assume TASK=gc or TASK=sudoku. Each method has its own config under config/$TASK/:

Config	Method
standard_sft.yaml	Standard SFT — plain token-averaged cross-entropy.
sft_reweight_fixed.yaml	Fixed reweighting — weight the perception (caption) loss by a fixed perception_loss_ratio ($\lambda$).
sft_reweight_ngdiff.yaml	NGDiff — tuning-free gradient-norm balancing; no $\lambda$ to set.

Each training target is a perception span (<caption>...</caption>) followed by a reasoning span. Perception occupies only a small fraction of the tokens, so the standard objective under-trains it — the two interventions rebalance the perception and reasoning loss to fix this.

Run from this directory:

# Standard SFT baseline
python src/train.py --config config/$TASK/standard_sft.yaml

# Fixed-ratio perception reweighting
python src/train.py --config config/$TASK/sft_reweight_fixed.yaml

# NGDiff dynamic balancing (tuning-free)
python src/train.py --config config/$TASK/sft_reweight_ngdiff.yaml

• Each run is configured for 4×80GB GPUs (e.g. 4× A100 80G).
• For fixed reweighting, tune perception_loss_ratio (default 0.2 in the config).