Skip to content

Commit cc7f5ac

Browse files
eval config cleanup
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
1 parent 3b19253 commit cc7f5ac

2 files changed

Lines changed: 12 additions & 11 deletions

File tree

examples/pruning/minitron/NVIDIA-Nemotron-Nano-9B-v2/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,7 @@ python /opt/Megatron-Bridge/examples/conversion/convert_checkpoints.py export \
225225

226226
### 4. Evaluation
227227

228-
The eval config in [nemo_evaluator.yaml](nemo_evaluator.yaml) is for Slurm-based evaluation — it submits a vLLM serving job and runs evals against it. For local model execution and evaluation, refer to the [NeMo Evaluator documentation](https://docs.nvidia.com/nemo/evaluator/latest/).
228+
The eval config xin [nemo_evaluator.yaml](nemo_evaluator.yaml) is for Slurm-based evaluation — it submits a vLLM serving job and runs evals against it. For local model execution and evaluation, refer to the [NeMo Evaluator documentation](https://docs.nvidia.com/nemo/evaluator/latest/) or this [blog](https://huggingface.co/blog/nvidia/nemotron-3-nano-evaluation-recipe).
229229

230230
Before running, update the following fields in the yaml:
231231

examples/pruning/minitron/NVIDIA-Nemotron-Nano-9B-v2/nemo_evaluator.yaml

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -160,16 +160,17 @@ evaluation:
160160
extra:
161161
num_repeats: 64
162162

163-
# 5. MATH500
164-
- name: AA_math_test_500
165-
env_vars:
166-
HF_TOKEN: HF_TOKEN
167-
nemo_evaluator_config:
168-
config:
169-
params:
170-
# limit_samples: 8
171-
extra:
172-
n_samples: 5
163+
# 5. MATH500 (Requires JUDGE_API_KEY)
164+
# - name: AA_math_test_500
165+
# env_vars:
166+
# HF_TOKEN: HF_TOKEN
167+
# JUDGE_API_KEY: JUDGE_API_KEY
168+
# nemo_evaluator_config:
169+
# config:
170+
# params:
171+
# # limit_samples: 8
172+
# extra:
173+
# n_samples: 5
173174

174175
# 6. IFEval
175176
- name: ns_ifeval

0 commit comments

Comments
 (0)