feat(metrics): add Perplexity metric to ignite.metrics.nlp

[steaphenai]

Closes #3742

Summary

• Add a new Perplexity metric implementation in ignite.metrics.nlp.perplexity.
• Export Perplexity from both ignite.metrics.nlp and top-level ignite.metrics.
• Add dedicated tests for correctness, accumulation behavior, reset behavior, return type, and invalid inputs.

Test plan

• python -m pytest tests/ignite/metrics/nlp/test_perplexity.py -v
• Smoke test:
  • python -c "from ignite.metrics.nlp import Perplexity; import torch; ppl = Perplexity(); ppl.reset(); ppl.update((torch.randn(2,5,3), torch.randint(0,5,(2,3)))); print('PPL =', ppl.compute())"

Files changed

• ignite/metrics/nlp/perplexity.py
• ignite/metrics/nlp/__init__.py
• ignite/metrics/__init__.py
• tests/ignite/metrics/nlp/test_perplexity.py

[Prathamesh8989]

Nice addition! Perplexity is definitely a useful metric for language modeling and it fits well under ignite.metrics.nlp.

The test coverage looks solid — especially the token-weighted accumulation test, which ensures correctness across batches with different sequence lengths.

One small suggestion: it might be useful to add a GPU test to ensure the metric behaves correctly when tensors are on CUDA devices, since many language modeling workloads run on GPU.

Something like:

def test_gpu_support():
    if not torch.cuda.is_available():
        pytest.skip()

Overall the implementation and tests look clean and consistent with existing Ignite metrics.

[steaphenai]

Good point, thanks. I’d like to keep this PR scoped to the Perplexity implementation and core correctness tests. We can add a dedicated CUDA test if maintainers want explicit GPU coverage.
@vfdev-5 thoughts?

[vfdev-5]

@steaphenai thanks for the PR, I made a quick pass and left few comments.

The tests look shallow and there is no reference implementation that we test against.
I suggest to check what we can use for reference implementation.
In terms of testing on accelerators check other tests like test_accuracy.py to inspire from.

[steaphenai]

@steaphenai thanks for the PR, I made a quick pass and left few comments.

The tests look shallow and there is no reference implementation that we test against. I suggest to check what we can use for reference implementation. In terms of testing on accelerators check other tests like test_accuracy.py to inspire from.

Thanks for the quick review, @vfdev-5.
I addressed the two code comments in the latest push:
detached tensors from the grad graph in Perplexity.update()
removed test_returns_float
I’ll also use existing metric tests (e.g., test_accuracy.py) as reference for accelerator-oriented test patterns as needed.

[steaphenai]

I added an explicit reference implementation check (_reference_perplexity) and validated both single-batch and multi-batch token-weighted accumulation against it and also I aligned test structure with existing Ignite metric patterns (as in test_accuracy.py): available_device parametrization, device assertions, and distributed-marked test layout.
Local run: python -m pytest tests/ignite/metrics/nlp/test_perplexity.py -m "not distributed" -v -> 10 passed, 12 skipped (CUDA/MPS skipped when unavailable).

[vfdev-5]

@steaphenai code style check is failing: https://github.com/pytorch/ignite/actions/runs/24830927009/job/72725024776?pr=3743

[steaphenai]

@vfdev-5 Could you approve the workflows to run? The required checks are pending approval.

[vfdev-5]

@steaphenai this failure is real: https://github.com/pytorch/ignite/actions/runs/24847148226/job/72879291595?pr=3743
Check other metrics how do they handle double dtype

[steaphenai]

@vfdev-5 I checked other metrics in ignite.metrics.nlp. I found that BLEU and ROUGE don't force dtype=torch.double on their accumulators. I've removed the explicit dtype=torch.double and dtype=torch.long from _sum_of_nll and _num_tokens in reset(), and removed dtype=torch.double from the .to() call in update() to match that pattern and fix MPS compatibility.

[TahaZahid05]

hi @steaphenai ! are you still working on this? if you are facing any issue, let us know!

[steaphenai]

Hi @TahaZahid05 ! Yes, I'm still working on this. This needs a workflow approval to get the required checks running. Thanks!

[TahaZahid05]

@steaphenai docs build failures are real. CI for unit tests are currently broken, are they passing locally for you?

[Copilot]

Pull request overview

This PR adds a new Perplexity NLP metric to ignite.metrics.nlp, exposes it via public imports (ignite.metrics.nlp and ignite.metrics), adds test coverage (including distributed integration), and updates the metrics documentation list.

Changes:

• Implement ignite.metrics.nlp.Perplexity with token-weighted NLL accumulation and distributed reduction support.
• Export Perplexity from ignite.metrics.nlp and top-level ignite.metrics.
• Add unit + distributed tests and include the metric in the docs metrics list.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
ignite/metrics/nlp/perplexity.py	Adds the new Perplexity metric implementation and its public API/docs.
ignite/metrics/nlp/init.py	Re-exports Perplexity from the NLP metrics namespace.
ignite/metrics/init.py	Re-exports Perplexity from the top-level metrics namespace.
tests/ignite/metrics/nlp/test_perplexity.py	Adds correctness, accumulation/reset, and distributed integration tests for Perplexity.
docs/source/metrics.rst	Adds Perplexity to the published metrics list.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[TahaZahid05]

@steaphenai some small changes required. You can ignore copilot's review as i have already incorporated them in my review. thanks!

[TahaZahid05]

@steaphenai thanks for the updates! All looks good, just apply the above suggestions and then we can merge.

[TahaZahid05]

@steaphenai code style checks are failing. Kindly follow steps given in CONTRIBUTING.MD