Skip to content

Please include a note for the workaround regarding lighteval when evaluating tasks with acc_norm #87

Description

@namlh201

Apparently there's a bug happening in lighteval when evaluating tasks with acc_norm right now: huggingface/lighteval#1170
And a possible fix is not merged yet (maybe will never be merged): huggingface/lighteval#1171

Right now, a workaround is to manually incorporate the fix above to the local installation of lighteval, as documented in:
#60

Please make this clear in README.md so that everyone has an idea what to do. Thanks!

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdocumentationImprovements or additions to documentation

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions