Apparently there's a bug happening in lighteval when evaluating tasks with acc_norm right now: huggingface/lighteval#1170
And a possible fix is not merged yet (maybe will never be merged): huggingface/lighteval#1171
Right now, a workaround is to manually incorporate the fix above to the local installation of lighteval, as documented in:
#60
Please make this clear in README.md so that everyone has an idea what to do. Thanks!
Apparently there's a bug happening in
lightevalwhen evaluating tasks withacc_normright now: huggingface/lighteval#1170And a possible fix is not merged yet (maybe will never be merged): huggingface/lighteval#1171
Right now, a workaround is to manually incorporate the fix above to the local installation of
lighteval, as documented in:#60
Please make this clear in README.md so that everyone has an idea what to do. Thanks!