-
The overlooked risk of over‑optimising for benchmark leaderboards
Research Idea
Many research projects are heavily oriented toward leaderboard metrics. But leaderboard performance does not always translate to real‑world robustness. What negative side‑effects have you observed from leaderboard‑driven research?
David Brown, Daniel Lopez and 21 others3 Comments-
Teams may tweak hyperparameters endlessly just to squeeze tiny metric gains.
2 -
Research questions get secondary to chasing higher numbers.
2 -
Leaderboards can create short‑term fashion around certain model architectures.
2
-