Benchmark contamination
Benchmark contamination: The presence in training data of examples identical or similar to those used for model evaluation. It can inflate results by measuring memorization instead of generalization. Overlap analysis, private benchmarks and temporal tests reduce the problem. Examples include Public questions present in pretraining, Near-duplicates in the test set, Private validation benchmark.
In italiano: Benchmark ContaminationLegge V · Overfitting, bias e collasso dei modelliIt can inflate results by measuring memorization instead of generalization. Overlap analysis, private benchmarks and temporal tests reduce the problem.
Examples
- Public questions present in pretraining
- Near-duplicates in the test set
- Private validation benchmark