Benchmarking data leakage and generalization in audio classification: an empirical analysis

Audio classification models are vulnerable to subtle forms of data leakage, which can lead to overly optimistic and unreliable performance results. This issue is particularly important in edge computing environments, where models are deployed on resource-limited devices and retraining after deployment is not practical. This study investigates data leakage, especially those related to data splitting and augmentation strategies. We propose a structured, leakage-aware evaluation pipeline that employs source-aware data splitting, followed by the application of data augmentation exclusively to the training set after partitioning. Using the UrbanSound8K benchmark dataset, our experiments show that neglecting this precaution can lead to up to a 25% inflation in model accuracy. These findings underscore the significant gap in generalization between naive and rigorously designed evaluation protocols. Our results emphasize the need for careful pipeline design to develop reproducible, trustworthy, and deployable audio classification models. To isolate and evaluate the impact of the pipeline design, we deliberately refrain from optimizing model performance, focusing instead on ensuring a fair and controlled comparison.