Automatically benchmarking biases in ML-based wafs

Web Application Firewalls (WAFs) are the first line of defense, inspecting incoming HTTP requests to detect malicious payloads. Distinguishing between malicious and benign payloads is complex, and universal criteria do not exist. In recent decades, Machine Learning (ML) has emerged as an alternative to rule-based WAFs, using trained models instead of manually crafted rules. However, the reliability of ML-WAFs is questionable, mainly due to recurrent biases and the lack of a standardized assessment framework. This paper introduces a benchmarking framework to systematically identify and measure biases in ML-WAFs, evaluating dataset diversity, training validity, and evaluation robustness. We conduct an in-depth case study on WafBrain and further validate the proposed framework by applying it to the Barracuda WAF, a widely deployed commercial product. Such biases determine inflated performances in controlled environments, but they make WAFs fragile in adversarial settings.