Robust AI-based DDoS Attack Detection in Complex Scenarios

The world is now pervasively connected, and billions of Internet of Things (IoT) devices are integrated across healthcare, energy, transportation, industry, and homes, so local faults can quickly escalate into systemic, cross-border risks. In this context, Distributed Denial-of-Service (DDoS) attacks can disrupt safety-critical services, degrade industrial control systems, and propagate through heterogeneous cloud/edge infrastructures. In fact, modern botnets execute multi-vector DDoS attacks, combining volumetric floods with application-layer attacks that overwhelm bandwidth and server resources while outpacing existing detection approaches. Artificial Intelligence (AI)-based detection is further constrained by scarce and imbalanced label datasets, distribution shifts across time and domains, strict latency and resource budgets at gateways and edge nodes, and privacy compliance limits on central aggregation. Meanwhile, the lowered barrier to data poisoning and model manipulation raises the stakes for robust, privacy-preserving AI. Existing approaches for DDoS detection span signature and rule-based systems, statistical anomaly detection, and classical Machine Learning (ML). Yet signatures and hand-tuned thresholds break under unseen variants and adaptive adversaries, while traditional models depend on brittle manual features. Deep Learning (DL) improves detection, but many studies rely on outdated datasets, lack principled model tuning, and struggle with minority attack classes. Centralized training also concentrates sensitive traffic, raising privacy and compliance concerns. Federated learning reduces data movement, but real deployments face non-IID data, device variability, inefficient random client selection, and exposure to poisoning, backdoor, and evasion attacks. The thesis aims to deliver a robust AI-based DDoS attack detection in complex scenarios by introducing an adaptive, scalable, privacy-preserving, and resilient framework. It examines how to maintain accuracy, efficiency, and trustworthiness at scale and addresses this with a four-part approach. First, a centralized deep-learning baseline standardizes preprocessing, mitigates class imbalance, reduces dimensionality, and applies systematic hyperparameter tuning to deliver accurate and adaptive detection at low false-positive rates. Second, TransferEdge reuses pretrained models with dataset- aware adaptation, enabling edge systems to sustain accuracy with fewer labels and lower computational complexity under distribution shift. Third, Federated Learning with Adaptive Client Selection (FELACS) introduces in federated learning an adaptive client- selection strategy that prioritizes high-impact participants to accelerate convergence and improve accuracy. Fourth, Federated Learning Isolation Forest with Robust Aggregation (FLIFRA) realizes a dual-layer poisoning defence that filters suspicious updates on devices and down weights suspect contributions at the server. Evaluation across multiple public datasets and stress conditions shows faster learning, higher accuracy, and stronger robustness under realistic constraints. The results obtained within the thesis indicate that a carefully tuned centralized baseline provides strong detection capability under realistic constraints; TransferEdge enables computationally efficient adaptation as environments shift; and, in distributed settings, FELACS speeds learning and improves accuracy by prioritizing the most informative clients, while FLIFRA strengthens resilience against adversarial manipulation without exposing raw data. Collectively, these components deliver a robust end-to-end, privacy- preserving, and attack-resilient AI framework for scalable DDoS detection in complex IoT ecosystems.