Towards Real-Time Network Intrusion Detection: An Enhanced Feature Selection Framework Using Variance Analysis and K-Means Clustering

Abstract

The cybersecurity threat landscape is increasingly dynamic, making Network Intrusion Detection Systems (NIDS) essential to cyber defense. NIDS face challenges such as network traffic imbalance, high false alarm rates, and suboptimal detection accuracy. To address these issues, we integrated variance analysis and k-means clustering, thereby enhancing computational efficiency across seven machine learning models evaluated on the UNSW-NB15 and NSL-KDD datasets. Key metrics included Accuracy, Precision, Recall, F1-Score, Model Training Time (MTT), and Inference Time (IT). Our feature selection technique achieved notably lower MTT and IT than existing methods, particularly with the XGB and RF models, which saw MTT drop by 80% and IT by 60% compared to the baseline. This efficiency is pivotal for real-time intrusion detection and resource-constrained NIDS, enabling swift threat response and optimized resource use. These computational gains emphasize the technique's potential for real-time network security.