AI-Driven Anomaly Detection in Cloud Based Data Pipelines
Main Article Content
Abstract
Cloud-based data pipelines are critical to handling vast amounts of information in the modern digital age; they are, however, prone to data quality-related issues and process disruptions. This research presents an artificial intelligence-driven framework for detecting anomalies in the context of cloud systems, where data is continually generated and processed. By incorporating advanced machine learning algorithms with statistical monitoring techniques, the proposed framework identifies anomalies and potential threats in real-time, hence reducing false positives and maximizing the overall system reliability. Empirical testing using real-world data sets confirms the scalability and robustness of the framework, pointing to its ability to scale with emerging cloud infrastructures. The findings confirm the ability of artificial intelligence to not only improve anomaly detection but also optimize resource usage and improve security mechanisms in modern cloud-based data pipelines
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The license allows re-users to share and adapt the work, as long as credit is given to the author and don't use it for commercial purposes.
References
• Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 1–58.
• Ahmed, M., Mahmood, A. N., & Hu, J. (2016). A survey of network anomaly detection techniques. Journal of Network and Computer Applications, 60, 19–31.
• Xu, X., Li, J., & Zhao, Y. (2018). Machine learning techniques for cloud-based anomaly detection: A review. IEEE Cloud Computing, 5(6), 58–66.
• Zhang, Y., Li, X., & Wang, Q. (2019). Anomaly detection in cloud systems using unsupervised deep learning. IEEE Transactions on Cloud Computing, 7(4), 1037–1048.
• Kumar, P., & Singh, A. (2020). Hybrid machine learning models for anomaly detection in distributed data pipelines. Journal of Big Data Analytics, 4(1), 25–39.
• Lee, S., Park, J., & Kim, H. (2021). Autoencoder-based deep anomaly detection in cloud computing environments. International Journal of Advanced Computer Science and Applications, 12(6), 375–384.
• Patel, R., Gupta, M., & Sharma, N. (2022). LSTM networks for anomaly detection in time-series data from cloud infrastructures. IEEE Access, 10, 2100–2111.
• Garcia, M., & Chen, H. (2023). Hybrid supervised-unsupervised learning for robust anomaly detection in cloud-based systems. Journal of Cloud Computing, 12(1), 1–18.
• Liu, X., Zhao, L., & Sun, Y. (2017). Real-time anomaly detection using deep neural networks. In Proceedings of the IEEE Conference on Big Data (pp. 567–576).
• Verma, S., & Gupta, N. (2018). Scalable anomaly detection in cloud infrastructures. IEEE Transactions on Services Computing, 11(2), 287–299.
• Wang, J., Li, W., & Hu, F. (2019). Data preprocessing techniques for anomaly detection in heterogeneous cloud data. Journal of Data and Information Quality, 10(3), 1–22.
• Li, Y., Zhang, F., & Chen, R. (2020). Continuous learning and feedback in AI-based anomaly detection systems. International Journal of Data Science, 8(2), 110–127.
• Chen, L., Zhou, D., & Huang, X. (2021). Explainable AI for anomaly detection in cloud computing. Journal of Intelligent Systems, 30(4), 655–670.
• Ahmed, F., Khan, S., & Ali, M. (2022). Cloud monitoring and anomaly detection using hybrid machine learning. IEEE Transactions on Network Science and Engineering, 9(1), 12–24.
• Miller, D., Thompson, R., & Jones, E. (2019). Distributed data pipeline management and anomaly detection in the cloud. Journal of Cloud Engineering, 5(3), 45–59.
• Silva, T., & Rodrigues, M. (2020). Federated learning approaches for secure anomaly detection in cloud environments. IEEE Cloud Computing, 7(1), 34–42.
• Kumar, R., & Zhao, Y. (2021). A review of anomaly detection techniques in cloud computing. Journal of Computer Networks, 18(2), 159–174.
• Smith, J., Brown, K., & Wilson, A. (2018). Adaptive anomaly detection with reinforcement learning in cloud systems. ACM SIGKDD Explorations, 20(2), 50–59.
• Patel, V., & Lee, H. (2020). Anomaly detection using statistical methods and machine learning in cloud pipelines. Journal of Big Data, 7(1), 33–47.
• Robinson, M., Carter, S., & Nguyen, T. (2022). Future trends in AI-driven anomaly detection for cloud infrastructure. IEEE Internet Computing, 26(5), 20–28.