AI-Driven Optimization of Data Ingestion and Transformation in Cloud Systems
Main Article Content
Abstract
The rapid evolution of cloud systems has underscored the need for more efficient methods of data ingestion and transformation. This study introduces an AI-driven framework designed to optimize these critical processes by leveraging advanced machine learning techniques. By dynamically adapting to varying workloads and heterogeneous data formats, the proposed approach streamlines resource allocation and minimizes latency, thereby enhancing overall system performance. Comprehensive experiments demonstrate significant improvements in processing speed and data quality, validating the framework’s ability to meet the demands of modern cloud environments. These findings pave the way for more resilient and scalable cloud architectures that can autonomously manage complex data pipelines, ultimately contributing to more efficient data-driven decision-making in diverse application domains
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The license allows re-users to share and adapt the work, as long as credit is given to the author and don't use it for commercial purposes.
References
• Calheiros, R. N., Ranjan, R., Beloglazov, A., De Rose, C. A., & Buyya, R. (2011). CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software: Practice and Experience, 41(1), 23–50.
• Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), Article 15.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
• Li, Y., Chen, M., & Li, K. (2017). Data ingestion and transformation techniques for cloud-based big data analytics. Journal of Cloud Computing, 6(1), 15–28.
• Patil, P., & Deshpande, D. (2018). AI-enabled resource management in cloud computing: A survey. International Journal of Cloud Applications and Computing, 8(3), 45–67.
• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.
• Zhang, H., Huang, Z., & Li, Q. (2019). Optimization strategies for data pipelines in cloud computing environments. IEEE Transactions on Cloud Computing, 7(2), 345–357.
• Zhao, X., Li, F., & Chen, Y. (2020). Integrating AI for dynamic data processing in cloud systems. In Proceedings of the International Conference on Cloud Computing and Big Data (CCBD) (pp. 112–119).
• Kumar, S., & Patel, R. (2019). Machine learning approaches for anomaly detection in data transformation pipelines. Journal of Data Science and Analytics, 12(4), 289–305.
• Nguyen, T. M., & Tran, Q. (2020). A framework for AI-based resource scaling in cloud environments. International Journal of Computer Science, 18(2), 123–137.
• Roberts, J., & Martinez, L. (2018). Big data analytics in the cloud: Challenges and solutions. Journal of Cloud Computing Research, 4(1), 45–59.
• Wang, X., Liu, S., & Chen, P. (2021). Real-time data ingestion using deep learning: A cloud-based approach. IEEE Access, 9, 45678–45689.
• Silva, A., & Souza, R. (2019). Adaptive cloud resource management using machine learning techniques. International Journal of Cloud Technology, 7(3), 210–226.
• Gupta, A., & Kumar, N. (2021). Enhancing data quality in cloud systems with AI-driven anomaly detection. Journal of Information Systems, 15(2), 89–102.
• Mendez, F., & Torres, E. (2020). Hybrid approaches for data transformation in heterogeneous cloud environments. IEEE Journal of Cloud Computing, 8(4), 301–312.
• Rahman, M., & Alam, S. (2022). Evaluating the performance of AI-based optimization in cloud data processing. Proceedings of the International Conference on Cloud and Big Data Computing, 33(2), 100–115.
• Perez, L., & Garcia, R. (2019). Dynamic resource allocation in cloud computing using reinforcement learning. Journal of Advanced Cloud Systems, 10(3), 234–248.
• Chen, Y., & Zhao, Q. (2021). Deep learning applications for optimizing cloud data ingestion. In Proceedings of the IEEE International Conference on Data Engineering (ICDE) (pp. 78–85).
• Thomas, E., & Lin, W. (2020). Cost optimization in cloud computing using AI: A case study approach. Journal of Cloud Economics, 5(2), 117–130.
• Rodriguez, M., & Kim, S. (2022). Future directions in AI-driven cloud systems: Challenges and opportunities. IEEE Transactions on Emerging Topics in Computing, 10(1), 12–25.