Modern Trends in Automating ETL Pipelines in Azure
Keywords:
ETL, Azure, automation, Microsoft Fabric, Azure Data Factory, Azure Databricks, Data Lakehouse, ELT, data pipeline, serverlessAbstract
The study is aimed at systematizing and analyzing contemporary trends in the automation of ETL pipelines within the Microsoft Azure cloud ecosystem. The objective of the work is to identify key paradigms, toolsets and architectural approaches, as well as to develop a scientifically grounded model for selecting an optimal technology stack depending on the specifics of business tasks. The methodological basis includes a comprehensive analysis of current scientific publications, technical documentation and industry reports, as well as a comparative evaluation of the leading Azure services: Data Factory, Databricks and the newest Microsoft Fabric platform. As a result of the study, the dominant trends have been identified: the shift from classical ETL to ELT, large-scale adoption of serverless architectures, active development of low-code/no-code solutions and the emergence of the Data Lakehouse concept as a universal data repository. Within the framework of the work, a decision matrix for selecting an automation tool is proposed, based on the criteria of transformation complexity and the need for an integrated analytics platform. It is concluded that the evolution of automation tools in Azure is progressing from a set of disparate services toward fully integrated platform solutions, which fundamentally changes the methodology of data lifecycle design and management. The results of the study are of practical value for data architects and engineers, as well as for IT department leaders responsible for developing and implementing data management strategies in a cloud environment.
References
[1]. Reinsel, D., Gantz, J., & Rydning, J. (2018). The digitization of the world – From edge to core. IDC. https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf (date of request: 05.05.2025).
[2]. Nambiar, A., & Mundra, D. (2022). An overview of data warehouse and data lake in modern enterprise data management. Big Data and Cognitive Computing, 6(4), 1–24. https://doi.org/10.3390/bdcc6040132
[3]. Pavao, A., Abid, A., Racah, E., Roesch, K., Yosinski, J., & Vartak, M. (2023). Codalab competitions: An open source platform to organize scientific challenges. Journal of Machine Learning Research, 24(198), 1-6.
[4]. Armbrust, M., Das, T., Zhu, S., Sen, R., Saha, A., Xin, R. S., ... & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. Proceedings of CIDR, 8, 1–8.
[5]. Shutaywi, M., & Kachouie, N. N. (2021). Silhouette analysis for performance evaluation in machine learning with applications to clustering. Entropy, 23(6), 1–17. https://doi.org/10.3390/e23060759
[6]. Tirupati, K. K., Hussain, M. A., Alam, M. M., Rauf, H. T., & Alshazly, H. (2023). Advanced techniques for data integration and management using Azure Logic Apps and ADF. International Journal of Progressive Research in Engineering Management and Science, 3(12), 460–475.
[7]. Singu, S. K. (2021). Designing scalable data engineering pipelines using Azure and Databricks. ESP Journal of Engineering & Technology Advancements, 1(2), 176–187.
[8]. Borra, P. (2024). Microsoft Fabric review: Exploring Microsoft's new data analytics platform. International Journal of Computer Science and Information Technology Research, 12(2), 34–39. https://doi.org/10.5281/zenodo.11502585
[9]. Habib, G., Hussain, M., Faheem, M., Rauf, H. T., Alshazly, H., & Alharbi, A. (2022). Blockchain technology: Benefits, challenges, applications, and integration of blockchain technology with cloud computing. Future Internet, 14(11), 1–22. https://doi.org/10.3390/fi14110341
Downloads
Published
Issue
Section
License
Copyright (c) 2025 American Scientific Research Journal for Engineering, Technology, and Sciences

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who submit papers with this journal agree to the following terms.