To work as a Data Engineer where I can use my skill set in software engineering to deliver dependable and scalable solutions that meet organization goals and expectations.
Designing, developing, and implementing Azure-based data solutions, including data lakes, data warehouses, and data processing systems.
Building and optimizing data pipelines for extracting, transforming, and loading (ETL) data from various sources into Azure environments.
Collaborating with data architects, data scientists, analysts, and other stakeholders to understand data requirements and translate them into technical solutions.
Extracting data from multiple data sources and creating end-to-end ETL pipelines using Azure data factory.
Creating pipelines to perform various data loads like Batch and Bulk load, Sequential load, Incremental extracts and CDC.
Transforming the data and using Control flow activities on the data as per the business needs and requirements.
Writing SQL queries and filtering data for various data retrieval needs.
Processing Batch data using Azure databricks when required.
Visualizing the data using Databricks notebooks and Power BI for end-user analysis and reporting.
Ensuring data security, integrity, and performance by implementing appropriate Azure data services and best practices.
Monitoring and troubleshooting data solutions to identify and resolve performance issues and bottlenecks.
Also Collaborating with the infrastructure team to ensure the overall health and scalability of the data platform.
Create an ETL Pipeline from On Premise SQL Database to Gold Storage in Data Lake House.
Data profiling and verification of data on-premise database.
Designed and developed a Medallion Architecture for ETL Pipeline.
Created and configured needed Azure ETL resources of Link services, Self-Hosted integration runtime, Auto integration run time, Dataset, Azure Data Lake storage Gen 2, Delta Tables, Databricks Clusters/Compute power, Workspace.
Worked on an Azure copy data to load data from an on-premise SQL server to Azure Data Lake Gen2.
Mounted /Integrated ADLS Gen2 to Databricks for data transformation and loading.
Created 3 tier storage of Bronze, Silver and Gold in Databricks.
Moved data from ADLS Gen2 to Bronze storage in Data Lake House.
Moved data from Bronze storage into Silver Storage for data transformation and processing.
Loaded processed and useable data into the Gold Storage for Analytics, Business intelligence, Machine Learning, Artificial Intelligence purposes.
Created a complete (Extraction, Transform, Load) pipeline in ADF.
Data extracted from numerous sources including AWS S3 bucket, transformed and then loaded into Cloud Storage (ADLS Gen2).
Participated in the development of Azure data modelling.
Designed, implemented and supported on-premise and cloud-based data infrastructure that is resilient to disruptions and failures.
Designed and implemented data acquisition, curation and consumption pipelines using Azure technologies including AWS S3 Bucket, Azure ADLS, and Azure SQL.
Participated in designing and building Azure data platforms supporting both batch and real-time (event-based) architecture.
Understood current application infrastructure and suggested Cloud based solutions which reduced operational cost, required minimal maintenance but provided high availability with improved security.
Create and develop Extraction, Transformation and Loading data pipeline for both batch and incremental loading in Azure ADF.
Provided technical guidance and support to project teams throughout the data engineering lifecycle.
Collaborated with cross functional teams to analyze data requirements and implement data solutions.
Identify and resolve issues concerning data management to improve data quality.
Implemented automation and streamline processes to optimize the entire data and analytics platform, ensuring efficient throughput and high-performance outcomes.
Designing, implementing, and managing data extraction, transformation, and loading (ETL) processes.
Optimize data storage and retrieval mechanisms to ensure the efficient processing of large datasets.
نظام التوظيف التشغيلي المعتمد على الذكاء الاصطناعي أولاً، الذي يوحّد ويؤتمت دورة التوظيف بالكامل للمؤسسات الحديثة حول العالم.
د.جوب منطقة حرة ذ.م.م. 2026 © جميع الحقوق محفوظة
