Senior Data Engineer

Senior Data Engineer with 5 years of experience

Mandatory Skills:

-Extensive experience in developing ETL pipelines

- Databricks

-scheduling using Azure Data Factory (ADF) and/or Apache Airflow

- performance tuning using SQL.

- SQL Python Spark PySpark and Azure.

- Fabric experience preferred.

Key Responsibilities:

• ETL Pipeline Development:

• Design develop and maintain ETL pipelines to extract transform and load data from various sources into Azure data platforms.
• Optimize ETL processes for performance and scalability.
• Implement data quality and validation checks within ETL processes.

• Scheduling and Orchestration:

• Develop and manage data integration workflows using Azure Data Factory (ADF) and/or Apache Airflow.
• Schedule and monitor ETL jobs to ensure timely data processing and availability.

• Performance Tuning:

• Optimize SQL queries and data processing tasks for performance.
• Utilize notebooks for development and performance tuning of ETL processes.

• Data Integration and Management:

• Develop and maintain data models and schemas.
• Ensure data integrity consistency and security across data platforms.
• Collaborate with data engineers solution architects and other stakeholders to understand business requirements and translate them into technical solutions.

Qualifications:

• Bachelors degree in computer science Information Technology or a related field.
• 5-6 years of experience as a Data Engineer in developing ETL pipelines and scheduling using ADF and/or Apache Airflow.
• Strong technical skills in SQL Python Spark PySpark and Azure.
• Proven experience in performance tuning and optimization of SQL and ETL processes.
• Strong understanding of Data Architecture and Data Modeling.
• Experience in healthcare and pharmaceutical industries is preferred.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Preferred Skills:

• Knowledge of big data technologies (Hadoop Hive Kafka).
• Familiarity with data visualization tools (Power BI Tableau).
• Microsoft Fabric experience.

Required Experience:

Senior IC