Senior Data Engineer

Job Description

ROLE:

We are seeking an experiencedSenior Data Engineerto join our data engineering this role you will design develop and optimize large-scale data pipelines and architectures using modern cloud technologies. You will work with cross-functional teams to build robust ETL/ELT solutions manage data lakes and lakehouses and ensure high-performance data processing systems that support critical business intelligence and analytics initiatives.

Key Responsibilities

• Design and implement scalable data pipelines using Azure Data Factory and Azure Databricks
• Develop and optimize ETL/ELT processes for large-scale distributed data processing
• Build and maintain data lake and lakehouse architectures on Azure cloud platform
• Write efficient SQL and PySpark code for data transformation and analysis
• Implement Lakeflow Declarative Pipelines for streamlined data workflows
• Perform performance tuning and optimization of data processing systems
• Manage version control using Git and implement CI/CD pipelines for data solutions
• Collaborate with data scientists analysts and stakeholders to understand data requirements
• Ensure data quality security and governance standards are met across all pipelines
• Mentor junior team members and contribute to technical best practices
• Required Skills & Experience
• Mandatory Requirements
• Azure Data Factory (ADF):Strong hands-on experience in designing and managing data pipelines
• Azure Databricks:Proficiency in building and optimizing distributed data processing solutions
• Apache Spark:In-depth knowledge of Spark architecture and distributed computing concepts
• Lakeflow Declarative Pipelines:Practical experience with declarative pipeline development
• SQL:Expert-level proficiency in writing complex queries and data manipulation
• PySpark:Strong programming skills in PySpark for data transformation
• Azure Cloud Platform:Hands-on experience with Azure data services and cloud infrastructure
• ETL/ELT Concepts:Strong understanding of data integration patterns and methodologies
• Data Warehousing Principles:Knowledge of dimensional modeling schema design and best practices
• Distributed Data Processing:Experience working with large-scale data systems and handling big data challenges
• Performance Tuning & Optimization:Ability to identify bottlenecks and optimize data pipeline performance
• Data Lake & Lakehouse Architectures:Understanding of modern data architecture patterns and implementation
• Git & CI/CD:Experience with version control systems and continuous integration/deployment practices
• Preferred Skills & Experience
• Experience with Delta Lake and Unity Catalog for data governance
• Knowledge of streaming frameworks and real-time data processing
• Exposure to Azure Synapse Analytics or Microsoft Fabric
• Working experience in Agile/Scrum development environments
• Familiarity with data governance frameworks and security best practices

Qualifications

Bachelors degree in Computer Science Engineering Mathematics or related field (or equivalent professional experience)
Minimum 5 years of experience as a Data Engineer or similar role
Proven track record of delivering production-grade data solutions
Strong problem-solving and analytical skills
Excellent communication and collaboration abilities
Ability to work independently and as part of a team

Required Skills:

AZURE DATA FACTORYAZUREApache