Lead Data Engineer Azure Databricks & DevOps
Share
Job Location:
Monthly Salary:
Posted on:
30+ days ago
Vacancies:
1 Vacancy
Job Summary
We are looking for a highly skilled Senior Data Engineer with a strong background in Azure Databricks Apache Spark Python SQL and DevOps practices. The ideal candidate will design and implement robust scalable and high-performing data solutions while ensuring smooth deployment and automation through CI/CD frameworks.
Responsibilities
Design develop and optimize data pipelines and ETL workflows using Azure Databricks and Apache Spark (PySpark).
Build scalable data ingestion and transformation frameworks integrating multiple structured and unstructured data sources.
Implement and manage DevOps practices for data workflows including CI/CD version control and automation.
Collaborate with cross-functional teams to deploy and monitor data applications in Azure environments.
Write efficient and reusable Python and SQL scripts for data cleaning transformation and analytics.
Implement data lakehouse architectures using Azure Data Lake and Delta Lake.
Ensure performance tuning cost optimization and adherence to data security best practices.
Troubleshoot monitor and resolve production issues related to data pipelines and infrastructure
Qualifications :
6 years of experience in Data Engineering or related domains.
Strong hands-on experience in Azure Databricks Apache Spark and PySpark.
Advanced proficiency in Python and SQL for data engineering tasks.
Experience with DevOps tools and concepts including:
CI/CD pipelines (Azure DevOps GitHub Actions or Jenkins)
Version control (Git)
Infrastructure as Code (ARM templates Terraform preferred)
Solid understanding of Azure Data Factory Azure Data Lake and Azure Synapse.
Proven experience in data pipeline orchestration monitoring and automation.
Strong analytical problem-solving and debugging skills.
Ability to work in a fast-paced collaborative environment with cross-functional teams.
Remote Work :
No
Employment Type :
Full-time
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Blend360 is an award-winning provider of data, analytics, and talent solutions for Fortune 500 companies. The company has made the Inc. 5000 list of Fastest Growing Companies every year they have been in business and has been awarded a world-class ranking in client satisfaction for th ... View more
