Sr. Data Engineer Azure Databricks
Share
Job Location:
Monthly Salary:
Posted on:
Vacancies:
Job Summary
Position Summary:
We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing building and maintaining data pipelines using a variety of technologies with a focus on the Microsoft Azure cloud platform.
Responsibilities:
- Design develop and implement data pipelines using Azure Data Factory (ADF) or other orchestration tools.
- Write efficient SQL queries to extract transform and load (ETL) data from various sources into Databricks.
- Utilize PySpark and Python for complex data processing tasks on large datasets within Azure Databricks.
- Databricks is Mandatory
- Collaborate with data analysts to understand data requirements and ensure data quality.
- Hands-on experience in designing and developing Datalakes and Warehouses
- Implement data governance practices to ensure data security and compliance.
- Monitor and maintain data pipelines for optimal performance and troubleshoot any issues.
- Develop and maintain unit tests for data pipeline code.
- Work collaboratively with other engineers and data professionals in an Agile development environment.
Preferred Skills & Experience:
- Good knowledge of PySpark & working knowledge of Python
- Full stack Azure Data Engineering skills (Azure Data Factory DataBricks and Synapse Analytics)
- Experience with large dataset handling
- Hands-on experience in designing and developing Datalakes and Warehouses
New Vision is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees
Required Experience:
Senior IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
