Data Engineer
Job Location:
Monthly Salary:
Posted on:
21 hours ago
Posted on:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
Position Summary:
We are seeking a talented Databricks Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing building and maintaining data pipelines using a variety of technologies with a focus on the Microsoft Azure cloud platform.
Responsibilities:
- Design develop and implement data pipelines using Azure Data Factory (ADF) or other orchestration tools.
- Write efficient SQL queries to extract transform and load (ETL) data from various sources into Azure Synapse Analytics.
- Utilize PySpark and Python for complex data processing tasks on large datasets within Azure Databricks.
- Collaborate with data analysts to understand data requirements and ensure data quality.
- Hands-on experience in designing and developing Datalakes and Warehouses
- Implement data governance practices to ensure data security and compliance.
- Monitor and maintain data pipelines for optimal performance and troubleshoot any issues.
- Develop and maintain unit tests for data pipeline code.
- Work collaboratively with other engineers and data professionals in an Agile development environment.
Must Have tech skills
- Databricks
- Fabric
- Sql
- Pyspark
- Python
Must Have competencies
- Pipeline - Data Ingestion and data quality transformation Databricks ADF
- Consumption of rest api
- Optimization of pyspark job sql procedure views
Nice to have ( These are design and admin skills)
- Warehouse and lake house creation
- Modeling
- Semantic model design
- Refresh management
- Logging and monitoring
- Rls and permissions
New Vision is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees
Required Experience:
IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
