Azure Data Engineer
Share
Job Location:
Monthly Salary:
Posted on:
4 hours ago
Vacancies:
1 Vacancy
Job Summary
Job Title- Azure Data Engineer
Location - Toronto ON
Fulltime position
Skills Required:
Strong understanding of data warehousing data lakes and best practices for data quality security and performance.
Design and Development Design develop deploy and maintain scalable data pipelines and ETL
ELT processes using Azure Data Factory (ADF) Azure Databricks and other Azure data services (e.g. Azure Data Lake Storage Azure Synapse Analytics).
Python PySpark Programming Implement efficient reusable and testable code for data transformation cleansing and analysis using Python and PySpark within Databricks notebooks and jobs.
Orchestration and Integration Use ADF pipelines to orchestrate data movement and trigger Databricks notebooks or jobs ensuring seamless integration between various data sources and destinations.
Data Management Manage data storage ensure data integrity optimize data processing and troubleshoot performance issues related to Databricks and ADF solutions.
Collaboration Collaborate closely with data scientists data analysts and business stakeholders to gather requirements and translate business needs into technical specifications and data solutions. Monitoring Security Implement monitoring error handling and security best practices across all data workflows and ensure compliance with data governance standards.
Documentation Create and maintain comprehensive technical documentation for data architectures pipelines and processes.
Required Skills and Qualifications Experience Proven experience in data engineering data warehousing or similar
Strong understanding of data warehousing data lakes and best practices for data quality security and performance.
Design and Development Design develop deploy and maintain scalable data pipelines and ETL
ELT processes using Azure Data Factory (ADF) Azure Databricks and other Azure data services (e.g. Azure Data Lake Storage Azure Synapse Analytics).
Python PySpark Programming Implement efficient reusable and testable code for data transformation cleansing and analysis using Python and PySpark within Databricks notebooks and jobs.
Orchestration and Integration Use ADF pipelines to orchestrate data movement and trigger Databricks notebooks or jobs ensuring seamless integration between various data sources and destinations.
Data Management Manage data storage ensure data integrity optimize data processing and troubleshoot performance issues related to Databricks and ADF solutions.
Collaboration Collaborate closely with data scientists data analysts and business stakeholders to gather requirements and translate business needs into technical specifications and data solutions. Monitoring Security Implement monitoring error handling and security best practices across all data workflows and ensure compliance with data governance standards.
Documentation Create and maintain comprehensive technical documentation for data architectures pipelines and processes.
Required Skills and Qualifications Experience Proven experience in data engineering data warehousing or similar
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
