Azure Data Engineer

Hyderabad - India

Monthly Salary: Not Disclosed

Posted on: 7 hours ago

Vacancies: 1 Vacancy

Job Summary

1. Validate pre-requisites for each data source including schema authentication and connectivity.

2. Duplicate and configure ingestion pipelines in Azure Data Factory (ADF) for new data sources.

3. Execute ingestion pipelines to land raw/JSON files into Azure Data Lake Storage Gen2 bronze layer.

4. Monitor ingestion jobs troubleshoot errors and re-run failed pipelines as needed.

5. Document ingestion run metrics (duration volume error logs) for each source.

6. Provision access permissions and share datasets with the Quality team for validation.

7. Collaborate with the Quality team to address ingestion gaps; repeat ingestion if validation fails.

8. Maintain ingestion activity logs operational dashboards and progress reports.

9. Work closely with source system owners to resolve access or data format issues.

10. Ensure adherence to established standards for pipeline configuration monitoring and logging.

Roles & Responsibilities

1. Gather and analyze business requirements for data ingestion and transformation processes.

2. Design and develop end-to-end data pipelines based on source system requirements.

3. Create and maintain Databricks notebooks for data processing transformation and validation.

4. Integrate and execute Databricks notebooks using Azure Data Factory (ADF) pipelines.

5. Develop configure and manage ADF pipelines for multiple data sources.

6. Monitor pipeline executions and troubleshoot failures to ensure reliable data delivery.

7. Optimize pipeline performance and implement best practices for scalability and reliability.

8. Actively track work items by updating JIRA tickets and maintaining Azure DevOps tasks.

1. Validate pre-requisites for each data source including schema authentication and connectivity. 2. Duplicate and configure ingestion pipelines in Azure Data Factory (ADF) for new data sources. 3. Execute ingestion pipelines to land raw/JSON files into Azure Data Lake Storage Gen2 bronze layer. 4. ...