Azure Data Engineer

• 6-7 years of experience in data engineering working with Azure Databricks PySpark and SQL.
• Proficiency in Python for data processing and transformation.
• Expertise in machine learning for processing large data set.
• Strong expertise in Azure Data Factory (ADF) for building and managing ETL processes.
• Familiarity with GitHub and Azure DevOps for version control and CI/CD automation.
• Design develop and optimize data pipelines usingAzure Databricks PySpark.
• Implement and maintainETL/ELT pipelinesusingAzure Data Factory (ADF)andAzure Synapsefor orchestration.
• Develop and optimize complexSQL queriesandPython-baseddata transformation logic.
• Work withversion control systems(GitHub Azure DevOps) to manage code and deployment processes.
• Implementing technical expertise and look for opportunities to reduce operational costs eradicating unnecessary processes and exploring opportunities for service efficiency and rationalising the service where possible.

Areas of responsibility

Gathering of requirements-Participate in gathering requirements for specific modules and ensure clear structured documentation. Work across multiple business processes to capture linkages dependencies and required changes.

Solution Design-Facilitate creation of application solution designs for a specific module under the guidance of the Project Manager ensuring alignment with business requirements and technical standards.

Coding and Configuration-Review the developed codes resolve technical queries of the team for the assigned module and create consistent replication patterns to ensure that the developed code aligns with the project with project managers through all phases of software development life cycle.

Perform root-cause analysis to identify and troubleshoot recurring technical issues. Modify software codes to resolve errors adapt to new hardware and software enhance performance and upgrade interfaces.

Implementation-Collaborate with technical teams to validate configurations facilitate integration of new applications and address dependencies during deployment. Prepare user training documents and related frameworks to ensure smooth adoption new applications and minimize transition risks.