ADB+ADF_ShivaniGautam

We are looking for a skilled and motivated Data Engineer with expertise in Azure Data Bricks (ADB) Azure Data Factory (ADF) Python PySpark and SQL to join our dynamic team. As a Data Engineer you will be responsible for designing building and maintaining robust data pipelines and working with large-scale datasets to enable efficient data processing storage and analytics.

• Design develop and optimize end-to-end data pipelines using Azure Data Factory (ADF) and Azure Data Bricks (ADB) to extract transform and load (ETL) data from various sources into structured and unstructured formats.

• Collaborate with Data Scientists Analysts and other engineers to ensure smooth data integration and data flow across multiple systems.

• Utilize Python and PySpark to process large datasets implement complex transformations and perform data wrangling tasks.

• Build and maintain scalable high-performance data processing solutions on Azure.

• Write and optimize SQL queries to interact with relational databases and data warehouses (e.g. Azure SQL SQL Server or other cloud-based databases).

• Work with cloud storage solutions like Azure Data Lake Azure Blob Storage and Azure SQL Data Warehouse.

• Automate data workflows monitor jobs and ensure data integrity and consistency throughout the data pipeline lifecycle.

• Participate in code reviews collaborate on best practices and work to continually improve the quality of the data engineering solutions.

• Troubleshoot and debug data-related issues identify root causes and implement fixes as needed.

• Proven experience as a Data Engineer or in a similar data engineering role with a focus on cloud-based data solutions (Azure preferred).

• Strong experience working with Azure Data Bricks (ADB) and Azure Data Factory (ADF) for data integration transformation and orchestration.

• Proficiency in Python for data processing automation and scripting.

• Solid understanding and hands-on experience with PySpark for big data processing and distributed data systems.

• Strong knowledge of SQL including advanced querying optimization and working with relational databases (e.g. Azure SQL SQL Server).

• Experience working with cloud storage solutions (Azure Data Lake Blob Storage etc.).

• Familiarity with data warehousing concepts and tools especially in an Azure environment.

• Ability to troubleshoot and optimize data pipelines for scalability performance and reliability.

• Strong problem-solving skills and attention to detail.

• Excellent communication skills and ability to work collaboratively in cross-functional teams.

• Experience with Azure Synapse Analytics Azure Machine Learning or Power BI.

• Knowledge of CI/CD practices and version control (e.g. Git).

• Familiarity with data modeling data governance and data security best practices.

• Certification in Microsoft Azure (e.g. Azure Data Engineer Associate) is a plus.