Dataengineer- ADF

Job Title: Data Engineer

Job Summary:

Are you passionate about building scalable data pipelines optimizing ETL processes and designing efficient data models We are looking for a Databricks Data Engineer to join our team and play a key role in managing and transforming data in Azure cloud environments.

In this role you will work with Azure Data Factory (ADF) Databricks Python and SQL to develop robust data ingestion and transformation workflows. You ll also be responsible for integrating SAP ISAuto data optimizing performance and ensuring data quality & governance.

If you have strong experience in big data processing distributed computing (Spark) and data modeling we d love to hear from you!

Key Responsibilities:

1. Develop & Optimize ETL Pipelines:
   
• Build robust and scalable data pipelines using ADF Databricks and Python for data ingestion transformation and loading.
  
2. Data Modeling & Systematic Layer Modeling:
   
• Design logical physical and systematic data models for structured and unstructured data.
  
3. Integrate SAP ISAuto:
   
• Extract transform and load data from SAP ISAuto into Azurebased data platforms.
  
4. Database Management:
   
• Develop and optimize SQL queries stored procedures and indexing strategies to enhance performance.
  
5. Big Data Processing:
   
• Work with Azure Databricks for distributed computing Spark for largescale processing and Delta Lake for optimized storage.
  
6. Data Quality & Governance:
   
• Implement data validation lineage tracking and security measures for highquality compliant data.
  
7. Collaboration:
   
• Work closely with business analysts data scientists and DevOps teams to ensure data availability and usability.
  
8. Testing and Debugging:
   
• Write unit tests and perform debugging to ensure the Implementation is robust and errorfree.
  
• Conduct performance optimization and security audits.
  

Required Skills and Qualifications:

1. Azure Cloud Expertise:
   
• Strong experience in Azure Data Factory (ADF) Databricks and Azure Synapse.
  
2. Programming:
   
• Proficiency in Python for data processing automation and scripting.
  
3. SQL & Database Skills:
   
• Advanced knowledge of SQL TSQL or PL/SQL for data manipulation.
  
4. SAP ISAuto Data Handling:
   
• Experience integrating SAP ISAuto as a data source into data pipelines.
  
5. Data Modeling:
   
• Handson experience in dimensional modeling systematic layer modeling and entityrelationship modeling.
  
6. Big Data Frameworks:
   
• Strong understanding of Apache Spark Delta Lake and distributed computing.
  
7. Performance Optimization:
   
• Expertise in query optimization indexing and performance tuning.
  
8. Data Governance & Security:
   
• Knowledge of RBAC encryption and data privacy standards.
  

Preferred Qualifications:

1. Experience with CI/CD for data pipelines using Azure DevOps.

2. Knowledge of Kafka/Event Hub for realtime data processing.

3. Experience with Power BI/Tableau for data visualization (not mandatory but a plus).