Data Engineer Databricks & Microsoft FabricData Engineer Databricks & Microsoft Fabric

Job Description: Data Engineer Databricks & Microsoft Fabric

Location :Chennai India
Experience :6 to 8 years

Role Summary
We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks Microsoft Fabric PySpark SQL and cloud-based data engineering. The candidate will be responsible for designing developing and optimizing scalable data pipelines lakehouse solutions and analytics-ready data models.
The role requires strong experience in building end-to-end data engineering solutions working with structured and semi-structured data implementing data quality controls and supporting enterprise reporting and analytics platforms.
Key Responsibilities
Data Engineering and Pipeline Development
Design build and maintain scalable data pipelines using Databricks PySpark Spark SQL and Microsoft Fabric.
Develop batch and incremental data ingestion pipelines from multiple source systems.
Build and optimize Bronze Silver and Gold layer data models using lakehouse architecture.
Implement ELT/ETL workflows for data transformation enrichment validation and publishing.
Work with structured semi-structured and unstructured data formats such as CSV Parquet JSON Delta and XML.
Databricks Development
Develop notebooks jobs workflows and reusable components in Azure Databricks.
Implement Delta Lake features such as schema evolution merge/upsert time travel and optimized storage.
Optimize Spark jobs for performance scalability and cost efficiency.
Implement partitioning caching indexing and cluster optimization strategies.
Troubleshoot job failures performance bottlenecks and data quality issues.
Microsoft Fabric Development
Build data solutions using Microsoft Fabric Lakehouse Warehouse Data Factory Pipelines Notebooks and OneLake.
Develop and manage data pipelines in Fabric for ingestion transformation and orchestration.
Work with Fabric SQL endpoints semantic models and Power BI integration.
Support migration or modernization of existing data platforms into Microsoft Fabric.
Implement reusable data engineering patterns and framework-based development in Fabric.
Data Quality Governance and Security
Implement data validation reconciliation exception handling and audit controls.
Define and apply data quality rules including null checks duplicate checks referential checks and cross-field validations.
Maintain data lineage metadata source-to-target mapping and technical documentation.
Ensure data pipelines comply with enterprise security access control and governance standards.
Support integration with data governance tools such as Microsoft Purview where applicable.
DevOps and Production Support
Implement CI/CD practices for notebooks pipelines SQL scripts and configuration files.
Use Git-based version control and deployment processes across environments.
Monitor production jobs and resolve incidents within agreed timelines.
Prepare runbooks deployment guides operational support documents and handover materials.
Collaborate with architects business analysts data analysts and reporting teams to deliver reliable data solutions.
Required Skills
Technical Skills
Strong hands-on experience in Azure Databricks.
Strong experience in Microsoft Fabric components such as Lakehouse Warehouse Data Factory Pipelines Notebooks and OneLake.
Proficiency in PySpark Spark SQL Python and SQL.
Strong knowledge of Delta Lake lakehouse architecture and medallion architecture.
Experience with cloud storage and data platforms preferably Azure Data Lake Storage Azure SQL Synapse or Fabric OneLake.
Experience in data ingestion from databases APIs files SFTP cloud storage and streaming sources.
Good understanding of data modeling dimensional modeling and analytics-ready data structures.
Experience with performance tuning of Spark jobs and SQL queries.
Experience in job scheduling monitoring logging and error handling.
Knowledge of CI/CD Git Azure DevOps and deployment automation.
Preferred Skills
Experience with Power BI and semantic model integration.
Experience in migrating workloads from legacy ETL tools Synapse ADF or Databricks to Microsoft Fabric.
Knowledge of Microsoft Purview for data cataloging lineage and governance.
Experience in building reusable data engineering frameworks.
Exposure to real-time or near-real-time data processing.
Azure certifications or Databricks certifications are preferred.
Roles and Responsibilities Summary
Build and maintain scalable data pipelines using Databricks and Microsoft Fabric.
Develop lakehouse solutions using Bronze Silver and Gold architecture.
Perform data transformation validation reconciliation and publishing.
Optimize Spark workloads and SQL queries for performance.
Implement data quality audit monitoring and exception handling frameworks.
Support deployment production monitoring incident resolution and documentation.
Collaborate with cross-functional teams to deliver enterprise data and analytics solutions.
Required Experience
6 to 8 years of overall experience in data engineering ETL/ELT or data platform development.
At least 3 years of hands-on experience in Databricks / PySpark.
At least 1 year of hands-on experience or strong working knowledge of Microsoft Fabric.
Experience working in enterprise-scale data platforms and analytics projects.
Experience in Agile delivery models and production support environments.
Educational Qualification
Bachelors degree in computer science Information Technology Engineering Data Analytics or a related discipline.
Good to Have Certifications
Microsoft Certified: Fabric Analytics Engineer Associate
Microsoft Certified: Azure Data Engineer Associate
Databricks Certified Data Engineer Associate / Professional
Microsoft Certified: Azure Fundamentals
Key Deliverables
Production-ready data pipelines and notebooks.
Optimized Databricks and Fabric workloads.
Bronze Silver and Gold layer data models.
Data quality and reconciliation reports.
Source-to-target mapping and technical design documents.
Deployment guides runbooks and support documentation.

Recruiter Name:Srinija Adapa

Recruiter Email ID:

We transform. SAP solutions into Value

NTT DATA Business Solutions is a fast-growing international IT company and one of the worlds leading SAP partners. We are a full service provider delivering everything from business consulting to implementation of SAP solutions including hosting services and support.