Data Engineer – Databricks & Lakehouse (Power BI Environment)
Coimbatore - India
Job Summary
Position Summary:
The Data Engineer will design build and optimize scalable data pipelines and curated data products within a modern Lakehouse architecture (Databricks). This role is responsible for delivering high-quality business-ready datasets (gold layer) that power enterprise analytics and Power BI reporting. The ideal candidate has deep Databricks experience strong data modeling skills and a focus on building reliable testable and governed data solutions. Exposure to AI-driven development or AI agents is a plus.
Work Youll Do:
Data Engineering & Pipeline Development
Design develop and maintain scalable data pipelines using Databricks and Delta Lake
Build and manage transformations across bronze silver and gold layers
Optimize processing for performance reliability and cost
Integrate data from ERP CRM APIs and other enterprise systems
Data Modeling & Business Logic
Develop gold-layer datasets aligned to standardized business definitions
Translate business requirements into reusable data models
Ensure consistency of core metrics across reporting
Align Databricks outputs with Power BI semantic models
Data Quality Testing & Reliability
Implement automated data quality checks and validation rules
Build testable production-ready pipelines
Support impact analysis using lineage tools
Participate in CI/CD and deployment processes
Performance Optimization & Operations
Monitor and optimize pipeline performance
Troubleshoot issues across environments
Ensure data consistency between dev test and prod
Support high-volume data workloads
AI & Automation (Preferred)
Use AI-assisted tools for development (e.g. Copilot Databricks Agents)
Explore AI agents for testing lineage analysis and optimization
Contribute to AI-driven engineering practices
Collaboration & Stakeholder Engagement
Partner with BI and business teams
Support governance and cataloging efforts
Document data models and pipelines
Basic Qualifications:
Bachelors degree in relevant field
8 years data engineering experience
Strong Databricks Spark Delta Lake experience
SQL and Python proficiency
Experience with Power BI or similar tools
Preferred Qualifications:
Azure Data Factory Synapse or Data Lake experience
DevOps and CI/CD experience
AI agents or AI-assisted development exposure
Experience in large complex enterprise environments
Who we are:
Required Experience:
IC
About Company
Hillenbrand is a world-class industrial company with multiple market-leading brands serving a wide variety of industries across the globe.