Senior Data Engineer

Bajaj Finance

Job Location:

Pune - India

Monthly Salary: Not Disclosed

Posted on: 4 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Purpose	To design and deliver AI-powered Business Intelligence solutions by integrating Databricks Genie Semantic Models and Metric Views on top of a robust end-to-end data engineering stack. The role focuses on enabling self-service analytics using natural language (GenAI) ensuring business-friendly data consumption and building governed scalable and high-performance data ecosystems across batch and real-time pipelines.
Duties and Responsibilities	KEY ROLES / PRINCIPAL ACCOUNTABILITIES AI for BI & GenAI Enablement Design and implement GenAI-powered BI solutions using Databricks Genie or MS Fabric Create and manage Data Rooms for business users with curated datasets Define and maintain Instructions Layer (prompt engineering for business context) Enable natural language to SQL/insights workflows for self-service analytics Drive adoption of Databricks One & other platforms as a unified analytics interface Semantic Layer & Metrics Engineering Design and manage Semantic Models for business abstraction Develop reusable and governed Metric Views (KPIs aggregations business definitions) Ensure consistency across BI tools (Power BI / Genie) Align semantic layer with business glossary and data governance policies Data Engineering & Platform Development Build scalable pipelines using Azure Databricks (PySpark SQL) Develop and orchestrate ETL workflows using Azure Data Factory Work with Delta Lake architecture (BronzeSilverGold layers) Enable real-time and batch data processing pipelines Enable data exposure via APIs for BI and downstream systems CI/CD & DevOps Implement CI/CD pipelines for data and AI workflows Automate deployments across environments (Dev QA Prod) Ensure version control and reproducibility KEY RESPONSIBILITIES Translate business problems into AI-driven BI solutions Own end-to-end delivery: ingestion transformation semantic layer AI consumption Design scalable data architecture aligned with lakehouse principles Ensure data quality governance and metric consistency Collaborate with business teams to onboard them onto Genie-based analytics Optimize performance of queries pipelines and AI responses Establish best practices for AI in BI (prompting semantic tuning governance) Drive adoption of self-service BI with minimal dependency on tech teams
Key Decisions / Dimensions	KEY DECISIONS / DIMENSIONS Define semantic layer design and metric definitions Decide GenAI prompting strategies and instruction frameworks Prioritize data vs AI optimization trade-offs Handle production issues with RCA and long-term fixes Drive architectural decisions for lakehouse AI integration
Major Challenges	MAJOR CHALLENGES Ensuring accuracy and trust in AI-generated insights Driving adoption of GenAI-based BI over traditional dashboards Maintaining semantic consistency across multiple tools Balancing performance cost and scalability Managing dependencies across data engineering AI and business teams
Required Qualifications and Experience	REQUIRED SKILLS & EXPERIENCE Must Have Azure Databricks PySpark SQL Delta Lake Strong experience in Semantic Modeling & Metrics Layer design Hands-on with Databricks Genie / GenAI-based BI workflows Databricks One (Unified BI Experience) Prompt Engineering / Instruction tuning for AI systems Python (Pandas PySpark FastAPI) Azure Data Factory (ADF) for ETL pipelines Strong SQL and data modeling skills Good to Have Cosmos DB / MongoDB (NoSQL concepts) Azure Data Explorer (KQL) DATA STACK (MANDATORY FOR SCREENING) SNoData Platform / ConceptsAssociated Technologies 1Databricks LakehousePySpark SQL Delta Lake 2AI for BIDatabricks Genie Genie Rooms Instructions Agents 3Semantic LayerSemantic Models Metric Views 4ETL & OrchestrationAzure Data Factory 5ProgrammingPython C#/.NET 6Cloud PlatformAzure (Preferred) 10DevOpsCI/CD Pipelines Git SOFT SKILLS Strong analytical and problem-solving mindset Ability to bridge business data AI Effective stakeholder communication Ownership-driven and proactive approach Ability to work in fast-paced evolving AI landscape

Required Experience:

Senior IC

Job PurposeTo design and deliver AI-powered Business Intelligence solutions by integrating Databricks Genie Semantic Models and Metric Views on top of a robust end-to-end data engineering stack.The role focuses on enabling self-service analytics using natural language (GenAI) ensuring business-frien...

Job Purpose	To design and deliver AI-powered Business Intelligence solutions by integrating Databricks Genie Semantic Models and Metric Views on top of a robust end-to-end data engineering stack. The role focuses on enabling self-service analytics using natural language (GenAI) ensuring business-friendly data consumption and building governed scalable and high-performance data ecosystems across batch and real-time pipelines.
Duties and Responsibilities	KEY ROLES / PRINCIPAL ACCOUNTABILITIES AI for BI & GenAI Enablement Design and implement GenAI-powered BI solutions using Databricks Genie or MS Fabric Create and manage Data Rooms for business users with curated datasets Define and maintain Instructions Layer (prompt engineering for business context) Enable natural language to SQL/insights workflows for self-service analytics Drive adoption of Databricks One & other platforms as a unified analytics interface Semantic Layer & Metrics Engineering Design and manage Semantic Models for business abstraction Develop reusable and governed Metric Views (KPIs aggregations business definitions) Ensure consistency across BI tools (Power BI / Genie) Align semantic layer with business glossary and data governance policies Data Engineering & Platform Development Build scalable pipelines using Azure Databricks (PySpark SQL) Develop and orchestrate ETL workflows using Azure Data Factory Work with Delta Lake architecture (BronzeSilverGold layers) Enable real-time and batch data processing pipelines Enable data exposure via APIs for BI and downstream systems CI/CD & DevOps Implement CI/CD pipelines for data and AI workflows Automate deployments across environments (Dev QA Prod) Ensure version control and reproducibility KEY RESPONSIBILITIES Translate business problems into AI-driven BI solutions Own end-to-end delivery: ingestion transformation semantic layer AI consumption Design scalable data architecture aligned with lakehouse principles Ensure data quality governance and metric consistency Collaborate with business teams to onboard them onto Genie-based analytics Optimize performance of queries pipelines and AI responses Establish best practices for AI in BI (prompting semantic tuning governance) Drive adoption of self-service BI with minimal dependency on tech teams
Key Decisions / Dimensions	KEY DECISIONS / DIMENSIONS Define semantic layer design and metric definitions Decide GenAI prompting strategies and instruction frameworks Prioritize data vs AI optimization trade-offs Handle production issues with RCA and long-term fixes Drive architectural decisions for lakehouse AI integration
Major Challenges	MAJOR CHALLENGES Ensuring accuracy and trust in AI-generated insights Driving adoption of GenAI-based BI over traditional dashboards Maintaining semantic consistency across multiple tools Balancing performance cost and scalability Managing dependencies across data engineering AI and business teams
Required Qualifications and Experience	REQUIRED SKILLS & EXPERIENCE Must Have Azure Databricks PySpark SQL Delta Lake Strong experience in Semantic Modeling & Metrics Layer design Hands-on with Databricks Genie / GenAI-based BI workflows Databricks One (Unified BI Experience) Prompt Engineering / Instruction tuning for AI systems Python (Pandas PySpark FastAPI) Azure Data Factory (ADF) for ETL pipelines Strong SQL and data modeling skills Good to Have Cosmos DB / MongoDB (NoSQL concepts) Azure Data Explorer (KQL) DATA STACK (MANDATORY FOR SCREENING) SNoData Platform / ConceptsAssociated Technologies 1Databricks LakehousePySpark SQL Delta Lake 2AI for BIDatabricks Genie Genie Rooms Instructions Agents 3Semantic LayerSemantic Models Metric Views 4ETL & OrchestrationAzure Data Factory 5ProgrammingPython C#/.NET 6Cloud PlatformAzure (Preferred) 10DevOpsCI/CD Pipelines Git SOFT SKILLS Strong analytical and problem-solving mindset Ability to bridge business data AI Effective stakeholder communication Ownership-driven and proactive approach Ability to work in fast-paced evolving AI landscape

Required Experience:

Senior IC

Apply Now

About Company

Bajaj Finance

Bajaj Finserv is India’s most diversified non-banking finance company. Visit the website to learn about our products. Get instant approval on loans, shop on EMIs, make an investment, get insurance and pay your bills.

View Profile View Profile

AI AutoApply

Apply to 100+ jobs with one click