DATABRICKS ARCHITECT

DATABRICKS ARCHITECT (10-15 years) Chennai

Job Description

• We are seeking a highly skilled Databricks Architect to lead the design implementation and optimization of scalable cloud-based data platforms.
• This role requires deep expertise in Lakehouse architecture Databricks Apache Spark and Delta Lake along with proven experience delivering end-to-end enterprise data engineering solutions.

Roles & Responsibilities

• Architect and implement enterprise-grade Lakehouse solutions using and deliver end-to-end data engineering pipelines including batch and real-time streaming solutions.
• Lead implementation of:Cloud-based data lakehouse platforms integrating diverse data sources.
• Real-time data processing pipelines for operational and analytical use cases.
• Develop scalable ETL/ELT pipelines using PySpark Scala and advanced data modeling solutions including 3NF dimensional modeling and enterprise data warehousing strategies.
• Design and build incremental data loading frameworks and metadata-driven ingestion pipelines.
• Establish data quality frameworks and governance and manage Unity Catalog including fine-grained security and access controls.
• Leverage Databricks components such as:Delta Live TablesAutoloaderStructured StreamingDatabricks WorkflowsIntegration with orchestration tools (e.g. Apache Airflow)Drive CI/CD automation deployment strategies and DevOps best practices.
• Optimize performance of pipelines Spark jobs and compute resources.
• Provide architectural guidance and technical leadership across cross-functional teams.
• Engage with stakeholders and clients to translate business requirements into scalable technical solutions.
• Deep expertise in:Databricks and cloud-native storage/compute platformsApache Spark (batch & streaming)Delta Lake & Lakehouse architectureDistributed data processing systemsStrong hands-on programming skills in Python PySpark Scala and SQL.