Senior Data Engineer – Azure Databricks & Scala Spark

Bengaluru - India

Monthly Salary: Not Disclosed

Posted on: 2 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Description:

Design develop and maintain scalable batch data pipelines using Apache Spark on Azure Databricks
Work on Spark repositories to implement new features and enhancements
Perform version control activities using Git (branching merging pull requests)
Build Spark executable JARs using Maven and manage dependencies
Execute and validate Spark jobs using Spark Submit with custom runtime arguments
Configure and monitor Databricks Workflows Jobs and Pipelines
Implement and manage Delta Lake components (Delta Tables Delta Live Tables)
Work with Hive Metastore and Unity Catalog for governance and metadata management
Optimize Spark jobs for performance and cost efficiency
Troubleshoot failures and resolve performance bottlenecks
Collaborate with cross-functional teams to translate business requirements into technical solutions

Job Description: Design develop and maintain scalable batch data pipelines using Apache Spark on Azure Databricks Work on Spark repositories to implement new features and enhancements Perform version control activities using Git (branching merging pull requests) Build Spark executable JA...