67518-1 Microsoft AzureDigital Databricks
Share
Job Location:
Monthly Salary:
Posted on:
9 days ago
Vacancies:
1 Vacancy
Job Summary
Req ID-
Role Descriptions: Azure Databricks EngineerKey responsibilitiesKey ResponsibilitiesPlatform InfrastructureOversee Databricks platform configuration resource management workspace structuring and cluster and troubleshoot performance issues across clusters jobs notebooks and governance security compliance and data access control using Role-Based Access Control (RBAC) and Unity Development ArchitectureDesign and implement end-to-end data pipelines using PySpark SQL and Delta Lake within a medallion architecture using Data factory And Data real-time and batch DLT pipelines using Databricks Delta Live Tables with a focus on reliability and Lakehouse architecture for performance cost-efficiency and data data ingestion transformation and validation including support for streaming (Autoloader) and scheduled data transformations cleansing and validations using data quality rules for consistent and accurate data and monitor job orchestration ensuring efficient pipelines run and DevOpsDesign and maintain CICD pipelines for Databricks artifacts (notebooks jobs libraries) using tools such as Azure DevOps GitHub Actions Terraform or trunk-based development deployment workflows and infrastructure-as-code version control and automated testing using Git and related DevOps DeliveryCollaborate with product owners business stakeholders and data teams to gather requirements and translate them into technical the adoption of best practices in coding versioning testing deployment monitoring and thought leadership on the best practices in Data Engineering Architecture and Cloud OptimizationDeliver optimized spark jobs and SQL queries for large scale data partitioning caching and indexing strategies to improve performance and scalability of big data POCs for capacity planning and recommend appropriate infrastructure optimizations for cost Knowledge SharingCreated detailed documentations and review them for data workflows SOPs Architectural reviews junior team members and promote a culture of learning and the culture of optimization and cost saving and enable research driven QualificationsTechnical Expertise5 years in data engineering with a strong focus on Databricks and Azure hands-on experience with Data Factory Databricks Lakehouse Architecture Delta Lake PySpark and Spark job in Python SQL and optionally Scala for building scalable ETLELT SQL skills are essential with hands-on experience in SQL Server or other RDBMS experience in designing and optimizing DLT pi
Essential Skills: Azure Databricks Engineer
Desirable Skills:
Keyword:
Skills: Digital : Microsoft AzureDigital : Databricks
Experience Required: 6-8
Role Descriptions: Azure Databricks EngineerKey responsibilitiesKey ResponsibilitiesPlatform InfrastructureOversee Databricks platform configuration resource management workspace structuring and cluster and troubleshoot performance issues across clusters jobs notebooks and governance security compliance and data access control using Role-Based Access Control (RBAC) and Unity Development ArchitectureDesign and implement end-to-end data pipelines using PySpark SQL and Delta Lake within a medallion architecture using Data factory And Data real-time and batch DLT pipelines using Databricks Delta Live Tables with a focus on reliability and Lakehouse architecture for performance cost-efficiency and data data ingestion transformation and validation including support for streaming (Autoloader) and scheduled data transformations cleansing and validations using data quality rules for consistent and accurate data and monitor job orchestration ensuring efficient pipelines run and DevOpsDesign and maintain CICD pipelines for Databricks artifacts (notebooks jobs libraries) using tools such as Azure DevOps GitHub Actions Terraform or trunk-based development deployment workflows and infrastructure-as-code version control and automated testing using Git and related DevOps DeliveryCollaborate with product owners business stakeholders and data teams to gather requirements and translate them into technical the adoption of best practices in coding versioning testing deployment monitoring and thought leadership on the best practices in Data Engineering Architecture and Cloud OptimizationDeliver optimized spark jobs and SQL queries for large scale data partitioning caching and indexing strategies to improve performance and scalability of big data POCs for capacity planning and recommend appropriate infrastructure optimizations for cost Knowledge SharingCreated detailed documentations and review them for data workflows SOPs Architectural reviews junior team members and promote a culture of learning and the culture of optimization and cost saving and enable research driven QualificationsTechnical Expertise5 years in data engineering with a strong focus on Databricks and Azure hands-on experience with Data Factory Databricks Lakehouse Architecture Delta Lake PySpark and Spark job in Python SQL and optionally Scala for building scalable ETLELT SQL skills are essential with hands-on experience in SQL Server or other RDBMS experience in designing and optimizing DLT pi
Essential Skills: Azure Databricks Engineer
Desirable Skills:
Keyword:
Skills: Digital : Microsoft AzureDigital : Databricks
Experience Required: 6-8
