Azure Databricks Solutions Architect (Technical Lead)

Toronto - Canada

Monthly Salary: Not Disclosed

Experience Required: 5years

Posted on: 30+ days ago

Vacancies: 1 Vacancy

Job Summary

Technical Lead

Skills Required: Digital - Databricks Azure Data Factory

Job Description

Azure Databricks Architect designs implements and optimizes scalable secure and high-performance data solutions on the Azure cloud focusing on the Databricks Lakehouse Platform for advanced analytics and machine learning. Key responsibilities include architecting data lakes and Delta Lake implementations building robust data pipelines ensuring data governance and security leading ETL/ELT processes collaborating with cross-functional teams and guiding the migration and transformation of data to the Azure ecosystem.

Key Responsibilities

Data Architecture Design:

Design and implement enterprise-grade data platforms using Databricks including data lakes and Delta Lake to support structured and unstructured data.

Data Pipeline Development:

Lead the development of robust ELT / ETL pipelines for both batch and real-time data processing.

Platform Optimization:

Optimize compute storage and network resources for performance and cost-efficiency within Azure.

Stakeholder Collaboration:

Work with cross-functional teams including Data Engineers Data Scientists and Product teams to understand business requirements and translate them into technical solutions.

Data Governance and Security:

Implement and manage data governance policies security controls access management and compliance protocols.

Migration & Transformation:

Develop strategies and oversee the technical execution for migrating data from existing sources to Azure leveraging Azure Databricks and other Azure services.

Technical Leadership:

Provide technical leadership and mentorship to engineering teams guide architectural standards and ensure adherence to best practices.

ML/AI Workloads:

Architect and implement solutions to support machine learning business intelligence and data science workloads on Databricks.

Required Skills and Knowledge

Azure Services:

Strong proficiency in Azure cloud computing and big data technologies including Azure Data Lake Storage (ADLS) Azure Data Factory and Azure Analysis Services.

Databricks Platform:

In-depth experience with the Azure Databricks platform including designing and managing Databricks clusters and pipelines.

Big Data Technologies

Knowledge of traditional and modern data architectures data warehousing NoSQL databases and big data processing concepts.

Programming:

Hands-on experience with languages like PySpark and a strong understanding of data transformation and analysis.

DevOps:

Experience with Azure DevOps for CI/CD pipelines and infrastructure-as-code tools like Terraform or ARM templates is a plus.

Exp Required: 8

Technical Lead Skills Required: Digital - Databricks Azure Data FactoryJob DescriptionAzure Databricks Architect designs implements and optimizes scalable secure and high-performance data solutions on the Azure cloud focusing on the Databricks Lakehouse Platform for advanced analytics and machine le...