Azure Databricks Architect

New York City, NY - USA

Monthly Salary: Not Disclosed

Posted on: Yesterday

Vacancies: 1 Vacancy

Job Summary

Experience Minimum 12 years of hands-on experience architecting and delivering largescale data platforms using Azure and Databricks.

Architect and implement Azure based data solutions using Databricks Data Factory Synapse Analytics and Data Lake services.

Design build and optimize a Medallion architecture Bronze Silver Gold supporting both HR and enterprise data domains.

Develop data pipelines APIs and orchestration frameworks that enable seamless data ingestion transformation and delivery.

Establish and maintain data governance metadata and lineage frameworks using Databricks Unity CatLog and Azure Purview.

Build and maintain ETL frameworks data APIs and Realtime workflows connecting diverse HR and enterprise systems.

Leverage AI and ML capabilities within Databricks including ML flow feature stores and the Agent Framework Agent Bricks to build intelligent data agents for automation monitoring and analytics enrichment.

Design and implement AI agent pipelines that can reason over HR data models automate transformations and enhance analytical outcomes.

Apply best practices for performance optimization scalability and cost efficiency within the Azure ecosystem.

Integrate CICD pipelines and infrastructure as code approaches Terraform Bicep or Azure DevOps.

Serve as the technical authority for Azure and Databricks architecture within HR Technology

Partner with HR Analytics and Infrastructure leaders to define and execute the data and AI modernization roadmap

Establish standards patterns and reusable frameworks for data engineering AI agents and automation

Mentor engineers and champion a culture of innovation experimentation and delivery excellence.

Expertise in Azure Data Lake Data Factory and Synapse Analytics

Databricks Delta Lake Unity CatLog Spark optimization

Python SQL PySpark and API development REST GraphQL

AIML frameworks within Databricks including Agent Bricks and ML flow

Strong grounding in data governance security and compliance best practices

Excellent communication and collaboration skills ability to translate complex architecture into business outcomes

Experience Minimum 12 years of hands-on experience architecting and delivering largescale data platforms using Azure and Databricks. Architect and implement Azure based data solutions using Databricks Data Factory Synapse Analytics and Data Lake services. Design build and optimize a Medallion archit...