Azure Databricks Architect

Houston, MS - USA

Monthly Salary: Not Disclosed

Posted on: 7 hours ago

Vacancies: 1 Vacancy

Job Summary

Position - Azure Databricks Architect

Location: Houston TX- Hybrid

Level: 5

Exp: 10 Years

Core Responsibilities

Design and implement scalable data platforms and pipelines using Azure Databricks Apache Spark Pyspark Delta Lake and MLflow.

Design and implement data pipelines using Databricks Spark and Delta Lake for batch and streaming data.

Optimize Spark jobs for performance scalability and cost efficiency.

Develop and maintain Lakehouse architecture leveraging Databricks and cloud storage solutions.

Lead the migration from legacy platforms to Lakehouse architecture.

Develop batch and streaming data pipelines for ingestion transformation and analytics.

Establish standards for data governance quality and security.

Collaborate with stakeholders to align architecture with business goals.

Mentor data engineers and developers on Databricks best practices.

Integrate Databricks with tools like Power BI Tableau Kafka Snowflake and Azure Data Factory.

Required Skills & Experience

8 10 years in data engineering or architecture roles.

8 years of hands-on experience with Databricks and Azure.

Strong command of SQL Python Scala and Spark.

Experience with CI/CD pipelines DevOps and orchestration tools like Airflow or Data Factory.

Familiarity with Azure cloud platforms.

Deep understanding of distributed computing performance tuning and data security.

Preferred Qualifications

Bachelors or Masters in Computer Science Data Science or related field.

Certifications such as Azure Solution Architect Azure Data Engineer or Databricks Certified Data Engineer or Solutions Architect.

Experience with data mesh data fabric or enterprise data architectures.

Domain knowledge in finance healthcare or public sector is a plus.

Hadoop & Scala experience

Additional Insights from Internal Communications

Emphasis on DataOps data orchestration and data modeling.

Strong ERP knowledge (e.g. SAP Salesforce) is valued.

Candidates should be hands-on capable of self-exploration and able to drive analytics adoption across business teams.

A structured questionnaire is often used to assess candidates on areas like Unity Catalog Spark optimization partitioning techniques and data security.

Position - Azure Databricks Architect Location: Houston TX- Hybrid Level: 5 Exp: 10 Years Core Responsibilities Design and implement scalable data platforms and pipelines using Azure Databricks Apache Spark Pyspark Delta Lake and MLflow. Design and implement data pipelines using Databricks S...