Senior Data Architect – Databricks

Essentially need a resource who has led Databricks implementations previously

Senior Data Architect Databricks

Position Overview

We are seeking an experienced Senior Data Architect with deep expertise in Databricks architecture and implementations to lead the design development and modernization of enterprise data platforms.

The ideal candidate will have a strong background in data architecture cloud-based data engineering and hands-on experience implementing Databricks solutions from initial design through production deployment.

This individual will play a critical role in defining data architecture strategies implementing scalable data solutions and helping organizations maximize the value of their data through modern cloud technologies.

We are looking for someone who can operate at both the strategic and technical levelsdesigning enterprise solutions while remaining hands-on throughout implementation.

Key Responsibilities

Data Architecture & Strategy

• Design and implement scalable secure and high-performing enterprise data architectures leveraging Databricks.
• Lead end-to-end Databricks implementations including architecture design environment setup data integration migration and production deployment.
• Define architectural standards best practices and technical roadmaps for modern data platforms.
• Design and implement Lakehouse architectures supporting analytics business intelligence and advanced data processing.
• Evaluate existing data environments and recommend modernization strategies.
• Partner with business stakeholders engineering teams and technology leadership to translate business requirements into technical solutions.

Databricks Implementation & Engineering

• Lead the design configuration deployment and optimization of Databricks environments.
• Architect and implement data pipelines using Databricks Apache Spark PySpark Python and SQL.
• Design and implement Delta Lake solutions for reliable scalable data storage and processing.
• Establish data ingestion transformation orchestration and integration patterns.
• Implement medallion architecture (Bronze Silver Gold) and other appropriate data modeling frameworks.
• Configure and optimize Databricks workflows compute resources and data processing performance.
• Support migration of legacy data warehouses ETL processes and data platforms into Databricks.
• Implement CI/CD processes infrastructure automation and deployment best practices.

Data Governance Security & Optimization

• Design and implement data governance frameworks utilizing Databricks Unity Catalog.
• Establish data quality lineage access control and security standards.
• Ensure solutions meet enterprise security compliance and data management requirements.
• Optimize platform performance scalability reliability and cloud infrastructure costs.
• Develop architectural documentation technical standards and implementation guidelines.

Technical Leadership & Collaboration

• Serve as the subject matter expert for Databricks architecture and implementation.
• Provide technical leadership and guidance to data engineers developers and cross-functional teams.
• Lead architecture discussions design reviews and technical decision-making.
• Collaborate with cloud infrastructure DevOps analytics and business intelligence teams.
• Identify opportunities to improve data accessibility platform efficiency and overall data capabilities.

Required Qualifications

• 8 years of experience in data architecture data engineering data warehousing or related technical disciplines.
• 3 years of hands-on experience with Databricks including designing and implementing enterprise solutions.
• Demonstrated experience leading at least one end-to-end Databricks implementation from architecture and initial setup through production deployment.
• Strong understanding of Databricks Lakehouse architecture Delta Lake Unity Catalog and Apache Spark.
• Advanced proficiency in Python PySpark and SQL.
• Experience designing and implementing scalable ETL/ELT pipelines and data integration solutions.
• Strong experience with at least one major cloud platform: Azure AWS or Google Cloud Platform.
• Experience with cloud-based data storage data warehouses and distributed processing technologies.
• Strong understanding of data modeling data governance data security and performance optimization.
• Experience migrating or modernizing legacy data platforms into cloud-based architectures.
• Ability to develop architecture diagrams technical documentation and implementation roadmaps.
• Strong communication skills with the ability to collaborate with technical teams and business stakeholders.

Preferred Qualifications

• Databricks Certified Data Engineer Professional or Databricks Certified Data Engineer Associate.
• Databricks Certified Data Architect (if applicable to current certification offerings) or other advanced Databricks credentials.
• Experience implementing Databricks in multiple enterprise environments.
• Experience with Azure Data Factory Azure Data Lake Storage AWS Glue Amazon S3 or similar cloud services.
• Experience with Terraform infrastructure as code and automated deployment pipelines.
• Familiarity with dbt Airflow or other orchestration and transformation technologies.
• Experience integrating Databricks with Power BI Tableau or other analytics platforms.
• Understanding of streaming data architectures and real-time processing.
• Experience supporting machine learning AI or advanced analytics workloads.
• Experience with enterprise data platform modernization initiatives.

What Success Looks Like

• Successfully architect and implement a production-ready Databricks environment.
• Establish scalable repeatable and well-governed data architecture standards.
• Deliver reliable data pipelines and integration frameworks that support enterprise reporting and analytics.
• Improve data processing performance accessibility and operational efficiency.
• Provide technical leadership that enables engineering teams to effectively develop and maintain solutions within Databricks.

Ideal Candidate Profile

The ideal candidate is a hands-on architect who has personally designed built and implemented Databricks environments rather than someone whose experience is limited to maintaining or developing within an established platform.

They should be comfortable owning technical architecture decisions guiding implementation teams solving complex data challenges and translating business needs into scalable enterprise data solutions.