Lead Databricks Engineer

Position: Lead Databricks Engineer

Location: Houston TX - The Galleria (100% Onsite)

Duration: 12 months Contract

Must have Databricks certifications

Job Description:
A Lead Databricks Engineer to support large-scale data platform initiatives focused on data modernization cloud migration and advanced analytics. This is a hands-on senior role requiring deep Databricks expertise strong AWS experience and the ability to partner closely with business and technical stakeholders.

The ideal consultant has led complex Databricks implementations end-to-end can code at a high level and has experience modernizing legacy data platforms into scalable cloud-native architectures.

Key Responsibilities

• Lead the design and implementation of enterprise-scale Databricks solutions on AWS
• Drive data modernization initiatives including lift-and-shift and re-architecture of legacy data platforms
• Build and optimize data pipelines using Python PySpark and SQL
• Design and manage Delta Lake architectures and implement Unity Catalog for governance and access control
• Develop and support streaming and batch data workloads
• Configure and optimize Databricks clusters and serverless compute for performance and cost
• Integrate Databricks with upstream and downstream systems (APIs data sources analytics tools)
• Partner with stakeholders to gather user and business requirements and translate them into technical solutions
• Implement observability monitoring and cost controls including usage volume and pricing metrics
• Support data domains related to billing accounting pricing and volume metrics
• Provide technical leadership best practices and mentorship to engineering teams
  

Required Skills & Experience

• 8 years of experience in data engineering with recent hands-on focus on Databricks
• Strong experience deploying Databricks on AWS
• Advanced coding skills in Python PySpark and SQL
• Deep knowledge of Delta Lake Unity Catalog and Databricks workspace governance
• Experience with streaming data (Structured Streaming Kafka or similar)
• Strong understanding of Databricks cluster management serverless compute and performance tuning
• Databricks certifications
• Experience integrating Databricks with enterprise systems and data sources
• Proven ability to work directly with business and technical stakeholders
• Experience supporting financial data domains (billing accounting pricing usage metrics) is highly preferred
• Strong communication skills and ability to lead technical discussions
• Experience with AKS / Kubernetes environments
• AWS certifications
• Consulting or contracting background in large enterprise environments