Databricks Platform & Data Engineer

Title: Databricks Platform & Data Engineer

Location: Plano TX & Minneapolis MN (Hybrid)

Type: Contract

Role Summary:

• Client is seeking a senior Databricks Platform & Data Engineer to join our Databricks Center of Excellence (CoE). This role blends platform engineering architecture and hands-on data engineering with a focus on building and scaling enterprise-grade Databricks environments for large regulated clients.
• The ideal candidate will lead platform setup governance and adoption while partnering directly with clients to design and deliver high-impact use cases leveraging the latest Databricks capabilities (e.g. Dataflow Unity Catalog AI/BI Genie).
• This is a client-facing consulting role requiring strong technical depth communication skills and the ability to translate business requirements into scalable data platform solutions.

Key Responsibilities:

Platform Engineering & Architecture

• Design and deploy enterprise-scale Databricks Lakehouse platforms across AWS/Azure
• Establish secure governed environments using Unity Catalog role-based access controls and data lineage
• Define platform standards reusable patterns and guardrails for scalable adoption
• Optimize platform performance cost and reliability for production workloads
• Implement CI/CD environment promotion and DevOps automation for Databricks

Databricks Feature Enablement

• Lead adoption of modern Databricks capabilities including:
• Unity Catalog: Centralized governance access control lineage
• Dataflow / declarative pipelines: build scalable ingestion and transformation frameworks
• Genie / AI-assisted development: accelerate developer productivity and data accessibility
• Enable AI/BI dashboards model serving and advanced analytics use cases

Data Engineering & Use Case Delivery

• Build and optimize batch and streaming data pipelines using Spark and Delta Lake
• Develop data products and domain-oriented pipelines aligned to enterprise data strategies
• Lead end-to-end use case delivery from requirements to production deployment
• Drive data quality observability and pipeline reliability

Client Engagement & Advisory

• Act as a trusted advisor to client stakeholders (architecture data risk and business teams)
• Translate business requirements into technical architecture and delivery roadmaps
• Lead workshops solution design sessions and platform adoption strategies
• Support proposals solutioning and client innovations within regulated industries

CoE Contribution:

• Build and contribute to Client accelerators frameworks and reusable assets
• Define best practices reference architectures and playbooks for enterprise Databricks adoption
• Mentor junior engineers and support capability building across the CoE

Required Qualifications:

• 12 years of experience in data engineering platform engineering or data architecture
• 5 years hands-on experience with Databricks in large enterprise environments

Deep expertise in: -

• Apache Spark (Scala/Python)
• Delta Lake and Lakehouse architectures
• Databricks workspace setup cluster policies and job orchestration
• Strong experience with cloud platforms (AWS Azure or GCP)
• Experience implementing data governance security and compliance controls
• Proven ability to design and deliver scalable production-grade data platforms
• Strong client-facing and communication skills

Preferred Qualifications:

• Experience in financial services or other regulated industries
• Familiarity with data governance frameworks regulatory reporting and risk data environments
• Databricks certifications (e.g. Data Engineer Solutions Architect) Experience with:
• Real-time streaming (Kafka Structured Streaming)
• MLOps / Model Serving
• Data marketplace / data product architectures
• Exposure to AI-assisted development workflows and agent-based tooling