Senior Databricks Engineer

Job Title: Senior Databricks Engineer

Location: Malvern PA
Job Type: Full-Time
Experience: 10-15 Years

Job Summary

We are seeking a highly experienced Senior Databricks Engineer with deep expertise in Apache Spark Databricks Lakehouse Platform Delta Lake AWS and modern data engineering. The ideal candidate will design and build scalable secure and high-performance data pipelines while implementing enterprise-grade data governance CI/CD and MLOps practices. Experience with Generative AI LLMs and Vector Search is highly desirable.

Required Skills

Databricks & Apache Spark

• 10-15 years of experience in Data Engineering
• Expert in:
  • Databricks Lakehouse Platform
  • Apache Spark
  • Spark SQL
  • PySpark
  • DataFrame API
  • Distributed Data Processing
  • Spark Performance Tuning
  • Query Optimization

Delta Lake & Lakehouse Architecture

• Delta Lake
• Medallion Architecture (Bronze Silver Gold)
• Delta Live Tables (DLT)
• ACID Transactions
• Z-Ordering
• Data Optimization
• Time Travel
• Schema Evolution

Data Pipelines & Orchestration

• ETL / ELT Development
• Databricks Workflows
• Databricks Jobs
• Delta Live Tables (DLT)
• Workflow Automation
• Batch Processing
• Streaming Data Pipelines
• Error Handling
• Retry Mechanisms

Programming Languages

• Python
• PySpark
• SQL
• Scala (Preferred)

AWS Cloud & Storage

• Amazon S3
• Amazon Redshift
• AWS Glue
• Amazon Kinesis
• AWS Step Functions
• AWS IAM
• AWS KMS
• Cross-Account IAM Roles
• S3 Bucket Policies

Databricks Governance & Security

• Unity Catalog
• Data Lineage
• Row-Level Security
• Column-Level Security
• Data Governance
• Secure Data Sharing
• Compliance

Compute & Cost Optimization

• Cluster Policies
• Instance Profiles
• Spot Instances
• Compute Optimization
• Cost Management

CI/CD & DevOps

• Git
• Databricks Git Folders
• CI/CD Pipelines
• Version Control
• Deployment Automation

AI & MLOps

• MLflow
• Model Registry
• Experiment Tracking
• Feature Engineering
• Large Language Models (LLMs)
• Vector Search
• Generative AI Integrations

Key Responsibilities

• Design develop and maintain scalable ETL/ELT pipelines using Databricks Apache Spark and Delta Lake
• Build and optimize batch and streaming data pipelines processing data from multiple enterprise sources
• Develop high-performance PySpark and Spark SQL solutions optimizing distributed processing and execution plans
• Implement Medallion Architecture (Bronze Silver Gold) using Delta Lake best practices
• Design and automate resilient workflows using Databricks Workflows Jobs and Delta Live Tables (DLT)
• Configure Unity Catalog for enterprise data governance lineage and fine-grained access controls
• Integrate Databricks with AWS services including Amazon S3 Redshift Glue Kinesis and Step Functions
• Implement secure cloud architectures using IAM roles S3 bucket policies and AWS KMS customer-managed encryption keys
• Optimize Databricks compute resources through cluster policies instance profiles and Spot Instances
• Collaborate with Data Scientists ML Engineers Analysts and BI teams to support analytics machine learning and Generative AI initiatives
• Implement CI/CD pipelines Git-based development workflows and deployment automation
• Utilize MLflow for experiment tracking model management and MLOps lifecycle support

Preferred Qualifications

• Experience with real-time streaming architectures and event-driven data platforms
• Hands-on experience with Generative AI LLMs Vector Databases and Retrieval-Augmented Generation (RAG)
• Databricks Certified Data Engineer Professional or Associate
• AWS Certified Data Analytics or Solutions Architect certification
• Experience with Terraform or Infrastructure-as-Code (IaC)
• Experience supporting enterprise-scale cloud data platforms