Software Engineer III Senior Data Engineer

We are seeking a Senior Data Engineer to lead the design build and operation of scalable data pipelines and domain-aligned data products on Databricks using Python. This role is accountable for delivering governed discoverable high-quality datasets and services aligned to data mesh principles (domain ownership product thinking and federated governance) enabling analytics and ML use cases with strong reliability security and performance.

Key Responsibilities

• Architect and deliver end-to-end data engineering solutions on Databricks (Spark/Delta Lake) using Python and SQL.
• Build data products aligned to data mesh principles: clear contracts documentation metadata lineage access controls and measurable SLAs/SLOs.
• Develop and optimize batch and streaming pipelines with strong data quality checks and observability.
• Implement curated data layers and models (e.g. bronze/silver/gold as appropriate) including partitioning file sizing clustering strategies and query optimization.
• Establish engineering standards: code quality unit/integration testing CI/CD release governance and reusable frameworks.
• Partner with governance security and platform teams to ensure compliance with firm policies (RBAC encryption retention auditability) and smooth onboarding of consumers.
• Provide technical leadership across squads: architecture reviews mentoring and hands-on delivery for critical components.

AI-Enabled Engineering Expectations

• Be well-versed in using AI-assisted development tools to accelerate delivery while maintaining strong engineering rigor (secure coding testing and review discipline).
• Use AI tools for tasks such as code scaffolding refactoring test generation documentation and troubleshooting with human validation and adherence to internal controls.
• Promote responsible adoption patterns within the team (e.g. prompt hygiene avoiding sensitive data exposure and ensuring outputs meet quality/security standards).

Required Qualifications

• 812 years of experience in data engineering / software engineering with end-to-end delivery ownership.
• Strong hands-on expertise with AWS Databricks and Delta Lake.
• Strong Python engineering skills (modular design packaging testing dependency management).
• Strong SQL and data modeling fundamentals.
• Solid Spark fundamentals (joins shuffle partitioning caching skew handling cluster sizing).
• Experience implementing CI/CD and SDLC best practices (Git-based workflows code reviews automated testing).
• Proven stakeholder management and ability to drive cross-team delivery.