Senior Data Engineer

Role Overview

We are seeking an experienced and versatile Senior Data Engineer to lead organizational capability for the design development and implementation of scalable secure and high-performance data pipelines and data lakes across AWS and Azure platforms.

The ideal candidate will have a proven track record of building robust data architectures leading teams and providing end-to-end data solutions that support AI analytics and enterprise-grade use addition to strong technical expertise the Senior Data Engineer will play a key role in mentoring teams collaborating with sales and pre-sales and translating business requirements into effective technical solutions. This role will also be responsible for building and implementing organizational data engineering capability and overseeing execution across projects and teams.

Leadership & Solution Ownership

• Design and deliver comprehensive data solutions across AWS and Azure environments from ingestion and transformation to storage governance and analytics.
  
• Lead and guide data engineering teams to implement scalable maintainable and secure data architectures.
  
• Collaborate with business leaders to translate complex business problems into actionable data solutions.
  
• Drive architectural decisions that ensure performance scalability and reliability in enterprise data systems.
  
• Provide strategic direction in adopting modern tools frameworks and practices across the data ecosystem.
  
• Partner with sales and pre-sales teams to design and present technical solutions to clients supporting proposals RFPs and client demos.
  
• Participate in proof-of-concept (POC) initiatives to validate technical feasibility and demonstrate business value.
  
• Build and implement organizational data engineering capability including frameworks best practices and reusable components for scalability and consistency.
  
• Oversee execution of multiple data engineering related initiatives ensuring technical alignment on-time delivery and adherence to quality and compliance standards.
  
• Mentor and upskill team members by planning and executing structured training programs for junior engineers fostering a culture of learning innovation and technical excellence.
  

Core Engineering Responsibilities

• Build & Maintain Pipelines: Develop and optimize ETL/ELT pipelines using AWS Glue Lambda Batch PySpark Azure Data Factory Synapse and Databricks.
  
• Delta Lake Architecture: Implement multi-layer Delta Lake (Bronze Silver Gold) on AWS S3 and Azure ADLS with strong governance and lifecycle management.
  
• Data Transformation: Design and run high-performance data ingestion and transformation workflows using PySpark on Glue EMR and Databricks.
  
• Metadata & Governance: Manage metadata and data governance with AWS Glue Catalog and Azure Purview to ensure traceability compliance and security.
  
• Multi-Source Ingestion: Integrate data from APIs flat files relational and NoSQL databases and on-prem systems using AWS DataSync Transfer Family and Azure Data Factory connectors.
  
• Workflow Orchestration: Build event-driven and scheduled workflows with AWS EventBridge Step Functions SQS Azure Event Grid and Logic Apps.
  
• Graph Data Support: Collaborate with teams to support graph data integrations (Neo4j Cosmos DB Graph) for AI and knowledge graph use cases.
  
• Data Quality & Security: Implement strong DQ checks encryption IAM and access controls (AWS KMS CloudTrail Azure Key Vault Defender for Cloud).
  
• CI/CD & IaC: Automate data infrastructure deployment using CloudFormation CDK Terraform and Azure DevOps pipelines.
  
• Monitoring & Optimization: Continuously monitor and fine-tune data pipelines for performance cost-efficiency and reliability.
  

Required Skills & Experience

• 7 years of experience in data engineering and cloud data architecture with hands-on leadership in project delivery.
  
• Proven expertise in AWS (Glue Lambda S3 EventBridge Batch SQS) and Azure (Data Factory Databricks Synapse Event Grid ADLS).
  
• Advanced proficiency in PySpark Delta Lake and large-scale data processing frameworks.
  
• Strong skills in Python SQL and shell scripting for ETL automation and data validation.
  
• Experience designing end-to-end data solutions for analytics ML and AI applications.
  
• Deep understanding of data governance metadata management and compliance (HIPAA GDPR).
  
• Strong grasp of serverless event-driven and microservices-based data architectures.
  
• Exposure to graph databases (Neo4j Cosmos DB Graph).
  
• Expertise in data security IAM and encryption standards across AWS and Azure.
  
• Experience with CI/CD pipelines and Infrastructure as Code (CloudFormation CDK Terraform Azure DevOps).
  
• Excellent communication and leadership skills with the ability to collaborate across technical and non-technical teams.
  
• Proven ability to work with sales and business teams to define and deliver client-focused technical solutions.
  
• Demonstrated success in building and leading data engineering teams.
  
• Proven experience in establishing organizational data capability defining processes and overseeing execution across multiple projects.
  

Preferred Qualifications

• Masters degree in Computer Science Data Engineering or a related field.
  
• Certifications such as AWS Data Analytics Specialty Azure Data Engineer Associate or Databricks Certified Data Engineer.
  
• Experience in regulated industries (e.g. healthcare finance or government).
  
• Experience in building organizational data frameworks capability development and process standardization.