Enter a job title or keyword
Lead Cloud Data Engineer – Data Mesh & AI
Job Location:
Job Location:
San Francisco, CA - USA
Hourly Salary:
USD 90 - 90
Hourly Salary:
Posted:
8 September 2026 (12 hours ago)
Posted:
Application Deadline:
6 December 2026
Application Deadline:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
Software Guidance & Assistance Inc. (SGA) is searching for a Lead Cloud Data Engineer Data Mesh & AI for a contract assignment with one of our premier Financial Services clients in San Francisco CA.
Responsibilities :
We are seeking an experienced Lead Data Engineer to join our team and drive the development of modern data platforms and AI-augmented solutions. This role is critical to supporting ongoing operations of Large Data Warehouse architecting the Large Data Hub and preparing for the enterprise data mesh modernization program.
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race Color Sex Sexual Orientation Gender Identity Religion National Origin Disability Veteran Status Age Marital Status Pregnancy Genetic Information or Other Legally Protected Status. We are committed to providing access equal opportunity and reasonable accommodation for individuals with disabilities in employment and our services programs and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy.
Responsibilities :
We are seeking an experienced Lead Data Engineer to join our team and drive the development of modern data platforms and AI-augmented solutions. This role is critical to supporting ongoing operations of Large Data Warehouse architecting the Large Data Hub and preparing for the enterprise data mesh modernization program.
- Data Engineering & Pipeline Development
- Develop and maintain end-to-end ETL/ELT pipelines for data ingestion from multiple sources (databases APIs streaming) transformation quality validation data modeling and consumption layer optimization using AWS services (Glue EMR Lambda Kinesis Step Functions)
- Implement data governance policies metadata management data lineage tracking fine-grained access controls and automated data quality frameworks with validation rules anomaly detection and monitoring/alerting mechanisms
- Data Mesh Architecture
- Design and implement Data Mesh architecture on AWS with federated governance defining domain boundaries data product specifications and self-serve infrastructure patterns across Databricks Starburst Collibra and Immuta platforms
- Develop reusable frameworks accelerators and self-serve tools enabling domain teams to independently publish discover and consume data products while ensuring performance optimization and cost efficiency
- DevSecOps & Infrastructure
- Integrate DevSecOps practices including CI/CD pipelines infrastructure as code (Terraform/CloudFormation) automated security scanning compliance validation and disaster recovery strategies
- AI-Augmented Development
- Design and build agentic AI solutions RAG pipelines and intelligent automation using LLMs orchestration frameworks and AI-powered development tools (Copilot Claude Code) to enhance platform capabilities and accelerate SDLC
- Technical Leadership & Collaboration
- Provide L3 technical support and mentor engineering teams on data mesh principles and modern data technologies
- Collaborate with stakeholders to translate business requirements into technical solutions
- Lead architecture design reviews and create comprehensive technical documentation
- Data Mesh Architecture & AWS (5 years required):
- Deep experience designing distributed data architectures on AWS (S3 Glue Lake Formation EMR Redshift)
- Hands-on implementation of data mesh principles including domain-oriented data ownership data as a product and federated computational governance
- Modern Data Platform Technologies:
- Proficiency: Databricks (Unity Catalog Delta Lake) Starburst/Trino (federated query) Collibra (data governance) or Immuta (dynamic data access control)
- Ability to architect multi-technology solutions
- DevSecOps & Infrastructure as Code:
- Strong experience with CI/CD pipelines (GitLab/GitHub Actions Jenkins)
- Containerization expertise (Docker/ECS)
- Infrastructure automation (Terraform CloudFormation)
- Security compliance frameworks for AWS GovCloud environments
- AI-Augmented Development & Engineering:
- Demonstrated use of generative AI tools for accelerating SDLC activities including code generation testing and documentation
- Experience designing or implementing agentic AI solutions using Amazon Bedrock or similar frameworks
- Leadership & Soft Skills
- Proven ability to translate business requirements into technical solutions
- Experience leading technical design sessions and mentoring engineering teams
- Strong communication skills to convey complex architectural concepts to both technical and non-technical stakeholders
- Experience working across hybrid cloud/on-premises environments
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race Color Sex Sexual Orientation Gender Identity Religion National Origin Disability Veteran Status Age Marital Status Pregnancy Genetic Information or Other Legally Protected Status. We are committed to providing access equal opportunity and reasonable accommodation for individuals with disabilities in employment and our services programs and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy.
Required Experience:
IC
About Company
About the company SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say ... View more
