Enter a job title or keyword
Lead Cloud Data Engineer – Data Mesh & AI
Job Location:
Job Location:
San Francisco, CA - USA
Hourly Salary:
USD 90 - 105
Hourly Salary:
Posted:
8 September 2026 (10 hours ago)
Posted:
Application Deadline:
6 December 2026
Application Deadline:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
VARITE is looking for a qualified Lead Cloud Data Engineer Data Mesh & AI in San Francisco CA 94105
What does the client do
AcentralbankingsystemoftheUnitedStatesthatmanagesmonetarypolicyregulatesfinancialinstitutionsprovides bankingservicesandactsasalenderoflastresortduringfinancialcrises.
What do we do
Established in the Year 2000 VARITE is an award-winning minority business enterprise providing global consulting & staffing services to Fortune 1000 companies and government agencies. With 850 global consultants VARITE is committed to delivering excellence to its customers by leveraging its global experience and expertise in providing comprehensive scientific engineering technical and non-technical staff augmentation and talent acquisition services.
Job Title: Lead Cloud Data Engineer Data Mesh & AI
Duration: 18 Months
Location: San Francisco CA 94105
Mode: Onsite role
Pay rate: $90/hr. - $105/hr on W2
Work Authorization: Only USC or GC
Heres What Youll Do
**** Job Title is: Lead Cloud Data Engineer Data Mesh & AI*****
We are seeking an experienced Lead Data Engineer to join our team and drive the development of modern data platforms and AI-augmented solutions. This role is critical to supporting ongoing operations of Large Data Warehouse architecting the Large Data Hub and preparing for the enterprise data mesh modernization program.
Responsibilities:
Data Engineering & Pipeline Development:
Develop and maintain end-to-end ETL/ELT pipelines for data ingestion from multiple sources (databases APIs streaming) transformation quality validation data modeling and consumption layer optimization using AWS services (Glue EMR Lambda Kinesis Step Functions)
Implement data governance policies metadata management data lineage tracking fine-grained access controls and automated data quality frameworks with validation rules anomaly detection and monitoring/alerting mechanisms
Data Mesh Architecture
Design and implement Data Mesh architecture on AWS with federated governance defining domain boundaries data product specifications and self-serve infrastructure patterns across Databricks Starburst Collibra and Immuta platforms
Develop reusable frameworks accelerators and self-serve tools enabling domain teams to independently publish Client and consume data products while ensuring performance optimization and cost efficiency
DevSecOps & Infrastructure
Integrate DevSecOps practices including CI/CD pipelines infrastructure as code (Terraform/CloudFormation) automated security scanning compliance validation and disaster recovery strategies
AI-Augmented Development
Design and build agentic AI solutions RAG pipelines and intelligent automation using LLMs orchestration frameworks and AI-powered development tools (Copilot Claude Code) to enhance platform capabilities and accelerate SDLC
Technical Leadership & Collaboration
Provide L3 technical support and mentor engineering teams on data mesh principles and modern data technologies
Collaborate with stakeholders to translate business requirements into technical solutions
Lead architecture design reviews and create comprehensive technical documentation
Required Qualifications:
Data Mesh Architecture & AWS (5 years required):
Deep experience designing distributed data architectures on AWS (S3 Glue Lake Formation EMR Redshift)
Hands-on implementation of data mesh principles including domain-oriented data ownership data as a product and federated computational governance
Modern Data Platform Technologies:
Proficiency: Databricks (Unity Catalog Delta Lake) Starburst/Trino (federated query) Collibra (data governance) or Immuta (dynamic data access control)
Ability to architect multi-technology solutions
DevSecOps & Infrastructure as Code:
Strong experience with CI/CD pipelines (GitLab/GitHub Actions Jenkins)
Containerization expertise (Docker/ECS)
Infrastructure automation (Terraform CloudFormation)
Security compliance frameworks for AWS GovCloud environments
AI-Augmented Development & Engineering:
Demonstrated use of generative AI tools for accelerating SDLC activities including code generation testing and documentation
Experience designing or implementing agentic AI solutions using Amazon Bedrock or similar frameworks
Leadership & Soft Skills:
Proven ability to translate business requirements into technical solutions
Experience leading technical design sessions and mentoring engineering teams
Strong communication skills to convey complex architectural concepts to both technical and non-technical stakeholders
Experience working across hybrid cloud/on-premises environments
Benefits:
We offer a comprehensive benefits package designed to support the health well-being and financial security of our employees and their families. Eligible employees may receive:
Health Insurance: Medical dental and vision coverage
Retirement Plans: Participation in a company-sponsored retirement savings plan
Legal Service Plans Offering access to attorneys for legal advice and representation
If this opportunity interests you please respond by clicking on EasyApply.
In case you are not currently available or interested in pursuing this opportunity please forward this email among your circle who might be suitable and interested in any of our open requirements. VARITE has a robust Candidate Referral Fee plan where you will receive a one-time referral fee if the referred candidate successfully works with VARITE on an assignment for at least 1 month or 160 Hours.
VARITE is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race color religion gender gender identity or expression sexual orientation national origin genetics disability age or veteran status.
What does the client do
AcentralbankingsystemoftheUnitedStatesthatmanagesmonetarypolicyregulatesfinancialinstitutionsprovides bankingservicesandactsasalenderoflastresortduringfinancialcrises.
What do we do
Established in the Year 2000 VARITE is an award-winning minority business enterprise providing global consulting & staffing services to Fortune 1000 companies and government agencies. With 850 global consultants VARITE is committed to delivering excellence to its customers by leveraging its global experience and expertise in providing comprehensive scientific engineering technical and non-technical staff augmentation and talent acquisition services.
Job Title: Lead Cloud Data Engineer Data Mesh & AI
Duration: 18 Months
Location: San Francisco CA 94105
Mode: Onsite role
Pay rate: $90/hr. - $105/hr on W2
Work Authorization: Only USC or GC
Heres What Youll Do
**** Job Title is: Lead Cloud Data Engineer Data Mesh & AI*****
We are seeking an experienced Lead Data Engineer to join our team and drive the development of modern data platforms and AI-augmented solutions. This role is critical to supporting ongoing operations of Large Data Warehouse architecting the Large Data Hub and preparing for the enterprise data mesh modernization program.
Responsibilities:
Data Engineering & Pipeline Development:
Develop and maintain end-to-end ETL/ELT pipelines for data ingestion from multiple sources (databases APIs streaming) transformation quality validation data modeling and consumption layer optimization using AWS services (Glue EMR Lambda Kinesis Step Functions)
Implement data governance policies metadata management data lineage tracking fine-grained access controls and automated data quality frameworks with validation rules anomaly detection and monitoring/alerting mechanisms
Data Mesh Architecture
Design and implement Data Mesh architecture on AWS with federated governance defining domain boundaries data product specifications and self-serve infrastructure patterns across Databricks Starburst Collibra and Immuta platforms
Develop reusable frameworks accelerators and self-serve tools enabling domain teams to independently publish Client and consume data products while ensuring performance optimization and cost efficiency
DevSecOps & Infrastructure
Integrate DevSecOps practices including CI/CD pipelines infrastructure as code (Terraform/CloudFormation) automated security scanning compliance validation and disaster recovery strategies
AI-Augmented Development
Design and build agentic AI solutions RAG pipelines and intelligent automation using LLMs orchestration frameworks and AI-powered development tools (Copilot Claude Code) to enhance platform capabilities and accelerate SDLC
Technical Leadership & Collaboration
Provide L3 technical support and mentor engineering teams on data mesh principles and modern data technologies
Collaborate with stakeholders to translate business requirements into technical solutions
Lead architecture design reviews and create comprehensive technical documentation
Required Qualifications:
Data Mesh Architecture & AWS (5 years required):
Deep experience designing distributed data architectures on AWS (S3 Glue Lake Formation EMR Redshift)
Hands-on implementation of data mesh principles including domain-oriented data ownership data as a product and federated computational governance
Modern Data Platform Technologies:
Proficiency: Databricks (Unity Catalog Delta Lake) Starburst/Trino (federated query) Collibra (data governance) or Immuta (dynamic data access control)
Ability to architect multi-technology solutions
DevSecOps & Infrastructure as Code:
Strong experience with CI/CD pipelines (GitLab/GitHub Actions Jenkins)
Containerization expertise (Docker/ECS)
Infrastructure automation (Terraform CloudFormation)
Security compliance frameworks for AWS GovCloud environments
AI-Augmented Development & Engineering:
Demonstrated use of generative AI tools for accelerating SDLC activities including code generation testing and documentation
Experience designing or implementing agentic AI solutions using Amazon Bedrock or similar frameworks
Leadership & Soft Skills:
Proven ability to translate business requirements into technical solutions
Experience leading technical design sessions and mentoring engineering teams
Strong communication skills to convey complex architectural concepts to both technical and non-technical stakeholders
Experience working across hybrid cloud/on-premises environments
Benefits:
We offer a comprehensive benefits package designed to support the health well-being and financial security of our employees and their families. Eligible employees may receive:
Health Insurance: Medical dental and vision coverage
Retirement Plans: Participation in a company-sponsored retirement savings plan
Legal Service Plans Offering access to attorneys for legal advice and representation
If this opportunity interests you please respond by clicking on EasyApply.
In case you are not currently available or interested in pursuing this opportunity please forward this email among your circle who might be suitable and interested in any of our open requirements. VARITE has a robust Candidate Referral Fee plan where you will receive a one-time referral fee if the referred candidate successfully works with VARITE on an assignment for at least 1 month or 160 Hours.
VARITE is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race color religion gender gender identity or expression sexual orientation national origin genetics disability age or veteran status.
Required Experience:
IC
