Lead AI Engineer, Agentic Data Lineage

JD: Lead AI Engineer Agentic Data Lineage

What You Will Do:

• Define and drive the technical strategy for a Firmwide column-level data lineage platform built on AI reasoning agents.
• Architect parsers that build Abstract Syntax Trees (ASTs) from SQL across multiple dialects stored procedures Python Java Scala and Spark jobs then resolve column-level transformations from those trees.
• Design ingestion of no-code signals from ETL and transformation tools workflow orchestrators data catalogs and BI platforms normalizing them into a common lineage model.
• Build a multi-agent system in which specialized deep agents parse reason disambiguate and reconcile conflicting signals then assemble a coherent end-to-end lineage graph.
• Establish how agents handle ambiguity (dynamic SQL runtime configuration hand-written transformations) by reasoning over context rather than failing silently or guessing.
• Model and persist lineage as a queryable knowledge graph supporting impact analysis root-cause tracing and regulatory evidence generation.
• Build confidence scoring and validation so every lineage edge carries a provenance signal and a measure of certainty with low-confidence inferences surfaced for human review.
• Partner with the Firmwide Data Office Risk Compliance and the divisional technology teams to align the platform with the Firms data governance obligations including BCBS 239 CCAR GDPR and SOX.
• Lead mentor and grow a team of AI and data engineers setting engineering standards reviewing designs and owning delivery.
• Operate within Morgan Stanleys controls for AI safety data confidentiality and model governance.

Required Qualifications:

• 8 years of software and data engineering experience including a track record of leading complex platform builds end to end.
• Strong background in program analysis or compiler techniques: parsing ASTs intermediate representations and static analysis. Hands-on experience with tools such as ANTLR tree-sitter sqlglot or language-native AST libraries.
• Practical experience building LLM-powered agents or multi-agent systems including reasoning loops tool use retrieval and orchestration frameworks (LangGraph LlamaIndex AutoGen or equivalent in-house systems).
• Deep data engineering expertise: SQL fluency across dialects plus experience with Spark distributed pipelines and modern transformation tooling such as dbt.
• Experience modeling and querying graph data with a graph database such as Neo4j Amazon Neptune or Janus Graph.
• Strong software engineering fundamentals: testing CI/CD and production-grade systems at scale.
• Demonstrated technical leadership including mentoring engineers and influencing architecture across teams.

Preferred Qualifications:

• Experience in financial services capital markets or another heavily regulated data environment.
• Familiarity with data lineage and metadata standards and tools such as OpenLineage Marquez DataHub Apache Atlas Egeria Collibra or Alation.
• Working knowledge of BCBS 239 data aggregation and risk reporting principles and how they translate into engineering requirements.
• Experience harvesting metadata from enterprise ETL tools (Informatica Talend) and BI platforms (Tableau Power BI).
• Background in knowledge representation semantic reasoning or evaluation of LLM output quality and hallucination control.

Tech Stack:
Python SQL Spark and Scala or Java for parsing targets. LLM and agent frameworks for reasoning. Graph databases for lineage storage. Workflow orchestration for pipeline execution. Cloud and on-premise deployment with containerized services. OpenLineage-aligned metadata models.

Note: Momento USA is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race color religion sex pregnancy sexual orientation gender identity national origin age protected veteran status or disability status.