Lead AI Engineer, Agentic Data Lineage

Lead AI Engineer, Agentic Data Lineage

What You Will Do

• Define and drive the technical strategy for a Firmwide, column-level data lineage platform built on AI reasoning agents.
• Architect parsers that build Abstract Syntax Trees (ASTs) from SQL across multiple dialects, stored procedures, Python, Java, Scala, and Spark jobs, then resolve column-level transformations from those trees.
• Design ingestion of no-code signals from ETL and transformation tools, workflow orchestrators, data catalogs, and BI platforms, normalizing them into a common lineage model.
• Build a multi-agent system in which specialized deep agents parse, reason, disambiguate, and reconcile conflicting signals, then assemble a coherent end-to-end lineage graph.
• Establish how agents handle ambiguity (dynamic SQL, runtime configuration, hand-written transformations) by reasoning over context rather than failing silently or guessing.
• Model and ...