Lead Data Engineer
Lead Data Engineer
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
Job Title: Lead Data Engineer
Job Location: Houston TX
Job Type: Long Term Contract
Role Overview
As a Data Engineering Technical Lead you will design and deliver robust data platforms using Databricks (Delta Live Tables Structured Streaming PySpark) Python
and SQL on Microsoft Azure. You will work with to build optimise data pipelines for optimal performance and cost ensure governance and quality and collaborate with cross-functional teams to unlock strategic insights from trading data. You will be a key contributor to customers digital transformation enabling smarter trading decisions and supporting the energy transition through data-driven innovation.
Key Responsibilities
Technical Leadership
- Architect Design and implement scalable Azure Data Lakehouse solutions using ADF ADLS Synapse Stream Analytics and Databricks.
- Lead the development of real-time data pipelines using Databricks DLT and Structured Streaming for high-volume data ingestion and transformation.
- Optimise pipeline performance and cloud resource usage to ensure cost-effective operations.
- Leverage Unity Catalog for metadata management data lineage and governance.
- Domain Expertise
- Integrate and manage data from ETRM systems to support Crude & Products trading analytics.
- Apply functional knowledge of commodity trading (Oil & Gas) to align data solutions with business needs.
- Collaborate with trading desks analysts and IT teams to deliver actionable insights.
- Governance & Quality
- Establish and enforce data governance frameworks to ensure compliance security and data quality.
- Monitor pipeline performance and proactively resolve production issues.
- Stakeholder Engagement
- Communicate effectively with business and technical stakeholders translating complex technical concepts into clear business goal/value.
- Lead executive-level discussions and presentations to align data initiatives with strategic priorities.
Team & Capability Building
- Mentor and lead a high-performing team of data engineers and analysts.
- Drive continuous learning and capability development across Azure Databricks and streaming technologies.
Required Skills & Experience
Technical Expertise:
- Deep experience with Databricks DLT Structured Streaming PySpark Python and SQL.
- Strong knowledge of Azure data services (ADF ADLS Synapse Stream Analytics).
- Proven ability to optimise Databricks for high-volume real-time data processing.
- Experience with ETRM platforms and trading data workflows.
- Leadership & Communication:
- Strong stakeholder management and executive communication skills.
- Demonstrated ownership accountability and passion for delivering impactful solutions.
- Ability to lead cross-functional teams and drive strategic data initiatives.
- Education & Certification:
- Bachelors or Masters degree in IT or related discipline.
- 12 years of experience in data engineering and analytics.
- Certifications in Azure Data Engineering and Databricks (preferred).
Nice to Have
- Experience with Kafka Hadoop Airflow and PowerBI/QlikSense.
- Familiarity with AI/ML NLP and GenAI concepts.
- Exposure to Agile/Kanban methodologies and CI/CD practices.
Employment Type
Full-time
