Principal Data Engineer
Share
Job Location:
Houston, MS - USA
Monthly Salary:
Not Disclosed
Posted on:
Yesterday
Vacancies:
1 Vacancy
Job Summary
Join the ECI Groups Professional Technical Services Network as a Principal Data Engineer.This is a Houston based a six-month contract role with an anticipated March 30 2006 start date and ending on September 30 2026.
NOTE: You must be a US Citizen or Permanent US resident to qualify for the position.
Roles and Responsibilities:
The Principal Data Engineer is the one who conceptualizes and designs data foundations and end to end solutions including data platforms for the Shell Business to maximize value from data. The role helps create a data-driven thinking within the organization not just within IT teams but also in the wider business stakeholder community. A Principal Data Engineer is expected to be a subject matter expert and an active thought partner in the organizations data strategy and the key driver to define the Vison and Data Strategy for robust and future ready IT solutions
Role Objective
This isnota standard data engineer role.We are looking for adeeply technical hands-on individual contributorwho can:
Current Platform & Architecture
Data Flow:
Key Challenges the Role Is Meant to Solve
1. Data Latency
Key Characteristics
Portfolio: Shell IDT/Downstream & Renewables IDT/Trading & Supply/TS - Crude & Products/TS - Crude Trading/TS - Crude Investments
Program:
Project: Crude Shift WONA - Crude Data Exchange
Description: Considering the decision to pause the work for the DEX to Endur migration focus remains on advancing the Crude PDS Transformation with a key priority to establish a global view of data with standardized granularity across the T&S Crude Business. This funding request is to start work on modernizing the WONA Data Warehouse using Azure cloud technology with the target of cleaning and aligning the data between WONA and NA. Ambition to eventually move NA ODW into this same platform.
Key focus
Business case: Considering the decision to pause the work for the DEX to Endur migration focus remains on advancing the Crude PDS Transformation with a key priority to establish a global view of data with standardized granularity across the T&S Crude Business. This funding request is to start work on modernizing the WONA Data Warehouse using Azure cloud technology with the target of cleaning and aligning the data between WONA and NA. Ambition to eventually move NA ODW into this same platform.
Business Application: CRUDE AND PRODUCTS DATA EXCHANGE Crude Data Exchange
Project Type: IT Project
Project Class: Application
Please Note: The ECI Group uses AIassisted tools in the early stages of application review with all decisions guided by human oversight and fully compliant with General Data Protection Regulation (GDPR) and ECI Groups applicable data protection policies and Privacy Notice. Bottom of Form
NOTE: You must be a US Citizen or Permanent US resident to qualify for the position.
Roles and Responsibilities:
The Principal Data Engineer is the one who conceptualizes and designs data foundations and end to end solutions including data platforms for the Shell Business to maximize value from data. The role helps create a data-driven thinking within the organization not just within IT teams but also in the wider business stakeholder community. A Principal Data Engineer is expected to be a subject matter expert and an active thought partner in the organizations data strategy and the key driver to define the Vison and Data Strategy for robust and future ready IT solutions
Role Objective
This isnota standard data engineer role.We are looking for adeeply technical hands-on individual contributorwho can:
- Diagnoseperformance latency and cost issuesin a large-scale cloud data platform
- Take atop-down platform-level viewacross multiple projects
- Improvearchitecture efficiency and cost optimization not just write Spark code
- Act as atechnical problem-solver and mentor guiding other data engineers
Current Platform & Architecture
Data Flow:
- On-premise systems Cloud (Azure)
- Streaming ingestion Azure Data Lake Storage (ADLS)
- Data processed intotwo separate containers:
- Crude trading
- Product trading
- Qlik Replicate(formerly Attunity)
- Streaming data from on-prem to Azure
- Azure Data Lake Storage (ADLS)
- Databricks
- Delta Live Tables (DLT)
- Spark / PySpark
- Python
- SQL(complex queries and procedures)
Key Challenges the Role Is Meant to Solve
1. Data Latency
- High-volume streaming data
- End-to-end latency issues that need root-cause analysis
- DLT costs arefar higher than expected
- Known contributors:
- Very high data volume (expected)
- Inefficient lookup logic used to split data into separate containers
- The current solution works but isnot optimal
Key Characteristics
- Hands-on technical delivery leader
- Visible community leader for data engineers
- Conducts quality community & conference presentations/trainings
- Has vast array of technology skills with deep expertise in at least 2core technologies (Azure AWS SAP)
- A skilled communicator capable of speaking to both technical developers and business managers.
- Able to drive change and improvement through a deep understanding of the inter-relationship between People Process Data and Technology
- Coordinate the change management process incident management and problem management process.
- Drive implementation efficiency and effectiveness across the pilots and future projects to minimize cost increase speed of implementation and maximize value delivery.
- Drives & contributes to community building initiatives like CoE CoP.
- AWS/Azure/SAP Master
- ELT Master
- Data Modeling Master
- Data Integration & Ingestion Master
- Data Manipulation and Processing Skill
- GITHUB Action Azure DevOps Skill
- Data factory Databricks SQL DB Synapse Stream Analytics Glue Airflow Kinesis Redshift SonarQube PyTest Skill
- Strong Python(mandatory)
- PySpark(advanced not basic)
- Advanced SQL
- Complex queries
- Stored procedures
- Analytical logic
- Experience in project management running a scrum team.
- Experience working with BPC Planning.
- Exposure to working with external technical ecosystem.
- Managing small teams.
- MKDocs documentation
Portfolio: Shell IDT/Downstream & Renewables IDT/Trading & Supply/TS - Crude & Products/TS - Crude Trading/TS - Crude Investments
Program:
Project: Crude Shift WONA - Crude Data Exchange
Description: Considering the decision to pause the work for the DEX to Endur migration focus remains on advancing the Crude PDS Transformation with a key priority to establish a global view of data with standardized granularity across the T&S Crude Business. This funding request is to start work on modernizing the WONA Data Warehouse using Azure cloud technology with the target of cleaning and aligning the data between WONA and NA. Ambition to eventually move NA ODW into this same platform.
Key focus
Business case: Considering the decision to pause the work for the DEX to Endur migration focus remains on advancing the Crude PDS Transformation with a key priority to establish a global view of data with standardized granularity across the T&S Crude Business. This funding request is to start work on modernizing the WONA Data Warehouse using Azure cloud technology with the target of cleaning and aligning the data between WONA and NA. Ambition to eventually move NA ODW into this same platform.
Business Application: CRUDE AND PRODUCTS DATA EXCHANGE Crude Data Exchange
Project Type: IT Project
Project Class: Application
Please Note: The ECI Group uses AIassisted tools in the early stages of application review with all decisions guided by human oversight and fully compliant with General Data Protection Regulation (GDPR) and ECI Groups applicable data protection policies and Privacy Notice. Bottom of Form
Required Experience:
Staff IC
Key Skills
- Design
- Academics
- AutoCAD 3D
- Cafe
- Fabrication
- Java
About Company
ECI Group is a technology led engineering services company with expertise ranging from process plant design and development to waste recycling
