Data Engineer

Data Engineer (this is a potential role that is going to be opened soon)

Rate : 5558/hr C2C

Length of contract 6 months and they renew every 6 months ( Long Term)

Location (onsite remote or hybrid). Hybrid Cincinnati OH

Visa restrictions(if any). EAD/GC and above

Integrator/End Client(if known). Kroger

Top 3 skills needed: Azure Databricks Spark/PySpark SQL

Job Description:

We are seeking a highly skilled Senior Data Engineer to join our team in building a cuttingedge data platform for the warehouse and manufacturing domains within The Companys supply chain space. As a Senior Data Engineer you will be responsible for designing implementing and optimizing data pipelines to handle terabytes of data in Azure cloud environments. You will collaborate with crossfunctional teams to architect scalable data solutions using Azure Databricks Cosmos DB Synapse Analytics and SQL Hyperscale.

The ideal candidate for the Data Engineer role will possess strong technical proficiency in a variety of tools and technologies relevant to data engineering in Azure cloud environments.

This includes demonstrated expertise in Azure Databricks for building and optimizing data pipelines as well as experience with Azure Synapse Analytics for data warehousing and analytics.

Proficiency in SQL is essential for querying and manipulating data within these platforms along with programming skills in Python and PySpark for data processing and transformation tasks.

Additionally the candidate should have experience working with distributed computing frameworks and familiarity with cloudbased storage solutions such as Azure Blob Storage and Azure Data Lake Storage.

Knowledge of data modeling ETL processes and data security best practices is also crucial for designing and implementing scalable and reliable data solutions.

The ability to troubleshoot and optimize data processing workflows along with strong problemsolving skills is essential for ensuring the performance and efficiency of data pipelines in highthroughput environments.

Key Responsibilities

Design and implement robust data pipelines for highthroughput data ingestion transformation and storage.

Optimize data processing workflows for performance and scalability in Azure cloud environments.

Collaborate with data scientists and business analysts to understand requirements and deliver data engineering solutions that meet business objectives.

Ensure data consistency reliability and security in distributed data processing environments.

Monitor and troubleshoot data pipeline failures ensuring minimal downtime and maximum reliability.

Refine Requirements

Design and develop Informatica solutions.

Perform Unit and Integration Testing

Handle deployments

Draft architectural diagrams interface specifications and other design documents

Participate in the development and communication of data strategy and roadmaps across the technology organization to support project portfolio and business strategy.

Innovate develop and drive the development and communication of data strategy and roadmaps across the technology organization to support project portfolio.

Drive the development and communication of enterprise standards for data domains and data solutions focusing on simplified integration and streamlined operational and analytical uses.

Drive digital innovation by leveraging innovative new technologies and approaches to renovate extend and transform the existing core data assets including SQLbased NoSQLbased and Cloudbased data platforms.

Define highlevel migration plans to address the gaps between the current and future state typically in sync with the budgeting or other capital planning processes.

Lead the analysis of the technology environment to detect critical deficiencies and recommend solutions for improvement.

Mentor team members in data principles patterns processes and practices

Promote the reuse of data assets including the management of the data catalog for reference.

Draft and review architectural diagrams interface specifications and other design documents