Lead Data Engineer

Position: Lead Data Engineer
Location: Charlotte NC 28217 (2 days/week Onsite)
Duration: 6month contract to hire

Requirements:

• Python SQL and PySpark
• Azure preferred but open to other (GCP AWS)
• Databricks Snowflake and Fabric
• Batch and realtime streaming building pipelines
• Gathering requirements from the business
• Mentoring some of the Data Engineers on the team room for growth

Any nicetohave skills that would make them exceptional Certifications 2 yrs data architecture data engineering patterns git repo selfstarter and gogetter attitude software engineering background ML experience

Job Description: As the Technical Lead Data Engineer your primary responsibility will be to spearhead the design development and implementation of data solutions to empower our organization to derive actionable insights from intricate datasets. You will take the lead in guiding a team of data engineers fostering collaboration with crossfunctional teams and spearheading initiatives geared towards fortifying our data infrastructure CI/CD pipelines and analytics capabilities

Responsibilities:

• Apply advanced knowledge of Data Engineering principles methodologies and techniques to design and implement data loading and aggregation frameworks across broad areas of the organization.
• Gather and process raw structured semistructured and unstructured data using batch and realtime data processing frameworks.
• Implement and optimize data solutions in enterprise data warehouses and big data repositories focusing primarily on movement to the cloud.
• Drive new and enhanced capabilities to Enterprise Data Platform partners to meet the needs of product / engineering / business.
• Experience building enterprise systems especially using Databricks Snowflake and platforms like Azure AWS GCP etc
• Leverage strong Python Spark SQL programming skills to construct robust pipelines for efficient data processing and analysis.
• Implement CI/CD pipelines for automating build test and deployment processes to accelerate the delivery of data solutions.
• Implement data modeling techniques to design and optimize data schemas ensuring data integrity and performance.
• Drive continuous improvement initiatives to enhance performance reliability and scalability of our data infrastructure.
• Collaborate with data scientists analysts and other stakeholders to understand business requirements and translate them into technical solutions.
• Implement best practices for data governance security and compliance to ensure the integrity and confidentiality of our data assets.

Qualifications:

• Bachelors or masters degree in computer science Engineering or a related field.
• Proven experience (8) in a data engineering role with expertise in designing and building data pipelines ETL processes and data warehouses.
• Strong proficiency in SQL Python and Spark programming languages.
• Strong experience with cloud platforms such as AWS Azure or GCP is a must.
• Handson experience with big data technologies such as Hadoop Spark Kafka and distributed computing frameworks.
• Knowledge of data lake and data warehouse solutions including Databricks Snowflake Amazon Redshift Google BigQuery Azure Data Factory Airflow etc.
• Experience in implementing CI/CD pipelines for automating build test and deployment processes.
• Solid understanding of data modeling concepts data warehousing architectures and data management best practices.

SQL , DATA SOLUTIONS , CI/CD , METHODOLOGIES , AWS , ENGINEER , SPARK , DATA ARCHITECT , PYTHON , ML , AZURE