Sr. Lead Data Engineer

Huntsville, AL - USA

Monthly Salary: Not Disclosed

Posted on: 30+ days ago

Vacancies: 1 Vacancy

Job Summary

Job Title: Sr. Lead Data Engineer (Only W2)
Duration: 9 Months Contract with possibility to extension
LOCATION: ** PREFERRED: Home office in Huntsville TX. May work remotely but would need the capability to report to the office with advanced notice. **

Job Description:

We are seeking a highly skilled and experienced professional to lead the design implementation and management of end-to-end enterprise-grade data solutions. This role involves expertise in building and optimizing data warehouses data lakes and lakehouse platforms with a strong emphasis on data engineering data science and machine learning. You will work closely with cross-functional teams to create scalable and robust architectures that support advanced analytics and machine learning use cases while adhering to industry standards and best practices.

Education: Bachelors Computer Science Data Science Engineering or a related field.
Experience: Minimum 10 years in data engineering data architecture or a similar role with at least 3 years in a lead capacity.

Responsibilities Include:

Architect design and manage the entire data lifecycle from data ingestion
transformation storage and processing to advanced analytics and machine learning databases and large-scale processing systems.
Implement robust data governance frameworks including metadata management lineage tracking security compliance and business glossary development.
Identify design and implement internal process improvements including redesigning infrastructure for greater scalability optimizing data delivery and automating manual
processes.
Ensure high data quality and reliability through automated data validation and testing and provide high quality clean and usable data from data sets of varying states of disorder.
Develop and enforce architecture standards patterns and reference models for large-scale data platforms.
Architect and implement Lambda and Kappa architectures for real-time and batch data processing workflows along with strong data modeling capabilities.
Ability to identify and implement the most appropriate data management system and enable integration capabilities for external tools to perform ingestion compilation analytics and visualization.

REQUIRED SKILLS:

Proficient in SQL Python and big data processing frameworks (e.g. Spark Flink).
Strong experience with cloud platforms (AWS Azure GCP) and related data services.
Hands-on experience with data warehousing tools (e.g. Snowflake Redshift BigQuery) Databricks running on multiple cloud platforms (AWS Azure and GCP) and data lake technologies (e.g. S3 ADLS HDFS).
Expertise in containerization and orchestration tools like Docker and Kubernetes.
Knowledge of MLOps frameworks and tools (e.g. MLflow Kubeflow Airflow).
Experience with real-time streaming architectures (e.g. Kafka Kinesis).
Familiarity with Lambda and Kappa architectures for data processing.
Enable integration capabilities for external tools to perform ingestion compilation analytics and visualization.

PREFERRED SKILLS:

Certifications in cloud platforms or data-related technologies.
Familiarity with graph databases NoSQL or time-series databases.
Knowledge of data privacy regulations (e.g. GDPR CCPA) and compliance requirements.
Experience in implementing and managing business glossaries data governance rules metadata lineage and ensuring data quality.
Highly experienced with AWS cloud platform and Databricks Lakehouse.

Thanks & Regards

DISCLAIMER: The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review retransmission dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error please contact the sender and delete the material from any computer or if you want to be REMOVED please reply with REMOVE in the Subject line of this email.

Job Title: Sr. Lead Data Engineer (Only W2) Duration: 9 Months Contract with possibility to extension LOCATION: ** PREFERRED: Home office in Huntsville TX. May work remotely but would need the capability to report to the office with advanced notice. ** Job Description: We are seeking a highly skill...

Education: Bachelors Computer Science Data Science Engineering or a related field.
Experience: Minimum 10 years in data engineering data architecture or a similar role with at least 3 years in a lead capacity.

Responsibilities Include:

Architect design and manage the entire data lifecycle from data ingestion
transformation storage and processing to advanced analytics and machine learning databases and large-scale processing systems.
Implement robust data governance frameworks including metadata management lineage tracking security compliance and business glossary development.
Identify design and implement internal process improvements including redesigning infrastructure for greater scalability optimizing data delivery and automating manual
processes.
Ensure high data quality and reliability through automated data validation and testing and provide high quality clean and usable data from data sets of varying states of disorder.
Develop and enforce architecture standards patterns and reference models for large-scale data platforms.
Architect and implement Lambda and Kappa architectures for real-time and batch data processing workflows along with strong data modeling capabilities.
Ability to identify and implement the most appropriate data management system and enable integration capabilities for external tools to perform ingestion compilation analytics and visualization.

REQUIRED SKILLS:

Proficient in SQL Python and big data processing frameworks (e.g. Spark Flink).
Strong experience with cloud platforms (AWS Azure GCP) and related data services.
Hands-on experience with data warehousing tools (e.g. Snowflake Redshift BigQuery) Databricks running on multiple cloud platforms (AWS Azure and GCP) and data lake technologies (e.g. S3 ADLS HDFS).
Expertise in containerization and orchestration tools like Docker and Kubernetes.
Knowledge of MLOps frameworks and tools (e.g. MLflow Kubeflow Airflow).
Experience with real-time streaming architectures (e.g. Kafka Kinesis).
Familiarity with Lambda and Kappa architectures for data processing.
Enable integration capabilities for external tools to perform ingestion compilation analytics and visualization.

PREFERRED SKILLS:

Certifications in cloud platforms or data-related technologies.
Familiarity with graph databases NoSQL or time-series databases.
Knowledge of data privacy regulations (e.g. GDPR CCPA) and compliance requirements.
Experience in implementing and managing business glossaries data governance rules metadata lineage and ensuring data quality.
Highly experienced with AWS cloud platform and Databricks Lakehouse.

Thanks & Regards