Sr. Lead Data Engineer

Huntsville, AL - USA

Monthly Salary: Not Disclosed

Posted on: 7 hours ago

Vacancies: 1 Vacancy

Job Summary

We are seeking a Sr. Lead Data Engineer for a hybrid W2 position (Must work on our W2 and commute to Huntsville Texas when required). Will lead the design implementation and management of end-to-end enterprise-grade data solutions for our Huntsville Texas client. This role requires expertise in building and optimizing data warehouses data lakes and lake house platforms with a strong emphasis on data engineering data science and machine learning. You will work closely with cross-functional teams to create scalable and robust architectures that support advanced analytics and machine learning use cases while adhering to industry standards and best practices.

Responsibilities Include:

* Architect design and manage the entire data lifecycle from data ingestion

transformation storage and processing to advanced analytics and machine learning databases and large-scale processing systems.

* Implement robust data governance frameworks including metadata management lineage tracking security compliance and business glossary development.

* Identify design and implement internal process improvements including redesigning infrastructure for greater scalability optimizing data delivery and automating manual

processes.

* Ensure high data quality and reliability through automated data validation and testing and provide high quality clean and usable data from data sets of varying states of disorder.

* Develop and enforce architecture standards patterns and reference models for large-scale data platforms.

* Architect and implement Lambda and Kappa architectures for real-time and batch data processing workflows along with strong data modeling capabilities.

* Ability to identify and implement the most appropriate data management system and enable integration capabilities for external tools to perform ingestion compilation analytics and visualization.

Required:

* Bachelors Computer Science Data Science Engineering or a related field.

* Minimum 10 years in data engineering data architecture or a similar role with at least 3 years in a lead capacity.

* Proficient in SQL Python and big data processing frameworks (e.g. Spark Flink).

* Strong experience with cloud platforms (AWS Azure GCP) and related data services.

* Hands-on experience with data warehousing tools (e.g. Snowflake Redshift BigQuery) Databricks running on multiple cloud platforms (AWS Azure and GCP) and data lake technologies (e.g. S3 ADLS HDFS).

* Expertise in containerization and orchestration tools like Docker and Kubernetes.

* Knowledge of MLOps frameworks and tools (e.g. MLflow Kubeflow Airflow).

* Experience with real-time streaming architectures (e.g. Kafka Kinesis).

* Familiarity with Lambda and Kappa architectures for data processing.

* Enable integration capabilities for external tools to perform ingestion compilation analytics and visualization.

Preferred:

* Certifications in cloud platforms or data-related technologies.

* Familiarity with graph databases NoSQL or time-series databases.

* Knowledge of data privacy regulations (e.g. GDPR CCPA) and compliance requirements.

* Experience in implementing and managing business glossaries data governance rules metadata lineage and ensuring data quality.

* Highly experienced with AWS cloud platform and Databricks Lakehouse.

Required Experience:

Senior IC

We are seeking a Sr. Lead Data Engineer for a hybrid W2 position (Must work on our W2 and commute to Huntsville Texas when required). Will lead the design implementation and management of end-to-end enterprise-grade data solutions for our Huntsville Texas client. This role requires expertise in buil...