Senior Data Pipeline Engineer
Share
Job Location:
Huntsville, AL - USA
Monthly Salary:
Not Disclosed
Posted on:
20-09-2025
Vacancies:
1 Vacancy
Job Summary
This position requires an ACTIVE security clearance.
This position can be performed remote from anywhere but may require up to 15% travel.
As a Senior Data Pipeline Engineer you will play a crucial role in designing building and maintaining robust data pipelines for our customers. Your expertise will drive efficient collection storage processing and transformation of large-scale data sets in support of the U.S. Armys energy and water use metering program. The total solution is considered a system-of-systems with developed applications integrating with the Army enterprise cloud and data environment. You will work closely with cross-functional teams to ensure seamless integration and optimal performance.
Responsibilities:
Pipeline Development:
- Design develop and optimize end-to-end data pipelines for efficient data extraction transformation and loading (ETL) processes.
- Collaborate with cross-functional teams to understand data requirements and translate them into scalable pipeline solutions.
- Implement best practices for data integration ensuring high performance reliability and scalability.
Data Transformation and Quality:
- Transform raw data into usable formats ensuring data quality consistency and accuracy.
- Handle data validation cleansing and error handling to maintain data integrity.
- Monitor and proactively maintain data pipelines to ensure high service availability.
Performance Optimization:
- Continuously improve pipeline performance by identifying bottlenecks and implementing optimizations.
- Work with cloud-based technologies (e.g. AWS GCP Azure) to enhance scalability and efficiency.
Collaboration and Leadership:
- Partner with Data Scientists Analysts and other stakeholders to understand their data needs.
- Lead discussions on system enhancements process improvements and data governance.
- Mentor junior engineers and contribute to the growth of the data engineering team.
#qf
#qg
REQUIRED QUALIFICATIONS:
- Bachelors degree in Computer Science Engineering or a related field.
- 5 years of experience in data engineering with a focus on building and maintaining data pipelines.
- 1-2 years experience with building NiFI data flows or similar for Kafka and Hadoop-based NoSQL databases.
- Proficiency in ETL tools SQL and scripting languages (Python Scala etc.).
- Experience with Data Catalog and Accumulo indexes for information retrieval and discovery.
- Experience with API-led design.
- Experience with ELK stack a plus.
- Experience with cloud-based data platforms (e.g. AWS S3 Redshift Google BigQuery).
- Strong problem-solving skills and attention to detail.
- Excellent communication and collaboration abilities.
- Security Certification
- Active US Government Clearance at Secret level or higher
- Effective written and verbal communications skills for collaboration with both customers and fellow team members.
- Ability to sit for extended periods of time.
- Ability to regularly lift at least 25 pounds.
- Ability to commute to the designated onsite work location as required.
QBE is an equal opportunity/affirmative action employer. All qualified applicants will receive consideration for employment without regard to race color religion sex national origin sexual orientation gender gender-identity and/or expression age disability Veteran status genetic information pregnancy (including childbirth lactation or other related medical conditions) marital-status neurodivergence ethnicity ancestry caste military/uniformed service-member status or any other characteristic protected by applicable federal state local or international law.
Required Experience:
Senior IC
Key Skills
- Continuous Integration
- Docker
- Jenkins
- Kubernetes
- Build Automation
- S3
- ASME Codes & Standards
- Redshift
- Spark
- CI/CD
- Kafka
- Scala
About Company
POSSIBLE TO PROVEN Where some see problems, we see possibilities. QBE's unparalleled experience, in-depth insights and sought-after technical expertise allow us to mitigate mission-critical challenges into transformative solutions. Find Out More What We Do Cybersecurity Cyber risks ar ... View more
