Software Engineer - Big Data Ingestion and Processing
Software Engineer - Big Data Ingestion and Processing
Posted on :
13-06-2025
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Monthly Salary
Not Disclosed
Salary Not Disclosed
Vacancy
1 Vacancy
Posted on :
13-06-2025
Job Description
About the Organization
Now is a great time to join Redhorse Corporation. Redhorse specializes in developing and implementing creative strategies and solutions with private state and federal customers in the areas of cultural and environmental resources services climate and energy change information technology and intelligence services. We are hiring creative motivated and talented people with a passion for doing whats right whats smart and what works.
About the Role
Redhorse is transforming how government agencies leverage data and technology. We are seeking a highly skilled Software Engineer to join our team supporting a critical intelligence mission. You will play a vital role in ingesting processing and analyzing massive datasets directly impacting the Sponsors ability to address pressing intelligence questions. You will work with cutting-edge technologies in a dynamic collaborative environment directly contributing to national security.
Key Responsibilities
- Load large datasets into the Sponsors on-premises and Cloud environments.
- Develop and maintain ingestion algorithms and schemas for large datasets.
- Analyze new large-volume datasets to optimize the data ingest processes.
- Support the creation of Apache NiFi schemas for new data loads.
- Develop software tools that efficiently preprocess modify aggregate load index and archive large data collections into clusters in near real-time.
- Ensure proper access controls are implemented.
- Generate metrics to track data ingest statistics to maintain data integrity and provenance.
- Document the data-flows according to standards set by the Sponsor.
- Engage regularly with data scientists analysts and managers.
Required Experience/Clearance
- Demonstrated professional experience in Computer Science Computer Engineering Systems Engineering or closely related discipline.
- Demonstrated professional experience with AWS cloud services including long-term storage options and cloud-based database services.
- Demonstrated experience working with Databricks.
- Demonstrated experience understanding SQL database structures and mapping them between different SQL databases.
- Demonstrated professional experience working with Apache NiFi.
- Demonstrated professional experience working with large data and high-performance compute clusters such as Hadoop or similar.
- Demonstrated experience with API development techniques.
- Demonstrated experience developing and deploying ETL processes for large data sets.
- Demonstrated experience creating operating system level scripts to perform ETL operations on SQL databases.
- Demonstrated professional experience with version control systems preferably Git.
- Demonstrated experience testing the development of software solutions for the extraction transformation and loading of data using the most efficient languages for the task such as NiFi Python and SQL.
- Demonstrated experience implementing multiprocessing data-flows to parallelize ingest operations.
- Minimum 5-7 years of relevant experience.
Desired Experience
- Demonstrated experience with the Sponsors data environment.
- Demonstrated experience exhibiting strong coordination and collaboration skills.
- Demonstrated experience working with full-stack developers to deploy applications that leverage large data sets.
- Demonstrated experience communicating technical concepts to non-technical audiences.
Equal Opportunity Employer/Veterans/Disabled
Accommodations:
If you are a qualified individual with a disability or a disabled veteran you may request a reasonable accommodation if you are unable or limited in your ability to access job openings or apply for a job on this site as a result of your disability. You can request reasonable accommodations by contacting Talent Acquisition
Redhorse Corporation shall in its discretion modify or adjust the position to meet Redhorses changing needs.
This job description is not a contract and may be adjusted as deemed appropriate in Redhorses sole discretion.
Employment Type
Full-Time
Key Skills
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
