Senior Data Engineer
Share
Job Location:
Indianapolis, IN - USA
Monthly Salary:
Not Disclosed
Posted on:
Yesterday
Vacancies:
1 Vacancy
Job Summary
Overview
Knowledge Services is seeking a Senior Data Engineer for a contract through the end of Dec 2026 (potential for extension) in Indianapolis IN.
- This role will work a hybrid schedule and be onsite Tuesday Wednesday and Thursday each week in Indianapolis.
Responsibilities
Senior Data Engineer Overview:
This role focuses on supporting our clients enterprise Data Platform by building and supporting a Lakehouse architecture on AWS leveraging Amazon EKS for container orchestration.
The platform will enable secure scalable and high-performance data processing for diverse agricultural and scientific datasets.
It will integrate cloud-native technologies and distributed systems to manage data ingestion transformation and curation supporting advanced analytics machine learning and operational workflows.
Responsibilities:
The engineer will design and implement scalable cloud-native data solutions to support an enterprise Data Platform.
This includes building and maintaining data pipelines and infrastructure on AWS leveraging Amazon EKS for container orchestration S3 for storage and Trino for distributed query processing.
The role involves developing workflows using Dagster for orchestration and DBT for transformation ensuring efficient ingestion curation and delivery of data across the Lakehouse architecture.
The engineer will focus on performance optimization security and reliability while enabling advanced analytics and machine learning capabilities for diverse agricultural and scientific datasets.
Qualifications
Senior Data Engineer Requirements:
710 years in data engineering
Programming: Strong proficiency in Python and SQL
Data Platform & Lakehouse:
Experience with Lakehouse architecture and modern open-source data platforms
Familiarity with Apache Iceberg Delta Lake and distributed processing frameworks (e.g. Spark)
Ability to design and implement cloud-native data pipelines and infrastructure on AWS
Experience with Dagster for orchestration and workflow management
Knowledge of Trino for federated query processing across heterogeneous data sources
Familiarity with MinIO for object storage in Lakehouse environments
Data Virtualization experience for integrating multiple data sources seamlessly
Cloud Technologies: Hands-on experience with AWS services including:
S3
ECS
Lambda
CloudFormation or Terraform
EKS
API Gateway
SQS
CloudWatch
Networking:Solid understanding of networking fundamentals for secure and optimized data platform communication
Security & Access Control:
Experience implementing role-based access control (RBAC) and attribute-based access control (ABAC)
Knowledge of data security policies encryption and identity-based access management (IAM)
Containerization: Experience with Docker and Kubernetes (EKS)
Monitoring & Logging: Proficiency with Datadog
Data Warehousing: Solid understanding of modern solutions (e.g. Snowflake Redshift)
Infrastructure as Code: Proficiency with Terraform or CloudFormation
Stakeholder Collaboration: Ability to translate complex requirements into scalable data solutions
Soft Skills:Excellent problem-solving attention to detail and communicating with end users and troubleshooting
Preferred Skills
Data Platform Expertise:
Experience with Lakehouse architecture and technologies like Delta Lake or Apache Iceberg
Familiarity with AWS services (S3 Glue EMR Lambda) and Amazon EKS for container orchestration
Proficiency in Databricks Spark and distributed data processing frameworks
Knowledge of Infrastructure as Code tools such as Terraform for cloud deployments
Understanding of data governance security and compliance in cloud environments
Knowledge Services is an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race color religion sex sexual orientation gender identity or expression pregnancy age national origin disability status genetic information protected veteran status or any other characteristic protected by law.
INDIT
ISDPD
Required Experience:
Senior IC
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Knowledge Services is a technology and professional services company whose mission is to serve those who serve others. As a certified Woman-owned Business Enterprise (WBE) founded and established in 1994 by CEO, Julie Bielawski, Knowledge Services delivers innovative workforce solutio ... View more
