Platform Engineer Big Data
Share
Job Location:
Rockville, MD - USA
Monthly Salary:
Not Disclosed
Posted on:
13 hours ago
Vacancies:
1 Vacancy
Job Summary
Location: Rockville MD Tysons Corner VA or Woodbridge NJ or Jersey City NJ (3 days onsite per week)
Duration: 6 months (long-term extensions)
Notes:
Senior Platform Engineer Big Data (AWS EMR EKS)
- Build and modernize a large scale AWS big data platform (EMR S3 Athena Trino) supporting enterprise analytics
- Help drive platform evolution toward cloud native containerized workloads on AWS EKS (Kubernetes)
- Work at the intersection of software engineering big data and platform engineering not ETL only
- Design and operate Spark based data workloads optimizing performance reliability and cost
- Implement CI/CD and Infrastructure as Code (Terraform / CloudFormation) for data platforms
- Ideal for engineers with a strong backend or platform background who ve grown into big data
Job Description:
Overview
We are seeking a Senior Platform Engineer with deep Big Data experience to help design operate and modernize a large scale data platform on AWS. This role goes beyond traditional ETL or pipeline development it is focused on building and evolving the underlying data platform that supports analytics reporting and future AI/ML use cases.
The current environment is built primarily on AWS EMR and S3 with a strong query layer using Athena and Trino. The team is actively modernizing the platform and evaluating AWS EKS (Kubernetes) as part of a shift toward more cloud native containerized data workloads.
This role is ideal for an engineer with a software or platform engineering background who moved into big data rather than a pure ETL developer.
Key Responsibilities
- Design build and operate scalable big data platforms on AWS with S3 as the core data lake.
- Develop and optimize Spark based workloads on EMR including performance tuning and cost optimization.
- Support and enhance federated query engines such as Athena and Trino for large scale analytics.
- Contribute to the modernization of the data platform including evaluation and adoption of Kubernetes/EKS for data services and workloads.
- Build and operate data services and platform components using containerized deployments (Docker EKS).
- Implement and maintain Infrastructure as Code using Terraform and/or CloudFormation.
- Design and support CI/CD pipelines for data and platform workloads.
- Partner with data engineers analytics teams and stakeholders to ensure the platform is reliable performant and extensible.
- Monitor and troubleshoot platform issues across clusters pipelines and query engines using CloudWatch and related tooling.
- Continuously evaluate new technologies and propose improvements to the overall data architecture.
Required Qualifications
- 8 years of experience in Big Data Platform Engineering or Data Engineering roles.
- Strong hands on experience with AWS including:
- EMR
- S3
- Athena
- AWS Glue / Glue Data Catalog
- Solid experience with Spark (PySpark or Scala) and distributed data processing.
- Strong SQL skills particularly with large datasets (Athena Trino Presto etc.).
- Experience with Docker and containerized applications.
- Working knowledge of Kubernetes with exposure to AWS EKS strongly preferred.
- Experience implementing CI/CD pipelines (Jenkins GitHub Actions or similar).
- Infrastructure as Code experience using Terraform and/or CloudFormation.
- Strong scripting and programming skills (Python preferred).
- Ability to think at a platform and architecture level not just task execution.
Nice to Have
- Experience running Spark on Kubernetes (EKS).
- Trino/Presto performance tuning experience.
- Experience preparing data platforms for AI/ML workloads.
- Observability tooling experience (CloudWatch Grafana Prometheus).
- Background as a software engineer before moving into big data.
