Senior Big Data Engineer (Kubernetes AWS Spark)

Title: Senior Big Data Engineer (Kubernetes / AWS / Spark)

Location: Rockville MD or Mclean VA or NYC Metro area (Hybrid 3 days onsite per week)

Contract: 6 month with extension for long-term

Only Local candidates who can take Assessment and only who are in DC/VA/MD/NY/NJ who can got for F2F interview

Role Overview

• This role focuses on building and optimizing large-scale data platforms with a strong emphasis on Kubernetes-based infrastructure and big data processing in cloud environments.

Key Responsibilities

• Design develop and optimize data pipelines handling terabyte-scale datasets
• Work with complex algorithms to process and analyze large volumes of data efficiently
• Optimize code performance for scalability and high-throughput systems
• Build and maintain containerized serverless data platforms using Kubernetes
• Support migration efforts from EMR/EC2-based systems to Kubernetes-based architecture
• Maintain and enhance existing Kubernetes infrastructure
• Develop and manage CI/CD pipelines using tools like GitLab and Bitbucket
• Collaborate on requirements documentation system design and implementation
• Contribute to emerging initiatives involving:
• GenAI integration
• AI agents and automation frameworks
• Technologies such as Kiro (Amazon GenAI) and MCP (Model Context Protocol)

Technical Environment

• Cloud Platforms: AWS (current) with exposure to Google Cloud and open-source ecosystems
• Core Technologies:
• Kubernetes (primary focus)
• Elasticsearch
• Big Data frameworks (EMR distributed systems)
• Programming Languages: Python SQL Scala (flexible)
• DevOps & Tooling: GitLab Bitbucket CI/CD pipelines

Required Skills & Experience

• Strong experience with Kubernetes including deployment migration and infrastructure management
• Experience working with large-scale data (terabytes) and distributed systems
• Proficiency in at least one: Python SQL or Scala
• Understanding of data processing optimization techniques
• Familiarity with cloud-native architectures and containerization
• Experience with CI/CD pipelines and modern DevOps practices
• Exposure to AI/GenAI tools agents or related frameworks is a strong plus
• AWS and/or Kubernetes certifications preferred