Senior Data Engineer

This is a remote position.

We are seeking a highly skilled Senior Spark Engineer with expertise in Python Spark Databricks and AWS to join our dynamic team. The ideal candidate will possess a deep understanding of Spark architecture and be proficient in finetuning Spark jobs for optimal performance. Additionally strong knowledge of software engineering best practices DevOps principles including CI/CD pipelines Kubernetes and either Jenkins Airflow or SageMaker is essential for this role. Experience in MLOps is desirable but not mandatory.

Requirements

• Proficiency in Spark Python Databricks and AWS services.
  
• 710 years of experience
  
• Lead the design development and implementation of Sparkbased solutions for complex data processing tasks.
  
• Collaborate with crossfunctional teams to gather requirements and translate them into technical specifications.
  
• Implement best practices in software engineering DevOps and MLOps methodologies.
  
• Design and maintain data pipelines for efficient data processing and analysis.
  
• Perform data modeling and schema design to support business requirements.
  
• Mentor junior team members and provide technical guidance as needed.
  
• Stay updated with the latest advancements in Spark Python and related technologies and advocate for their adoption when appropriate.
  
• Bachelors degree in Computer Science Engineering or a related field. Masters degree preferred.
  
• Strong understanding of software engineering principles and best practices.
  
• Handson experience with DevOps tools and practices including CI/CD pipelines Kubernetes and either Jenkins Airflow or SageMaker.
  
• Experience with MLOps practices is a plus.
  
• Solid understanding of data engineering concepts and data modeling techniques.
  
• Excellent problemsolving skills and attention to detail.
  
• Ability to work independently and lead tasks effectively.
  
• Strong communication and interpersonal skills.