Data Engineer

Required Skills - Top 3-5 Key Words

Required Technical Skills: Python SQL AWS Web services (Glu S3 Lambda)

Core Programming Skills:

• Expert proficiency in Python with experience in building data pipelines and back-end
• systems.
• Advanced knowledge of SQL for querying and optimizing large datasets.
• AWS Cloud Services Expertise:
• DynamoDB S3 Athena GlueETL Lambda ECS Glue Data Quality EventBridge
• Redshift Machine Learning OpenSearch and RDS.

API and Resilience Engineering:

• Proven expertise in designing fault-tolerant APIs using Swagger/OpenAPI GraphQL
• and RESTful standards.
• Robust understanding of distributed systems load balancing and failover strategies.
• Monitoring and Orchestration:
• Hands-on experience with Prometheus and Grafana for observability and monitoring.

Job Duties

• Senior Data Engineer 7 Years of Experience
• We are seeking a highly experienced Senior Data Engineer with 7 years of expertise in
• designing building and optimizing robust data solutions. The ideal candidate must
• possess top-tier skills in Python AWS services API development and TypeScript and
• have significant hands-on experience with anomaly detection systems.
• The candidate should have a proven ability to work at both strategic and tactical levels
• from designing data architectures to implementing them in the weeds.

Job Requirements

Key Responsibilities:

• Data Pipeline Development
• Independently design build and maintain complex ETL pipelines ensuring scalability
• and efficiency for large-scale data processing needs.
• Manage pipeline complexity and orchestration delivering high-performance data
• products accessible via APIs for business-critical applications.
• Archive processed data products into data lakes (e.g. AWS S3) for analytics and
• machine learning use cases.
• Anomaly Detection and Data Quality

• Implement advanced anomaly detection systems and data validation techniques ensuring data integrity and quality.
• Leverage AI/ML methodologies including Large Language Models (LLMs) to detect and address data inconsistencies.
• Develop and automate robust data quality and validation frameworks.
• Cloud and API Engineering
• Architect and manage resilient APIs using modern patterns including microservices
• RESTful design and GraphQL.
• Configure API gateways circuit breakers and fault-tolerant mechanisms for distributed
• systems.
• Ensure horizontal and vertical scaling strategies for API-driven data products.
• Monitoring and Observability
• Implement comprehensive monitoring and observability solutions using Prometheus and Grafana to optimize system reliability.
• Establish proactive alerting systems and ensure real-time system health visibility.
• Cross-functional Collaboration and Innovation
• Collaborate with stakeholders to understand business needs and translate them into scalable data-driven solutions.
• Continuously research and integrate emerging technologies to enhance data engineering practices.

Required Skills :

Basic Qualification :

Additional Skills :

Background Check : No

Drug Screen : No