Senior Data Engineer
Share
Job Location:
Monthly Salary:
Posted on:
12 hours ago
Vacancies:
1 Vacancy
Job Summary
We are looking for a Senior Data Engineer specialized in Python PySpark AWS and data to join our clients dynamic team. You will play a key role in designing optimizing and improving data pipelines for the ingestion enrichment and exposure of classified and transactional data on AWS. You will work directly under the supervision of a Data Engineering Team Lead who will organize tasks and ensure the smooth delivery of the project.
Tasks
- Pipeline Optimization:
Analyze and improve existing data pipelines to optimize performance cost efficiency and scalability.
- Pipeline Transformation:
Transition batch/snapshot pipelines to delta-based data processing pipelines.
- Infrastructure as Code:
Develop and maintain Terraform modules for efficient infrastructure management on AWS.
- Pipeline Migration:
Migrate data pipelines from Google Cloud Platform to AWS while minimizing downtime and ensuring high reliability.
- Monitoring and Alerts:
Design and implement DataDog dashboards and alerting systems to enable proactive monitoring of data pipeline performance.
- Technology Watch & Innovation:
Stay up to date with emerging technologies and actively promote relevant innovations and improvements within the team.
- Mentoring & Team Support:
Support and guide junior team members share expertise and best practices and contribute to the overall growth of the team.
Requirements
Required Technical Skills:
- AWS:
Strong hands-on experience with AWS services such as S3 ECS etc. with particular emphasis on Lambda and Glue. - Python:
Advanced Python skills for data manipulation scripting and pipeline development. - PySpark:
Solid experience building scalable and distributed data pipelines using PySpark. - SQL:
Strong command of SQL for querying and transforming large datasets. - Terraform:
Experience designing and managing infrastructure using Terraform. Knowledge of Terragrunt is a plus. - CI/CD (CircleCI):
Experience configuring and maintaining pipelines to support automation and deployment workflows. - DataDog:
Knowledge of DataDog for monitoring alerting and dashboard creation.
Desired Qualities:
- Autonomous in project management with strong ownership of production infrastructure and data access.
- Open-minded and driven by innovation and continuous improvement.
- Pragmatic solution-oriented and able to adapt quickly.
- Strong focus on code quality and adherence to best practices.
- Ensures high-quality testing and maintains a customer-focused approach.
Join a fast-scaling tech platform revolutionizing marketplaces at scale. Work with elite engineering teams on mission-critical AI data and optimization challenges.
Extremely competitive comp equity. Bucharest or Belgrade offices.
Apply to build the future of intelligent platforms.
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
The rec hub is trusted by 120+ European companies, from ambitious start-ups to global enterprises, to deliver hiring solutions that actually work. On average, our clients see a 60% reduction in time-to-hire, helping them scale faster without compromising on quality. With zero lock-in ... View more
