Data Engineer sénior
Share
Job Location:
Daily Salary:
Posted on:
9 hours ago
Vacancies:
1 Vacancy
Job Summary
About the role:
We are looking for a result-driven Senior Data Engineer to join our core data engineering team . The successful candidate should have in-depth knowledge in data engineering with the ability to share experience with the team. The company has recently started the Data Transformation Program involving broad-scale modernization of Data Architecture Data Governance and Data Exposition. The ultimate goal for this role is to apply best engineering practices in building end-to-end data pipelines and support transformation of the core data platform according to Data Lakehouse approach.
Responsibilities:
- Provide high quality code aligned with software engineering best practices
- Share experience with the team; encourage creating guidelines / policies
- Create data collection / ingestion / validation pipelines for various sources
- Sync with DataOps team to leverage the required CI/CD pipelines and integrations
- Work together with Data Architect to validate the target data architecture design
- Lead POCs
- Help the team to refactor the existing orchestration design in Airflow
Required skills & experience:
- Hands on experience in implementation Data Lakehouse architecture
- Advanced hands on Python skills is a must
- Airflow is a must
- Spark - nice to have
- Experience with Apache Iceberg / Delta Lake- nice to have
- Hands on experience with AWS services
- Advanced SQL
- Deep understanding of Data modelling; knowledge of medallion architecture is a plus
- Terraform - nice to have
- Good understanding of Data Mesh principles
- Snowflake experience is nice to have
- Good understanding of Data Governance & Data Quality principles
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Collective is the platform to find freelance work Go and check us out we are the biggest platform in terms of jobs published
