Collaborate with stakeholders to identify business requirements and translate them into technical specifications
Design build monitor and maintain data pipelines in production including complex pipelines (Airflow Python eventdriven systems)
Develop and maintain ETL processes for ingesting and transforming data from various sources
Monitor and troubleshoot infrastructure issues such as Kubernetes and Terraform including data quality ETL processes and cost optimization
Collaborate closely with analytics engineers on CI and infrastructure management
Drive the establishment and maintenance of the highest coding standards and practices ensuring the development of efficient scalable and reliable data pipelines and systems
Participate in data governance initiatives to ensure data accuracy and integrity
Actively participate in the data teams routines and enhancement plans
Stay up to date with the latest developments in data technology and provide recommendations for improving our analytics capabilities
Qualifications :
At least 5 years of experience in data engineering or software engineering with a strong focus on data infrastructure
Handson experience in AWS (or equivalent cloud platforms like GCP) and data analytics services
Strong proficiency in Python and SQL
Good understandingof database design optimization and maintenance (using DBT)
Strong experience with data modeling ETL processes and data warehousing
Familiarity with Terraform and Kubernetes
Expertise in developing and managing largescale data flows efficiently
Experience with job orchestrators or scheduling tools like Airflow
At least an UpperIntermediate level of English
Would be a plus:
Experience managing RBAC on data warehouse
Experience maintaining security on data warehouse (IPs whitelist masking sharing data between accounts/clusters etc.)
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.