Data Engineer (MId)
Share
Job Location:
Monthly Salary:
Posted on:
3 hours ago
Vacancies:
1 Vacancy
Job Summary
Design implement and maintain scalable and reliable data pipelines to ingest transform and load data from multiple sources.
Orchestrate and schedule workflows using tools such as Apache Airflow ensuring correct dependencies retries alerting monitoring and performance.
Build and maintain data storage solutions data warehouses and/or data lakes ensuring data is properly modelled organized and accessible for analytics and ML use cases.
Collaborate with data scientists analysts and other stakeholders to understand data requirements and deliver clean high-quality data in usable formats.
Implement data validation monitoring and quality controls to guarantee data integrity consistency and reliability.
Contribute to productionizing machine learning pipelines from raw data ingestion through feature engineering training data preparation and integration with ML workflows.
Follow and promote CI/CD best practices for data code and infrastructure (version control automated tests deployments infrastructure as code etc.).
Qualifications :
Several years (e.g. 25) of professional experience in data engineering or a related role (ETL data pipelines data infrastructure).
Strong proficiency with data orchestration tools such as Apache Airflow (or similar).
Experience with cloud-based data platforms and warehouses especially Google BigQuery and/or Databricks
Excellent SQL skills and comfortable writing complex queries for large datasets.
Solid programming skills (e.g. Python) for data processing and transformation.
Experience working with cloud infrastructure ideally both on Amazon Web Services (AWS) and Google Cloud Platform (GCP) or willingness and ability to learn them.
Strong understanding of software engineering practices: version control (e.g. Git) modular code configuration management CI/CD for data workloads.
Strong problem-solving skills attention to detail and ability to handle large complex datasets with performance scalability and data quality in mind.
Nice-to-Have (Bonus) but not mandatory
Familiarity with data governance data cataloging metadata management data lineage ensuring data quality and traceability.
Previous exposure to machine learning workflows or model-training pipelines (feature engineering data versioning model data pipelines).
Experience with infrastructure-as-code automated deployments (Terraform CloudFormation etc.) or orchestration beyond just simple pipelines.
Exposure to cost optimization data partitioning/indexing performance tuning of big data workloads.
Additional Information :
Our Benefits
- Flexible working environment
- Volunteer time off
- LinkedIn Learning
- Employee-Assistance-Program (EAP)
About NIQ
NIQ is the worlds leading consumer intelligence company delivering the most complete understanding of consumer buying behavior and revealing new pathways to 2023 NIQ combined with GfK bringing together the two industry leaders with unparalleled global reach. With a holistic retail read and the most comprehensive consumer insightsdelivered with advanced analytics through state-of-the-art platformsNIQ delivers the Full View. NIQ is an Advent International portfolio company with operations in 100 markets covering more than 90% of the worlds population.
For more information visit
Want to keep up with our latest updates
Follow us on: LinkedIn Instagram Twitter Facebook
Our commitment to Diversity Equity and Inclusion
At NIQ we are steadfast in our commitment to fostering an inclusive workplace that mirrors the rich diversity of the communities and markets we serve. We believe that embracing a wide range of perspectives drives innovation and excellence. All employment decisions at NIQ are made without regard to race color religion sex (including pregnancy sexual orientation or gender identity) national origin age disability genetic information marital status veteran status or any other characteristic protected by applicable laws. We invite individuals who share our dedication to inclusivity and equity to join us in making a meaningful impact. To learn more about our ongoing efforts in diversity and inclusion please visit the Work :
No
Employment Type :
Full-time
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
NIQ is the worlds leading consumer intelligence company, delivering the most complete understanding of consumer buying behavior and revealing new pathways to growth. In 2023, NIQ combined with GfK, bringing together the two industry leaders with unparalleled global reach. With a holi ... View more
