Medior Data Engineer Extraction & Ingestion (Python)
Turn raw market noise into Europes most trusted curated foodindustry market intelligence insights
A-INSIGHTS turns raw market signals into clear actionable insights for decision-makers at names like McCain FrieslandCampina and Ahold Delhaize. Our data pipelines are the first mile of that journey. As our next Medior Data Engineer youll own a portfolio of connectors and ingestion services designing resilient scrapers and API connectors that keep fresh high-quality data flowing 24/7 while mentoring juniors and aligning engineering work with business goals.
Full-time Amsterdam
What you will be doing
You are co-responsible for the front door of our central data platform
Lead design & delivery of Python ingestion jobs (scrapers API connectors) with robust error handling idempotency and documented backfill strategies.
Define the first steps of ELT: validation normalization and schema mapping and agree on data contracts with downstream consumers.
Own observability & reliability: define Service Level Indicators and Objectives add logs/metrics/alerts drive and track error budgets.
Set standards & mentor: create connector templates/runbooks lead code reviews pair with juniors and coach on clean code and debugging.
Design for ethics & resilience: handle pagination throttling and anti-bot challenges respectfully; follow legal guidelines; secure secrets/PII via Azure Key Vault and least privilege access.
Embed AI enrichment (Azure OpenAI & Cognitive Services) to classify summarize and deduplicate content; set acceptance criteria for model outputs.
Projects youll tackle in the first year
Lead migration processes of legacy scrapers to Azure Functions with retry logic exponential back-off and structured logs plan phases risks and rollbacks.
Connector standardization: ship reusable templates runbooks and SLIs/SLOs for ingestion and guide the team to adopt them.
Document-to-insight pipeline: evolve extraction/vectorization/summarization at scale; improve throughput and cost per document.
AI-powered news crawler: raise precision/recall and define evaluation metrics for summarization/deduplication.
Ongoing work you will contribute to
Build & maintain Python ingestion jobs and first-step ELT pipelines with clear ownership boundaries.
Deploy and observe services through Azure DevOps and Docker; proactively optimize cost/performance in Azure.
Keep our data platform reliable scalable and secure; lead post-incident improvements.
Define and follow data-engineering best practices for design testing and documentation; guide their adoption.
Build and maintain cloud infrastructure in Azure to support scalable and secure data ingestion processes.
Tech stack
Daily tools
Python (including Pandas Selenium and internal SDKs)
API integration (REST GraphQL)
Azure DevOps (Git version control CI/CD pipelines)
Docker & Azure Container Registry
Azure Function Apps & Logic Apps
Youll also touch
Azure AI Services & AI Foundry
SQL (SQL Server Management Studio / Azure Data Studio)
Azure Key Vault
Azure Storage Services (Blob Table Queue)
Azure Monitor / Application Insights
Havent used every tool yet If youve mastered similar tech and love learning well back you.
Who youll be working with:
Data Ingestion Architect Python Data Engineers Insights Analysts Product Manager (Amsterdambased collaborative and handson)
Your Talent
46 years of professional software engineering experience with strong Python skills.
Proven end-to-end ownership of ingestion projects (from design to production) or major refactors in production.
Led design reviews wrote technical docs/runbooks.
Git power user branching rebasing PR reviews and CI checks are second nature.
Strong on clean code debugging architectural trade-offs and performance tuning in Azure.
Nice to have: web scraping at scale Selenium SQL optimization event-driven/streaming (e.g. Kafka/Event Hubs) Pandas/NumPy experience in mentoring junior team members
What we offer:
annually including 8% holiday allowance.
Performance bonus up to 10% of gross salary.
Hybrid work: 25 days/week in our Amsterdam HQ.
Personal growth plan and budget for both hard & soft skills.
A supportive team of experienced engineers (pairprogramming architecture sparring).
Regular company events dinners BBQs boat trips padel tournaments because fun fuels focus.
A-INSIGHTS
We enable the worlds biggest food companies like McCain Royal FrieslandCampina Ahold Delhaize Puratos Savencia and more to save time deepen insights and seamlessly share knowledge. With a strong product a clear mission and rapid growth ahead theres never been a better time to join us.
Our Amsterdam team is a vibrant mix of 15 nationalities who blend curiosity with craft (and occasionally turn each other into works of art).
Ready to be our next secret ingredient
Application process
Online SkillBased Assessment: If we see a match well invite you to take an online assessment.
Screening Call: If the assessment is successful you will receive a call from us with some followup questions.
InPerson Interview: Intro chat in our Amsterdam office to assess team and culture fit.
Case Preparation: Youll work on a case to prepare for the next interview.
InDepth Interview: We review your case and discuss your technical skills. Plenty of room for your questions.
Offer: If everything aligns well make you an offer you cant refuse.
We aim to finalize the hiring process in 36 weeks.
Were serious about inclusion
Research shows some groups only apply when they meet every requirement. If youre excited by this role but your experience doesnt line up perfectly please apply anywaywed love to hear your story.
Ready to be our next secret ingredient
Have Questions For any inquiries or additional information please contact Nofar Biran.
FD Gazellen 2018
Nr. 17 van NL
Sprout
Top 25 onder de 25
ECE & RSM
Top 250 Growth Companies
Deloitte - Fast 50
Technology top 50 in NL
